Long-term agent memory grows continuously with interaction, making full-memory prefill increasingly costly in terms of time to first token (TTFT) and redundant computation. KV-cache reuse could avoid repeatedly prefilling the accumulated memory, but KV caching relies on exact prefix matching. When memory is updated, reorganized, or assembled from reusable units, prefix mismatches prevent direct reuse of existing KV caches and force the model to recompute a long memory prefix. Existing methods largely assume that memory blocks are already given and address the prefix-matching limitation by concatenating their KV caches and then repairing the resulting mismatch. We therefore argue that the primary question for agent-memory KV reuse is where to segment the memory. We propose AD-CUT, an Attention-derived dependency Cut method for training-free, independence-oriented segmentation. AD-Cut computes attention-derived dependency signals over sentence-level atomic units, including an incoming signal Ain and an outgoing signal Aout, and selects boundaries where both signals are low. The signals exhibit stable positive associations with intervention-derived KV and NLL sensitivity, with Spearman ρ ranging from 0.615 to 0.868 and context-cluster bootstrap intervals remaining positive, supporting their use as predictive proxies. In block-KV serving for agent QA, AD-Cut achieves a 16.17× Online TTFT speedup with a QA Quality Retention of 0.7675. It also improves task quality over naive sentence-level reuse in a controlled agent-memory evaluation. These results show that AD-Cut substantially reduces online computation while mitigating the quality loss caused by KV-cache reuse.
- Open Access
- Article
Attention-Derived Dependency Cut for Independent KV-Cache Reuse in Agent Memory
- Yizhuo Zhao,
- Boxi Wu * and
- Xiaofei He
Author Information
Received: 01 September 2026 | Revised: 28 September 2026 | Accepted: 29 October 2026 | Published: 06 November 2026
Abstract
Keywords
agent memory | KV-cache reuse | attention dependency | memory segmentation | large language model serving
References
- 1.
Wu, D.; Wang, H.; Yu, W.; et al. LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory. In Proceedings of the 13th International Conference on Learning Representations, Singapore, 24–28 April 2025; pp. 86809–86836.
- 2.
Wu, D.; Ji, Z.; Kawatkar, A.; et al. LongMemEval-V2: Evaluating Long-Term Agent Memory toward Experienced Colleagues. arXiv 2026, arXiv:cs.cl/2605.12493.
- 3.
Yao, J.; Li, H.; Liu, Y.; et al. CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion. In Proceedings of the EuroSys’25: Twentieth European Conference on Computer Systems, Rotterdam, The Netherlands, 30 March–3 April 2025; pp. 94–109. https://doi.org/10.1145/3689031.3696098.
- 4.
Yang, J.; Hou, B.; Wei, W.; et al. KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse. In Proceedings of the 39th International Conference on Neural Information Processing Systems, San Diego, CA, USA, 2–7 December 2025.
- 5.
Beltagy, I.; Peters, M.E.; Cohan, A. Longformer: The Long-Document Transformer. arXiv 2020, arXiv:2004.05150.
- 6.
Dao, T.; Fu, D.; Ermon, S.; et al. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. In Proceedings of the 36th International Conference on Neural Information Processing Systems, New Orleans, LA, USA, 28 November–9 December 2022; Volume 35, pp. 16344–16359. https://doi.org/10.52202/068431-1189.
- 7.
Hu, J.; Huang, W.; Wang, W.; et al. EPIC: Efficient Position-Independent Caching for Serving Large Language Models. In Proceedings of the 42nd International Conference on Machine Learning, Vancouver, BC, Canada, 13–19 July 2025; Volume 267, pp. 24391–24402.
- 8.
Kwon, W.; Li, Z.; Zhuang, S.; et al. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the SOSP ’23: 29th Symposium on Operating Systems Principles, Koblenz, Germany, 23–26 October 2023; pp. 611–626. https://doi.org/10.1145/3600006.3613165.
- 9.
Zheng, L.; Yin, L.; Xie, Z.; et al. SGLang: Efficient Execution of Structured Language Model Programs. In Proceedings of the Advances in Neural Information Processing Systems 37, Vancouver, BC, Canada, 10–15 December 2024; pp. 62557–62583. https://doi.org/10.52202/079017-2000.
- 10.
Gim, I.; Chen, G.; Lee, S.S.; et al. Prompt Cache: Modular Attention Reuse for Low-Latency Inference. In Proceedings of the Machine Learning and Systems, Santa Clara, CA, USA, 13–16 May 2024; Volume 6.
- 11.
Ma, D.; Wang, Y.; Lan, T. Block-Attention for Efficient Prefilling. In Proceedings of the International Conference on Learning Representations, Singapore, 24–28 April 2025; Volume 2025, pp. 63774–63788.
- 12.
Chen, C.; Zhang, G.L.; Yin, X.; et al. KV Packet: Recomputation-Free Context-Independent KV Caching for LLMs. arXiv 2026, arXiv:cs.lg/2604.13226.
- 13.
Zhao, J.; Ji, Z.; Fan, Z.; et al. MoC: Mixtures of Text Chunking Learners for Retrieval-Augmented Generation System. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vienna, Austria, 27 July–1 August 2025; Volume 1, pp. 5172–5189. https://doi.org/10.18653/v1/2025.acl-long.258.
- 14.
Li, S.; Zhang, Z.; Wang, Y.; et al. Towards Generalization of Block Attention via Automatic Segmentation and Block Distillation. arXiv 2026, arXiv:cs.cl/2605.15913.
- 15.
Vaswani, A.; Shazeer, N.; Parmar, N.; et al. Attention Is All You Need. In Advances in Neural Information Processing Systems; Curran Associates, Inc.: Red Hook, NY, USA, 2017; Volume 30, pp. 5998–6008.
- 16.
Xiao, G.; Tian, Y.; Chen, B.; et al. Efficient Streaming Language Models with Attention Sinks. In Proceedings of the Twelfth International Conference on Learning Representations, Vienna, Austria, 7–11 May 2024; Volume 2024, pp. 21875–21895.
- 17.
Su, J.; Lu, Y.; Pan, S.; et al. RoFormer: Enhanced Transformer with Rotary Position Embedding. arXiv 2021, arXiv:2104.09864.
- 18.
Lambert, N.; Morrison, J.; Pyatkin, V.; et al. Tulu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv 2024, arXiv:2411.15124.
- 19.
Grattafiori, A.; Dubey, A.; Jauhri, A.; et al. The Llama 3 Herd of Models. arXiv 2024, arXiv:2407.21783.
- 20.
OpenAI. GPT-5.5 Model. Available online: https://developers.openai.com/api/docs/models/gpt-5.5 (accessed on 21 August 2026).
- 21.
Butler, U. Semchunk: A Python Library for Semantic Chunking. Available online: https://github.com/isaacus-dev/semchunk (accessed on 23 August 2026).
- 22.
Hearst, M.A. TextTiling: Segmenting Text into Multi-Paragraph Subtopic Passages. Comput. Linguist. 1997, 23, 33–64. https://www.aclweb.org/anthology/J97-1003.pdf.
- 23.
Riedl, M.; Biemann, C. TopicTiling: A Text Segmentation Algorithm Based on LDA. In Proceedings of the ACL 2012 Student Research Workshop, Jeju Island, Korea, 9–11 July 2012; pp. 37–42.
- 24.
LlamaIndex. Semantic Splitter. Available online: https://docs.llamaindex.ai/en/stable/api_reference/node_parsers/semantic_splitter/ (accessed on 21 August 2026).
- 25.
Chen, J.; Xiao, S.; Zhang, P.; et al. M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings through Self-Knowledge Distillation. arXiv 2024, arXiv:2402.03216.
- 26.
Minhas, B.; Nigam, S. Chonkie: The Lightweight Ingestion Library for Fast, Efficient and Robust RAG Pipelines. Available online: https://github.com/chonkie-inc/chonkie (accessed on 23 August 2026).
- 27.
Lewis, P.; Perez, E.; Piktus, A.; et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In Proceedings of the Neural Information Processing Systems, Online, 6–12 December 2020; Volume 33, pp. 9459–9474.
- 28.
Ho, X.; Duong Nguyen, A.K.; Sugawara, S.; et al. Constructing a Multi-Hop QA Dataset for Comprehensive Evaluation of Reasoning Steps. In Proceedings of the 28th International Conference on Computational Linguistics, Online, 8–13 December 2020; pp. 6609–6625. https://doi.org/10.18653/v1/2020.coling-main.580.
- 29.
Yang, Z.; Qi, P.; Zhang, S.; et al. HotpotQA: A Dataset for Diverse, Explainable Multi-Hop Question Answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, 31 October–4 November 2018; pp. 2369–2380. https://doi.org/10.18653/v1/d18-1259.
- 30.
Kwiatkowski, T.; Palomaki, J.; Redfield, O.; et al. Natural Questions: A Benchmark for Question Answering Research. Trans. Assoc. Comput. Linguist. 2019, 7, 453–466. https://doi.org/10.1162/tacl_a_00276.
- 31.
Joshi, M.; Choi, E.; Weld, D.; et al. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vancouver, BC, Canada, 30 July–4 August 2017; pp. 1601–1611. https://doi.org/10.18653/v1/p17-1147.
This work is licensed under a Creative Commons Attribution 4.0 International License.
