Memory Tax: Dissecting the Net Value of Agent Memory
Abstract
Large-scale evaluation reveals that dedicated memory systems often underperform naive retrieval-augmented generation, introducing a framework of memory benefit and memory tax to assess net value.
Memory is increasingly used in long-horizon agents, yet the net value of building memory over direct retrieval from raw history remains largely unclear. Through a large-scale evaluation across multiple benchmarks and model backbones, we find many dedicated memory systems underperform naive RAG in many cases, indicating a negative net value. Taking RAG as a reference, we dissect the net value of memory into the gains and losses over direct retrieval, termed memory benefit and memory tax. These gains and losses arise because memory process changes how the raw information is organized and accessed. We view such changes as the double-edged deviations from the raw history. They are originally designed and introduced to benefit the task, for example by reorganizing facts or distilling preferences, but some may also remove or obscure evidence, giving rise to memory tax. Since memory benefit reflects the expected gains of its design, we then focus more on the overlooked memory tax, study how it arises from the system and evaluation side, and generalize the concept to more application scenarios of memory. We hope the notion of memory tax can provide a new lens and shift more attention from standalone performance to the net value of memory, informing more reliable memory designs and evaluations. We will open-source the code and results to the community. Project Page: https://hexuan21.github.io/MemoryTax/
Community
Many agent memory systems can’t even beat a naive RAG? We conducted a comprehensive evaluation and find many underperform the naive RAG, even at higher cost. To better understand this, we introduce “Memory Tax”, see more in our paper.
Really useful framing, and I appreciate how carefully the RAG baseline is specified (BM25 + Qwen3-Embedding-0.6B, reranked by Qwen3-Reranker-0.6B), since that whole stack fits easily on a home GPU. Table 1 makes the point clearly: with Qwen3.6-27B on LongMemEval-s, two-round RAG reaches 81.4 versus 76.6 for MemOS and 78.2 for HippoRAG2, and the fact tracing in Table 2 shows how different construction losses can be, from 98.4% no-loss for MemoryOS to 38.0% of sampled facts with no memory unit at all for OpenViking. The recovery path in Section 6.1, retrieving from the raw history alongside memory and reranking jointly, feels like the most practical takeaway for people building agents today. How much extra token cost and latency does that hybrid path add compared with plain one-round RAG, and does it close most of the gap to the Gold Context scores?
Table 18 suggests a workload threshold. With GPT-5.6 Luna, LightMem pays 5.22M build tokens across 100 histories (about 52.2k per history), then 0.09M QA tokens versus 0.27M for one-round RAG (about 0.9k versus 2.7k per query). Linearly extrapolating those rates, LLM-token cost crosses near 29 queries per history. Accuracy still trails RAG, 77% versus 82%, so that cost crossover does not erase memory tax. Could you show amortization curves alongside accuracy for the other systems?
Get this paper in your agent:
hf papers read 2610.34073 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 7
Browse 7 datasets citing this paperSpaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper