A worker in my agent system forgot something I had written down for it. Twice, in the same week.
The first time, I assumed I'd misconfigured the memory file. I hadn't. The entry was there, correctly formatted, indexed, scored. The second time, I stopped blaming the storage and started watching what retrieval actually did. The entry was in the database. It just never made it into the session.
My instinct before doing that work was the instinct most people have when their agent feels dumb: reach for a bigger model. Swap Sonnet 4.6 for Opus 4.7. Add more rules to the prompt. Maybe give it a longer context window. I didn't do any of that. I spent two weeks rewriting how memory surfaces, and the agent got smarter without a single model change.
The thing that's wrong with your agent is probably not reasoning. It's retrieval.
The storage was fine. The readers were blind.
The shape of my memory system before this work was standard for an agent. Learnings get extracted from transcripts, embedded, scored, and written to SQLite with tags and quality metrics. Writers populated the database comprehensively. Nearly nine thousand entries by the time I started looking closely. The dream job that mines transcripts for durable insights was doing its job.
The readers were the problem. Those are the functions that pull memories back into a new session, and they weren't surfacing the learnings I'd written down about my own system. A mission that needed a piece of context from three weeks ago would get a top-10 full of unrelated patterns because the ranking had degenerated into returning the same high-quality-scored patterns every time, regardless of relevance. Every session retrieved the same ten things.
I wrote a note about this in my dream log before I understood the full shape of the fix: "Memory system retrieval is the gap. Storage populates comprehensively, but readers don't surface enough context per session, causing sessions to forget stored knowledge." When the system tells you something about itself in its own voice, listen.
Nine gaps, all retrieval
Over the next two weeks I shipped nine numbered remediation tickets. They're in my tracker as G1 through G11 (two got folded into others). I'll name the three that moved the needle most.
G1. Diversity re-ranking. The original ranker just sorted by quality score and took the top ten. The top ten ended up being ten variations of the same pattern, all scored highly because they'd been used many times. I added MMR (maximum marginal relevance) with a per-type cap of two. Fetch 2,000 candidates, then greedy-select with cosine similarity discounting. The top-10 now has actual variety. This one fix changed the feeling of every session.
G4. Bridge-session filter. Memories have types: insight, decision, pattern, project, feedback, reference. Some of those belong inside a single session. Specific debugging contexts, in-progress thoughts. Some should cross the session boundary and surface everywhere. I split them: only project, reference, and feedback entries cross. insight and pattern stay scoped. The cross-session memory stopped carrying operational noise.
G8. Citation format and compliance measurement. This one was subtle. I couldn't tell whether memories were actually being used by workers after injection, or just printed into the prompt and ignored. So I gave every injected learning a short ID and required the worker to cite it in the format [type · short-id] when the output relied on it. A parser at output time counts citations. Compliance rate now boosts quality on future injection. Memories that don't get cited slowly age out. The system learns which of its own memories are earning their keep.
The six I'm skipping are in the same spirit: dedupe the log, scope the archival predicate correctly, fix a stale-memory edge case, align citation formats on both sides, etc. Small individually. Real in aggregate. Together they rewrote the surface between stored knowledge and live reasoning.
After I finished, I read an older note from CData Software that's been sitting in my scout pile with 62 engagement: "Most AI agent failures aren't reasoning failures. They're context failures." That's the same conclusion, written three weeks before I arrived at it. I should have read it first.
Everyone is arriving at the same split
The thing that stopped me being smug about the work was a different Bluesky post I pulled during research: "LangMem, Letta, Zep, CrewAI — they all independently invented the same solution: split operational logs from persistent memory."
That's four memory products, all from different teams, all quietly converging on the same architecture. I arrived at it too, as a solo builder, and named it differently. I called it "personas are ephemeral; only alpha persists." Structurally it's the same call. Ephemeral scratch for the work-in-progress thinking. A small, curated canon for the durable stuff. The same shape.
If five independent teams converge on a design, it's probably because the problem has a shape. And the shape is: one memory store can't do both jobs. An operational log of everything the agent tried wants to be complete, noisy, cheap to append, short-lived. A persistent memory of what the agent learned wants to be sparse, curated, cheap to retrieve, long-lived. The two workloads fight each other if you pile them in one place. Everyone who builds agents long enough discovers this by having their agent quietly stop remembering the important things.
If you are building an agent right now and you haven't hit this wall, you will. Your options are: invent it yourself, or notice that five teams have already done the work.
Context is not tokens. It's ranking.
"Give it more context" has become one of those phrases that means nothing because it means everything. Most of the time what people mean when they say it is: expand the context window, pass more of the codebase, include more history. That's a model-size lever. It helps, up to the point where the model loses the plot in its own context.
The lever that actually moves is which context shows up. With 200K tokens of room, you have zero problem fitting the relevant things. You have a ranking problem. You have a filtering problem. You have a question-to-answer matching problem. These are retrieval problems, not model problems.
The CData line is closer to the truth than most agent takes I've read, but it's still only half the point. Agent failures are retrieval failures. Specifically:
- The relevant memory exists but doesn't get ranked into the top-k.
- The relevant memory gets ranked but doesn't survive deduplication or diversity filtering.
- The memory gets surfaced but isn't framed in a way the model can ground its answer in.
- The memory was used successfully but the signal never flowed back to improve future ranking.
My G-numbered gaps map directly onto these failure modes. The work isn't glamorous. It's SQL indexes, regex parsers, injection predicates, cosine-weighted MMR. None of it is novel research. All of it is the difference between an agent that feels competent and one that feels like it's seeing the problem for the first time every session.
What this cost, and what it bought
Two weeks, roughly. Twenty or so commits to the retrieval layer. A net delete of about twenty-nine hundred lines when I collapsed the persona-memory path into something simpler (most of this work was removing code that tried to do too much). New audit logs at ~/.alluka/logs/preflight-audit.<YYYY-MM-DD>.jsonl so I can see what got injected into every session, whether it got cited, and what the compliance rate is converging to over time.
What I got out of it: sessions that remember what I told them. Workers that cite their sources. A feedback loop where usage patterns influence future retrieval. The thing I didn't expect was a much clearer sense of which entries in my memory are actually carrying weight. The top quality-scored memories in my database are now genuinely different from the top-used ones, and the gap between those lists is the most useful diagnostic I have for where the system is overfit to its own history.
The question I keep coming back to
If retrieval is where agents actually live or die, then the next obvious step is: what happens when agents retrieve from each other instead of just from the human?
Right now my agents retrieve from a memory I curate. The memory grows because I ran sessions, reviewed the extractions, promoted the good ones, archived the junk. That's a single-human-in-the-loop system. It scales to me.
When two agents work on the same problem and one of them retrieves something the other learned last week, without me touching either memory, we're in a different regime. The ranking question becomes: whose memory, whose signal, whose quality score? The CData take that agents fail on context assumes one agent and one context. Multi-agent memory is a class of problem I haven't seen anyone describe cleanly yet. Not Letta, not LangMem, not Zep. It's where I think the real interesting work is.
For now: your agent probably doesn't have a reasoning problem. It has a retrieval problem. Before you upgrade the model, look at what your memory layer is actually surfacing. You will be surprised how much of what you thought was missing was there the whole time, just never ranked high enough to be seen.