The question that broke it was: how did I solve the orders-lumen bulkExport issue.
I knew the answer was there. I had spent an afternoon on it in Cursor a few weeks earlier, and the whole point of what I had just built was that my agent should be able to tell me about it without being asked. It told me nothing.
Not a wrong answer. Nothing at all. Which, once I went looking, turned out to be the more interesting failure.
Nonsense scored higher than the right answer
Here is the measurement, run against my own index of 42,611 passages. I embedded that question and scored it against everything, then did the same with randomly generated nonsense words, to see where the floor was.
meaningless nonsense, best match 0.525 the actual correct answer 0.505
The correct conversation scored below gibberish.
There is no threshold that fixes that. Any number low enough to catch the real answer catches every piece of noise in the corpus with room to spare. I had spent a day tuning a floor to sit above the noise band, and the thing I most wanted to find was underneath it.
The cause is unglamorous. orders-lumen and bulkExport are the two words in that sentence that actually identify what I am asking about. They are also the two an embedding model has never seen. It breaks them into subword fragments, and fragments of an unfamiliar identifier average out to something close to a generic vector, which is by construction a bit similar to everything and properly similar to nothing.
So the more distinctive a term is to a person, the less distinctive it is to the model. That is backwards from how anyone reasons about this, and it is invisible until you measure it, because the failure looks identical to "we just don't have that conversation".
What keyword search did with the same question
Nothing either. Zero results.
Which surprised me more than the first part, because those are exact strings sitting in the index. The reason is that my keyword query joined every word in the question with AND, so it went looking for a single message containing "how" and "did" and "I" and "solve" and "orders-lumen" and "bulkExport" and "issue". No such message exists.
Strip the question down to the two terms that matter and it is instant:
"how did i solve the orders-lumen bulkExport issue" -> 0 results "bulkExport excludePaths" -> ranks it 1st, 2nd, 3rd
So both halves of a normal hybrid search were blind, in opposite directions, to the same question. Semantic could not see the identifiers. Keyword could not see past the sentence wrapped around them.
Picking the words that carry the meaning
What I do now is pull the distinctive terms out of the request before searching for them, and the definition of distinctive is just rarity: a term that appears in more than zero and at most one percent of messages.
That needs no stopword list, which is the part I like. "How" and "issue" are discarded because they are everywhere, not because they are on a list somebody wrote. And it adapts per person. "Kubernetes" is a distinctive term for someone who has mentioned it twice and a useless one for someone who discusses nothing else, and the same code gets both right without knowing anything about either of them.
The threshold is the whole feature
The thing I underestimated is how lopsided the costs are once memory is proactive rather than searched.
A bad search result costs you a glance. You scan it, you ignore it, you move on. But this text goes into an agent's context before it starts work, without anyone asking for it. A wrong one is not noise a person filters out. It is a confidently stated piece of context that steers real work, and when the agent then does something daft because of it, that is traceable to me rather than to the model.
So it has to be much stricter than search, and the number cannot be a guess. Mine sits at 0.70 against a measured noise ceiling of 0.525, which is deliberately clear of it rather than just above it, because forty samples of nonsense cannot show you the real tail.
The cost of that is silence. It stays quiet on roughly three requests in four. I think that is the right trade and I would rather say so plainly than pretend the feature fires all the time: staying silent leaves you exactly where you were, and a false alarm leaves you worse off than before you installed anything.
Something I was wrong about
Before measuring, I was confident that a chunk of my noise problem was structural text. Directory listings, config blocks, base64, the stuff that is dense with rare subwords and therefore lands on that same generic vector. Roughly 21% of my corpus is that. My theory was that filtering it out would drag the noise floor down and let me lower the threshold.
It does not. I measured it both ways and the ceiling for nonsense was 0.525 with those chunks and 0.525 without them, because the highest-scoring accident was against ordinary prose all along. Excluding a fifth of the index bought nothing and very slightly reduced how often real questions matched.
Worth writing down, since it was a confident and wrong prediction, and I would probably have shipped it as an optimisation and believed it helped.
What I would tell anyone building this
Most writing about agent memory is about architecture. Which store, which tiers, episodic against semantic, which vector database. I found none of that to be where the difficulty was.
The difficulty is that retrieval and injection are not the same problem, and it is very easy to build the second while thinking about the first. Retrieval optimises for finding something. Injection has to optimise for being right, which means most of the engineering goes into deciding when to say nothing. Every hard bug I hit was a version of that.
And measure it against a real corpus before you tune anything. I had 120 synthetic test prompts that all passed. The failure was found by one actual question I asked in the course of my own work, which is not a coincidence: real questions contain the names of real things, and synthetic ones almost never do.
Where this ended up
This is all from building Inventory, which indexes the conversations your AI coding tools already write to disk and, as of this release, tells them when you have solved something before. Most usefully, when you tried something and it did not work, which is the one thing that leaves no commit and no file behind and therefore cannot be recovered from your repository at all.
Everything above runs on your own machine. That is not a positioning decision, it is why the rarity trick works: distinctive is measured against your history, and there is no shared corpus for it to be measured against instead.