The Verdict Index

Every search over an AI conversation history ranks by similarity, whether that is BM25 keyword scoring, embedding cosine distance, or the hybrid of both that most modern tools, including Inventory until this release, actually run. Cursor's own semantic search work confirms the hybrid is the right default — their agent "makes heavy use of grep as well as semantic search, and the combination of these two leads to the best outcomes" — and Russ Cox's trigram-index work is the classic statement of the pattern underneath both: a cheap index narrows a huge corpus to a small candidate set, and something more expensive ranks what is left.

None of that ranking asks the one question that actually matters for a debugging history: did this answer work? A conversation where the first suggestion was wrong and the second fixed it produces two passages that are both highly relevant to the same query. Similarity cannot tell them apart, because it was never given anything to tell them apart with. Handed to an agent, the confident, well-worded, wrong answer is exactly as likely to surface as the one that actually worked, and an agent will cheerfully re-apply it.

Supermemory's LongMemEval results are the clearest evidence this is a real gap and not a theoretical one. Across their benchmark, "Knowledge Update" was the category where a full-context baseline did worst (78.2%) and their outcome-aware retrieval did best (99%) — the hardest thing for a memory system to get right is knowing which fact superseded which. Their benchmark is chat memory for a single assistant. Ours is different in a way that makes the problem sharper: six tools, the same debugging session sometimes spread across more than one of them, and the "which fact superseded which" question has a very concrete answer — did the error come back or not.

Every assistant message gets a verdict: confirmed, rejected, or open (no signal either way, the default and by far the most common case). Two independent passes produce it, and neither is a model call — both run entirely on-device, in the same pass that already parses each conversation.

Explicit. The message immediately after an answer is checked against two short, hand-written phrase lists — confirming ("that's it", "found it", "you were right") and rejecting ("still failing", "same error", "didn't work"). Rejection is checked first, because "that worked, but it's still failing on retries" is a rejection, not a partial credit.

Implicit. This is the one we think is genuinely new. Most real replies contain no verbal verdict at all — people do not narrate outcomes, they act on them. So when a user's message carries a pasted error, the answer that preceded it is checked against what comes after: if the same error signature appears again later, the fix is read as rejected. If it never appears again while the user goes on to paste different errors — proof they kept working rather than closing the laptop — the fix is read as confirmed. Error signatures strip anything that changes between retries (attempt counts, durations, hex ids) so the same failure collapses to the same signature across repeated attempts.

Both are deliberately biased toward silence. A missed verdict costs a small ranking nudge; a false one promotes a wrong answer to the top of a search and — worse, over MCP — tells an agent the matter is settled. Confirmed and rejected are stored with a confidence weight (explicit counts double an inferred one), and ranking shifts by at most 1.5 points on a bm25 scale that already spans several points — enough to resolve a near-tie in favor of the answer that worked, not enough to override a search that is not close.

Everything below comes from one run of a small, throwaway measurement tool against 500 real conversations already on a real Mac — 17,192 assistant messages across Claude Code, Cursor, Codex, Zed, Kiro and Antigravity. It reads each source tool's own plaintext files directly, the same files Inventory's indexer always reads, and never touches Inventory's own encrypted index. Nothing about this required a benchmark harness or synthetic data; it is the honest answer to "does this fire on real conversations, and is it right when it does."

Read that table for what it says, not more. 2.5% coverage sounds small next to a benchmark like LongMemEval's 95% recall — but that number measures retrieval over a labelled question set, and this one measures how often unprompted human behavior in ordinary use contains a detectable outcome at all. The honest comparison is not "our number vs. their number." It is that most conversations simply never state or exhibit an outcome, and for the ones that do, we now use it, where every other tool in this category discards it identically to everything else.

The chart is the finding, not just an illustration of the table: the two lighter bars — fixes read from the error never recurring, failures read from it recurring — outnumber the two solid bars by roughly 8 to 1. Almost everything this feature knows, it knows from watching what happened next rather than from being told.

The first version of the explicit matcher looked correct: fifteen hand-written test cases, all passing. Running it against real conversations found bugs the hand-written cases never would have, because real phrasing is stranger than anything we thought to write ourselves. Three examples, in the order we found and fixed them:

• "how can i found it in logs"

  Matched as a confirmation, because it contains "found it." It is a question. Fix: two short marker words ("found it", "fixed it") only count at the literal start of a reply now, where genuine declarations overwhelmingly sit — not anywhere in the first 60 characters, which is where questions mentioning the same words also live.

• "that's it? what about httproutes?"

  Matched as a confirmation. Also a question — the punctuation said so, but the matcher had already stripped it before looking. Fix: check the original text for a question mark before checking the normalized text for a phrase, not after.

• a task-notification block injected by tooling

  Agent transcripts contain machine-generated messages that are not the user speaking at all. One was being read as a verdict on the answer before it. Fix: anything shaped like injected tooling output is excluded from both passes before any matching runs.

Each of these is now a permanent regression test, not a note in a changelog. The measurement also surfaced a subtler finding worth stating plainly: on this dataset, rejection phrasing in real usage ("still failing," "same error," "no luck") was reliably unambiguous, while confirmation phrasing was not — "that's it" is used at least as often to mean "just this, nothing more" as it is to mean "this fixed it." That asymmetry is why the implicit, behavior-based pass carries most of the real coverage: 380 of 428 total verdicts, against 48 from words alone.

A comparison table with invented percentages against tools we have not benchmarked would be worse than no comparison at all. This one is architectural — what each system is actually built to do, sourced from what each team has published, not from a shared test we all ran.

The "knows if an answer worked" column is the actual claim, and it is a narrow one on purpose. We are not claiming better retrieval than Cursor's trained embeddings — we have not run their benchmark and would not trust a number we produced on it ourselves. We are claiming a signal that, as far as we can find in what each of these teams has published, none of them extract: whether the thing that was suggested actually happened to work, read from the conversation's own aftermath rather than inferred by a model.

2.5% coverage is low in absolute terms, and the reason is structural rather than a bug to fix: most real conversations simply do not contain a stated or actable outcome. Someone drops a thread, moves to a different problem, or closes the laptop, and there is nothing left to read. No phrase list or behavioral heuristic invents a signal that was never there.

The explicit matcher is phrase-based, not semantic — a confirmation phrased in a way that resembles neither list is invisible to it, on purpose, since the alternative is a model call we do not want to make locally for every message. The implicit pass only understands pasted errors specifically; a UI bug confirmed by a screenshot, or a design decision confirmed by "I shipped it," produce no behavioral trace at all today.

This measurement is one machine's real history, not a held-out benchmark with labelled ground truth, and we have not run a blind human evaluation of precision on the confirmed set — every example in the false-positive section above was found by us reading our own output, which is a real check but not an independent one. If you run this against your own history and it gets something wrong, we want to know; the fixes above all came from exactly that process.

• Russ Cox, "Regular Expression Matching with a Trigram Index"
• Cursor, "Semantic Search"
• Cursor, "Dynamic Context Discovery"
• Cursor, "Secure Codebase Indexing"
• Supermemory, "LongMemEval Results"
• turbopuffer, turbopuffer.com