Testing memory in the real world.

2026-09-03 · by Dennis Gubsky · memory-testing arc · ~13 min read

Unit tests are consistent by construction. Each one sets up its data, exercises one code path, and asserts an equality. Every test suite that says "green" has been consistent with itself. Whether it was consistent with the world it claims to model is a different question, and one only real corpora can answer.

Over the past month loomcycle grew three benchmark harnesses. Each one loaded real content into the memory subsystem and asked "what does the runtime think happened here?" Each one found a defect that no unit test had caught, because the defect was in the seam between two consistent halves. This post tells that story chronologically, and it is worth reading in that order: none of what follows was predicted from the code.

The versions covered: v1.60.0 through v1.75.0. Roughly a month of releases.

The starting question

Loomcycle's memory subsystem had been shipping richer surfaces for months by the time this arc started. Facts, notes, and documents in one search. Bi-temporal validity. A background consolidator turning transcripts into distilled facts. Provenance on every write. A hybrid vector + full-text retrieval with Reciprocal Rank Fusion.

Every one of those had unit tests. Every one of those tests passed. And every operator who deployed it on real content ran into the same kind of report: "I asked the agent about a person we discussed last month and it said it had no record." The agent had a record. The retrieval could not find it. Or the record was written but the entity graph could not walk to it. Or the extraction had produced facts but they were filed against the wrong subject.

None of these were reachable by a test that constructs its own row. All of them were reachable by a benchmark that loads a real conversation and asks "did the runtime read this correctly?" So the first thing was to write the benchmarks.

Three harnesses, three questions

LoCoMo: the memory-retrieval baseline (v1.60.0)

LoCoMo is a public evaluation corpus for long-term conversational memory: scripted multi-session dialogues with a labelled question set. The right answer to each question is stated somewhere in the transcript; the runtime's job is to find it later, in a different session. That is exactly what an agentic memory subsystem should be measured on.

The v1.60.0 harness loads a LoCoMo run, feeds it turn-by-turn through a chat agent whose memory is the tenant's own store, and then poses the question set to the same agent in a fresh session. The number that comes out is a straight accuracy on labelled answers.

What LoCoMo bought immediately was a way to talk about memory quality with a real number attached, rather than a phrase like "seems to work." It also became the reference corpus for the extraction eval that had been shipping since RFC BL P4a.

Wikidata: knowledge-updates, ontology-typing, cross-lingual (v1.67.0)

LoCoMo tests memory of a conversation. It does not test whether a runtime keeps up when the world changes. Wikidata's edit history does. The v1.67.0 harnesses use Wikidata's own claim history:

• Knowledge updates. Take a fact whose value changed between two Wikidata snapshots. Feed the pre-change statement, then feed the post-change statement. Ask the runtime which value is current. This is the corpus for RFC CL's bi-temporal columns: valid_at, invalid_at, and the as_of predicate that was landing in the same release window.
• Ontology typing. Take a subject with a known Wikidata type. Feed a claim about it. Check what type the loomcycle extractor assigns.
• Cross-lingual recall. Feed a claim in language A. Ask a question in language B. Check whether the fact still surfaces.

Plus a bulk fact-corpus builder and importer, because none of these harnesses is useful at 20 rows. They needed thousands.

RFC CS long-horizon: context retention across many turns (v1.68.0)

LoCoMo scripts sessions. Wikidata is one-fact-at-a-time. Neither tests the case that matters most for chat agents: one long conversation, where the answer to turn 200 depends on something stated at turn 5, and the model's context window closes long before then. RFC CS defines a synthetic long-horizon corpus with planted answers at increasing depths, and measures whether the runtime keeps track.

This is the harness that measures the value of context distillation. The retention modes shipping in RFC CR (recap, stateful, compaction) each drop something on the way. RFC CS is what lets us say whether the drop cost the answer, and how far back the retained context can reach.

What the harnesses found

None of the three harnesses landed on a clean run. Each of them surfaced a defect on its first non-fixture data, and each defect had a shape unit tests could not have caught. In chronological order:

Finding #1: five subjects for one person, and 94% of claims on multiply-typed subjects (v1.68.0 → v1.69.0)

Wikidata's ontology-typing harness ran on a 142-claim scratch corpus. It reported that one specific person existed five separate times in the store, filed under five different entity types: event, location, object, organization, and person. Between them the five subjects held 89 claims. 94% of all claims in the store hung off a multiply-typed subject.

"What else do we know about her" is the question the entity tier exists to answer. This corpus could reach at most one-fifth of what it stored.

The cause was straightforward once measured. A subject node's natural key is type + ":" + slug, so the type is part of the identity. Each extraction call picked a type fresh, with no knowledge of how that subject had been typed before, and nothing reconciled them. The extractor for a claim about "Ada's presentation at the conference" might reach for event; a claim about "Ada's office" reaches for location; the same person, filed under two names.

The fix (v1.69.0) is one rule: the type a subject already has wins over the current call's guess, for both the natural key and the type field. First write wins, deliberately, because any rule that can change a subject's type later re-partitions every fact already filed under the old one. A wrong-but-stable type keeps a subject's facts together; that is what matters.

A separate half of the same investigation: the extractor sometimes invents entity types the tenant has not declared. The old upsert_chunk gate refused those writes, so the fact landed in key/value with no graph presence at all. The message the gate itself printed was that an undeclared type "becomes a node nobody can find." But refusing produced no node, which lost the subject entirely. The fix files the invented type as an inert ontology candidate (an operator can accept it later) and writes the subject under a fallback type. An invented kind becomes something a person can adopt rather than a silent loss.

Finding #2: the extractor never said who THEM was (v1.70.0)

A two-speaker corpus fed through v1.68.1. The extraction prompt carried a self-guard rule: "a fact about THEM takes the subject user." The rule was worse than no rule.

The declared identity names live in the user-root document, reachable through {{memory:user_info}}. That template does not reach the extractor's system prompt. So the model was asked "which speaker is THEM?" with no data to answer with. It guessed the more prominent speaker.

In a scope whose profile declared Dave as the owner, the model made user = Calvin. It inverted the self-guard: it protected Calvin's facts as if they were the owner's, and left Dave's (the actual owner's) placeable. Protecting the wrong person while exposing the right one is worse than doing neither.

The v1.70.0 fix threads the declared names into the extractor's own context. Context op=self now reports self_names, parsed server-side by the one function that owns the column-0 rule that stops the template's own indented example from naming every profile after it. Names are omitted rather than empty when nobody declared any, so a caller can tell "nobody said" from "said nothing." The prompt then asks for consistent spelling rather than inviting the model to identify an owner it cannot know.

The measurement that closed the finding: on the same corpus that had produced zero facts on the local extractor, a cloud model produced 74 facts on the same input. So the earlier "typing is inconsistent" reading had been measuring a model barely functioning, not a design defect. With a capable extractor: 62/62 subject linkage, 0% of claims on a multiply-typed subject, repeat-subject consistency 1.00 against 0.00 before. Two independent bugs had been stacked on top of each other and pattern-matched to one cause.

Finding #3: the consolidator held the mechanism but not the grant (v1.71.0)

RFC CQ (v1.68.0) shipped ontology-declared memory placement: an entity type in the tenant ontology can declare which memory scope facts about that kind of thing belong in. A fact about a shared service goes to the tenant plane; a fact about the user stays in the user scope. Mechanism, versioned config, operator control, propose-adopt for undeclared types.

One problem: nothing was ever placed in any deployment. The mechanism worked, the ontology declarations parsed, the resolver returned the right decision. The consolidator held memory_scopes: [agent, user] and no sql_scopes at all. A placed fact is stored twice (a k/v row that recall searches, and a typed chunk that a graph walk reaches), so tenant has to be on both grants or the two halves cannot land in the same scope. The bundle had neither.

The absence had been deliberate, argued as "an unused grant is the capability an injected instruction reaches for." That argument is written for a prompt-driven agent, where model output steers control flow. The consolidator is provider: code-js: a deterministic body in which model output is data that gets written, never code that gets run. What bounds the tenant write is the placement resolver, which fails closed on every uncertainty. The grant is inert on its own until an operator declares a type-scope pairing. So the fix (v1.71.0) is to grant both.

The measurement on a two-user corpus after the fix: duplication fell from 30 duplicate copies to 23 (cross-user duplication eliminated, replaced by tenant-owner duplication, so the honest figure is 23% reduction, not elimination). The real payoff is sharing: 94 facts readable by both users where 0 were before. A second-order effect nobody predicted: cross-user type stability went from 4 for the user who wrote first to 91 for the second, whose extractor kept proposing new types for subjects already on file in the shared plane.

Finding #4: the benchmark accepted a rigged partition (v1.75.0)

The most recent finding, and the one that shows why benchmarks need guards of their own. The answer axis of a memory benchmark already refused an empty store: an accuracy of 0.0000 on empty is a fact about plumbing wearing the costume of a fact about memory.

That guard was not enough. With the corpus tenant's ontology declaring a tenant memory scope for its own entity types, the consolidator did the right thing and placed most facts into the tenant scope. The answerer, on the same run, recalled from the user scope only. Enough rows remained in the user scope for the empty check to pass, and the run scored 0.0216 with 95% abstention. Three plausible mechanisms were reasoned on top of that number before anyone compared the writer's counter to the store.

The new guard compares them: how many facts the pass reported writing against how many the benchmark can actually reach on the answer path. When the two disagree by more than a small tolerance, the benchmark refuses to report a score: the partition is rigged and the number would misrepresent the memory subsystem. The scaffolding is now what refuses; a design change is what fixes.

The one-tenant-mapping fix from the same release is the other half of the same class of bug: four separate implementations of the "" → "default" tenant mapping, each with a comment naming a different source of truth. They agreed, byte for byte, which is why the defect stayed latent. Divergence is not cosmetic, and the erasure call site already spelled out why: a DropScope built from a raw "" tenant matches nothing, so a single-tenant deployment's subject erasure would leave the subject's entire SQL Memory database in place while reporting success. The rule now lives once, in internal/sqlmem.

The pattern

Four defects, four fixes. The shared shape is worth naming.

Each defect lived in a seam between two consistent halves. The extractor was self-consistent. The user-root document schema was self-consistent. The join point that would have carried Identity names from one to the other did not exist. The consolidator's memory-write path was self-consistent. Its sql_scopes gate was self-consistent. The requirement that a tenant placement needs both was not enforced anywhere; it was implicit in the two-halves-of-a-fact design.

A unit test could have covered each half. A unit test would have missed each seam.

Each defect surfaced on the first real corpus. Not the second. Not the tenth. The very first Wikidata run showed multiply-typed subjects. The very first two-speaker LoCoMo run showed the extractor guessing at the owner. The very first ontology-typed corpus showed placements missing. This is a general property of real content: it is composed of many small joint distributions the tests never sampled from, and one of them is almost always adversarial to the code.

Each fix is small. One rule (first-write-wins for subject type). One added field (self_names on Context op=self). One config change (grant the consolidator sql_scopes: [agent, user, tenant]). One counter comparison (writer versus store on the benchmark answer path). None of them changes the design. Each of them changes what the design does when it meets real data.

The story is that testing memory is not the same as testing a data structure. A data structure has correctness. A memory has correctness relative to the world it was told about. Constructing that world is what a benchmark does, and it is what a suite of unit tests cannot do by definition, because a unit test is precisely a test whose world it constructed itself.

What runs now

As of v1.75.0, three harnesses are wired into the eval track:

Plus a bulk Wikidata fact-corpus builder and importer, because the harnesses need thousands of rows to be useful.

What isn't yet measured

The honesty section, because a testing story that ends "and now everything is measured" is a bad one.

Ontology placement's value is still unmeasured on a store whose typing was human-audited. The 23% duplication drop was measured on a two-user corpus after RFC CQ's placement gate was known to fail closed on every uncertainty. Whether the payoff generalises to a store where a human decides the typing is a different measurement, and it needs a different corpus.

The self-guard's promise is defeated in a multi-party corpus. Each user played one speaker. The guard correctly kept a speaker's own facts in their scope. The other user, for whom that speaker is a third party, published the same facts to the tenant plane. So placement left the owner's facts more exposed than baseline, which is precisely what the guard exists to prevent. Not a bug: the consequence of per-scope decisions with no global view.

Extraction yield on longer conversations is unmeasured for the local extractor. 1,136 turns of input across two users produced four facts on the shipped local model; the same input on a cloud model produced dozens. What extraction rate a real deployment gets depends on which extractor it runs against, and the harness surfaces that dependency but does not currently score it.

Those three gaps are the material for the next month.

The bridge

The memory story here is one half of a two-post arc. This post is the chronology: what benchmarks landed, what they found, what got fixed. The companion post is the shape: what the memory subsystem now looks like as a system, with structure diagrams of the planes, provenance envelopes, bi-temporal validity, ontology placement, and how a fact harvested from context distillation (RFC CT P2) becomes a durable memory row.

And the context-distillation story that feeds those harvested facts is a third post: how the reasoning-recap, structured-state, and compaction modes discard work by design, how the Recall tool builds a run-scoped index over what was evicted, and how RFC CT P2's context.harvest_to_memory banks those evicted spans onto the consolidation queue so a distillation drop can outlive the run.

Companion reading: agentic memory (v1.33-v1.49 story), the external data plane (v1.54), PR #1000 milestone.