Data retention arrives.

2026-07-23 · by Dennis Gubsky · v1.32.0 · ~13 min read

Loomcycle accumulated data forever in several places. Some of that was fine (an audit log should not silently disappear); some of it was less fine (a retired agent's memory outlived it forever). The sharpest case: every one of the nine versioned def stores could soft-retire a version, but none of them could hard-delete one. A tenant that had spent a year iterating on an agent had, by design, an immortal record of every intermediate version. Storage grew monotonically.

v1.32.0 ships RFC BM, the data-retention subsystem. It is a first-class, config-driven sweeper that ages out data across three families. Every family defaults off. Every deletion goes through export-then-delete: if the JSON archive fails to write, the row does not delete. And a post-merge review over the whole line confirmed no data-loss bugs and caught two P3 completeness bugs before ship. This post walks the design and the review findings.

The shape: a periodic goroutine, not a schedule

The natural intuition is to hang retention off the loomcycle scheduler. The scheduler already fires periodic work and it already has cluster coordination for singleton sweepers. The problem is that the loomcycle scheduler only fires agent runs. It is the surface an operator uses to say "every night at 3 AM, run this AgentDef." A retention sweep is not an agent run; it is substrate maintenance that should not consume the scheduler's admission slots or show up as spurious agent activity.

So retention is a goroutine, not a schedule. Its cadence is LOOMCYCLE_RETENTION_INTERVAL_MS. Cluster coordination reuses the same advisory-lock singleton machinery the other sweepers use, so on a multi-replica cluster exactly one replica sweeps per tick.

The three families sweep independently, each with its own mode (off | prune | export+prune) and its own age cutoff. So an operator can enable defs-purge without touching chats, and enable chats without touching memory. Every family defaults off. Nothing is removed until the operator opts in.

Modes work how they read. off: the sweep does nothing for this family. prune: delete rows that qualify. export+prune: write a JSON archive to LOOMCYCLE_RETENTION_EXPORT_DIR, then delete. The order matters. Export-then-delete means a failed export never deletes the row. An operator who wants a durable record of what left the DB gets one; an operator who does not can pick prune and skip the archive.

P1: retired def-version purge

Nine of loomcycle's substrate def stores share the same shape. AgentDef, SkillDef, TeamDef, MCPServerDef, ScheduleDef, A2AServerCardDef, A2AAgentDef, WebhookDef, MemoryBackendDef. Each holds versioned records: an active pointer per (tenant, name), plus a chain of historical versions linked by parent_def_id. Retiring a version does not delete it; it sets a flag and moves the pointer. So every one of these stores had grown monotonically since loomcycle shipped its content-addressed versioning.

The two new methods and the safety properties

P1 adds two table-parameterized store methods that operate over all nine families with one implementation.

ListPurgeableRetiredDefVersions returns the set of versions that qualify for purge. A version qualifies only when all four of these hold: it is retired; its created_at is older than the cutoff; it is not the active pointer for its (tenant, name); and it is beyond keep_last_n for its (tenant, name) (default 5, so the five most recent versions of any def line survive purge regardless of age).

One more property, and this is the correctness anchor of the whole design: the query is leaf-only. A version is excluded while any version still references it as a parent_def_id. So a retired chain drains leaf-first over successive ticks. Purging a retired parent while a live child still points at it as ancestor is impossible by construction; the self-FK is never tripped; a surviving child's lineage is never orphaned.

DeleteDefVersions takes a list of IDs and deletes them in a single transaction. Every one of the nine tables is allowlisted; a caller cannot pass an arbitrary table name and delete from anywhere. The allowlist plus the leaf-only enumerator plus the four qualification conditions is a small, verifiable surface.

No migration. The queries and deletes run against the existing tables. Env: LOOMCYCLE_RETENTION_ENABLED, _INTERVAL_MS, _EXPORT_DIR, _DEFS_MODE, _DEFS_MAX_AGE_MS, _DEFS_KEEP_LAST_N.

P2: aged chat retention

A chat is a session (session → runs → events, the same shape the History tool operates on). A session accumulates as agents run against it. Some sessions are one-off; some sessions are long-running and go dormant; a few sessions are ones the operator has pinned because they are load-bearing knowledge.

P2 archives aged chat sessions. A session qualifies when all its runs are terminal (no in-flight or paused runs) and its latest completed_at is older than the cutoff. On qualification the sweep exports the session plus its runs plus its events as one JSON bundle, then calls DeleteSessionCascade.

Why the prune boundary is a session, not a run

This one is worth naming because the naive design gets it wrong. The tempting shape is "prune aged individual runs." That would fail for a real reason: the continuation path (a new run against an existing session_id) replays the whole-session transcript to reconstruct the model's view of the conversation. If a session has ten runs and you prune the four oldest as aged, the eleventh continuation run replays a transcript with holes in it. The model sees a mangled history. The transcript is corrupt as a data structure.

So the prune boundary is a session. Either the whole session goes (all runs, all events, one bundle) or it stays. This is why P2's qualification predicate insists on every run being terminal, not just some subset.

Pinned exemption

A pinned session is never pruned by any path. The exemption is in PrunableAgedSessions at the store layer, not at the retention-sweep layer. That placement matters: the pre-existing RFC AV usage archiver (which also cascade-deletes aged sessions) reads the same enumerator. So the pinned exemption protects both paths automatically.

In fact, the legacy RFC AV usage-run-retention archiver is subsumed here. Its LOOMCYCLE_USAGE_RUN_RETENTION_* config aliases into retention.chats, but only when it was genuinely delete-enabled: an operator who had the archiver disabled keeps it disabled. main.go makes exactly one path prune, so a session is never cascade-deleted twice. Exact back-compat: no operator sees a behavior change unless they turn a retention family on.

Env: _CHATS_MODE, _CHATS_MAX_AGE_MS.

P3: retired-agent memory reclamation

This is the hardest phase, because the target data lives across three different stores under two different tenant-keying schemes. An agent's accumulated per-scope data includes its SQL-Memory scope (per-scope SQL tables plus document structure), its dirents (Path names it registered), and its base-memory key-value store (learned facts plus document chunk bodies). Reclaiming a retired agent's data means finding all three, honoring both keying schemes, and never touching data that a living agent still needs.

What "retired" means for this reclaim

A retired agent, for P3's purposes, is one where AgentDefListNames reports LiveVersionCount == 0. Every version of the agent's name is retired. This is stricter than "the active pointer is retired" and by design: as long as one version of the def is still active in any tenant, the agent's name is in use, and its memory could still be legitimately consulted.

The two keying schemes and how P3 handles each

The SQL-Memory scope and the dirents are keyed by the triple (tenant, "agent", name). That is the standard tenant-qualified keying every other loomcycle primitive uses. On reclaim, P3 calls a new sqlmem.Manager.DropScope that drops the SQL tables plus document structure for that triple, and calls DirentDeleteUnder to remove the dirents. Both are tenant-safe by construction: a same-named agent in a different tenant is untouched.

Base-memory is different. The memory table has no tenant column. It is keyed by the bare agent name. Two tenants running same-named agents (which happens more than you would expect, because bundle names like chat/medium are the same for every tenant that loads the bundle) share the row. So base-memory can only be dropped when the name is retired in every tenant. P3's base-memory reclamation calls a new store.MemoryDeleteScope, but gates on the global-death predicate: no tenant anywhere still has a live version of this name. Embeddings cascade via FK.

That gate is P3's most important safety property. A tenant whose chat/medium is retired locally does not lose the base-memory shared with the tenant whose chat/medium is still active. Only when the last tenant retires the name does base-memory get reclaimed.

Sweep order matters: mem before defs

The mem sweep runs before the defs purge on the same tick. If defs ran first and keep_last_n were set to zero (an aggressive operator's choice), the last remaining def row for a fully-retired agent could be deleted before the mem reclaim keys off it. The mem reclaim would then have nothing to enumerate; the agent's data would silently persist forever with no live def to point at it.

Ordering mem before defs closes that. The mem sweep reads live def counts, does its reclaim, and then the defs sweep purges. Even at keep_last_n=0, no agent's memory is orphaned by same-tick sweep order.

Env: _MEM_MODE, _MEM_MAX_AGE_MS.

Why Document age-retention was dropped

An earlier draft included a fourth family: age-retention over Documents. It was cut before ship for two reasons.

First, the loomcycle Document store is now the primary home for the RFCs, the guides, and the operator research. Age-retention on those would be actively dangerous: a document nobody has touched for six months might be a load-bearing reference the next agent needs.

Second, documents do not have a cross-scope age enumeration surface. The Document store enumerates by chunk relations and by Path; it does not know how to answer "give me every document nobody has read since a cutoff." Adding that surface just to enable a retention family we did not want was a bad trade.

Explicit delete_document remains. Operators who want to purge a specific document call the tool directly.

The /v1/_retention read surface

GET /v1/_retention is the operator's window into the retention subsystem. Two audiences read it. A tenant operator sees the config: which families are enabled, at what cutoff, in what mode. An admin sees more: a per-family dry-run purge preview (what would be purged on the next tick if the sweep ran now, without actually purging), plus the separate always-on SQL-Memory GC that operates orthogonally to RFC BM.

No adapter twin. The endpoint is HTTP-only; the operator surface is a read view, not a wire method a client needs to call on the hot path. So @loomcycle/client and Python are unchanged (1.30.0 and 1.25.0 respectively).

The post-merge review

Deletion code deserves the strictest possible review, because a false negative on a retention rule means "your data stayed longer than you wanted" while a false positive means "your data is gone." A full-feature code review ran over P1 + P2 + P3 as a single unit. The good news up front: no data-loss bugs across all three phases. Every destructive path was correctly scoped; every defect the review found failed toward under-reclamation, not over-reclamation.

Two of those under-reclamation defects were significant enough to name.

Base-memory reclamation was gated on MemoryListScopeIDs. That function has a hardcoded ORDER BY updated_at DESC LIMIT 200. Nothing on the surface said "this is a windowed read"; the function name implies it enumerates. So the P3 gate silently skipped exactly the stale retired agents it was targeting: agents whose updated_at was old enough to fall outside the 200-newest window. The fix rewrites the enumerator to be genuinely exhaustive, with a fail-before regression test that stages more than 200 retired agents and confirms all of them are reclaimed.

The defs-before-mem sweep order could orphan memory under keep_last_n=0. Documented above; the fix is to run mem before defs. The regression test stages an agent with keep_last_n=0, retires every version, runs a single tick, and confirms both the memory and the defs are gone.

Two lower-severity fixes shipped too. An export truncated at the key cap no longer proceeds to delete more than it exported: if the archive writer hit its cap partway through a batch, the sweep now stops the delete at the same watermark rather than deleting rows that never made the archive. And the usage-archiver reconciliation only claims chats ownership when the retention chats sweep will actually run: a config-only alias without an enabled sweep does not disable the legacy archiver, so an operator halfway through the migration does not accidentally stop pruning.

What's next

Three things are queued around the retention line.

Per-tenant retention configuration. Today _MODE and _MAX_AGE_MS are process-level. A future revision lets a tenant operator set their own retention policy for their own data, gated by admin approval where it needs to be.

Retention on Documents behind a cross-scope enumeration surface. If the Document store gains a proper "list documents by last-read timestamp" op, an age-retention family for Documents becomes safe. Not queued for a specific version; blocked on the enumeration op.

A read view that surfaces the last N tick outcomes: how many rows purged, how many exported, how many refused because they held a live child ref. Today GET /v1/_retention reports a dry-run preview but not a historical record. Small change; queued as a v1.32.x point release.

Companion reading: the Document viewer for the reading surface that made age-retention on Documents an active decision; the History tool for the pinned-session model P2's exemption piggybacks on; the cost ledger for the RFC AV usage archiver P2 subsumes; and docs/CONFIGURATION.md for the full LOOMCYCLE_RETENTION_* env reference.