Warm containers, cloneable agents, resident sub-agents.

2026-07-21 · by Dennis Gubsky · v1.23.2 → v1.28.0 · ~17 min read

Three arcs shipped this week that read separately in the changelog and read as one thing in the wild. The sandbox sidecar grew workspaces, run tagging, keepalive, and bulk teardown. The Library gained a Clone action that lets an operator derive a personalized agent whose tool list can be edited and widened. The Agent tool learned three new ops that keep a helper sub-agent resident across turns instead of re-spawning it every time. Put those three primitives on the same run and a compile-test-fix loop that used to lose its container between commands now keeps everything warm: the child stays alive, its build cache survives, and its parent never has to restate what the compiler is being asked to do.

This post walks the three arcs in order (sandbox hardening, agent cloning, resident sub-agents) and then shows what they add up to.

Part 1: the sandbox sidecar grows up

RFC BI P1 shipped in v1.23.0 as an isolated builder sidecar: a standalone service loomcycle drives over HTTP-MCP, spawning per-session containers with --network none, read-only rootfs, tmpfs /work, every Linux capability dropped, and a pluggable runc / runsc / kata runtime. It solved the isolation problem. It left three things unfinished for real-world use.

First, the workspace was always ephemeral. A tmpfs /work is right for a single-shot run of a Python script; it's wrong for a Go module that ships with a 400 MB build cache the compiler wants to reuse across turns. Second, sessions were untagged. If an operator wanted to say "close every sandbox this run opened," there was no way for the sidecar to answer the question, because it did not know which sessions belonged to which run. Third, the sidecar and loomcycle had to come up in the right order at boot: if the sidecar started second, its mcp__sandbox__* tools were absent from every fresh model prompt for the lifetime of the loomcycle process.

P2 addresses all three across v1.23.2, v1.24.0, and the two follow-up patches.

Durable workspaces

sandbox_open gains an optional workspace name. When set, /work is bind-mounted from <SANDBOX_WORKSPACE_ROOT>/<principal>/<name> instead of a tmpfs. The path is charset-gated (no dots, no slashes inside the name), symlink-escape-checked, and always under the caller's own principal subtree, so one tenant can never open another tenant's workspace by guessing the name. Reopen the same name later (same principal) and the same directory is remounted: your checkout, your build artifacts, your Go module cache, your Cargo target directory, your npm store are all where you left them.

{
  "tool": "sandbox_open",
  "input": { "workspace": "kvflow-checkout" }
}
// container is disposable; the work at /work persists on the host disk under
// SANDBOX_WORKSPACE_ROOT/<principal>/kvflow-checkout

Everything else about the P1 hardening still applies. The container is still per-session, still runs with --cap-drop=ALL, still has no network, still runs non-root. Only /work is bind-mounted; the rest of the filesystem stays ephemeral and read-only. And durable workspaces are off by default: unset SANDBOX_WORKSPACE_ROOT on the sidecar and every session is tmpfs-only, byte-identical to v1.23.0.

The sidecar also learned to redirect build-cache environment variables (GOCACHE, CARGO_HOME, and the equivalents for npm) into the workspace itself, so caches accrue in /work and follow the workspace across container reap and sidecar restart. The Go module cache is what makes this concretely worth it: a workspace with 300 MB of module cache and go build intermediate objects is the difference between a 12-second incremental rebuild and a 3-minute cold rebuild.

Run tagging and bulk teardown

The sidecar now tags each session with the loomcycle run tree that opened it. Not by taking a run ID as a tool argument (a compromised or confused model could lie about that) but by reading an X-Loom-Root-Run HTTP header that loomcycle sets on every MCP request. The header is server-set, propagated from the run's own identity, and the model never sees it: it can neither read the value nor influence it. The sidecar reads the header, stamps the tag on the session record at sandbox_open, and treats it as immutable for the session's lifetime.

Two new ops use the tag. sandbox_touch {session_id} resets a session's idle timer without executing a command. It's a keepalive for a caller that knows it will come back but not immediately. And sandbox_close_run {root_run_id} bulk-closes every session the caller owns under a given run tree, principal-scoped so cross-tenant close is refused, empty-guarded so a bad root_run_id can't cascade beyond its owner, idempotent. A team-orchestrator agent finishing a workflow can now close every sandbox its team opened without tracking session IDs by hand.

Non-secret run identifiers also propagate to any operator MCP server through header substitution. The MCP HTTP client resolves ${run.root_run_id} and ${run.tenant_id} in mcp_servers headers at request time, substituting empty when the run carries none (never dropping the header). Any MCP server the operator points at can attribute a request to a run tree or a tenant without loomcycle having to add a bespoke integration.

Static MCP re-probe at run start

This one is a bootstrap-order fix that mattered more than expected. When loomcycle starts, it dials every declared MCP server during a bounded 30-second window and registers whatever tools it learns. If the sandbox sidecar (or any other static MCP server) is not yet reachable, loomcycle logs "skipped" and moves on. The tools that server would advertise are never registered for the process lifetime, which means every fresh model prompt for the rest of the run is missing them. A restart fixes it. Waiting does not.

Dynamic substrate servers and A2A peers already got a bounded run-start re-probe for a different reason (F33). v1.24.0 extends the same mechanism to static YAML servers. On a run start, if the run's agent references a skipped server, the sidecar retries once under the run's tenant, honoring the operator's tools: filter: cached tools when warm, one bounded handshake otherwise. A server that comes up after loomcycle recovers on the next run. No restart. Servers the run does not reference are never dialed; healthy boot-registered servers are never re-dialed. The observed case that motivated it was clean: the builder sidecar starting a moment after loomcycle left mcp__sandbox__* absent from Context op=tools until reboot.

The three small fixes that made the sandbox usable in practice

Field use exposed three defects that would have been much harder to find without real workloads driving the sandbox. Naming them because "the sandbox works" turns out to be a specific claim.

The tmpfs was root-owned (v1.25.1). The default ephemeral session mounted /work as mode=0700 under root. The container ran as a non-root user. So every sandbox_write and every sandbox_exec file operation returned "permission denied," and only inline python3 -c commands ran. Docker's --tmpfs does not reliably honor uid=, so the portable knob is the mode: mode=0777. The mount is container-private and ephemeral, so world-writable is fine at this scope. Pull the v1.25.1 sidecar image; the fix lives there, not in the loomcycle binary.

The auth env var had two names (v1.25.1). The shared secret was named LOOMCYCLE_SANDBOX_TOKEN on the loomcycle side and SANDBOX_AUTH_TOKEN on the sidecar side. A mismatch surfaced only as a 401 on the MCP handshake and left an operator debugging their bearer for an hour. Unified on SANDBOX_AUTH_TOKEN for both sides. The sandbox bundle's Authorization header dropped its ${run.user_bearer} prefix: the sidecar authenticates on the shared secret, so a per-run bearer could only 401.

Login shell was resetting PATH (v1.25.2). The sidecar exec'd session commands through bash -lc. A login shell sources /etc/profile, which resets PATH to the Debian default and discards the session image's ENV PATH=/usr/local/go/bin:/usr/local/cargo/bin:.... So the toolchains under /usr/local/* (Go, Cargo, Rustc) fell off PATH, and go build returned exit 127 despite the sidecar's prompt correctly stating Go is preinstalled. Switched to bash -c (non-login) for exec, write, and read. docker exec inherits the container environment, so the image's ENV PATH plus the sidecar's --env overrides are both honored. Also drops the login-shell $HOME/.bash_profile permission warnings that were cluttering session logs.

Individually small. Together they are the difference between "the sandbox is deployed and no agent can use it" and "the sandbox runs a Go compile end to end."

Part 2: RFC BJ, clone an agent, keep the conversation

Loomcycle's bundled chat/* agents are opinionated and useful. They are also, by design, ceiling-locked to a specific tool set. An operator who wants to give one of them a new capability (say, the ability to reach a private MCP server, or the ability to run code via mcp__sandbox__*) has been stuck. Forking a def can narrow tools but not widen them, because a fork could otherwise be a self-privilege-escalation vector: an in-run LLM tricked into forking itself into a broader ceiling would be a real problem. So the answer for widening had to come from somewhere else.

RFC BJ shipped that answer across five merges in v1.25.0. The security invariant that only an operator on the bearer may widen a tool ceiling stays intact; the operator gets a first-class flow to actually do the widening.

The Clone action

A new per-row Clone action on Library → Agents. Clicking Clone opens a create-form pre-filled with the source agent's fields, including its tool list, which is now editable and may add tools that the source didn't have. Distinct from fork: fork creates a lineage version of the same identity that may only narrow tools; clone creates a brand-new independent agent with no lineage tie to the source, and can widen. The security seam holds because clone runs through the same authorization path as any other agent creation: the operator's bearer authenticates the widening, and an in-run LLM cannot trigger the flow through a tool call.

The Library's React package (@loomcycle/library) bumps to 0.2.0 for the Clone button and the widen-capable form.

Self-service skills for chat agents

Chat agents get SkillDef alongside Skill in their bundled tool grant. That means a chat agent can save a procedure it just executed as a reusable named skill, and load that skill on later turns via the on-demand Skill tool from RFC BA. The security invariant here is that the new skill's tools: must stay a subset of the authoring agent's own tools; a skill authored by chat/medium can only reach into chat/medium's ceiling, never outside it. So a chat agent can save its own workflows without escalating.

The fork ceiling check is now glob-aware

A subtle correctness fix that unlocks a real workflow. Before v1.25.0, when an operator forked an agent that had been granted a whole MCP server via a glob (tools: [mcp__sandbox__*]), the fork ceiling check ran against the resolved tool set (the concrete mcp__sandbox__open / exec / write / read / close / list). If the fork's proposed tools: also carried the glob (mcp__sandbox__*), the check refused: "concrete list cannot cover a broader wildcard." The result was that an agent with a glob grant could not be cleanly forked; every fork had to explicitly enumerate the concrete tools.

v1.25.0's ceiling check (assertToolsSubset, shared by AgentDef and SkillDef) is now glob-aware. It reads the raw tool patterns instead of the resolved set, so a fork keeping the same mcp__sandbox__* glob passes. A fork proposing a broader glob (say, mcp__*) is still refused: the invariant is preserved by the root-covers logic; only same-or-narrower wildcards pass.

Context replay across all transports

Cloning an agent is only half the story. The other half is bringing the conversation with you. If the source chat agent has 40 turns of context and you clone it to add mcp__sandbox__*, restarting from scratch throws that context away.

A new endpoint solves it: POST /v1/sessions/{id}/replay {agent, compress?}. It copies the source session's transcript into a new session bound to a (possibly different) target agent. The transcript is persisted as the new session's opening events, so continuation runs replay it exactly as if the conversation had happened in the new session originally. Provider-specific reasoning is stripped at the boundary (so an Anthropic thinking block does not leak into an OpenAI-bound clone). Tenant isolation is preserved (a cross-tenant replay folds to opaque not-found). And compress: true collapses a long history to a summary plus a recent tail using the same summarizer the History tool's recap op uses.

POST /v1/sessions/sess_abc/replay
{ "agent": "chat/medium-with-sandbox",
  "compress": true }
→ 200 OK
{ "session_id": "sess_new", "events_copied": 41, "compressed": true }

Every transport picks it up: HTTP as above, gRPC as ReplaySession, @loomcycle/client.replaySession(), Python replay_session. The wire method is new, which is why adapter versions bump to 1.25.0.

The operator algorithm, made explicit

RFC BJ Phase 1 landed CUSTOMIZING_AGENTS.md and a matching Context op=help topic=adding-capabilities topic. The algorithm names three cases explicitly:

• A skill within the ceiling is free. An agent can save a procedure as a SkillDef and reload it later, as long as the skill's tools stay a subset of the agent's own.
• A new tool needs a clone. Widening the ceiling requires an operator action (the Clone flow). An in-run LLM cannot trigger it.
• A dynamic agent can be widened in place. Dynamic AgentDefs (not bundled, tenant-scoped) accept operator-authenticated updates to their tool list without needing a clone.

The agent-side help topic exists so an in-run LLM asking "how do I get access to X?" reads the same algorithm the operator does, and answers "delegate to a sub-agent" or "ask the operator" instead of trying to widen its own tools via a tool call.

Part 3: RFC BK, resident sub-agents

A parent agent that delegates to a helper sub-agent has always been able to spawn_run and get a result. What it could not do, until v1.26.0, was keep the sub-agent alive across turns. Every spawn_run was a fresh child: a fresh interactive-run setup, a fresh loop, and (as the sandbox work made painfully concrete) a fresh sandbox container every time. A compile-test-fix loop that wanted to keep its container hot had to pass a session_id around by hand, hope the sub-agent honored the "reuse this session" contract, and pray that the operator's TTL didn't reap the container in the gap between spawns.

RFC BK adds three ops to the Agent tool that keep a child resident. The child stays alive between sends. Its sandbox container stays warm. Its conversation state stays loaded.

Phase 1: open, send, close

Agent {op:"open", name, prompt} starts a persistent interactive sub-run, runs its first turn, parks it at awaiting_input, and returns a child_run_id plus that turn's output. The child is a real interactive run under the covers, running in a detached goroutine under context.WithoutCancel, so it survives the parent's tool call returning to the model.

Agent {op:"send", child_run_id, prompt} steers the next turn and blocks until the child parks again, returning that turn's output. The child never lost its context, so the parent never has to restate what the sub-agent is doing.

Agent {op:"close", child_run_id} finalizes the run. Idempotent. Cascades to grandchildren (a resident child that itself opened residents gets its whole subtree torn down).

The reuse of the existing interactive-run engine is worth naming. A resident child is an ordinary interactive run: it has a real run ID, a real session, real events, real streaming, real turn-cancel. Anything that already works on an interactive run (SSE streaming to observers, cross-replica turn-cancel via the RFC BH machinery, snapshot integration) works on a resident child for free. The RFC BK plumbing is a small in-band API on the parent's Agent tool that talks to the same steer registry and awaits the same awaiting_input boundary that POST /v1/sessions/{id}/messages uses.

Bounded by design. A per-run cap (LOOMCYCLE_MAX_INTERACTIVE_CHILDREN, default 8) refuses new opens past the ceiling; an idle-TTL sweeper (LOOMCYCLE_INTERACTIVE_CHILD_IDLE_TTL_MS, default 30 minutes) reaps abandoned children; per-open override via idle_ttl_seconds for cases where a child is legitimately quiet for longer. A parent-teardown backstop closes a run's resident children when the parent ends, so nothing outlives its owner.

Single-replica for P1. A resident child is always co-located with its opener; the address only resolves within the opener's tenant on its own replica. That is fine for the primary use case (sub-agent-as-tool inside the same run tree) and defers the harder P3 question of cross-replica handoff.

Phase 2: timeout_ms, poll, cancel

P1's send blocked until the child parked. That is right for the fast-path case (send a prompt, get an answer, continue) and wrong when the child is doing something slow: a long compile, a stuck loop, a network call that has hung. The parent has no way to say "check in on it" or "give up on this turn." v1.27.0 fixes that with three additions.

send gains a timeout_ms argument. Zero (the default) is the P1 blocking behavior. Greater than zero returns early with {state:"running", output:"<partial>"} if the turn is still in flight. So a parent can bound how long it's willing to wait, get whatever the child produced so far, and decide what to do next.

poll is a new op. Check a resident child's state without sending new input. Returns current output plus state. timeout_ms:0 is a non-blocking snapshot; greater than zero waits up to that long for the child to park. Pair it with an early-returning send to await a slow turn without holding a synchronous send block open.

cancel is a new op. Turn-cancel the child's current turn (the RFC BH mechanism from v1.22.0 applied inward) and re-park it. The child stays alive, distinct from close. For a turn that is stuck or no longer worth finishing. A no-op if the child is already parked.

P2 is deliberately small. Because a resident child is an ordinary interactive run, much of the original P2 wishlist (streamed output for observers, external turn-cancel, cross-replica routing) was already delivered by P1 through the interactive-run engine's existing surface. P2 is only the in-band API surface: timeout_ms on send, poll, cancel. No speculative cross-replica plumbing.

Phase 3: Web UI visibility + operator control

An operator watching a run should be able to see its resident children and, if something is wrong, close or turn-cancel them without asking the parent agent nicely to do it. v1.28.0 adds the surface.

GET /v1/_resident is a tenant-scoped listing of live resident sub-agents: child_run_id, agent name, state (running / awaiting_input / completed / failed), parent, idle TTL, last-used timestamp. Per-replica, because the registry is in-process; a resident child is always co-located with its opener. A tenant operator sees their own tenant's; admin sees all (or focuses one via ?tenant=). Cross-tenant reads fold to opaque not-found, the same fold every other tenant-gated endpoint uses.

Two action endpoints: POST /v1/_resident/{run_id}/close to terminate a resident child, and POST /v1/_resident/{run_id}/cancel to turn-cancel its current turn while leaving the child alive. Both are tenant-gated with the same opaque-404 posture.

The Web UI's Activity view picks up the new fields. The runs listing and the agent detail pane badge a live resident child with resident · <state>, and the detail pane gains "cancel turn" and "close resident" buttons. The listing itself gains additive resident and resident_state fields for external consumers.

One quiet observation on team/orchestrator. It already had the Agent tool and was already a long-lived interactive run, so it could already open a helper and drive it across states with send / poll / cancel. What v1.28.0 adds for it is a new Context op=help topic=resident-sub-agents section that names the pattern explicitly ("Keeping a helper resident across a long task"), so the agent's own help system points at what already worked but was undocumented.

How the three arcs compose

The three RFCs were shipped in overlapping windows because they compose. A concrete example: a chat agent that wants to build and test a Go module.

1. The operator clones chat/medium into chat/medium-go-dev, editing the tool list to add mcp__sandbox__*. That is RFC BJ: the clone action widened the ceiling under the operator bearer. The chat agent now has sandbox tools.
2. Rather than adding sandbox tools directly to the chat agent (which would put the chat agent's turn in the same run as the sandbox activity, blocking its main loop), the operator keeps the chat agent thin and adds an Agent tool grant. The chat agent will delegate.
3. Mid-conversation, the chat agent opens a resident sub-agent: Agent {op:"open", name:"dev/sandbox"}. That is RFC BK P1: the sub-agent stays alive across the chat agent's turns.
4. The sub-agent's first act is sandbox_open {workspace:"kvflow"}. That is RFC BI P2a: the workspace /work is bind-mounted from the host, so the Go module cache from the last session is already there.
5. The chat agent sends "build main.go" through Agent {op:"send"}. The sub-agent runs sandbox_exec {cmd:"go build"}. Because the workspace is warm, the build finishes in 12 seconds instead of 3 minutes.
6. The chat agent sends "run the tests." Same sub-agent, same warm container, same session. sandbox_exec {cmd:"go test ./..."}. Two tests fail.
7. The chat agent sends "fix the failing tests and run them again." The sub-agent iterates. The workspace, the module cache, and the compiler state all survive across the parent's tool calls.
8. When the chat agent's conversation ends, its resident child is closed by the teardown backstop, and sandbox_close_run is called with the chat agent's root_run_id. That is RFC BI P2b: every sandbox session the run tree opened is closed in one call. The workspace on the host disk survives for the next conversation.

None of the three arcs would have carried this workflow on its own. RFC BJ without RFC BK would have put the sandbox activity inside the chat agent's own turn, blocking it and mixing its context with tool output. RFC BK without RFC BI's workspace path would have kept the sub-agent resident but let its container idle-reap between turns, losing the build cache. RFC BI without RFC BK would have offered warm containers but no clean way to hold onto one across a multi-turn conversation. The composition is where the value lands.

What's next

Three follow-ups are queued in the RFC BK / BI lines.

Cross-replica resident children. Today a resident child is single-replica: address it only from the opener's replica. That is fine for the sub-agent-as-tool pattern but does not compose with multi-replica load balancing when a parent's turn happens to land on a different replica than its child. RFC BK P4 is drafted around the same shared-store coordinator the turn-cancel work uses.

Snapshot/resume of resident children. Snapshot integration exists for ordinary interactive runs (RFC X phase 2). Extending it to a resident-child registry means a resident child can survive a runtime restart or a cross-instance move. Queued for later in the RFC BK line.

Prompt sandbox teardown on close. Today closing a resident child does not proactively close its sandbox container; the sidecar's own TTL reaps it eventually. Wiring close to a sandbox_close_run at teardown time makes the cleanup immediate and observable. Small change; queued as a v1.28.x point release.

Companion reading: safe code execution (RFC BI P1) for the sandbox architecture this post extends; agent teams (RFC AP + RFC BD) for the team orchestrator that gets resident sub-agents for free; turn-scoped cancellation (RFC BH) for the machinery the resident cancel op reuses; the History tool (RFC BE) for the recap summarizer the replay compress option shares; docs/CUSTOMIZING_AGENTS.md for the operator algorithm; and docs/SANDBOX.md for the sandbox operator reference including the workspace shape and the sidecar deploy story.