Input webhooks — the signed-by-default front door for external events.

2026-05-30 · by Dennis Gubsky · ~7 min read

RFC H shipped in v0.14.1 — the fifth substrate primitive. WebhookDef joins AgentDef, SkillDef, MCPServerDef, and ScheduleDef as a content-addressed, versioned, signed-by-default thing operators register and the substrate owns. The shape: external systems (GitHub, Stripe, Linear, n8n cloud, an internal service) sign and POST an event to /v1/_webhooks/{name}, and loomcycle either spawns an agent run or wakes an agent already parked on a channel — the trigger shape that ScheduleDef handles for cron, generalized to the HTTP-POST half of the same problem.

The honest engineering of this release isn't the feature — it's two trust-boundary bugs the review caught. Both would have been silent in production. Both shaped the final shipped shape. Most of this post is about them.

Why a substrate primitive, not glue code

Without WebhookDef, every operator wires the same receiver: decode the GitHub payload, verify X-Hub-Signature-256 against a per-route secret, reshape the body into a run request, POST to /v1/runs with the operator's bearer. That glue moves credential handling out of the substrate (where the env-allowlist gate is the trust boundary) into bespoke code, and it gets re-implemented per source.

Making this a primitive gets three things at once. Versioning + lineage — every webhook is a content-addressed Def with the same fork/get/list/retire CRUD as every other substrate primitive. Single trust boundary — signing secrets and per-event credentials resolve through the same env-allowlist gate the scheduler uses, so an operator who already trusts that gate for cron runs doesn't grow a second authority. Symmetric mental model — scheduled runs and webhook-triggered runs are the two halves of one problem (autonomous run creation from a non-interactive source), and treating them with the same shape means operators learn one model rather than two.

The shape

Off by default. Operators turn it on with one env var, and the receiver mounts at POST /v1/_webhooks/{name} — deliberately not behind the global bearer auth middleware, because per-Def signature verification IS the authentication:

LOOMCYCLE_WEBHOOKS_ENABLED=1
LOOMCYCLE_SCHEDULER_ENV_ALLOWLIST=LOOMCYCLE_*

A webhook is a yaml block or a substrate fork of an existing WebhookDef:

webhooks:
  github-pr-review:
    enabled: true
    delivery: spawn                       # spawn (default) | channel
    agent: pr-review-agent
    auth:
      kind: hmac                          # hmac (default) | bearer
      header: "X-Hub-Signature-256"
      signing_secret_env: "LOOMCYCLE_GITHUB_WEBHOOK_SECRET"
      delivery_id_header: "X-GitHub-Delivery"
    rate_limit: { requests_per_minute: 60, burst: 10 }
    body_size_limit_bytes: 1048576
    user_credentials_from_env:            # operator-owned, env-allowlist-gated
      github: "LOOMCYCLE_GITHUB_TOKEN"
    payload_mapping:
      goal:    "$.pull_request.title"
      user_id: "$.sender.login"
      user_credentials.github: "$.installation.access_token.value"
    on_complete:
      - { kind: channel.publish, channel: "_system/pr-reviews", payload: { text: "reviewed" } }

Two delivery modes, deliberately distinct. delivery: spawn turns the inbound POST into a runner.RunOnce against the named agent — autonomous run creation, with credentials and metadata projected from the payload. delivery: channel publishes the projected payload to a named channel — used when an agent is already parked on Channel.subscribe() waiting for an external callback (think CI "build done" notifications). Channel mode carries no credentials by design: a channel publish has no run identity to attach per-user bearers to, and accepting them would leak secrets onto the message bus. The validator refuses credentials in channel mode at admin-write time.

Verify before parse

Every request goes through one shared front-half before the spawn-vs-channel fork:

1. Resolve the active WebhookDef by name from the URL. Unknown or disabled → opaque 404. The URL is the only routing source — never the body.
2. Read raw body under body_size_limit_bytes (default 1 MiB) via MaxBytesReader. No streaming-decode of untrusted bytes.
3. Verify signature over the raw bytes with crypto/hmac.Equal (constant-time compare). Three HMAC envelopes auto-detected from the header value: Stripe (t=<unix>,v1=<hex> over <t>.<body>, ±5-minute window), GitHub (sha256=<hex> over the body), and bare-hex (the whole value is the hex MAC, no prefix — Linear and many custom sources). Plus a bearer fallback for systems that can't sign.
4. Replay/dedup — Layer 1, in-memory, keyed by the configured delivery_id_header or a body-hash fallback.
5. Project via JSONPath. Strict subset only — $.a.b, $.a[0], no wildcards, no filters, no recursion, no eval surface. An absent path resolves to empty plus a tracing note, never a failure.
6. Rate limit via per-Def token bucket → 429 + Retry-After.
7. Deliver. Spawn → build a RunInput, run it on a detached ctx; async 202 or ?sync=true blocks on the run-state bus to terminal/504. Channel → publish + notify.

Verify before parse is the discipline. The signature is HMAC over the raw bytes — any middleware that re-serializes the body, normalizes whitespace, or canonicalizes JSON before our handler runs breaks verification. We read raw bytes, verify, then parse. This is the standard LINE-channel pattern from prior art, made explicit.

The two bugs the review caught

Whole-feature code review of the receiver surfaced two real correctness defects, each with regression-grade tests on the fix. They're worth telling in order because together they're the case for trust-boundary code reviews on hot paths.

Bug 1 — dedup burning IDs on the unhappy path

The first cut of the receiver recorded delivery_id in the dedup cache at the guard step — right after signature verification, before rate-limit + mapping-validation + spawn-setup. The intuition was sound: "we've seen this delivery, don't process it twice." But the failure shape was nasty.

Concrete scenario: GitHub sends a signature-valid webhook. Loomcycle verifies, records the delivery_id in the dedup cache, then hits the per-Def rate limit and returns 429. GitHub honors the Retry-After and retries 60 seconds later. The signature is still valid; the rate-limit budget is now available — but the delivery_id is in the cache. The retry is dropped as a replay. The sender did everything right and we silently swallowed their legitimate event.

Decision 9 of the RFC is named "never silently degrade" for exactly this kind of failure. The fix splits the dedup cache's API into two operations: seen() checks (read-only), record() marks on actual acceptance. A signature-valid request that's then rate-limited or mapping-rejected or hits a transient 503 stays retryable; the delivery_id only burns when we've actually accepted and dispatched. Regression test pins the invariant.

Bug 2 — trusted-text where untrusted-block belonged

The second bug is smaller in code but bigger in implication. The receiver projected the payload's mapped goal field into the run's prompt as trusted-text — the same shape loomcycle uses for operator-authored prompt content from the yaml config. That sounds fine until you remember what the value is: a GitHub PR title, a Stripe event type, a Linear issue body. External, attacker-influenceable input. Marking it trusted-text bypassed the loop's <untrusted> fence — the prompt-injection boundary that tells the model "anything inside these tags is data, not instructions."

The fix is a one-line type change: untrusted-block{kind:webhook_payload}. Now a webhook-spawned run wraps the projected payload in <untrusted kind="webhook_payload">…</untrusted>, the model treats it as the attacker-influenceable content it is, and a malicious PR title saying "ignore your previous instructions" gets the same defense the loop provides for any other untrusted input. Regression test pins the type; if a future refactor reverts it, the test fails.

Two layers of idempotency

The first bug was about Layer 1 — the in-memory dedup cache. Layer 2 is a different thing entirely: a durable, cross-replica guarantee on top.

A new column on the runs table — idempotency_key — with a UNIQUE partial index (only enforced when the key is non-null). The receiver sets the key to the webhook's delivery_id before calling runner.RunOnce. A second delivery of the same event — past the Layer-1 cache's TTL, or on a different replica — does a BEFORE-spawn RunByIdempotencyKey lookup, finds the existing run, and returns 202 deduped instead of spawning twice. A concurrent-insert race (two pods see "no existing run" at the same time, both call RunOnce, one wins the unique-constraint check) gets the same answer: the loser re-looks-up the winner and returns 202 deduped, not a 503.

Layer 1 stays the cheap fast path for the common case. Layer 2 is the cross-replica + past-TTL backstop. The general primitive — runs.idempotency_key with the unique partial index — is reusable: future RFC G push notifications can ride it, a general POST /v1/runs dedup story can ride it, anything that needs "same key = same run, never twice."

What this looks like for real senders

Five copy-paste recipes ship in the help topic, each verified against the receiver's signature handling. Quick survey:

A sixth recipe — channel-mode for CI build callbacks — shows the wake-a-parked-agent pattern. Honest about residual scope: not yet supported are base64-encoded HMAC digests (Shopify's X-Shopify-Hmac-Sha256) and custom signed-payload constructions like Slack's v0:<ts>:<body> with the timestamp in a separate header. The bearer fallback covers many of those today; a future PR adds the specific envelopes.

Never silently degrade

Every failure mode is distinct, and none are silent:

Two admin-bearer-gated triage endpoints help debug a webhook that looks like it's failing silently from the sender's side but isn't from ours. GET /v1/_webhooks/{name}/recent-deliveries shows a bounded ring buffer of the last 50 invocations with delivery_id, verdict (accepted / rejected_sig / rejected_replay / rejected_rate / unresolvable_secret / …), received_at, and run_id. POST /v1/_webhooks/{name}/test is a dry-run validator: POST a sample body + signature, get back {would_accept, verdict, run_input_preview}, no run is actually created, no dedup record is written. The preview lists credential key names only, never values.

What you can do with it today

On v0.14.1, with LOOMCYCLE_WEBHOOKS_ENABLED=1 set and a yaml entry like the GitHub recipe above, a real POST from GitHub lands on your pr-review-agent within milliseconds. The agent can call your authorized GitHub MCP server using ${run.credentials.github} — either the operator-env-allowlisted org token or the per-event installation token from the payload, whichever you configured. The full operator picture lives in Context.help input-webhooks on any post-v0.14.1 build.

One caveat to name loudly: this is single-replica v1. The Layer-1 dedup cache + rate-limit buckets are per-replica today; the durable runs.idempotency_key is the cross-replica backstop for spawn mode. Cluster-wide dedup + rate-limit is a later concern, not a v1 hole.

Companion reading: Loomcycle speaks A2A (RFC G, the previous v0.13.x ship — the substrate-primitive pattern carries through, and A2A's push notifications will eventually ride WebhookDef's outbound-event surface), Scheduled runs at 30,000 fires (RFC E — the cron half of the same problem), Three MCP tokens in one run (RFC F — the credential channel webhook-spawned runs ride through).