Providers as a YAML block.

2026-07-15 · by Dennis Gubsky · v1.21.0 · ~13 min read

Adding an LLM provider to loomcycle used to be a code change. The provider list was hardcoded in Go, the driver constructors sat in cmd/loomcycle, and pointing loomcycle at a self-hosted OpenAI-compatible endpoint meant editing the binary, rebuilding, and shipping a new release. The runtime does most things by config, but this one specific piece was source-level.

That was fine while there were six providers and no reasonable pressure to add more. It stopped being fine the moment operators asked for self-hosted vLLM behind a company VPN, a second Ollama pointed at a different GPU, a Together.ai endpoint, a Groq endpoint, or a home-lab llama.cpp. Every one of those is nominally an OpenAI-compatible endpoint. Loomcycle already had the OpenAI driver. What was missing was the config surface to declare a new entry, point it at a base URL, and let the resolver pick it up.

v1.21.0 ships that config surface. LLM providers are now a top-level YAML block backed by a driver registry, so any endpoint whose wire shape matches a compiled-in driver is a config edit away. The six built-ins keep working with byte-identical behavior because an embedded default-providers layer supplies them under the hood. And the same block gained two things worth having anyway: a max_concurrent cap that finally lets a single local GPU host a stable batch, and a capabilities override that re-enables vision on a multimodal model behind an OpenAI-compatible URL.

The providers: map

A minimal entry, declaring a self-hosted vLLM behind a company URL that talks the OpenAI dialect:

providers:
  vllm-internal:
    driver: openai
    base_url: https://vllm.corp.internal/v1
    api_key_env: VLLM_INTERNAL_API_KEY

The map key (vllm-internal) is the id agents reference in their provider: field. The driver is a compiled-in wire dialect: v1.21.0 ships with anthropic / openai / gemini / deepseek / ollama / mock / code-js. Any OpenAI-compatible endpoint uses driver: openai. The base_url is the endpoint. The api_key_env is an env-var name that resolves server-side (this matters, and there's a section on it below).

That's the whole thing. An agent that lists provider: vllm-internal now routes to that endpoint. The runtime does not need a rebuild.

Adding a second Ollama is the same shape:

providers:
  ollama-gpu-b:
    driver: ollama
    base_url: http://192.168.0.42:11434
    max_concurrent: 2

Or a Together.ai endpoint:

providers:
  together:
    driver: openai
    base_url: https://api.together.xyz/v1
    api_key_env: TOGETHER_API_KEY

Or a Groq endpoint. Or a hosted vLLM at Modal. Or a llama.cpp server on a workstation. The wire shape is what matters; the driver handles it.

Keyless third-party providers

A self-hosted vLLM or llama.cpp server usually has no auth at all. It sits behind a VPN or on a local network and doesn't ask for a key. In an earlier RC of the RFC BF work, a config-declared provider with no api_key_env was silently treated as "not configured" and the driver refused to build. The headline use case of the whole change (pointing loomcycle at a keyless self-hosted endpoint) didn't work.

The v1.21.0 review fix flipped that behavior. A declared provider with no api_key_env is now enabled. Declaring it in providers: IS the opt-in; the driver calls base_url with no key. A keyed provider whose env var is unset stays disabled, byte-identical to before. None of the built-ins reach this path (ollama-local / mock / code-js are special-cased in the resolver), so only operator-declared keyless providers newly enable.

providers:
  llamacpp-local:
    driver: openai
    base_url: http://localhost:8080/v1
    # no api_key_env: declaring the entry is the opt-in
    max_concurrent: 1

That's the RFC's headline shape working end to end.

api_key_env is a name, not a value

One detail worth naming: api_key_env holds the name of an env var, not the value. Loomcycle does not ${VAR}-interpolate the field at YAML parse time. The name flows through to the driver, and the driver resolves the value at request time from the server's environment.

That gets you two things. First, a tenant using RFC AR CredentialDefs to bring their own provider key can override the resolution: their CredentialDef named after the same env var name wins over the operator's host key at request time. Second, secrets never appear in the config file, so a checked-in loomcycle.yaml does not leak. The driver resolves the value; the config carries only the pointer.

There was a bug here worth naming, because it's the kind of thing that would silently make CredentialDef overrides target the wrong var. The anthropic, gemini, and ollama drivers had a hardcoded KeyEnvName() that ignored the config's api_key_env, while openai and deepseek forwarded it correctly. So an entry like {driver: anthropic, api_key_env: CUSTOM_ANTHROPIC_KEY} read the host key from CUSTOM_ANTHROPIC_KEY, but a tenant's CredentialDef override was looking up ANTHROPIC_API_KEY. Fixed. The three drivers each carry a settable keyEnvName field now, defaulted in New() to their canonical name and overridable by the config. Tests fail on the pre-fix code.

Per-provider max_concurrent

The second thing the providers: block gained is a concurrency cap. Set max_concurrent: 2 on an entry and no more than two runs can be in flight against that provider at once. The rest queue inside loomcycle; on timeout a queued run returns 429 provider_concurrency_exhausted over HTTP or ResourceExhausted over gRPC.

providers:
  ollama-local:
    driver: ollama
    base_url: http://localhost:11434
    max_concurrent: 2

The motivating case is a local model on a single GPU. Fan out ten queries at qwen3.6, they all hit the same 780M iGPU, and Ollama has to context-swap KV cache between them. Throughput craters. Set max_concurrent: 2 and the fan-out drains in pairs of two: each pair runs to completion before the next pair starts, so the KV cache stays resident for its whole turn. Total wall clock drops because the swap thrashing is gone.

The gate is acquired before the global concurrency slot. That order matters. A saturated provider (all its slots held) never starves runs targeting other, uncapped providers. The provider gate returns fast; the global slot then admits the run against a different provider. Without that order, ten pending Ollama runs would eat the global slot pool and stall anthropic-bound runs waiting behind them.

A provider with max_concurrent: 0 (or unset) gets no gate at all; Acquire returns a shared no-op release. The common uncapped case has zero contention.

Sub-agents and the deadlock carve-out

There's one subtle case to handle. A fan-out parent is a run that spawns N sub-agents concurrently. If the parent holds provider P's slot and the sub-agents also want P, they'd queue behind their own parent forever: the parent can't release P's slot until the sub-agents finish; the sub-agents can't start until the parent releases P's slot. Self-deadlock.

v1.21.0 handles it with a deadlock carve-out. Every run (top-level or sub-agent) gates on its resolved provider. Except: a run whose ancestor in the same run tree already holds that provider's slot skips the gate entirely. So the fan-out parent holds P's slot; its children want P; they detect that an ancestor holds P; they run without acquiring P's gate. No deadlock. The cap holds at the top level (only the parent's slot counts against P's max_concurrent).

The held-set is carried on the run's context and is copy-on-add. Copy-on-add matters because parallel spawn children run concurrently off the same parent context; a shared mutable map would race and cross-contaminate siblings' carve-out decisions. Each spawned child gets an independent superset.

One correctness gap worth naming. If a carve-out sub-agent (already ungated because an ancestor holds P) fails over to a different capped provider Q on retry, it stays ungated on Q too. The swap intentionally doesn't acquire Q's gate at fallback time: doing so would reintroduce the parent-awaits-descendant deadlock the carve-out exists to prevent. The v1.21.0 fix keeps the deadlock-safe behavior but makes the escape observable: a WARN log names the transient over-subscription so an operator watching Q's provider gauge can see the spike. Full enforcement across the carve-out plus fallback boundary needs a mutable subtree-scoped held-set, which is a bigger change, tracked for a later RFC-BF revision.

Capability overrides

Some endpoints advertise less than they support. A multimodal model behind a self-hosted OpenAI-compatible URL usually accepts image parts, but the openai driver's default capability set marks vision as off by default because generic OpenAI-compat servers don't guarantee it. A capabilities: block on the provider entry re-enables it:

providers:
  llava-local:
    driver: openai
    base_url: http://gpu-box.local:8080/v1
    capabilities:
      supports_vision: true

Overrides are applied inside the driver, so the driver's optional behaviors (KeyedProvider, ThinkingDowngrader) are preserved. Other overridable flags: native_prompt_cache, max_context_tokens, and the rest of the capability struct. An override that lies about the endpoint's real behavior surfaces as a provider error at request time, which is the same failure mode you'd get without the override, just visible earlier.

The embedded default-providers layer

Backward compatibility here is not optional. Every deployment of loomcycle before v1.21.0 relied on the six built-ins existing with specific ids (anthropic, openai, gemini, deepseek, ollama, ollama-local), each with a specific driver, base URL default, and env var. If the migration to a config-driven registry required every operator to add nine YAML entries to their config just to keep working, the change would be a compatibility break.

It doesn't. The runtime ships an embedded providers.default.yaml layer that declares every built-in the pre-1.21.0 hardcoded resolver built. That layer is prepended to every config as the unconditional stack base. So a config with no providers: block resolves the built-ins exactly as before. An unkeyed provider returns the identical "not configured" error. Byte-identical.

An operator entry with the same id as a built-in deep-merges over the default. Override anthropic's base_url to point at a proxy without restating the driver or the env var; the default supplies those. Override just max_concurrent on ollama-local; the default supplies the driver, base URL, and everything else.

Operators who want a strictly-declared config with no built-in surface can drop the whole default layer with LOOMCYCLE_NO_DEFAULT_PROVIDERS=1. In that mode the operator's providers: block is the entire registry.

CLI parity

One quiet fix worth naming. The CLI subcommands that inspect config (validate, agents, doctor) used to assemble their view of providers without the embedded default layer that cmd/loomcycle serve prepends. They leaned on a config-package floor instead. So a CLI's picture of "what providers are available" could drift from what the server would actually build at boot. Rare, but real, and confusing when it happened.

v1.21.0's loadLayeredConfig now prepends embedded.DefaultProviders() for CLI paths too (honoring LOOMCYCLE_NO_DEFAULT_PROVIDERS, matching the server). A missing --config still errors: the explicit path check now compares against a baseLayers watermark rather than len(layers)==0, so a missing user config file still surfaces file-not-found instead of being silently satisfied by the default layer. And a provider_priority config error now names the actual known provider set including any operator-declared ones.

Queue-depth and timeout knobs

The per-provider queue is bounded. The bound and the per-acquire timeout are tunable via env:

• LOOMCYCLE_PROVIDER_QUEUE_DEPTH: max queued runs waiting for a slot on any provider. Defaults to the global MaxQueueDepth.
• LOOMCYCLE_PROVIDER_QUEUE_TIMEOUT_MS: how long a queued acquire waits before returning provider_concurrency_exhausted. Defaults to the global QueueTimeout.

Both fall back to the global concurrency-block defaults if unset, so an operator who doesn't touch them gets sensible behavior. The gate itself is per-replica (advisory across a multi-replica deployment); enforcement across replicas is a P4 candidate for the RFC BF line.

What this does to the run event stream

Nothing new on the wire. A run that trips the provider gate returns the existing error envelope: HTTP 429 with code: provider_concurrency_exhausted; gRPC ResourceExhausted; the same error surface every other quota-shaped refusal uses. That's why @loomcycle/client and Python bump to 1.21.0 in lockstep with no adapter code change: nothing new for the adapter to expose. The adapter version tracks the runtime it targets, not the surface it needs to add.

What's next

Two things queued for the RFC BF line. First, cross-replica gate enforcement: today the cap is advisory per replica, so two replicas each honoring max_concurrent: 2 can put four runs on the same provider at once. A shared-store gate rides the same shape as the multi-replica session steer coordinator and lands as a later RFC BF revision. Second, richer model auto-tracking: RFC BG P1 (which shipped in v1.22.0 alongside the turn-cancel work) added model_pattern so a config can auto-track a family (claude-haiku-*) rather than pin a specific snapshot, and the resolver picks the newest matching model from the provider's live catalog. Together those two land the last pieces of "the config never needs an edit when a provider ships something new."

Companion reading: turn-scoped cancellation (RFC BH) for the interruption + park work that shipped in v1.22.0 on top of this; agent teams (RFC AP + RFC BD) for how the deadlock carve-out interacts with parallel handlers in a team; local LLMs on TrueNAS for the field report that made the local-GPU story concrete; and docs/PROVIDERS.md for the operator reference including the config-driven registry section.