A run that waits instead of being refused.
Two releases, three mechanisms, one story. v1.113 shipped a run queue so a new run that cannot start at once gets a row and a cancel path instead of a 429. v1.114 moved provider caps to count model calls, in a lane per model kind, and taught a waiting run to lend its global slot to another run that could use it. Together they close the gap between "a run that is going" and "a run that is waiting," and they stop a saturated local provider from starving every run in the deployment.
The short version. A run with no slot gets status queued, its id, its cancel registration and its first stream frames before it waits. A provider's max_concurrent now counts model calls, in a lane per model kind. A single model inside a lane can be capped at one in-flight call. A run whose model call waits on a busy local provider lends its global run slot to the next run in line.
A run no longer gets a 429
Before v1.113 the behaviour on an overflowed queue was to wait up to queue_timeout_ms (30 seconds by default) and then refuse the request with HTTP 429. A caller got no row, no run id, no events, nothing to cancel. A burst lost work and a short local spike had to be re-tried by the client. A run already over its user cap was refused at once with the same shape.
v1.113 moved every start route through a queue that does not time out. The run gets its row with the new status queued, its id, its cancel registration, and its first stream frames before it waits. Two new events arrive on the stream:
queued, withadmission.reasonset touser_caporglobal_capand a position in the queue. A run can wait on its user's cap first, then on the global slot pool.admitted, withwait_ms. The row moves torunning.
Every start path funnels through the same queue: POST /v1/runs, the gRPC run RPCs, MCP's spawn_run and spawn_runs, batches, configured-run starts, schedules, webhooks, A2A. v1.114 extended the queue to session continuations (POST /v1/sessions/{id}/messages) and to the LLM gateway (/v1/chat/completions, /v1/embeddings): they wait in the same queue, round-robin with the runs, with no timeout.
A queued run is cancellable, and its row ends cancelled with no model call made. The old answer stays on request: pass admission: "reject" and the start refuses with 429 the way it used to. The old concurrency.queue_timeout_ms field is now ignored; a config that still sets it loads unchanged.
Two bounds keep the queue from becoming a backdoor to unbounded memory: max_queue_depth (now 32, was 16) bounds every waiting run together, and max_queued_runs_per_user (default 16) bounds one user's places so one caller cannot fill the queue. Over the bound: 429 backpressure, the one case where a run still refuses rather than waits. A scheduled fire the full queue refuses is fired again on the next tick, under the same run key, and does not count against max_fires; before, the slot was lost.
Provider caps now count model calls
A provider's max_concurrent used to count runs. A run that spent one second on a model call and twenty seconds on a tool call held a provider slot the whole time, even though the provider was idle for nineteen of those twenty seconds. A deployment with four runs talking to one local Ollama could have the Ollama waiting on nothing while four slots sat held.
v1.113 and v1.114 moved the cap to count model calls, in a lane per model kind. Each provider now has one lane per tier:
chat:anthropic:default decision:anthropic:default embedder:anthropic:default chat:ollama:default decision:ollama:default embedder:ollama:default chat:deepseek:default ...
A run holds a slot in the right lane only while its model call is in flight. Between calls (running a tool, waiting for a sub-agent, waiting for a human) the lane slot is free for another run. A tool's own wait inside a run (a Decision call that queues, an embedding that queues, a sub-agent that queues to start) reports as a tool wait, and the run can be shown it: the call_queued and call_admitted events carry a lane name and a wait time.
v1.114 added per-model lane caps. A single Ollama model that computes one request at a time can be capped at 1 inside a lane that caps the whole lane at 4, so the model gets fair share without the lane getting one-queue-per-model problems. The field is max_models_in_flight on the lane, set in the provider config.
A run that lends its slot
A run whose model call is waiting for a saturated local provider used to hold its global run slot the whole time. A few such runs could keep a run bound for an idle cloud provider from starting at all: ten runs waiting on a local Ollama held ten slots out of the twelve the deployment allowed, and two Anthropic runs were all that could start.
v1.114 lets a run lend its global slot while its own turn call waits. Before the call computes, the run takes its slot back ahead of every new run. Lent runs count against max_queue_depth with the queued ones, so runs alive stay within max_concurrent_runs + max_queue_depth. At that bound a run waits holding its slot, as before.
Lending only happens for the run's own turn call. A tool's own decision or embedding, a compaction summary, or a sub-agent's call still waits holding the parent's slot: lending them would let the parent run free its slot and then be unable to take it back without the child, which is a deadlock on the first lane that was full in both directions.
The cost of taking a slot back is one wait for an admitted global slot. If every slot is held by runs that are not computing (a parked interactive run, a long tool call), that wait lasts until one of them ends. GET /v1/_concurrency/stats reports lent alongside running and queued so an operator can see how many runs are in each state.
Routing around a busy provider
The tier cascade used to take the first available candidate, however long its queue. A run could wait minutes for a capped local model while a cloud candidate of the same tier sat idle. v1.114 lets an agent, or one run, skip a busy provider for an idle candidate of its tier:
- Per agent, in the
AgentDefunderprovider_routing: queue_aware, as the baseline for every run. - Per run, as an override on the request. The override gets one skip at the start, and one more if the run loses its place after lending a slot.
The two skips are deliberate. A run that waited at the back of the lane deserves to try a different provider once; a run that lent its slot and found the lane still full deserves one more chance before it is sent to the back again. More than two would get an inner loop that would probably skip around a transient blip and change what the run meant to call.
What this is not
Not a priority queue. Waiting runs are served first in first out within one (tenant, user), and round-robin across (tenant, user) pairs, so a user who has one run already going does not jump a user who has none. A resumed run is served first before any new ones: that is the one exception, because a paused run has already paid for its row.
Not a cross-replica queue. The queue is per-replica. A slot in-flight on another replica frees up on that replica first; a run queued here does not take that slot. Cross-replica fairness is a different machinery (the Postgres LISTEN/NOTIFY coordination the deployment runs for HA): it balances who gets the next replica, not who gets the next slot within one.
Not a bypass for budgets. Every queued run is still checked against the caller's token budget before it may queue and again when it starts. A caller already over the hard budget is refused at the queue door, not after a long wait. v1.112's strict enforcement extends the check to in-flight runs; the queue work does not loosen it.
Where to read more
The per-release detail lives in the release notes: see REVISIONS.md § v1.113.0 for the queue mechanics and the queued event payload, and REVISIONS.md § v1.114.0 for the lane and lending work. The configuration guide (docs/CONFIGURATION.md) carries the operator-level knobs under "Concurrency and queueing".