Circuit breakers for inference

The timeout that isn’t a deadline#

The Anthropic and OpenAI Python SDK versions used here both default to a 600-second read timeout. To find out what that timeout would actually stop, I ran Anthropic SDK 1.13.0 against a small local server with timeout=3.0:

1. stream, a token every 1.5s, timeout=3.0s: completed in 18.0s, no timeout (12 tokens)
2. not streamed, 20 bytes every 1.5s, timeout=3.0s: completed in 16.5s, no timeout
3. stream, a 4s stall after 3 tokens, timeout=3.0s: httpx2.ReadTimeout after 7.5s; an anthropic.APIError? False; an anthropic.APITimeoutError? False
4. stream, a token every 1.5s, timeout=3.0s: deadline 5s: stopped after 5.0s with 3 of 12 tokens
5. no response at all, timeout=3.0s: anthropic.APITimeoutError after 10.3s, 3 attempts (2 retries)

These are wall-clock measurements. The SDK is real; the server deliberately controls when each chunk arrives.

The second run is easy to overlook. Turning streaming off doesn’t fix the problem: a response body can arrive slowly too. The HTTP client sees a sequence of reads either way.

Both pinned SDKs use httpx2, a successor to httpx by the same author, with the same timeout model. The httpx documentation puts it in one sentence: “The read timeout specifies the maximum duration to wait for a chunk of data to be received”. Whenever a chunk arrives in time, reading the next one gets another wait. There’s no total duration in that rule.

Run three has a separate trap. The timeout fires after the response headers have arrived, while the SDK is consuming the stream. The exception is httpx2.ReadTimeout, not anthropic.APITimeoutError, and not any other anthropic.APIError. A handler catching the SDK’s timeout exception misses it. The SDK doesn’t retry that interrupted stream either.

With the silent server, the failure happens earlier. The SDK makes three attempts of three seconds each, waits between them, then raises its own APITimeoutError. Those two paths deserve separate tests even if the application reports both as “the model timed out.” Claude’s errors documentation makes the broader point too: “When receiving a streaming response over server-sent events (SSE), an error can occur after the API returns a 200 response.”

An elapsed-time deadline stops the trickling response:

1
async with asyncio.timeout(deadline):
2
    async with client.messages.stream(**ASK) as s:
3
        async for _ in s.text_stream:
4
            got += 1

That’s run four, with a deadline of five seconds: cancellation at 5.0 seconds, with three tokens received. It doesn’t need a long gap between tokens to fire.

The placement of that wrapper matters. Here it encloses the streaming operation, including the wait to start receiving it. Any retries performed inside that operation also consume the five seconds. A fresh deadline around each attempt would allow a longer total. A limit around the task has a different scope again.

The pinned OpenAI Agents SDK exposes this idea as ModelSettings.timeout, described in its source as bounding “the complete model attempt”. The default is None. The read timeout and the attempt deadline are separate settings.

The budget nobody chose#

The defaults are easier to reason about once their scopes are written down. These came from the installed packages:

anthropic 1.13.0: timeout read=600s connect=5s, max_retries=2, backoff 0.5s doubling to 8s, retry-after obeyed up to 4,294,967s
openai 3.28.0: timeout read=600s connect=5s, max_retries=2
openai-agents 0.23.1: max_turns=10, ModelSettings().timeout=None
langgraph 1.2.14: recursion_limit=10007

For a server that accepts the connection and sends nothing, the arithmetic is roughly thirty minutes per SDK call: three ten-minute waits, plus backoff. If the agent loop catches that failure and starts another turn, ten failed turns can take five hours.

That last step is an assumption about the loop. The Anthropic SDK has no turn limit, and the ten-turn default belongs to the OpenAI Agents SDK. An application that exits on the first failed call won’t make another nine. The five-hour case is what a loop that keeps going permits, rather than a claim that every framework runs that way.

A successful trickle needs no retries. At two tokens a second, a 4,096-token response takes about 34 minutes. Ten such calls take 5.7 hours. The read timeout never fires; max_tokens eventually ends each answer.

Then there’s retry-after. The pinned Anthropic SDK accepts a value of up to 4,294,967 seconds, about fifty days. That ceiling comes from a platform limit on sleeping. It says very little about how long an application ought to wait.

None of these individual settings expresses a support chat’s latency budget. To get that budget, someone has to decide how much time the conversation has and how much of it retries are allowed to consume.

1
DEFAULT_RECURSION_LIMIT = int(getenv("LANGGRAPH_DEFAULT_RECURSION_LIMIT", "10007"))

Four ways a model call goes wrong#

A conventional circuit breaker counts failures on a dependency. Enough failures open the circuit, so calls fail fast or take another route. After a cooling-off period, a half-open state permits a trial request. If it works, the circuit closes. Martin Fowler’s description is a good reference for those mechanics.

The definition of a failure is where inference gets awkward. A 429 means the caller is being rate-limited, and a 529 means the provider is overloaded:

A status-only breaker can see the first row. The other three may complete without an exception. Even a perfectly reliable HTTP connection can spend minutes carrying answers that don’t help the application.

The support-chat simulation makes this concrete. Each conversation needs four good model calls: interpret the question, look up the order, look up the delivery, and answer. The primary provider has five bad minutes, from second 300 to second 600. These are invented failure rates, not measurements of a provider.

Conversation 350 starts during that incident. The trace records elapsed time from the start of the conversation:

defaults: conversation 350 started at 350 s, ended failed after 203.6 s (turn limit), 60,200 tokens
   150.8 s  primary  slow
   151.0 s  primary  overloaded
   157.2 s  primary  ok
   163.0 s  primary  loop
   168.8 s  primary  loop
 ...
   203.6 s  primary  loop

The first call takes 150.8 seconds. The next is refused, but its retry works. Then the model repeats its tool call eight times, until the agent reaches its ten-turn limit. Only one request returned an error, while the slow response and the repeated calls all returned 200. In the simulation’s default policy, the conversation ends without an answer or a handover, after three and a half minutes and 60,200 tokens.

Detecting that kind of failure takes information from several layers:

The SDK’s retries are a reasonable place to handle a transient refusal. They don’t know that the third successful tool call was identical to the previous two. The application has to make that judgment and decide what to do with it.

Retries on a metered API#

An agent can inherit retries from the HTTP client, SDK, framework and gateway, then add another loop around the whole turn. Those retry counts multiply unless a shared budget stops them.

The AWS Builders’ Library article on timeouts, retries and backoff with jitter puts it bluntly: “Retries are “selfish.” In other words, when a client retries, it spends more of the server’s time to get a higher chance of success.” In its example of five layers retrying three times each, “If each layer retries independently, the load on the database will increase 243x, making it unlikely to ever recover.” Its advice is to “retry at a single point in the stack”, which was useful before anyone put a model in the system and remains useful afterwards.

Google’s SRE chapter on overload gives retries explicit limits: “a per-request retry budget of up to three attempts”, plus a per-client rule that “A request will only be retried as long as this ratio is below 10%.” These are examples of policies that account for repeated work across requests, rather than leaving every request free to retry on its own.

A metered model adds a cost to application-level retries. An answer with an unusable tool call may already be billable. Asking again generates another answer, and sends the prompt and accumulated history again too. A retry can succeed and still make the task substantially more expensive.

Some failures won’t improve with another attempt. Claude’s errors documentation says “A tier spend-cap 429 has no retry-after header and keeps failing until access resumes.” Treating that as ordinary temporary overload only adds waiting.

There’s also the question of what the previous turn already did. If it sent a tool call before failing, replaying the turn can repeat the operation. The operation needs an identity that survives retries; changing an attempt ID is no protection against doing the same thing twice. That distinction is covered in When the user changes their mind mid-task.

Budgets that trip the breaker#

So the breaker needs more inputs than status codes. Here are the five checks the simulation can switch on, each at the layer that can see its signal:

1. Status breaker. Five consecutive failures on the primary open it for thirty seconds. Calls go to a smaller fallback model during that time, followed by one trial call to the primary.
2. Stream-rate check. Three seconds into a stream, check whether it is delivering at least fifteen tokens a second. If it isn’t, abort it and retry on the fallback.
3. Validity check. An answer whose tool call won’t parse counts as a failure, including for the breaker. Two in one conversation trigger a handover.
4. Loop guard. Three identical tool calls in a row trigger a handover.
5. Conversation budget. Sixty seconds and 40,000 tokens in total. With this policy enabled, any conversation that ends without an answer is handed to a person, with the customer told.

The breaker still has its ordinary three states:

The state transitions stay the same. The additional checks change which outcomes count as failures.

The important implementation detail is the set of outcomes passed to breaker.record:

1
if on("breaker") and model == "primary":
2
    seen = {"overloaded"} | ({"aborted"} if on("stream") else set()) | ({"malformed"} if on("valid") else set())
3
    breaker.record(task.now, outcome in seen)

With only the breaker enabled, it sees overload errors. With the stream check enabled, it also sees responses aborted for being too slow. Turn on validity checking and malformed answers join the count. The slow stream and malformed answer can both arrive as HTTP 200; the application now has a reason to record a failure anyway.

The primary-model condition matters as well: this count describes the primary. The fallback has its own behaviour. A conversation’s loop guard and total budget answer a different question, about whether to continue that conversation at all.

These thresholds make the checks easy to compare in the demo. They need tuning for an actual workload. Fifteen tokens a second could be an appropriate floor for one model and a poor choice for another; three repeated calls are only a useful signal when the task doesn’t legitimately require that repetition.

The same arithmetic applies to any agent’s settings, so I turned it into a small tool. Pick your stack, which fills in that SDK’s read timeout, retries and turn limit, then add any attempt deadline, max_tokens, and how long the task may take. It shows the worst case for one task and the setting that brings it within the wait, then, for a silent, trickling and overloaded provider, how long each takes per attempt, per call and per task and what stops it, and a chart of how the worst case grows from one attempt to one call to the whole task. Start with the SDK defaults, then try the 30-second deadline sample: it fixes the trickle and still leaves the task far over two minutes, because the retries and turns multiply it.

Five bad minutes, three policies#

All five checks answered fewer conversations than the defaults: 867 against 899. That’s the trade-off worth keeping in view while looking at the improvements in latency and handover.

The simulation starts 1,000 conversations, one a second. On a normal day, the primary’s calls are 1% overloaded, 1% slow, 2% malformed and 2% looping. During seconds 300 to 600, those rates become 40%, 20%, 6% and 2%. The fallback keeps its separate, unchanged failure profile. A seed is the number the simulation’s random choices start from, so the same seed always gives the same run. This is seed 7:

python sim.py. The p99, the time that only the slowest 1% of conversations reached or exceeded: 323.8 s, 193.2 s and 29.6 s. Tokens counted as billed: 18.78 M, 20.24 M and 17.08 M, respectively.

The status breaker opens six times. It answers another 32 conversations and nearly halves the number waiting over two minutes. It also consumes about 8% more tokens: conversations that would have failed are now able to continue. Improved availability can increase the bill.

Its benefit varies across seeds, and seed 7 is its best case. On seeds 1 to 5, the change in completed conversations ranges from six fewer to 33 more, and the reduction in long waits ranges from 16% to 45%. The p99 stays above 310 seconds on those seeds, compared with 193.2 seconds here. A breaker helps with refusals, but it leaves slow successful responses running.

With all five checks, the slowest conversation on seed 7 takes 34.8 seconds. No conversation waits over two minutes or ends silently. The other tested seeds also have no waits over two minutes and no silent failures, though their answer and handover counts differ.

Those limits stop some conversations that would eventually have been answered, minutes later. For a support chat, I’d accept that trade if the handover works. For a batch job, a late answer may be perfectly useful. The right budget depends on what the application is waiting for.

The lab contains every combination of the five checks under three incident types. Pick an incident, then switch checks on one at a time. The tall grey bars show the slowest conversation starting in each twenty-second interval; the red bars underneath show conversations that failed without a handover. The tall bars outside the incident come from the normal day’s 1% of slow responses: at two tokens a second, one slow answer takes about 150 seconds before the conversation’s other calls are counted.

Three configurations are worth trying on purpose:

• Only the conversation budget. Every failed conversation is handed over, but 248 still take more than two minutes. The sixty-second budget is checked between calls, so it can’t stop a 150-second response that’s already running. For a hard elapsed-time limit, cancellation needs to reach the call in progress.
• Only the stream-rate check. Long waits disappear, but silent failures rise to 187. Aborting slow calls uses up retry attempts sooner. Without the handover policy, exhausting those attempts just ends the conversation. Stopping wasted work doesn’t, by itself, produce a useful outcome for the customer.
• “Mostly slow,” with only the status breaker. On this run the breaker never opens, and every result is identical to the defaults. It has no signal for the condition causing most of the waiting. Switch on the stream-rate check and aborted responses start contributing to its failure count.

Falling back, and the case against it#

The fallback lets the breaker do more than fail fast. It’s also a different model, with different failure modes. In the simulation it keeps its normal failure rates during the primary’s incident, but is worse at valid tool calls and more likely to loop.

A real fallback may change answer quality, wording, price and latency. Its capacity may also be needed by other systems at exactly the same time. The simulation’s results depend on its independent failure profile; they don’t establish that another model will stay healthy during a real incident.

The AWS Builders’ Library article on avoiding fallback argues that “Distributed fallback strategies often make the outage worse.” A path that runs only when something else is broken is often the least exercised part of the system, and the article prefers “code paths that are exercised in production continuously rather than rarely.”

For a model fallback, that’s a reason to send it some normal traffic and evaluate the results before relying on it. Apply the same validity and progress checks to its answers. Moving calls away from a sick provider is useful only if the replacement can still do the work.

What the customer sees#

The defaults leave 101 conversations with no answer and nobody told. With all five checks, 133 conversations are handed to a person and none end silently. That change depends on the handover being an explicit part of the policy, not just on the breaker opening.

There still has to be someone available to take the work. The simulation doesn’t model staffing or queue times, and “handed over” isn’t the same as “resolved.” For a support product, those are part of deciding whether this trade is acceptable.

The software also needs to preserve enough state for the handover to mean something: which order was found, which operations completed, what the agent attempted next and why it stopped. Otherwise the person picking it up starts from scratch, and the customer may have to explain everything again.

The degraded state in What a state machine buys an agent gives that handover somewhere to land. Its customer message is written from known state, rather than improvised by the model that just failed. The checks here decide when to enter that state.

A status breaker remains useful. What’s missing is a limit on the successful-looking work around it: the call that keeps streaming, the answer that can’t be used, the next turn that repeats the last one. Deadlines, validation and progress checks make those failures visible. A conversation budget decides how much more work to allow, and a defined end state tells the application what happens next.

Run it yourself#

The scripts are in mainlog-code/agent-cost-control. timeouts.py runs the SDK against fake_server.py, defaults.py reads the limits from the pinned packages, and sim.py runs the support conversations.

cd mainlog-code/agent-cost-control

python3 sim.py              # no install needed
python3 sim.py --task 350   # one conversation, each policy
python3 sim.py --seeds      # compare seeds

python3 -m venv .venv
.venv/bin/pip install -r requirements.txt
.venv/bin/python timeouts.py
.venv/bin/python defaults.py

The timeout experiment takes about fifty seconds and calls only the local server. CI checks the timing ranges, the defaults in the pinned versions, and the simulation results and lab data. A package update that changes an assumption should fail a check.

For an existing agent, the slow-stream test is where I’d start. It’s small enough to run locally, and it answers a concrete question: when the model keeps sending data, does the deadline actually stop the call?