LLM timeout check

/tools/llm-timeout-check/

How long one agent task can run before your SDK and loop settings stop it, when the model is silent, trickling or overloaded, and what to set when that is too long.

How it works

LLM timeout check#

What it answers#

How long can one agent task run before anything in its settings stops it, and what should you set if that’s too long? It checks three ways a model call goes wrong:

Failure	What the provider does	What stops it by default
Silent	Accepts the connection and sends nothing	The read timeout, once per attempt, then the retries
Trickling	Answers 200 and streams slowly	Only `max_tokens`: every token restarts the read timeout
Overloaded	Refuses at once with 429 or 529	The SDK’s backoff between retries

Each is worked out for one attempt, one call with its retries, and one task of up to the turn limit, and judged against how long the task may take: a person in a chat, or a job’s own deadline. Where it’s too long, the tool names the one setting that fixes it.

Stack fills in your SDK’s defaults, so you only type what you changed:

Stack	Read timeout	Retries	Turns
Anthropic or OpenAI Python SDK, 1.13.0 and 3.28.0	600 s	2	left as it is: neither SDK has a turn limit, so enter your loop’s
OpenAI Agents SDK 0.23.1	600 s	2, from the OpenAI client it creates	10, its `max_turns`
Custom

The two Python SDKs share one choice because their defaults are the same. Edit any of the numbers afterwards and Stack switches to Custom, or to whichever stack the numbers still match. None of the three sets an attempt deadline.

How to use it: an example#

Say you run a support chat. Your agent uses the Anthropic Python SDK and you never changed its timeout. It makes up to 10 model calls to answer one question, and a customer will wait about 2 minutes before giving up.

1. Enter what you run.

Input	You enter	Why
Stack	Anthropic or OpenAI Python SDK	It fills Read timeout with `600` and Retries with `2`, the SDK’s defaults
Attempt deadline	`0`	You don’t wrap the call in your own time limit
Turns	`10`	Your agent stops after 10 calls
max_tokens	`4096`	What you pass to the API
Acceptable wait	`120`	2 minutes, the customer’s patience

2. Press Run, and read the top line.

plaintext

1
Worst case for one task
2
5.7 h
3
A model that trickles its answer can keep a 10-turn task running this long, which is over the 2 min you can wait.

In plain words: if the model starts answering very slowly, your agent can keep a customer waiting almost six hours, and nothing in your settings stops it.

3. Read what to set, worst first:

Failure	What to set	Worst case	What ends it
Trickling	Set an attempt deadline of 3 s or less.	5.7 h	Only max_tokens ends it, because every token restarts the read timeout.
Silent	Set an attempt deadline of 3 s or less.	5 h	The read timeout ends each attempt, then the SDK retries.
Overloaded	Nothing. It fits the wait.	15 s	The SDK gives up after its retries and their backoff.

A model that says nothing is cut off after 600 s and retried twice, on every turn. A slow answer is ended only by its length. A model that refuses at once costs only the short waits between retries, so that one is fine. The fix for both slow cases is the same: give each attempt its own time limit.

4. Read the numbers under it. For the worst case, trickling:

plaintext

1
One attempt                     34.1 min   4,096 tokens at 2 tokens a second.
2
One call, with its retries      34.1 min   Not retried, because the provider answered 200.
3
Cost of the worst-case trickle  $1.01      10 calls of 4,700 input tokens, at $4 and $20 per million tokens, ...

5. Read the chart to see why. Look at the Trickling column:

the diamond, one call, sits at about 2,000 s: one slow answer of 4,096 tokens at 2 tokens a second takes 34 minutes;
the triangle, the whole task, sits at about 20,000 s: 10 calls like that is 5.7 hours;
both are far above the red dashed line, your 2 minutes.

The read timeout never helps here. It only fires when the model goes quiet for 600 s, and a slow stream is never quiet that long: every token restarts the clock.

6. Try the fix and Run again. Put 3 in Attempt deadline. Every mark drops under the red line, the worst case becomes 1.8 min, and the top line turns green.

If 3 seconds is too short for your model to answer, that tells you something too: 10 turns with 2 retries each can’t fit in 2 minutes. Lower Turns to 4 and Retries to 1, and the deadline it suggests rises to 14 s.

For a background job where an hour is fine, set Acceptable wait to 3600. The same defaults are still far too long, 5.7 hours, but now an attempt deadline of about 2 minutes is enough.

How it counts#

The read timeout limits each wait for the next chunk, not the call. In the anthropic and openai Python SDKs, timeout=600 is an httpx2 timeout whose read part is the longest wait for the next chunk. Measured against a local server with anthropic 1.13.0, timeout=3.0 let a stream of 12 tokens 1.5 s apart run 18 s without an error.
Silent: each attempt lasts the read timeout, or the attempt deadline if that is shorter. Between attempts the SDK waits 0.5 s doubling to 8 s. The SDKs shorten each wait by up to a quarter at random; the worst case is the unshortened wait.
Trickling: fixed at 2 tokens a second, so one call lasts max_tokens / 2 seconds unless the attempt deadline cuts it. A cut attempt is retried like a timeout, the conservative reading. A read timeout shorter than the gap between tokens fires mid-stream, after the 200, as a transport error the SDK doesn’t retry.
Overloaded: the refusals are immediate, so the call is just the backoff between retries.
A task is up to turns calls at the worst case. That assumes the agent loop starts another turn after a failed call, as many hand-written loops do. A loop that stops at the first failure takes one call.
The suggested deadline is the longest attempt deadline that fits the whole task, retries and backoff included, into the acceptable wait: (wait - turns x backoff) / ((retries + 1) x turns), rounded down.
Cost is a sense of scale only: the worst-case trickle, at 4,700 input tokens a call and Claude Opus 5.5’s list price on 2026-10-11. The lower figure bills only calls that finish; the higher bills every cut-off attempt for what it streamed. No provider documents which happens.

What it assumes#

Worst cases, not forecasts. The point is which limit, if any, is doing the work.
One call at a time. No parallel tool calls and no queueing in front of the provider.
The SDK’s own backoff. A retry-after header from the provider replaces it, and the anthropic SDK accepts one of up to 4,294,967 s.
Connect timeouts are left out: they bound only reaching the server, 5 s in both SDKs.

References#

httpx timeouts: “The read timeout specifies the maximum duration to wait for a chunk of data to be received”. The SDKs use httpx2, its successor, with the same model.
anthropic-sdk-python 1.13.0, _constants.py: DEFAULT_TIMEOUT = httpx2.Timeout(timeout=10 * 60, connect=5.0), DEFAULT_MAX_RETRIES = 2, backoff 0.5 s to 8 s.
openai-agents-python 0.23.1, run_config.py: DEFAULT_MAX_TURNS = 10.
AWS Builders’ Library, Timeouts, retries and backoff with jitter.
Written for the post Circuit breakers for inference: what HTTP 200 misses, whose companion code, mainlog-code/agent-cost-control, measured the SDK behaviour above.

Source and references

index.ts335 lines, TypeScript

1
/**
2
 * LLM timeout check: how long one agent task can take before anything in its settings stops it, for the three ways
3
 * a model call goes wrong, judged against how long the caller can wait.
4
 *
5
 *  - **Silent.** The provider accepts the connection and sends nothing. Each attempt lasts the read timeout (or the
6
 *    attempt deadline, if shorter), and the SDK retries, waiting its backoff between attempts: 0.5 s doubling to 8 s
7
 *    in the anthropic and openai Python SDKs, before jitter. Jitter only shortens a wait, so the worst case is the
8
 *    unshortened one.
9
 *  - **Trickling.** The provider answers 200 and streams slowly. The read timeout bounds each gap between tokens,
10
 *    not the call, so a stream that keeps sending runs until `max_tokens`, unless an attempt deadline cuts it. A cut
11
 *    attempt is a failure and is retried, the conservative reading. A gap longer than the read timeout fires it
12
 *    mid-stream, after the 200, where the anthropic SDK raises the transport's own ReadTimeout and does not retry.
13
 *  - **Overloaded.** The provider refuses at once with 429 or 529. Each retry waits the backoff.
14
 *
15
 * A task is up to `turns` calls, each at its worst case, which assumes the agent loop starts another turn after a
16
 * failed call, as many hand-written loops do. Fixed, and stated in the inputs: a trickle of 2 tokens a second, the
17
 * rate of the slow streams in the post this tool came from.
18
 *
19
 * The output leads with the worst case for one task, as a banner only when it is over the reader's wait, then the
20
 * three failures side by side, the chart, and what it assumes. The cost of the worst-case trickle comes last, for one
21
 * input size and one price, as a sense of scale. Whether a
22
 * provider bills an attempt that was cut off is not documented by any provider checked (Anthropic, OpenAI, AWS,
23
 * Google, on 2026-10-11), so it gives both bounds.
24
 */
25
import type { Cell, Output, Tool } from "@toolbench/sdk";
26

27
type Input = {
28
  /** Which SDK's defaults filled the form. The presets live in tool.json; the numbers below are what counts. */
29
  stack: string;
30
  read: number;
31
  retries: number;
32
  deadline: number;
33
  turns: number;
34
  maxtokens: number;
35
  patience: number;
36
};
37

38
class InputError extends Error {
39
  readonly input: string;
40
  constructor(message: string, input: string) {
41
    super(message);
42
    this.input = input;
43
  }
44
}
45

46
/** The anthropic and openai Python SDKs' backoff before jitter. */
47
export const BACKOFF = 0.5;
48
export const BACKOFF_MAX = 8;
49
/** The trickle's rate, tokens a second. */
50
export const TRICKLE = 2;
51
/** For the cost line only: input tokens per call, and Claude Opus 5.5's list prices on 2026-10-11, $ per million. */
52
export const INPUT_TOKENS = 4700;
53
export const PRICE_IN = 4;
54
export const PRICE_OUT = 20;
55

56
/** The waits before retries 1..n, before jitter. */
57
export function backoffs(retries: number): number[] {
58
  return Array.from({ length: retries }, (_, n) => Math.min(BACKOFF * 2 ** n, BACKOFF_MAX));
59
}
60

61
export type Shape = {
62
  name: "Silent" | "Trickling" | "Overloaded";
63
  /** One attempt, seconds; 0 when the refusal is immediate. */
64
  attempt: number;
65
  /** One call with its retries. */
66
  call: number;
67
  /** One task of up to `turns` calls. */
68
  task: number;
69
  /** What ends one call. */
70
  endedBy: string;
71
  /** The same, in a few words, for a table column: "max_tokens only", "retry limit". */
72
  stop: string;
73
  /** Output tokens one attempt streams. */
74
  streamed: number;
75
  /** Attempts per call that stream output. */
76
  attempts: number;
77
};
78

79
export function shapes(i: Input): Shape[] {
80
  const waits = backoffs(i.retries).reduce((a, b) => a + b, 0);
81
  const tries = i.retries + 1;
82
  const cut = i.deadline > 0 ? i.deadline : Number.POSITIVE_INFINITY;
83

84
  const hang = Math.min(i.read, cut);
85
  const silent: Shape = {
86
    name: "Silent",
87
    attempt: hang,
88
    call: tries * hang + waits,
89
    task: 0,
90
    endedBy:
91
      cut < i.read
92
        ? "The attempt deadline ends each attempt, then the SDK retries."
93
        : "The read timeout ends each attempt, then the SDK retries.",
94
    stop: `${cut < i.read ? "attempt deadline" : "read timeout"}${i.retries ? ", then retries" : ""}`,
95
    streamed: 0,
96
    attempts: tries,
97
  };
98

99
  const full = i.maxtokens / TRICKLE;
100
  let trickle: Shape;
101
  if (1 / TRICKLE >= i.read) {
102
    trickle = {
103
      name: "Trickling",
104
      attempt: i.read,
105
      call: i.read,
106
      task: 0,
107
      endedBy: "The read timeout fires mid-stream, and the SDK does not retry it.",
108
      stop: "read timeout, not retried",
109
      streamed: 0,
110
      attempts: 1,
111
    };
112
  } else if (cut < full) {
113
    trickle = {
114
      name: "Trickling",
115
      attempt: cut,
116
      call: tries * cut + waits,
117
      task: 0,
118
      endedBy: "The attempt deadline cuts each attempt, then the SDK retries.",
119
      stop: `attempt deadline${i.retries ? ", then retries" : ""}`,
120
      streamed: Math.floor(cut * TRICKLE),
121
      attempts: tries,
122
    };
123
  } else {
124
    trickle = {
125
      name: "Trickling",
126
      attempt: full,
127
      call: full,
128
      task: 0,
129
      endedBy: "Only max_tokens ends it, because every token restarts the read timeout.",
130
      stop: "max_tokens only",
131
      streamed: i.maxtokens,
132
      attempts: 1,
133
    };
134
  }
135

136
  const overloaded: Shape = {
137
    name: "Overloaded",
138
    attempt: 0,
139
    call: waits,
140
    task: 0,
141
    endedBy: i.retries > 0 ? "The SDK gives up after its retries and their backoff." : "The first refusal ends it.",
142
    stop: i.retries > 0 ? "retry limit" : "first refusal",
143
    streamed: 0,
144
    attempts: 0,
145
  };
146

147
  return [silent, trickle, overloaded].map((s) => ({ ...s, task: s.call * i.turns }));
148
}
149

150
/** The one setting that brings this shape under the acceptable wait, or that it already fits. */
151
export function advice(i: Input, s: Shape): { text: string; ok: boolean } {
152
  if (s.task <= i.patience) return { text: `It stops after ${dur(s.task)}, inside the wait.`, ok: true };
153
  if (s.name === "Overloaded") return { text: "Use fewer retries or turns.", ok: false };
154
  const waits = backoffs(i.retries).reduce((a, b) => a + b, 0);
155
  const perAttempt = (i.patience - waits * i.turns) / ((i.retries + 1) * i.turns);
156
  if (perAttempt >= 1) return { text: `Set an attempt deadline of ${dur(Math.floor(perAttempt))} or less.`, ok: false };
157
  return { text: "Use fewer retries or turns, then set an attempt deadline.", ok: false };
158
}
159

160
/** 4 s, 2.5 min, 3 h, 28.4 days: the unit a person would say it in. */
161
export function dur(s: number): string {
162
  if (s < 60) return `${trim(s)} s`;
163
  if (s < 3600) return `${trim(s / 60)} min`;
164
  if (s < 86400 * 2) return `${trim(s / 3600)} h`;
165
  return `${trim(s / 86400)} days`;
166
}
167
const trim = (x: number) => (Math.abs(x - Math.round(x)) < 0.05 ? Math.round(x).toString() : x.toFixed(1));
168
const int = (x: number) => Math.round(x).toLocaleString("en-US");
169
const money = (x: number) => (x === 0 ? "$0" : x < 0.01 ? "under $0.01" : `$${x.toFixed(2)}`);
170
const price = (input: number, output: number) => (input * PRICE_IN + output * PRICE_OUT) / 1e6;
171

172
/** Who causes each worst case, as the verdict says it. */
173
const WHO: Record<Shape["name"], string> = {
174
  Silent: "A model that sends nothing",
175
  Trickling: "A model that trickles its answer",
176
  Overloaded: "A model that refuses every call",
177
};
178
const RANGES: [Exclude<keyof Input, "stack">, number, number, string][] = [
179
  ["read", 0.1, 86400, "The read timeout is between 0.1 s and a day."],
180
  ["retries", 0, 20, "Retries are between 0 and 20."],
181
  ["deadline", 0, 86400, "The attempt deadline is between 0 (none) and a day."],
182
  ["turns", 1, 1000, "Turns are between 1 and 1,000."],
183
  ["maxtokens", 1, 1000000, "max_tokens is between 1 and 1,000,000."],
184
  ["patience", 1, 86400, "The wait is between 1 s and a day."],
185
];
186

187
export default {
188
  run(raw: Input) {
189
    try {
190
      const i = { ...raw };
191
      for (const [k, lo, hi, msg] of RANGES) {
192
        const v = Number(i[k]);
193
        if (!(v >= lo && v <= hi)) throw new InputError(msg, k);
194
        i[k] = v;
195
      }
196
      if (!Number.isInteger(i.retries)) throw new InputError("Retries are a whole number.", "retries");
197
      if (!Number.isInteger(i.turns)) throw new InputError("Turns are a whole number.", "turns");
198

199
      const sh = shapes(i);
200
      const worst = sh.reduce((a, s) => (s.task > a.task ? s : a));
201
      const over = worst.task > i.patience;
202
      const turns = `${i.turns} turn${i.turns === 1 ? "" : "s"}`;
203

204
      /*
205
       * ⚠️ A banner only when the task can outlast the wait the reader typed: that limit is theirs, so its tone is
206
       * earned. It carries the worst case and the one setting that bounds it, and nothing else; the table under it
207
       * says which failure and why. Within the wait, the same line is the quiet answer.
208
       */
209
      const fix = advice(i, worst);
210
      /* A failure the worst case's setting does not fix gets its own clause, so no row over the wait goes unanswered. */
211
      const also = sh
212
        .filter((s) => s.task > i.patience && advice(i, s).text !== fix.text)
213
        .map((s) => ` ${s.name}: ${advice(i, s).text.charAt(0).toLowerCase()}${advice(i, s).text.slice(1)}`);
214
      const head: Output = over
215
        ? {
216
            kind: "fields",
217
            role: "verdict",
218
            fields: [
219
              {
220
                label: `Over your ${dur(i.patience)} wait`,
221
                role: "verdict",
222
                value: `Up to ${dur(worst.task)} for one ${i.turns}-turn task`,
223
                note: `${fix.text}${also.join("")}`,
224
                tone: "bad",
225
              },
226
            ],
227
          }
228
        : {
229
            kind: "fields",
230
            role: "answer",
231
            fields: [
232
              {
233
                label: "Worst case for one task",
234
                value: dur(worst.task),
235
                note: `${WHO[worst.name]} can keep a ${i.turns}-turn task running this long, inside the ${dur(i.patience)} you can wait.`,
236
              },
237
            ],
238
          };
239

240
      /*
241
       * The three failures side by side, worst first; a stable sort keeps silent ahead of trickling on a tie. One
242
       * attempt, one call with its retries, one task, and what stops it in a few words, so the columns fit a phone.
243
       * Only the task column is toned, and only against the reader's wait.
244
       */
245
      const rows: Cell[][] = [...sh]
246
        .sort((a, b) => b.task - a.task)
247
        .map((s) => [
248
          s.name,
249
          { text: s.attempt > 0 ? dur(s.attempt) : "0 s", mono: true },
250
          { text: dur(s.call), mono: true },
251
          { text: dur(s.task), mono: true, ...(s.task > i.patience ? { tone: "bad" as const } : {}) },
252
          s.stop,
253
        ]);
254
      const matrix: Output = {
255
        kind: "table",
256
        role: "comparison",
257
        caption: `Each failure, for one task of up to ${turns}, worst first`,
258
        columns: [
259
          { label: "Failure", wrap: "prose", rowHeader: true },
260
          { label: "Attempt", align: "end", mono: true, wrap: "none" },
261
          { label: "Call", align: "end", mono: true, wrap: "none" },
262
          { label: "Task", align: "end", mono: true, wrap: "none" },
263
          { label: "Stopped by", wrap: "prose" },
264
        ],
265
        rows,
266
      };
267

268
      const tr = sh[1]!;
269
      const finished = tr.streamed === i.maxtokens && tr.attempts === 1;
270
      const billed = tr.streamed > 0 ? i.turns : 0;
271
      const low = finished ? price(i.turns * INPUT_TOKENS, i.turns * i.maxtokens) : 0;
272
      const high = price(billed * tr.attempts * INPUT_TOKENS, billed * tr.attempts * tr.streamed);
273
      const basis = `${i.turns} call${i.turns === 1 ? "" : "s"} of ${int(INPUT_TOKENS)} input tokens, at $${PRICE_IN} and $${PRICE_OUT} per million tokens, Claude Opus 5.5's list prices on 2026-10-11.`;
274
      /* What the numbers assume, and the cost as a sense of scale, last and quiet. */
275
      const method: Output = {
276
        kind: "fields",
277
        role: "method",
278
        fields: [
279
          {
280
            label: "Assumes",
281
            mono: false,
282
            value: `A trickle of ${TRICKLE} tokens a second, and a loop that starts another turn after a failed call.`,
283
          },
284
          {
285
            label: "Cost of the worst-case trickle",
286
            value: low === high ? money(low) : `${money(low)} to ${money(high)}`,
287
            note:
288
              low === high
289
                ? basis
290
                : `${basis} The higher figure also bills the attempts that were cut off. No provider says whether it does.`,
291
          },
292
        ],
293
      };
294

295
      const pt = (v: number) => (v > 0 ? v : null);
296
      const chart: Output = {
297
        kind: "series",
298
        chart: {
299
          xLabel: "Failure",
300
          yLabel: "Worst case",
301
          yUnit: "s",
302
          yScale: "log",
303
          x: sh.map((s) => s.name),
304
          series: [
305
            {
306
              label: "One attempt",
307
              shape: "points",
308
              points: sh.map((s) => pt(s.attempt)),
309
              notes: sh.map((s) => (s.attempt > 0 ? dur(s.attempt) : "refused at once")),
310
            },
311
            {
312
              label: "One call, with retries",
313
              shape: "points",
314
              points: sh.map((s) => pt(s.call)),
315
              notes: sh.map((s) => dur(s.call)),
316
            },
317
            {
318
              label: `One task, ${turns}`,
319
              shape: "points",
320
              points: sh.map((s) => pt(s.task)),
321
              notes: sh.map((s) => dur(s.task)),
322
            },
323
          ],
324
          /* The reader's wait, and nothing else: a fixed one-hour line was a threshold nobody set. */
325
          thresholds: [{ y: i.patience, label: `you can wait ${dur(i.patience)}`, tone: "bad" }],
326
        },
327
      };
328

329
      return { kind: "group", parts: [head, matrix, chart, method] };
330
    } catch (error) {
331
      if (error instanceof InputError) return { kind: "error", message: error.message, input: error.input };
332
      throw error;
333
    }
334
  },
335
} satisfies Tool<Input>;

2 items

Oct 19, 2026 21 min read AI

Circuit breakers for inference

Timeouts, retries and budgets for model calls that finish too slowly, produce unusable answers or keep repeating themselves. With the real Anthropic SDK, timeout=3.0 let an 18-second stream finish. In a simulated support chat, with invented failure rates, five checks took the conversations waiting over two minutes from 249 to 0.

AI & agentic architecturesai-agentsreliability
9 posts Series

AI & agentic architectures

What it takes to make agents reliable enough to run in production.

AI

Circuit breakers for inference

Timeouts, retries and budgets for model calls that finish too slowly, produce unusable answers or keep repeating themselves. With the real Anthropic SDK, timeout=3.0 let an 18-second stream finish. In a simulated support chat, with invented failure rates, five checks took the conversations waiting over two minutes from 249 to 0.

ai-agentsreliability
Oct 2026
Series

AI & agentic architectures

What it takes to make agents reliable enough to run in production.

9 posts