Timeout budget checker

/tools/timeout-budget/

A call chain's per-try timeouts and retries, top first: the worst case when the bottom hangs, whether it fits the caller's deadline, and which retries can never finish in time.

How it works

Timeout budget checker#

Every call in a chain has a per-try timeout and some retries, and they are usually set by different people at different times: one in the service’s code, one in a client library, one left at the mesh’s default. Nothing checks that they agree. When they do not, a caller gives up on a try while the service it called is still retrying further down, and those retries are work nobody is waiting for, on a system that is already struggling.

Write the chain one call per line, top first:

text

frontend -> checkout    none   2
checkout -> pricing     1s     2
pricing -> inventory    500ms  2

That is a caller, the service it calls, the per-try timeout (ms, s, or none), and the number of retries. A # starts a comment. Press Run, and it answers, for the worst case of the bottom service hanging:

A banner, only when the chain cannot finish inside the deadline you set: how long it can take, against how long whatever calls the top of the chain waits. When it fits, the answer is one quiet line instead: the worst case, inside the deadline, with every retry able to finish.
The calls whose tries cannot fit, the deadline first. A call is listed when its tries, with their backoff, do not all fit inside one try of the call above it, or inside the deadline for the top call. Each row says what the caller allows, what the call’s tries need, how many of them fit, and the longest per-try timeout that would make them fit. The caption counts the rows.
Other changes: the other ways out for each of those calls (fewer retries, or more room above), and a row for each call with no timeout, because it waits forever on a hung callee; Istio sets no HTTP timeout by default. Fix from the top down: a shorter timeout on one call changes what fits under it, so run it again after each change.
The chain, drawn top to bottom, as you wrote it. Each arrow is one call with its per-try timeout and tries, and each service says how long its own calls can take. A call whose tries do not fit is red and thick, and the first of them is highlighted.
A table of each call: per try, tries, worst case, and how many of its own tries fit inside one try of its caller (“Fit above”), toned on the same call the second part lists.
What it assumes, last: the bottom hangs and every layer retries a timed-out try, the backoff, and the tries a fast failure would make at the bottom if every layer retried it. That last number is the product of everyone’s tries, which is the retry amplification tool’s question. Istio’s default retry policy does not retry an HTTP 503, so there it is 1.

The rule it checks is the one in the SRE book’s advice on deadlines: each layer’s tries, with their backoff, should fit inside one try of the layer above, so timeouts shrink going down the chain.

How to use it: an example#

The chain above, with the app at the top giving up after 2 seconds and 25 ms between tries, Istio’s base backoff. Paste it, set Top caller waits to 2000 ms and Backoff between tries to 25 ms, and press Run (or press the sample By hand, all retried).

The answer, in red, because 9.20 s overruns the 2 s you set:

When the bottom hangs: Can take 9.20 s; the caller gives up after 2.00 s

Then the calls whose tries cannot fit:

Call	Allowed	Needs	Fit	Set per-try to
`frontend -> checkout`	2.00 s	9.20 s	0/3	650 ms or less
`pricing -> inventory`	1.00 s	1.55 s	1/3	316 ms or less

And the other ways out, under Other changes. Fix the top one first, then run again.:

Call	Change
`frontend -> checkout`	Or let the caller wait at least 9.20 s.
`pricing -> inventory`	Or do not retry it, or give checkout -> pricing at least 1.55 s per try.

The chain draws frontend’s call and pricing’s call in red. The detail table under it gives each call’s per try, tries, worst case and fit, and the last part notes that a bottom failing fast instead would see 27 tries if every layer retried the error.

In plain words: when inventory hangs, pricing tries it three times at 500 ms each, which takes 1.55 s, but checkout gives pricing only 1 s per try, so two of pricing’s three tries run for nobody. Checkout’s three tries of pricing take up to 3.05 s, and frontend, with no timeout, tries checkout three times: 9.2 s for a request the app gave up on after 2. The 650 ms and 316 ms are what make each call fit on its own. To make the whole chain fit, shrink the timeouts going down: the sample Shrinking timeouts uses 1.8 s with no retry, 800 ms with one, and 225 ms with two, for a worst case of 1.48 s, which the tool gives as one quiet line with no banner and no table of changes.

Where it stops being true#

The worst case assumes every layer retries a try that timed out. A layer only retries what its policy names. Envoy retries a per-try timeout only when retry_on includes gateway-error, 5xx or reset, and Istio’s default includes none of them: set a perTryTimeout, leave retryOn alone, and the timed-out try is simply not retried. Measured with Envoy 1.39.3 in mainlog-code’s service-mesh/retry_chain.py: 1 try with Istio’s default, 3 with gateway-error added.
Backoff is one fixed wait before each retry. Envoy’s is random and grows from a 25 ms base: 0 to 24 ms before the first retry, and before the second 0 to 74 ms by its documentation, 0 to 49 ms in the 1.39.3 code. So leave a little room rather than fitting a chain exactly.
The services’ own work is not counted: only time spent waiting on the call below.

Source and references

index.ts333 lines, TypeScript

1
/**
2
 * Do a call chain's timeouts and retries fit inside each other?
3
 *
4
 * Every call in a chain has a per-try timeout and some retries, usually chosen by different people at different
5
 * times: one in the service's code, one in a client library, one by the mesh's defaults. Nothing checks that they
6
 * agree. The failure they produce is quiet: a caller gives up on a try while the service it called is still
7
 * retrying further down, so those retries are work that nobody is waiting for, on a system that is already in
8
 * trouble.
9
 *
10
 * The worst case this checks is the bottom service hanging: it accepts a request and never answers. Then each try
11
 * at the bottom runs to its timeout, the call above sees a failure only after all of them, and so on up. It lists
12
 * each call whose tries do not fit inside one try of its caller, with the per-try timeout that would make them fit,
13
 * then the chain, then how long each call can take and how many of its tries fit above it.
14
 *
15
 * ⚠️ It assumes every layer retries a try that timed out. Envoy only does that if `retry_on` names gateway-error,
16
 * 5xx or reset, and Istio's default names none of them, so with the defaults a timed-out try is not retried at all.
17
 * Set a per-try timeout and leave retryOn alone, and you get the timeout without the retry.
18
 *
19
 * The same arithmetic as `service-mesh/timeout_budget.py` in mainlog-code; the fixtures are its output.
20
 */
21
import type { Cell, GraphEdge, GraphNode, Output, Tool } from "@toolbench/sdk";
22

23
type Input = { chain: string; deadline: number; backoff: number };
24

25
class InputError extends Error {
26
  readonly input: string;
27

28
  constructor(message: string, input: string) {
29
    super(message);
30
    this.name = "InputError";
31
    this.input = input;
32
  }
33
}
34

35
type Call = { caller: string; callee: string; timeout: number; retries: number };
36

37
function parse(text: string): Call[] {
38
  const calls: Call[] = [];
39
  for (const raw of text.split("\n")) {
40
    const line = raw.split("#")[0]!.trim();
41
    if (!line) continue;
42
    const m = /^(.+?)\s+(none|[\d.]+\s*(?:ms|s)?)\s+(\d+)$/.exec(line);
43
    if (!m)
44
      throw new InputError(`Cannot read "${raw.trim()}": expected "caller -> callee  timeout  retries".`, "chain");
45
    const [, name, t, r] = m as unknown as [string, string, string, string];
46
    const ends = name.split(/\s*(?:->|→)\s*/);
47
    if (ends.length !== 2 || !ends[0] || !ends[1])
48
      throw new InputError(`"${name.trim()}" should be "caller -> callee".`, "chain");
49
    let timeout: number;
50
    if (t === "none") timeout = Number.POSITIVE_INFINITY;
51
    else if (t.endsWith("ms")) timeout = Number.parseFloat(t) / 1000;
52
    else if (t.endsWith("s")) timeout = Number.parseFloat(t);
53
    else timeout = Number.parseFloat(t) / 1000;
54
    if (!(timeout > 0))
55
      throw new InputError(`The timeout in "${raw.trim()}" must be more than zero, or none.`, "chain");
56
    calls.push({ caller: ends[0].trim(), callee: ends[1].trim(), timeout, retries: Number(r) });
57
  }
58
  if (calls.length === 0) throw new InputError("Give at least one call, one per line, top first.", "chain");
59
  if (calls.length > 20) throw new InputError("At most 20 calls.", "chain");
60
  for (let i = 1; i < calls.length; i++) {
61
    if (calls[i]!.caller !== calls[i - 1]!.callee) {
62
      throw new InputError(
63
        `Line ${i + 1} starts at ${calls[i]!.caller}, but line ${i} ends at ${calls[i - 1]!.callee}. Each call should be made by the service the line above called.`,
64
        "chain",
65
      );
66
    }
67
  }
68
  return calls;
69
}
70

71
function fmt(s: number): string {
72
  if (!Number.isFinite(s)) return "unbounded";
73
  return s < 1 ? `${(s * 1000).toFixed(0)} ms` : `${s.toFixed(2)} s`;
74
}
75

76
export default {
77
  run(input: Input) {
78
    try {
79
      const calls = parse(String(input.chain ?? ""));
80
      const deadline = Number(input.deadline) / 1000;
81
      const backoff = Number(input.backoff) / 1000;
82
      if (!(deadline > 0)) throw new InputError("The deadline must be more than zero.", "deadline");
83
      if (!(backoff >= 0)) throw new InputError("The backoff cannot be negative.", "backoff");
84

85
      /* Worst case per call, from the bottom up: each try lasts its timeout or until the call below has failed. */
86
      const worst = new Array<number>(calls.length).fill(0);
87
      let below = Number.POSITIVE_INFINITY;
88
      for (let i = calls.length - 1; i >= 0; i--) {
89
        const c = calls[i]!;
90
        const oneTry = Math.min(c.timeout, below);
91
        worst[i] = (1 + c.retries) * oneTry + c.retries * backoff;
92
        below = worst[i]!;
93
      }
94

95
      /* How many of the callee's tries finish inside one of the caller's. */
96
      const fit: (number | null)[] = calls.map((outer, i) => {
97
        const inner = calls[i + 1];
98
        if (!inner) return null;
99
        const innerTry = Math.min(inner.timeout, i + 2 < calls.length ? worst[i + 2]! : Number.POSITIVE_INFINITY);
100
        if (!Number.isFinite(outer.timeout)) return 1 + inner.retries;
101
        if (!Number.isFinite(innerTry)) return 0;
102
        return Math.min(1 + inner.retries, Math.floor((outer.timeout + backoff) / (innerTry + backoff) + 1e-9));
103
      });
104

105
      /*
106
       * The top call is checked against the deadline as if the deadline were one try of a call above it: the
107
       * app's own timeout is the first layer of the budget, and a top call that can't finish inside it wastes
108
       * every try it makes.
109
       */
110
      const topTry = Math.min(calls[0]!.timeout, calls.length > 1 ? worst[1]! : Number.POSITIVE_INFINITY);
111
      const topFit = Number.isFinite(topTry)
112
        ? Math.min(1 + calls[0]!.retries, Math.floor((deadline + backoff) / (topTry + backoff) + 1e-9))
113
        : 0;
114

115
      let attempts = 1;
116
      for (const c of calls) attempts *= 1 + c.retries;
117
      const top = worst[0]!;
118
      const tries = (i: number) => 1 + calls[i]!.retries;
119
      const name = (c: Call) => `${c.caller} -> ${c.callee}`;
120
      /* `broken[i]`: call i's tries do not all fit inside one try of the call above it, or the deadline for the top. */
121
      const broken = calls.map((_, i) => (i === 0 ? topFit < tries(0) : (fit[i - 1] ?? tries(i)) < tries(i)));
122
      const fitOf = (i: number) => (i === 0 ? topFit : fit[i - 1]!);
123

124
      /*
125
       * The failing boundaries, one row per call whose tries do not all fit inside one try of the call above it,
126
       * or inside the deadline for the top call. Each row is the call itself, so the timeout to set and the
127
       * numbers that say why are on the same row. The old detail table put "1 of 3" on checkout -> pricing while
128
       * its fix named pricing -> inventory, because it showed each call's callee rather than the call.
129
       */
130
      const plural = (n: number, one: string, many: string) => `${n} ${n === 1 ? one : many}`;
131
      const retriesFix = (f: number) => (f <= 1 ? "do not retry it" : `retry it at most ${plural(f - 1, "time", "times")}`);
132
      const failing: Cell[][] = [];
133
      const others: Cell[][] = [];
134
      /* Calls whose row already says what timeout to set, so a "set a timeout" row would only repeat it. */
135
      const timedFix = new Set<number>();
136
      const order = calls.map((_, i) => i).filter((i) => broken[i]);
137
      /* The deadline first, then the rest top down: a fix above changes what fits below. */
138
      for (const i of order) {
139
        const c = calls[i]!;
140
        const f = fitOf(i);
141
        const room = i === 0 ? deadline : calls[i - 1]!.timeout;
142
        const maxTry = (room - c.retries * backoff) / tries(i);
143
        if (maxTry > 0) timedFix.add(i);
144
        const tone = i === 0 || f === 0 ? ("bad" as const) : ("warn" as const);
145
        failing.push([
146
          { text: name(c), mono: true },
147
          { text: fmt(room), mono: true },
148
          { text: fmt(worst[i]!), mono: true },
149
          { text: `${f}/${tries(i)}`, mono: true, tone },
150
          { text: maxTry > 0 ? `${fmt(Math.floor(maxTry * 1000) / 1000)} or less` : "no timeout fits", mono: true },
151
        ]);
152
        /* The other ways out, which the table's last column does not already say. */
153
        const ways = [
154
          ...(f >= 1 ? [retriesFix(f)] : []),
155
          ...(!Number.isFinite(worst[i]!)
156
            ? []
157
            : i === 0
158
              ? [`let the caller wait at least ${fmt(worst[0]!)}`]
159
              : [`give ${name(calls[i - 1]!)} at least ${fmt(worst[i]!)} per try`]),
160
        ];
161
        if (ways.length) {
162
          const said = ways.join(", or ");
163
          others.push([{ text: name(c), mono: true }, { text: `Or ${said}.` }]);
164
        }
165
      }
166
      calls.forEach((c, i) => {
167
        if (Number.isFinite(c.timeout) || timedFix.has(i)) return;
168
        others.push([
169
          { text: name(c), mono: true },
170
          { text: `Set a per-try timeout. With none, a hung ${c.callee} holds the call until something above gives up.` },
171
        ]);
172
      });
173
      /* Back into chain order, so the rows read top down whichever kind each one is. */
174
      const at = (r: Cell[]) => calls.findIndex((c) => name(c) === (r[0] as { text: string }).text);
175
      others.sort((x, y) => at(x) - at(y));
176

177
      const fitsDeadline = top <= deadline;
178
      const parts: Output[] = [];
179
      /*
180
       * ⚠️ A banner only when the chain cannot finish inside the deadline the reader typed. That is the one result
181
       * here a limit the reader supplied decides; anything else leads with the table, or with the quiet answer
182
       * when every retry fits.
183
       */
184
      if (!fitsDeadline)
185
        parts.push({
186
          kind: "fields",
187
          role: "verdict",
188
          fields: [
189
            {
190
              label: "When the bottom hangs",
191
              role: "verdict",
192
              mono: false,
193
              tone: "bad",
194
              value: Number.isFinite(top)
195
                ? `Can take ${fmt(top)}; the caller gives up after ${fmt(deadline)}`
196
                : `Nothing bounds it; the caller gives up after ${fmt(deadline)}`,
197
            },
198
          ],
199
        });
200
      if (failing.length)
201
        parts.push({
202
          kind: "table",
203
          role: "comparison",
204
          caption:
205
            failing.length === 1
206
              ? "1 call cannot finish its tries inside its caller's time"
207
              : `${failing.length} calls cannot finish their tries inside their caller's time`,
208
          columns: [
209
            { label: "Call", wrap: "none", rowHeader: true },
210
            { label: "Allowed", align: "end", mono: true, wrap: "none" },
211
            { label: "Needs", align: "end", mono: true, wrap: "none" },
212
            { label: "Fit", align: "end", mono: true, wrap: "none" },
213
            { label: "Set per-try to", align: "end", mono: true, wrap: "none" },
214
          ],
215
          rows: failing,
216
        });
217
      else
218
        parts.push({
219
          kind: "fields",
220
          role: "answer",
221
          fields: [
222
            {
223
              label: "Worst case when the bottom hangs",
224
              value: fmt(top),
225
              note: `Inside the caller's ${fmt(deadline)}, and every call's tries fit inside one try of the call above.`,
226
            },
227
          ],
228
        });
229
      if (others.length)
230
        parts.push({
231
          kind: "table",
232
          caption: failing.length ? "Other changes. Fix the top one first, then run again." : "Changes",
233
          columns: [
234
            { label: "Call", wrap: "none", rowHeader: true },
235
            { label: "Change", wrap: "prose" },
236
          ],
237
          rows: others,
238
        });
239
      parts.push(chainGraph(calls, worst, broken, deadline), {
240
        kind: "table",
241
        caption: "Each call, top first, when the bottom hangs",
242
        columns: [
243
          { label: "Call", wrap: "none", rowHeader: true },
244
          { label: "Per try", align: "end", mono: true, wrap: "none" },
245
          { label: "Tries", align: "end", mono: true, wrap: "none" },
246
          { label: "Worst case", align: "end", mono: true, wrap: "none" },
247
          { label: "Fit above", align: "end", mono: true, wrap: "none" },
248
        ],
249
        rows: calls.map((c, i) => [
250
          { text: name(c) },
251
          {
252
            text: Number.isFinite(c.timeout) ? fmt(c.timeout) : "none",
253
            mono: true,
254
            ...(Number.isFinite(c.timeout) ? {} : { tone: "warn" as const }),
255
          },
256
          { text: String(tries(i)), mono: true },
257
          { text: fmt(worst[i]!), mono: true },
258
          { text: `${fitOf(i)}/${tries(i)}`, mono: true, ...(broken[i] ? { tone: "bad" as const } : {}) },
259
        ]),
260
      });
261
      /*
262
       * What the check assumes, last and quiet. The product of everyone's tries is a different failure, the bottom
263
       * answering fast with an error, so it is here as a caveat rather than beside the answer.
264
       */
265
      parts.push({
266
        kind: "fields",
267
        role: "method",
268
        fields: [
269
          { label: "Assumes", value: "The bottom hangs, and every layer retries a try that timed out.", mono: false },
270
          { label: "Backoff", value: `${fmt(backoff)} between tries` },
271
          {
272
            label: "If the bottom fails fast instead",
273
            value: `${plural(attempts, "try reaches", "tries reach")} it, when every layer retries the error.`,
274
            mono: false,
275
            note: "Istio's defaults do not retry an HTTP 503.",
276
          },
277
        ],
278
      });
279
      return { kind: "group", parts };
280
    } catch (error) {
281
      if (error instanceof InputError) return { kind: "error", message: error.message, input: error.input };
282
      throw error;
283
    }
284
  },
285
} satisfies Tool<Input>;
286

287
/**
288
 * The chain top to bottom, from whatever calls the top to the service that hangs. Each arrow is one call with its
289
 * per-try timeout and tries; each service says how long its own calls can take when the bottom hangs. A call whose
290
 * tries do not fit inside the try above it is drawn red and thick, and the first of them, the deadline's if that is
291
 * broken, is highlighted.
292
 */
293
function chainGraph(calls: Call[], worst: number[], broken: boolean[], deadline: number): Output {
294
  const nodes: GraphNode[] = [{ id: "caller", label: "Caller", note: `waits ${fmt(deadline)}`, shape: "pill" }];
295
  const edges: GraphEdge[] = [{ from: "caller", to: "n0", line: "dotted" }];
296
  const services = [calls[0]!.caller, ...calls.map((c) => c.callee)];
297
  services.forEach((s, j) => {
298
    const bottom = j === calls.length;
299
    nodes.push({
300
      id: `n${j}`,
301
      label: s,
302
      note: bottom
303
        ? "hangs"
304
        : Number.isFinite(worst[j]!)
305
          ? `its calls take up to ${fmt(worst[j]!)}`
306
          : "its calls can wait forever",
307
      ...(bottom ? { tone: "warn" as const } : broken[j] ? { tone: "bad" as const } : {}),
308
    });
309
  });
310
  calls.forEach((c, i) => {
311
    const timed = Number.isFinite(c.timeout);
312
    edges.push({
313
      from: `n${i}`,
314
      to: `n${i + 1}`,
315
      label: `${timed ? `${fmt(c.timeout)} per try` : "no timeout"}, ${1 + c.retries} ${c.retries === 0 ? "try" : "tries"}`,
316
      ...(broken[i] ? { tone: "bad" as const, width: "thick" as const } : !timed ? { tone: "warn" as const } : {}),
317
      ...(timed ? {} : { line: "dashed" as const }),
318
    });
319
  });
320
  const first = broken.indexOf(true);
321
  return {
322
    kind: "graph",
323
    graph: {
324
      label: "The chain when the bottom hangs, with each call's per-try timeout and tries",
325
      layout: "layered",
326
      /* Top to bottom, as the chain is written: a phone has room for a column of five boxes, not a row. */
327
      direction: "TB",
328
      nodes,
329
      edges,
330
      ...(first >= 0 ? { highlight: [`n${first}`, `n${first + 1}`] } : {}),
331
    },
332
  };
333
}

2 items

Oct 7, 2026 28 min read Systems

What a service mesh costs

Your shop now runs 14 services in three languages, and security wants every call between them encrypted and identified. A mesh does that without touching the code, for about a fifth of a millisecond per hop and a tenth of a core per proxy per 1,000 requests a second, measured here with real Envoy. It also brings retry and queue defaults nobody chose. Over plain HTTP they retry less than most people expect; over gRPC, or after one common edit, a failing service gets 27 calls for one checkout.

Down to the wireinfrastructuremicroservicesnetworking
5 posts Series

Down to the wire

Distributed systems at the point where the theory meets the pager.

Systems

What a service mesh costs

Your shop now runs 14 services in three languages, and security wants every call between them encrypted and identified. A mesh does that without touching the code, for about a fifth of a millisecond per hop and a tenth of a core per proxy per 1,000 requests a second, measured here with real Envoy. It also brings retry and queue defaults nobody chose. Over plain HTTP they retry less than most people expect; over gRPC, or after one common edit, a failing service gets 27 calls for one checkout.

infrastructuremicroservices
Oct 2026
Series

Down to the wire

Distributed systems at the point where the theory meets the pager.

5 posts