Tools

Retry amplification calculator

/tools/retry-amplification/

Retries per layer and how deep the chain goes: how many calls reach the bottom service for one user request when every layer retries what it gets, and the extra load on it while it fails.

How it works

Retry amplification calculator#

A retry count in a config file looks like a small number: 2 here, 3 there. Down a chain of calls it is an exponent. A layer that retries wraps everything below it, so code that makes 3 attempts through a proxy that makes 3 makes 9, and the next call down sees all 9 of them, each making its own.

Give it how deep the chain is and how many retries each layer makes, then press Run. Mesh fills the proxy retries with a mesh’s default: Istio over gRPC 2, Linkerd 1 (once retries are turned on for a route; it makes none until then), None 0. Change the number and the select shows Custom. It answers in this order:

  1. Calls at the bottom: for one user request, when the bottom service is failing every call. (1 + code retries) x (1 + proxy retries) per call, to the power of the depth, times the client’s tries.
  2. What to change, the biggest cut first. Each row is the same arithmetic run again with that one change made, so it says how many calls the change leaves at the bottom: a retry budget, retrying only at the top of the chain, retrying in the code or the proxy but not both, and a client that stops retrying the whole request. A change that would not lower the number is not offered.
  3. The numbers beside it: tries per call; the calls that reach the bottom on average when it fails only some share of calls, which is the extra load on a service that is already failing (at 50% failure, 3 deep with 2 retries each, it doubles); and the share of user requests that still fail, if failures were independent. They rarely are: a failing service fails many calls in a row, so treat that as the best case.
  4. The tree of every try one user request causes. Only the first try at each service is followed down, because every other try sends exactly the same calls below it; each of those says how many it sends. Past four tries at one service, the rest are drawn as one node.
  5. A table of what each service in the chain receives.

How to use it: an example#

Three gRPC calls in a row, frontend to checkout to pricing to inventory, on a mesh with Istio’s default of 2 retries per call, and inventory is down. Enter Calls deep 3, Mesh Istio over gRPC, which fills Proxy retries per call with 2, everything else 0, Bottom fails 100%, Retry budget None, and press Run (or press the sample Istio gRPC, bottom down).

The answer:

Calls at the bottom: 27 calls reach the bottom service for one user request while it is down. 3 tries per call, 3 calls deep, if every layer retries the error it gets.

What to change:

ChangeCalls at the bottomWhy it works
Turn on a retry budget of 20%, Envoy’s default share once it is enabled.1.728 calls reach the bottom instead of 27.A budget allows retries only while they are a small share of a layer’s requests. Envoy’s always allows 3 at once, so at low traffic it does not bind.
Retry only at the top of the chain, and let every call below it fail fast.3 calls reach the bottom instead of 27.3 calls in a row each retry, so their tries multiply: 3 to the power of 3.

And beside it: Tries per call 3, When the bottom fails 100% 27 calls, User requests that still fail 100%. The tree shows checkout tried 3 times, pricing 9 and inventory 27, and the table lists 1, 3, 9 and 27.

In plain words: two retries per call looks harmless in one config file, but three of them in a row turn one request into 27 at a service that is already down, and every one of them fails. The retries add load exactly when the bottom can least take it, and they cannot help, because it is down. A budget, or retries at one layer only, takes it back to a handful. Try App retries in the mesh to see the same chain with a client library that also retries twice: 9 tries per call, 729 at the bottom. Or set Mesh to Linkerd: 1 retry per call, 2 tries, and 8 calls reach the bottom instead of 27.

Retry layers that are easy to miss#

  • The mesh. Istio retries twice by default, on connection failures, refused streams and gRPC UNAVAILABLE. Not on an HTTP 503, unless you add it. So over plain HTTP only the proxy in front of a down service retries, and its callers get a 503 they pass up: 3 calls reach the bottom, not 27, and a 503 from the bottom reaches it once. Linkerd retries nothing until a route is configured to, and then once, unless the configuration gives a count.
  • A client library with its own retry policy, inside a service’s code.
  • The app or browser at the top, retrying the whole request.
  • A load balancer or gateway in front of it all.

Where it stops being true#

  • Failures are independent, and only at the bottom. Real failures are correlated, and an overloaded service in the middle fails on its own too.
  • A layer retries only failures it sees. If a service in the middle turns a downstream failure into something its caller’s policy does not retry (an HTTP 503 under Istio’s defaults, say), the multiplication stops there. This is the worst case: every layer passes the failure up in a form the next one retries.
  • The retry budget is a simple cap of 1 + budget per layer. Envoy measures it over concurrent requests, and the SRE book’s version over a client’s recent requests, so the capped numbers are a ceiling.
  • Timeouts are not modelled. The timeout-budget tool covers whether the retries can fit in the time at all.

Source and references

index.ts335 lines, TypeScript
TypeScript
/**
 * How many calls reach the bottom of a call chain when every layer retries.
 *
 * Retries look additive in a config file, "2 here, 3 there", and they are not. A layer that retries wraps the one
 * below it, so it multiplies: code that makes 3 attempts through a proxy that makes 3 makes 9, and every call further
 * down sees all of them. With depth `d` and `n` attempts per call, the bottom sees `n^d` calls for one user request
 * when it is failing, which is exactly when it can least afford them.
 *
 * The arithmetic, layer by layer from the bottom up: a layer making `n` attempts over something that fails with
 * probability `f` makes `1 + f + ... + f^(n-1)` attempts on average, and fails with probability `f^n`. Failures are
 * independent and only at the bottom; nothing in the middle fails on its own.
 *
 * A retry budget, as Envoy and the SRE book describe it, caps each layer's retries at a share of its requests, so
 * no layer can multiply by more than `1 + budget`. Here it is that cap and nothing more: real budgets are measured
 * over a window or over concurrent requests, so read the capped numbers as a ceiling.
 *
 * The same arithmetic as `service-mesh/retry_fanout.py` in mainlog-code; the fixtures are its output.
 *
 * The answer comes first, with its multiplication, then the chain it multiplies along, then what to change, the
 * biggest cut first. Each change is scored by running the same arithmetic with that change made, so "turn on a
 * budget" says how many calls it leaves, not just that it helps.
 */
import type { Cell, Field, GraphEdge, GraphNode, Output, Tool } from "@toolbench/sdk";


/**
 * `platform` is the Mesh preset select. It only fills `mesh` in the form, so the arithmetic reads the number and never the
 * name: a reader who picks Istio and then types 5 gets 5.
 */
type Input = { depth: number; platform?: string; mesh: number; app: number; client: number; fail: number; budget: string };


class InputError extends Error {
  readonly input: string;


  constructor(message: string, input: string) {
    super(message);
    this.name = "InputError";
    this.input = input;
  }
}


type Layer = { call: number; who: "proxy" | "code" | "client"; attempts: number };


/** The retry layers from the bottom up. Call 1 is the deepest. */
function layersOf(depth: number, mesh: number, app: number, client: number): Layer[] {
  const out: Layer[] = [];
  for (let c = 1; c <= depth; c++) {
    out.push({ call: c, who: "proxy", attempts: 1 + mesh });
    if (app) out.push({ call: c, who: "code", attempts: 1 + app });
  }
  if (client) out.push({ call: depth + 1, who: "client", attempts: 1 + client });
  return out;
}


/** Fixed places, without a trailing ".00" on whole numbers, so 27 reads as 27 and 1.728 as 1.728. */
function num(x: number, places = 2): string {
  if (Number.isInteger(x)) return x.toLocaleString("en-US");
  return Number(x.toFixed(places)).toLocaleString("en-US", { maximumFractionDigits: places });
}


function chance(f: number): string {
  if (f === 0) return "0%";
  if (f >= 0.0001) return `${num(f * 100, 2)}%`;
  return `1 in ${Math.round(1 / f).toLocaleString("en-US")}`;
}


/** Fewest calls at the bottom, worst case, for a chain of retry layers given as attempts each, after `cap`. */
const product = (attempts: number[], cap: (m: number) => number) => attempts.reduce((w, n) => w * cap(n), 1);


/**
 * Calls at the bottom for one user request, walking the layers up from the bottom: the worst case, when every try
 * fails; the average, when each call to the bottom fails with probability `p`; and the chance the user's request
 * still fails. Exported so the tests can check the exact arithmetic behind a number the page rounds.
 */
export function bottomCalls(
  depth: number,
  mesh: number,
  app: number,
  client: number,
  p: number,
  budget: number | null,
): { worst: number; expected: number; stillFails: number } {
  const cap = (m: number) => (budget === null ? m : Math.min(m, 1 + budget));
  let worst = 1;
  let expected = 1;
  let f = p;
  for (const layer of layersOf(depth, mesh, app, client)) {
    worst *= cap(layer.attempts);
    let m = 0;
    for (let j = 0; j < layer.attempts; j++) m += f ** j;
    expected *= cap(m);
    f = f ** layer.attempts;
  }
  return { worst, expected, stillFails: f };
}


/**
 * A count of calls as a reader should read it. Whole numbers are exact; anything else comes from a budget or an
 * average, which are models, so one decimal and "about": 1.728 is a cap computed as if the budget always binds,
 * and three decimals claimed a precision the model does not have.
 */
function calls(x: number): string {
  if (Number.isInteger(x)) return `${x.toLocaleString("en-US")} ${x === 1 ? "call" : "calls"}`;
  return `about ${num(x, 1)} calls`;
}
const bare = (x: number) => (Number.isInteger(x) ? x.toLocaleString("en-US") : `about ${num(x, 1)}`);


export default {
  run(input: Input) {
    try {
      const depth = Number(input.depth);
      const mesh = Number(input.mesh);
      const app = Number(input.app);
      const client = Number(input.client);
      const p = Number(input.fail) / 100;
      if (!(Number.isInteger(depth) && depth >= 1 && depth <= 12))
        throw new InputError("Depth is a whole number of calls, 1 to 12.", "depth");
      for (const [v, key, label] of [
        [mesh, "mesh", "Proxy retries"],
        [app, "app", "Code retries"],
        [client, "client", "Client retries"],
      ] as const) {
        if (!(Number.isInteger(v) && v >= 0 && v <= 10))
          throw new InputError(`${label} must be a whole number from 0 to 10.`, key);
      }
      if (!(p >= 0 && p <= 1)) throw new InputError("The failure share is a percentage from 0 to 100.", "fail");
      const budget = input.budget === "none" ? null : Number(input.budget) / 100;
      if (budget !== null && !(budget > 0 && budget < 1)) throw new InputError("Unknown budget.", "budget");


      const layers = layersOf(depth, mesh, app, client);
      const cap = (m: number) => (budget === null ? m : Math.min(m, 1 + budget));
      const { worst, expected, stillFails } = bottomCalls(depth, mesh, app, client, p, budget);


      const perCall = (1 + mesh) * (app ? 1 + app : 1);
      const perCallWorst = cap(1 + mesh) * (app ? cap(1 + app) : 1);
      const clientMul = client ? cap(1 + client) : 1;


      /* Calls each service receives, top first: the top service gets the user's tries, the bottom gets the most. */
      const services: { name: string; worst: number }[] = [];
      for (let level = 0; level <= depth; level++) {
        services.push({
          name: level === 0 ? "Top service" : level === depth ? "Bottom service" : `Service ${level + 1}`,
          worst: clientMul * perCallWorst ** level,
        });
      }


      /*
       * ⚠️ The answer comes first, said once. It used to be said four times, in a banner, a tile, the tree and a
       * table, and the number a reader came for was the same in all four. No tone: a count is not a verdict, and
       * the post this embeds in is about what the count means.
       */
      const key: Output = {
        kind: "fields",
        role: "key",
        fields: [
          {
            label: "Calls at the bottom service",
            value: calls(worst),
            note:
              budget === null
                ? "For one user request, while it is down."
                : `At most, for one user request while it is down, if the ${num(budget * 100)}% budget binds.`,
          },
        ],
      };


      /*
       * The multiplication behind it. With a budget the factors are the cap, not the retries, because the cap is
       * what multiplies.
       */
      const factors =
        budget === null
          ? [...(client ? [`${1 + client} from the client`] : []), ...Array.from({ length: depth }, () => String(perCall))]
          : layers.map(() => num(1 + budget, 2));
      const formula =
        budget === null
          ? `${perCall} ${perCall === 1 ? "try" : "tries"} per call${app ? ` (${1 + app} in the code, each through ${1 + mesh} in the proxy)` : ""}, ${depth} ${depth === 1 ? "call" : "calls"} deep: ${factors.join(" x ")} = ${bare(worst)}`
          : `${layers.length} retrying ${layers.length === 1 ? "layer" : "layers"}, each capped at ${num(1 + budget, 2)} times its traffic: ${factors.join(" x ")} = ${bare(worst)}`;


      /*
       * Each change, scored by the same arithmetic with it made. Only changes that lower the worst case are offered,
       * and the biggest reduction comes first. The reason each works is in the method notes, so the table stays
       * two columns that fit a phone.
       */
      const proxyOnly = (n: number) => Array.from({ length: depth }, () => n);
      const withClient = (xs: number[]) => (client ? [...xs, 1 + client] : xs);
      const candidates: { what: string; after: number; why: string }[] = [];
      if (app && mesh) {
        candidates.push({
          what: "Retry in the code or the proxy, not both",
          after: product(withClient(proxyOnly(Math.max(1 + app, 1 + mesh))), cap),
          why: `The code's ${1 + app} tries each go through the proxy's ${1 + mesh}, so every call makes ${perCall}.`,
        });
      }
      if (depth > 1 && perCall > 1) {
        candidates.push({
          what: "Retry only at the top",
          after: product(withClient([perCall]), cap),
          why: `${depth} calls in a row each retry, so their tries multiply. Retrying once, at the top, and failing fast below it stops that.`,
        });
      }
      if (budget === null) {
        candidates.push({
          what: "A 20% retry budget",
          after: product(
            layers.map((l) => l.attempts),
            (m) => Math.min(m, 1.2),
          ),
          why: "Retries are allowed only while they are a small share of a layer's requests: 20% is Envoy's default once a budget is turned on. Envoy's always allows 3 at once, so at low traffic it does not bind.",
        });
      }
      if (client) {
        candidates.push({
          what: "No client retries",
          after: product(
            layersOf(depth, mesh, app, 0).map((l) => l.attempts),
            cap,
          ),
          why: `Each of the client's ${1 + client} tries goes down the whole chain again. Waiting for a Retry-After has the same effect.`,
        });
      }
      const changes = candidates.filter((c) => c.after < worst * 0.95).sort((a, b) => a.after - b.after);


      const parts: Output[] = [
        key,
        { kind: "text", role: "formula", mono: true, text: formula },
        chain(depth, client, perCallWorst, services, budget),
      ];
      if (changes.length) {
        parts.push({
          kind: "table",
          role: "comparison",
          caption: "What each change leaves at the bottom",
          columns: [
            { label: "Change", wrap: "prose" },
            { label: "Calls at the bottom", align: "end", mono: true, wrap: "none" },
          ],
          rows: changes.map((c): Cell[] => [c.what, { text: bare(c.after), mono: true }]),
        });
      }


      /*
       * The partial-failure numbers only mean something when the bottom is not simply down: at 100% the average is
       * the worst case and every request fails, which says nothing the answer has not.
       */
      if (p < 1) {
        parts.push({
          kind: "fields",
          role: "metrics",
          fields: [
            {
              label: `When the bottom fails ${num(p * 100, 2)}%`,
              value: calls(expected),
              note: "The average for each user request: extra load on a service that is already failing.",
            },
            ...(budget === null
              ? [
                  {
                    label: "User requests that still fail",
                    value: chance(stillFails),
                    note: `If failures were independent. Real ones come in runs, so this is the best case. Without retries, ${chance(p)}.`,
                  },
                ]
              : []),
          ],
        });
      }


      const method: Field[] = [
        {
          group: "How this was worked out",
          label: "Every layer retries the error it gets",
          value: "As a gRPC service that passes UNAVAILABLE up does",
          mono: false,
          note: "Failures are independent and happen only at the bottom; nothing in the middle fails on its own.",
        },
      ];
      if (budget !== null) {
        method.push({
          group: "How this was worked out",
          label: "The budget",
          value: `Each layer at most ${num(1 + budget, 2)} times its traffic`,
          mono: false,
          note: "Real budgets are counted over a window or over requests in flight, so read the capped numbers as a ceiling.",
        });
      }
      for (const c of changes) method.push({ group: "Why each change works", label: c.what, value: c.why, mono: false });
      parts.push({ kind: "fields", role: "method", fields: method });


      return { kind: "group", parts };
    } catch (error) {
      if (error instanceof InputError) return { kind: "error", message: error.message, input: error.input };
      throw error;
    }
  },
} satisfies Tool<Input>;


/**
 * The chain, one node per service, top to bottom, each edge labelled with how many times it multiplies.
 *
 * ⚠️ Not a tree of every try. The earlier tree followed the first try down and stood the others in as "try 2: 9 more
 * below, not drawn", which described the renderer rather than the system. One node per service with its total, and
 * "x3" on every hop, says the only thing the picture is for: it multiplies at every hop.
 */
function chain(
  depth: number,
  client: number,
  perCallWorst: number,
  services: { name: string; worst: number }[],
  budget: number | null,
): Output {
  const times = (x: number) => `x${num(x, 2)}`;
  const nodes: GraphNode[] = [{ id: "user", label: "User request", shape: "pill" }];
  const edges: GraphEdge[] = [];
  services.forEach((s, level) => {
    const id = `s${level}`;
    nodes.push({ id, label: s.name, note: calls(s.worst), role: level === depth ? "bottom" : "service" });
    const from = level === 0 ? "user" : `s${level - 1}`;
    const by = level === 0 ? (client ? services[0]!.worst : 1) : perCallWorst;
    edges.push({ from, to: id, ...(by !== 1 ? { label: times(by) } : {}) });
  });
  return {
    kind: "graph",
    graph: {
      label:
        budget === null
          ? "Calls each service receives for one user request while the bottom is down"
          : `Calls each service receives for one user request while the bottom is down, with a ${num(budget * 100)}% budget`,
      layout: "layered",
      direction: "TB",
      spacing: "compact",
      nodes,
      edges,
    },
  };
}

Related posts

2 items
  • Oct 7, 2026 28 min read Systems

    What a service mesh costs

    Your shop now runs 14 services in three languages, and security wants every call between them encrypted and identified. A mesh does that without touching the code, for about a fifth of a millisecond per hop and a tenth of a core per proxy per 1,000 requests a second, measured here with real Envoy. It also brings retry and queue defaults nobody chose. Over plain HTTP they retry less than most people expect; over gRPC, or after one common edit, a failing service gets 27 calls for one checkout.

    Down to the wireinfrastructuremicroservicesnetworking
  • 5 posts Series

    Down to the wire

    Distributed systems at the point where the theory meets the pager.

  • Systems

    What a service mesh costs

    Your shop now runs 14 services in three languages, and security wants every call between them encrypted and identified. A mesh does that without touching the code, for about a fifth of a millisecond per hop and a tenth of a core per proxy per 1,000 requests a second, measured here with real Envoy. It also brings retry and queue defaults nobody chose. Over plain HTTP they retry less than most people expect; over gRPC, or after one common edit, a failing service gets 27 calls for one checkout.

    infrastructuremicroservices
    Oct 2026
  • Series

    Down to the wire

    Distributed systems at the point where the theory meets the pager.

    5 posts