What a state machine buys an agent

What the state machine controls#

A free-form loop gives the model the tools and a system prompt describing how to use them. At each step, the model decides what to do next. The prompt might say to check availability before reserving and to confirm before paying, but those instructions still have to be followed on each call.

In this state machine, the task moves through gather, search, confirm, book and notify. Two more states handle trouble: report tells the user what happened after a booking exists, and degraded handles the model becoming unavailable. The current state determines which tools may execute. Code decides when the task can move on.

An airport kiosk is a useful comparison. Its current screen determines what you can do, and some steps, such as printing a boarding pass once the details are known, need no decision from you. The booking workflow has the same distinction between a step needing interpretation and a step whose inputs are already available.

Three terms make the implementation precise:

• A state records where the task stands. Being in confirm means something different from being in notify: in the latter, a booking already exists.
• An event is something code can observe, such as a tool result, the user’s confirmation or a restaurant webhook. slot_free, user_yes and slot_gone are events. A sentence in the model’s reply isn’t automatically one.
• A transition defines what an event does in a particular state. The model can work within the state, but it doesn’t choose the next state in this design.

Anthropic’s Building effective agents draws the same line: “Workflows are systems where LLMs and tools are orchestrated through predefined code paths,” while “Agents, on the other hand, are systems where LLMs dynamically direct their own processes and tool usage.” A booking can use both: language understanding where it helps, with a workflow around the parts that have definite prerequisites.

How I measured it#

The seven steps are: read the request, check availability, ask for confirmation, reserve, pay the deposit, look up the group and text them. At each step, every design receives the same short history of what has happened so far.

The steps are scripted. Model replies are recorded, but they don’t steer the next request. The free-form loop also gets an eighth, closing call to recognize that the work is done. This measures the model work each design asks for on that path; it doesn’t measure how often an autonomous booking agent would complete the task correctly.

I sent the requests to Claude Opus 5.5 twice and Claude Sonnet 5 once. The API supplied the token counts, including its tool-use instructions and cache usage. Input-token counts repeated exactly. Output and wall-clock time varied between runs.

The first comparison keeps a model call at each of the seven steps and changes how the request is built. These are the relevant branches in booking.py:

1
def request(design: str, i: int) -> dict:
2
    # ...
3
    state = RUN[i][0]
4
    instr, allowed = STATES[state]
5
    if design == "free":
6
        return {"tools": list(TOOLS.values()), "system": FREE_SYSTEM, "user": history(i)}
7
    if design == "states":
8
        return {"tools": [TOOLS[t] for t in allowed],
9
                "system": f"You are a dinner-booking assistant. Current step: {state}. {instr}", "user": history(i)}
10
    if design == "fixed":
11
        # ...
12
        return {"tools": list(TOOLS.values()), "system": FREE_SYSTEM,
13
                "user": f"{history(i)}\n\nCurrent step: {state}. {instr} Allowed tools: {', '.join(allowed)}."}
14
    # ...

What the model sees in each design#

Code like that is easy to misread, and the model never sees the code. It sees the request. Here is what each design sends at step 6, looking up the group, with the booking and the deposit already done. The history is identical in all three:

user: Friday at 7 for four, and tell the group
tool: Fri 19:00 free for 4, deposit $20
user: yes, book it and pay the deposit
tool: committed R1 Fri 19:00 x4
tool: receipt P1 $20

Free-form sends all eight tools and the same long system prompt at every step. The rules for ordering live in that prompt, and the model has to follow them on every call:

tools:  check_availability, reserve_table, cancel_reservation, lookup_reservation,
        pay_deposit, get_contacts, text_group, notify_user

system: You are a dinner-booking assistant acting for the user. Book restaurant tables,
        handle deposits, and tell the user's group. Rules: never reserve before checking
        availability; never reserve or pay without the user's explicit confirmation of the
        exact slot and amount; text the group only after the booking is committed, and only
        once; if a booking outcome is unknown, look it up before trying again; never cancel a
        reservation unless the user asked. Ask the user when anything is ambiguous. When the
        task is complete, say so and stop.

Per-state sends only the tools the current state allows, and a one-line instruction for that state. There are no ordering rules, because the order is the state machine’s job:

tools:  get_contacts, text_group

system: You are a dinner-booking assistant. Current step: notify. Tell the dinner group the
        confirmed booking, once.

Followed through the whole booking, each state carries its own instruction and its own short tool list. In the final version of the machine, described further down, code runs five of the seven steps, so only gather and notify send the model an instruction at all:

Fixed sends exactly what free-form sends, all eight tools and the long system prompt, and adds the state to the end of the user message. Code refuses any call outside the allowed list:

Current step: notify. Tell the dinner group the confirmed booking, once. Allowed tools: get_contacts, text_group.

Each tool in those lists is a block of text too. This is one of the eight, as the model receives it:

1
{
2
  "name": "reserve_table",
3
  "description": "Books a table at the restaurant for the party, day and time given. This commits a real reservation and may charge a no-show fee, so only call it after the user has confirmed the exact slot. Send the idempotency key so a retry never books twice. Returns the reservation ID.",
4
  "input_schema": {
5
    "type": "object",
6
    "properties": {
7
      "day": {"type": "string"},
8
      "time": {"type": "string"},
9
      "party": {"type": "integer"},
10
      "key": {"type": "string", "description": "Idempotency key"}
11
    },
12
    "required": ["day", "time", "party", "key"]
13
  }
14
}

The fixed design looks redundant until caching enters the comparison: it keeps the start of every request identical, which is what the cache needs.

Costs below are relative units. One uncached input token costs 1; an output token costs 5. That is Anthropic’s published input-to-output ratio: 4 USD and 20 USD per million tokens for Opus 5.5, and 2 USD and 10 USD for Sonnet 5. Other platforms publish their own prices, so read these as ratios, not an invoice. “Input cost” excludes output; “total” includes it.

Where the tokens go#

A tool definition is part of the model’s input: its name, description and argument schema. Eight carefully described tools carry a fair amount of text before the conversation history even begins. When tools are present, the API also adds its own tool-use instructions. Anthropic’s pricing documentation lists 286 tokens of that overhead for Opus 5.5.

The free-form design’s fixed part, the tools, system prompt and API-added instructions, was 1,561 tokens.¹ The history specific to each step was only 93 to 189 tokens. Most of each request was material the model had already seen.

Without caching, the per-state design has the expected advantage: it sends fewer tool definitions and a shorter instruction. With caching, the service charges different rates for different parts of that input.

The cache changes the arithmetic#

Prompt caching reuses an already processed request prefix. Under the pricing used here, writing the prefix costs 1.25 times the normal input rate. Reading it costs 0.05 times on Opus 5.5 and 0.1 times on Sonnet 5. Reusing a long prefix can therefore be cheaper than processing a much shorter one afresh.

The prefix is everything up to a cache breakpoint, a marker in the request. In the demo, it sits on the system prompt:

1
kwargs = dict(model=PREFIX + model, max_tokens=4000,
2
              system=[{"type": "text", "text": f"[{tag}] {req['system']}", "cache_control": {"type": "ephemeral"}}],
3
              messages=[{"role": "user", "content": req["user"]}])
4
if req["tools"]:
5
    kwargs["tools"] = req["tools"]

The API builds the prefix in a fixed order: “Cache prefixes are created in the following order: tools, system, then messages.” So at step 6, the three designs’ requests, and the part of each one billed as a cache read, look like this:

Two details determine whether that works:

1. Tools come first in the prefix. The cache documentation says “Modifying tool definitions (names, descriptions, parameters) invalidates the entire cache.” A different per-state tool list changes the prefix before the rest of the request is considered.
2. There is a minimum cacheable length. It was 512 tokens on Opus 5.5 and 1,024 on Sonnet 5. A shorter prefix doesn’t produce an error; it simply isn’t cached.

The free-form loop’s 1,561-token prefix is identical over the seven steps. It is written once and read six times. Each per-state tool list has a different prefix, used only once or twice, and most of them are short. Subtracting each step’s history from what the per-state request sent gives its prefix on Opus 5.5:

On Sonnet 5, with its 1,024-token minimum, none was cached.

Laid out step by step, that is one prefix written once and read six times, against a prefix per state, of which only one clears the 512-token line:

Step six shows the effect without needing a whole-run total. Per-state tools send 666 tokens, all charged at the normal rate. Free-form sends 1,729 tokens, but 1,561 are a cache read. Its input cost is about 168 + 1,561 × 0.05 = 246. The larger request costs substantially less. Over the seven steps, this is what the recorded run gives:

== Claude Opus 5.5, the seven scripted steps, repetition 1
  free-form                  sent 11,876  cached-read  9,366  input cost  3,368.6  total 11,218.5
  per-state tools            sent  4,675  cached-read    761  input cost  4,142.3  total 10,922.3
  fixed tools + code check   sent 12,236  cached-read  9,378  input cost  3,717.7  total  9,277.6
  rep 1 per-state tools: 61% fewer tokens sent; input cost +23% step for step, +14% counting the closing call; total -2.6%
  rep 2 per-state tools: 61% fewer tokens sent; input cost +23% step for step, +14% counting the closing call; total -3.7%

Per-state input costs 23% more over the matched steps. Include the free-form loop’s closing call in the comparison and that becomes 14% more. Sonnet 5 shows the same pattern: 30% more step for step, or 19% more when the closing call is included.

Output changes the size of the overall difference. It accounts for about 70% of the free-form seven-step total, and prompt caching doesn’t discount it. Over those seven steps, per-state tools were 2.6% and 3.7% cheaper in the two Opus runs, and 11% cheaper on Sonnet. The token reduction is real, but it doesn’t support a large total-cost claim once the cache is working.

Keep the tools, restrict the call#

The fixed design separates the tool definitions sent to the model from the calls the application will execute. Every request carries all eight definitions, preserving the cached prefix. The current state and allowed tools go in the message, and code refuses a call outside that set. In kiosk terms, every button stays on the screen, but the ones that don’t belong to this step do nothing when pressed.

OpenAI’s function-calling guide describes this directly: “You might want to configure an allowed_tools list in case you want to make only a subset of tools available across model requests, but not modify the list of tools you pass in, so you can maximize savings from prompt caching.” Anthropic’s documented tool_choice options are auto, any, tool and none; none expresses a subset. On Opus 5.5, according to the tool-use documentation, “any and tool return a 400 error.” The per-state restriction in this design is enforced by application code.

Its Opus input cost was about 3,718 units, 10% above free-form over the seven steps. With output included, its total was 17% lower on Opus but 31% higher on Sonnet. What the model chose to write moved that result considerably.

The consistent benefit is the execution check. A tool definition can remain in the cached prompt without granting permission to run that tool in every state. That permission is decided at the point of execution.

The real saving: steps with no model#

Keeping a model call at every step preserves a lot of work that doesn’t need a model. Once the day, time and party size have been extracted, checking availability is a tool call with known arguments. Reserving uses the confirmed slot and its operation key. Paying uses the amount the user approved.

Five steps can run from facts the application already holds. The free-form loop asks the model at each one anyway, then makes another call to notice the task is finished.

On the first Opus run, that difference looks like this:

Confirmation is a particularly useful step to make deterministic. The runtime already holds a pending action: the exact work it will perform if the user says yes. Writing the confirmation question from that record means the user approves what will actually run, including the deposit. That is the arrangement in the earlier runtime post.

Let the state machine execute those five steps directly, and the complete Opus booking looks like this:

Costs rounded to whole relative units. These are model-call times, not end-to-end booking latency: the tools in the experiment are fakes and add no time. Neither of the two remaining requests is long enough to cache, so tokens sent and input cost are the same.

The two-call design gets no cache saving, and still reduces the total by 72% and 73% across the two Opus runs. Sonnet’s single run went from 30.4 to 8.2 seconds of model time, with a total of 16,069 versus 4,300 units: about 73% less.

Each eliminated call removes both its input and whatever output the model would have generated. That is a much larger change than adjusting the tool list on a call that still happens.

Two calls is a property of this path, not every dinner request. The first call extracts all three required values at once. If something is missing, the task goes through gather again. The design also chooses to use the model for the group message. The useful question is whether each step needs interpretation or can proceed from facts already available to code.

The lab puts the four designs beside each other. Start with per-state tools and switch caching off, then on. Compare input cost with total cost. Next try fixed tools with the code check, and finally the design that runs five steps without the model. The model selector shows how those comparisons changed between Opus and Sonnet:

From thirteen arrows to 119 decisions#

Here is the booking as a state machine, with the thirteen transitions of its first drawing and the four into and out of degraded that the table added later:

The booking’s first diagram had thirteen transitions: the normal path and the obvious branches. It looked reasonably complete. Turning it into a state-event table left 106 cells undecided.

Some event names carry important distinctions. unknown means the reservation request timed out and its result is not known. The task stays in book and looks up the reservation by its key before trying again. refused means the restaurant actually said no. committed means the table is reserved, but the task stays in book, because the deposit is still due; paid means the deposit is settled, or that none was due, and only that moves the task on to notify. told means the message has been shown to the user.

The table puts all seven states down one side and all seventeen observable events across the top. That makes 119 combinations. A diagram records the transitions you drew; the table also asks about the event arriving in a state that wasn’t expecting it.

Every cell gets an explicit outcome:

• Move: enter another state.
• Stay: the event is expected here and doesn’t change the state.
• Defer: hold the event until the work in flight resolves, then handle it.
• Reject: the event shouldn’t arrive here; log it and take no action.
• Ignore: it’s irrelevant in this state; drop it without logging. The booking doesn’t use this outcome, but the state machine analyzer supports it.

Most of the missing cells are straightforward. A slot_free result while the agent is texting the group is a result that state never requested. Fifty-nine cells in this table are rejections.

Three event columns need different treatment. A late committed reply is evidence that a reservation exists, even if the task has moved away from the state that requested it. Rejecting the reply doesn’t undo the reservation; it loses track of it. Outside book, the table sends that event to report, and a late paid goes there too, because the money has already moved. An unexpected sent result also needs accounting for, because the text has already gone out. Here are two cells as the demo prints them:

search    committed          move   report    a reservation exists that this state didn't make; it is real, so report it
report    user_revises       defer  report    a booking exists; finish telling the user, then revise it rather than start again

The second rule is easy to miss in a diagram. Before a reservation exists, “make it Saturday” can send the task back to gather. In report, the booking already exists. Starting the original workflow again can create another reservation. The revision must operate on what the earlier work left behind.

Four events can arrive across the workflow, and had no arrows in the initial drawing:

The revision and webhook cases are worked through in When the user changes their mind mid-task and When a turn doesn’t start with the user. The table puts those cases next to every state, so they have to be decided before an incident supplies the missing event.

The demo checks that every cell declares an outcome and a reason:

1
def test_every_cell_has_a_declared_outcome():
2
    t = booking.table()
3
    assert len(t) == len(booking.STATES) * len(booking.EVENTS) == 119
4
    for (state, event), (kind, nxt, why) in t.items():
5
        assert kind in KINDS, (state, event)
6
        assert why, (state, event)
7
        # ...

Completeness is only part of the check. Another test walks the table and fails if a late committed, paid or sent event is rejected. That encodes a specific requirement: an acknowledged external effect must not disappear just because its reply arrived in an inconvenient state.

There is one thing the table can’t do: decide an event nobody listed. What happens if the deposit is declined? The task would sit in book, holding a reservation, with no event to move it, and every cell in the table would still be decided. The grid checks every combination of the states and events you wrote down. It can’t tell you which event is missing. That question still needs someone to walk the real failure cases, here the restaurant, the payment and the user, and ask what each can send back.

You can run the same check on your own machine. Write each state as a header, with what each event does beneath it, and the tool counts how many state and event pairs are decided, shows the grid with the least-decided events on top and a red ? wherever no rule says what happens, draws the states with the moves between them, and lists the states that can’t be reached or left and any cell two rules disagree about. Run the booking as first drawn: 106 of 119 cells are ?, and degraded can’t be reached or left. Then run the version with every cell decided, and “Two people edited this one”, where one cell reads “2 rules”. Set “Also write” to markdown to take the grid into your own docs:

The pytest skeleton, under “Also write”, gives each state-event pair a case, with an undecided value where a decision is still missing. It checks the table’s specification; the application’s handlers still need to be connected to those expected outcomes.

The state the breaker trips into#

degraded is where the task waits when it can’t use the model. The provider might be unavailable, or a budget might have stopped further calls. Circuit breakers for inference covers when to stop. The state machine defines what the application can still do afterwards.

If task progress is represented only in a conversation history, reporting what happened requires interpreting that history. With an explicit state and recorded booking and payment facts, code can produce the message itself. A failure in notify doesn’t have to turn into an ambiguous “something went wrong”:

Booked: Friday 19:00 for 4 (R1). Deposit paid: $20 (P1).
Not done yet: texting the group. I'll send it when I can, or you can send it yourself.

The reservation and payment references come from stored results. The unfinished action comes from the state. None of that requires asking the unavailable model to explain itself.

Only two states in this design call the model: gather and notify. The other five can keep running during a provider outage. An outage during book doesn’t prevent the already authorized reservation or payment, because those operations don’t need the model.

Recovery also has a defined destination. When model_back arrives in degraded, the task returns to the state it left. Resuming notification after a paid reservation is different from restarting the booking.

What the free-form loop relies on instead#

The free-form design in this demo keeps the ordering rules in its system prompt. The recorded calls did not reserve before checking availability or text before a booking existed. The experiment doesn’t show those rules failing.

It does show how much room the prompt is being asked to constrain. Eight tools, callable at every step, allow 8⁴ = 4,096 sequences of four calls. Of those, 1,400 reserve a table before the first availability check, whether that check happens later or never.

That 34% is a count of expressible sequences, not a measured failure rate or a claim that the model chooses tools randomly. In the enforced state machine, those sequences cannot execute: reserve_table isn’t permitted until book, and book is not reachable until confirmation.

That still leaves ordering and repeated actions within a state, as the allowed-list examples showed. State restrictions, operation keys and step-specific checks cover different parts of the workflow. A state machine is useful because those rules can be enforced and tested without relying on a particular model reply.

When a state machine is the wrong tool#

A booking, checkout, refund or onboarding flow usually has stages that can be written down. An open-ended research task or coding session may not. If the only honest state name is “work on the problem,” a one-row table adds little.

Anthropic’s guidance is to start simple: “When building applications with LLMs, we recommend finding the simplest solution possible, and only increasing complexity when needed.” A state machine can control a bounded operation inside a more open-ended agent without describing everything that agent will ever do.

There are also designs that keep explicit states while allowing the model to decide some transitions. StateFlow is one example. It reports that it “achieves 13% and 28% higher success rates compared to ReAct in InterCode SQL and ALFWorld benchmark, with 5x and 3x less cost respectively.” Those are results for that approach on those benchmarks, rather than measurements from this booking experiment.

The 12-factor agents control-flow guidance states the design principle plainly: “Build your own control structures that make sense for your specific use case.”

There is a practical limit in the other direction too. On a real-time media platform I worked on, each application had acquired its own hand-written state machine. The useful change was to replace those separate implementations with a deterministic lifecycle contract the applications could share.

Splitting the dinner agent into Bookings, Calendar and Messaging, as the handoff and event-bus posts do, can produce the same problem. The agents need different task states, but they can share what counts as an event, what defer and reject mean, and what must be recorded when entering degraded. That gives their tables a common meaning without forcing every agent through the same workflow.

Run it yourself#

The demo is in mainlog-code/agent-state-machines. booking.py contains the tools, prompts, scripted path and transition table. live.py makes the model requests and records usage. demo.py prints the recorded results without calling a model.

cd mainlog-code/agent-state-machines

python3 demo.py
python3 demo.py --table

python3 -m pip install pytest==8.3.5
python3 -m pytest -q tests

To collect a new live run, python3 live.py makes 66 model calls and needs an API key. The times and output will differ from the recorded run.

CI never calls a model. It runs the tests, checks the lab’s data against the recorded results and verifies the totals quoted in the post. The cost comparison can be reproduced from that record; the state-event decisions can be inspected and tested locally.