Introducing toolbench: a tiny framework for interactive web tools

A function and a manifest#

The smallest tool takes a name and says hello. This is the whole function:

1
import type { Output, Tool } from "@toolbench/sdk";
2

3
type Input = { name: string };
4

5
export default {
6
  run({ name }): Output {
7
    const who = name.trim();
8
    if (who === "") return { kind: "error", message: "Type a name.", input: "name" };
9
    return { kind: "text", text: `Hello, ${who}.` };
10
  },
11
} satisfies Tool<Input>;

run receives plain values and returns a result tagged with its kind: a line of text, or an error that names the input at fault. The import is types only, so the tool has no runtime dependency on toolbench, and nothing in it knows about the DOM.

This is its manifest:

1
{
2
  "sdk": 1,
3
  "id": "hello",
4
  "name": "Hello",
5
  "blurb": "Says hello to a name.",
6
  "version": "1.0.0",
7
  "capabilities": ["pure"],
8
  "runtime": { "entry": "index.ts" },
9
  "kinds": ["text", "error"],
10
  "inputs": [{ "id": "name", "type": "text", "label": "Name", "default": "Ahmed" }],
11
  "help": "README.md"
12
}

inputs describes the form, and kinds lists the result kinds the function may return. sdk: 1 is the version of the contract the tool is written against. pure says the answer depends on the declared inputs and nothing else: no clock, no unseeded randomness, no network and no files. The validator insists on error in kinds, because every tool can be given bad input, which is why even this one has an empty-name check.

For those two files the runtime draws a labelled field, a Run button that also answers to Enter, and the result. When the function returns an error naming name, the runtime marks that field invalid and ties the message to it with aria-describedby, the attribute a screen reader uses to read a description with its field.

A third file, cases.json, holds the tool’s known answers: inputs paired with the exact results they must produce, here for the default name, a name with spaces around it, and an empty one. The example is in mainlog-code, runnable, with its test and the page in the screenshot.

The choices, and what each costs#

My first plan for mainLog was a Svelte island per tool, an island being an interactive component the page hydrates on its own. It is the quickest way to ship one tool, and it makes every later tool a small front-end project of its own.

The serious alternative was the middle ground: shared form and result components, with each tool’s computation pulled out into a plain function. That is close to where toolbench ended up, and a good shared wrapper would centralise much of the glue. The difference is a deliberate restriction. In toolbench a tool author can only describe inputs and return data, so no tool can grow a UI of its own, nothing in a tool imports a component, and one harness can check every tool the same way. For small tools whose value is the computation, I think that restriction is worth more than the freedom it takes away. The price is a vocabulary the framework has to define and keep, a handful of input types and a closed list of result kinds, where a component could draw anything.

An iframe per tool keeps tools apart in a different way. Each one is a second document with its own styles, so following the page’s theme needs a bridge between the two documents, and sizing it to its content needs another. Both could be shared code, but I didn’t want every tool in a post to be a page inside a page.

I also chose a published package over a shared folder inside the blog. A folder is cheaper on day one, but nothing stops code in it from importing the blog’s own modules. Toolbench lives in its own repository and depends on nothing, so reaching for the blog’s code isn’t an option, and an assumption about the site has to be made explicit in the contract or not made at all. The blog pins a version, so when something breaks there is a version to point at. The cost is the work of releasing it, a version for every change and notes a reader can follow. An upgrade can also change how every tool looks at once, because the contract covers what a result means, not its pixels.

For the form, a JSON Schema form generator such as react-jsonschema-form or JSON Forms is the obvious starting point. They solve the input half and stop there: no runner, no result rendering, no known answers, and each brings a UI framework with it. Toolbench’s contract covers inputs, running and results together, and its validator holds tools to rules a general schema leaves optional. A number input must declare min and max, because a tool’s running time often grows with a number, and a declared range means the author has decided the largest input the tool must handle. A JSON Schema can say the same, but nothing requires it to. Likewise a select needs at least two options with its default among them. I wrote that validator by hand, because its messages are the first thing a new tool author meets. This is what it says about a timeout on a tool that runs on the main thread:

is only meaningful with runtime.thread = "worker". On the main thread there is nothing to terminate, so a timeout here would be a field that lies. Either move the tool to a worker or bound its input.

The last choice was the shape of the runtime: a component for one framework, or a custom element, an HTML tag whose behaviour the page registers itself, with a shadow root, a DOM subtree whose styles are isolated from the page in both directions. I chose the custom element because a page needs no framework to use one. The toolbench test bench is a plain Vite page for that reason, and the hello world runs in plain HTML. Keeping the runtime free of dependencies also means it builds its DOM by hand, with createElement and three small helpers, and a page can style a tool only through the hooks it is given.

What the website writes once#

The host is the website that embeds toolbench. Its part is small, and it exists because only the site’s own bundler can find the site’s tool files.

The two packages split along the same line. @toolbench/sdk is what a tool author and a site’s build import: the types, manifest validation, version migration, seed computation and the fixture harness. It never touches the DOM, so all of it runs in Node. @toolbench/runtime is what the page imports: the <tool-host> element, the runner that calls the function, and one renderer per kind of result.

The one file every host writes is a registry, and its heart is two file patterns:

1
const manifests = import.meta.glob<Manifest>("../tools/*/tool.json", { eager: true, import: "default" });
2
const modules = import.meta.glob<ToolModule>("../tools/*/index.ts");

import.meta.glob is how Vite, the bundler here, finds files by pattern at build time. The registry includes every manifest, so it can show a tool’s name and inputs before loading its code. The modules are lazy, so the bundler turns each tool into its own chunk, a separate JavaScript file the browser fetches only when asked. Both patterns have to be literal strings, because the bundler resolves them when the site is built.

1
import { RegistrySource } from "@toolbench/runtime";
2
import type { Manifest, ToolModule } from "@toolbench/sdk";
3

4
const manifests = import.meta.glob<Manifest>("../tools/*/tool.json", { eager: true, import: "default" });
5
const modules = import.meta.glob<ToolModule>("../tools/*/index.ts");
6

7
const idOf = (key: string) => key.split("/")[2]!;
8

9
export const loaders = Object.fromEntries(Object.entries(modules).map(([key, load]) => [idOf(key), load]));
10

11
export const source = new RegistrySource(
12
  Object.fromEntries(
13
    Object.entries(manifests).map(([key, manifest]) => [idOf(key), { manifest, load: loaders[idOf(key)]! }]),
14
  ),
15
);

The hello world page registers the element with one call:

1
import { defineToolHost } from "@toolbench/runtime";
2
import { source } from "./registry.ts";
3

4
// The whole of a host's setup for a tool that runs on the main thread: where the tools are.
5
defineToolHost({ source });

That defines the <tool-host> custom element. The page itself contains one <tool-host tool="hello" mode="page"> and no other toolbench markup.

A site with tools that run in a worker also writes a six-line worker entry, for the same reason: the worker has to import tool modules, and only the host’s bundler can see them. The rest of what a host can pass, page URLs, a stylesheet and a code highlighter, is in the host reference.

How the pieces fit#

Code and data live in three places: the site’s build, the host’s code written once, and the reader’s browser. One boundary runs through all three. The manifest is data and can be read at any time, while the tool’s code has to be present when the site is built.

Four things in that picture carry the design:

• A manifest is upgraded and validated before anything else sees it, so every part downstream deals only with current, well-formed data.
• A tool’s code reaches the browser only through the host’s bundler. The browser fetches chunks the build made, and never code from anywhere the build didn’t see, which keeps bundling, tree shaking and a strict Content-Security-Policy working.
• The worker is optional and per tool. The runner’s interface is the same either way, so once a host has its worker entry, a tool moves to a worker by changing one line of its manifest.
• Results are upgraded on the way out, so a tool written against an older contract is drawn by today’s renderers without knowing it.

Inside the browser, <tool-host> is the hub. It asks the registry for manifests and code, asks the runner for results, and asks the renderers for markup, and it reaches a worker only through the runner. The architecture docs follow each of those paths in detail.

A form from ten input types#

There are ten input types. Five were there from the start: text, textarea, number, select and toggle. A number must declare its range and can show a slider, and a select’s options can fill in other inputs, so one choice such as an SDK’s name can set its timeout and retry defaults. The other five came in 0.15.0: date, color, secret for a password that stays out of links, multiselect, and rows, a list the reader adds to and reorders. A select with a few short options can also be drawn as a row of segments. The Fan-out tail latency calculator on mainLog is a form built from them, three short inputs and a textarea, each with its description under it and a row of samples below. It works out how often a page that waits for many backend calls is held up by one slow call:

Press a sample, then Run. The answer is a headline number, a formula, a sentence and a chart, and the tool’s code drew none of it: the labels, the descriptions, the sample buttons, Run and the Reset beside it, and the chart’s axes and hover readout all come from the runtime.

So does the link icon at the end of the Run row. It copies this page’s address with the tool’s inputs in it, each named after the tool, as in fan-out-tail.calls=50, so the three tools in this post never read each other’s, and the link opens the tool with those inputs. An input the manifest marks "share": false, such as a pasted packet or a password, stays out of every link.

An input’s description is drawn under the field, not as a placeholder, because placeholder text disappears the moment someone starts typing, which is when they need it. It can hold a short list, code and bold, and a long one can keep the one line that matters there and the rest in details, behind a “More” that stays closed until the reader opens it.

Samples are example inputs in the manifest, drawn as a row of buttons under the form. They belong to the tool rather than to the page, because the tool is what knows which inputs are interesting. They came into the contract when I wanted example packets beside a decoder, and the most useful one turned out to be a malformed sample, because it shows a reader how the tool reports a mistake before they make one.

The error in that screenshot is an ordinary return value, { kind: "error", message, input, at, len }. input names the control at fault, and at and len say which characters, so the runtime can mark the field aria-invalid and point at the problem.

Bad input and a bug are different failures

A returned error means the input was wrong and the tool worked, and the reader sees the tool’s own message on the field. A thrown exception means the tool has a bug, and the reader sees “This tool hit a bug and stopped” while the stack goes to the console.

A result is data, not markup#

A tool never draws its answer. It returns a value whose kind says what it is, and the runtime has one renderer for each kind. The list is closed on purpose, so a tool can’t invent a shape that nobody can draw. The results reference has every field of every kind.

The STUN message decoder on mainLog shows what a group is for. STUN is the small protocol WebRTC uses to find out which public address a computer is seen at. For one message the decoder returns a group: the answer as labelled fields, a table of the decoded attributes, and the raw bytes with each attribute’s range marked. Abridged, the value it returns for its default message is this:

1
{
2
  kind: "group",
3
  parts: [
4
    { kind: "fields", role: "answer", fields: [
5
      { label: "Message", value: "Binding success response", note: "0x0101" },
6
      { label: "Mapped address", value: "192.0.2.10:54321", note: "Where the server saw the sender." },
7
      // ...
8
    ] },
9
    { kind: "table", role: "decoded", columns: [/* Offset, Attribute, Value, ... */], rows: [/* one per attribute */] },
10
    { kind: "bytes", bytes: [1, 1, 0, 12, 33, 18, 164, 66, /* ... */], highlight: [
11
      { at: 0, len: 4, label: "Type and length" },
12
      { at: 20, len: 12, label: "XOR-MAPPED-ADDRESS" },
13
      // ...
14
    ] },
15
    // ...
16
  ],
17
}

Nothing in it says how anything looks. This is the decoder itself, running in the post:

Press Run on the default message. It decodes to a Binding success response from a server that saw the sender at 192.0.2.10:54321, and the table and the byte view show where in the message each of those facts came from. The samples under the form are other real messages, one of them copied out of Wireshark.

Because the decoder only returns data, the same function runs in Node for its known answers, in a worker, or during a site’s build. The contract asks that run never touch the DOM, and none of those places has one, so a tool that reached for it on a path its known answers exercise would fail before it reached a page.

The closed list is enforced at both ends. The renderer’s switch over the kinds ends in a TypeScript exhaustiveness check, const _exhaustive: never = output;, so a kind added to the SDK without a renderer doesn’t compile. The fallback after it covers what a compiler can’t see, an older runtime cached in someone’s browser meeting a newer tool, and draws a message asking for a newer runtime rather than an empty box.

Nothing runs until the reader presses Run#

By default, opening a tool paints the form and stops. The function runs when the reader presses Run, or presses Enter in a one-line input, or Cmd or Ctrl with Enter in a textarea.

I made that the default because most of these tools take several inputs that only make sense together: a packet and the view to decode it in, an arrival rate and a service time and a target. Running on open and again on every keystroke would compute answers for half-typed inputs, and it makes the Run button meaningless, because the answer is already there by the time anyone looks at it.

When an input changes after a run, the last answer stays on screen, dimmed, and the status line says so. I keep it because it is still the true answer to the previous inputs, and having it there is what lets a reader compare before and after.

The progress bar follows the same idea. It appears only if a run is still going after 400 ms, because a bar that flashes for twenty milliseconds on every fast run tells the reader nothing.

A tool that is genuinely instant can opt back into running as the reader types, with "autoRun": true. The validator refuses it on a worker tool. Worker tools are the ones that can be slow, and I don’t want the slowest tools launched on every keystroke.

Code arrives when it is about to be used#

A tool can sit on a page in three modes. A card is the compact form, with the name, a short description, the main inputs and the first part of the answer, and it fetches the tool’s code only when the reader opens it. page shows everything, and embed is the same without the title, for the middle of an article. Those two fetch the code when they come within two screens of the viewport, except that a tool running in a worker leaves its code to the worker, at the first Run. Until then a tool is a facade, a term from web performance work for a cheap stand-in that the real thing replaces when the reader asks for it; Lighthouse recommends the pattern for heavy embeds such as video players.

A closed card doesn’t have to be empty. A seed is the tool’s answer for its default inputs, computed during the site’s build by the SDK’s seed() and written into the page as data, so a card can show a real answer before any of the tool’s own code has loaded. seed() throws if a tool crashes on its own defaults or rejects them, so a broken tool fails the site’s build instead of shipping a card with an error baked into it. Because the seed is computed by the same function on every build, it can’t drift from what the tool answers, which a hand-written one can.

The tools in this post open as forms, without a seed.

On mainLog, a page with no tool downloads none of toolbench, and a tool’s own code is a few kilobytes that arrive only on demand. The second part has the measured numbers, including the part that costs more than it should.

The page decides how a tool looks#

Since the page’s CSS can’t reach into a tool’s shadow root, a tool would look the same on every page, which means foreign on most of them. CSS custom properties are toolbench’s theme interface, because they inherit across the boundary. mainLog maps toolbench’s properties onto its own design tokens:

1
tool-host {
2
  --tb-bg: var(--bg-1);
3
  --tb-fg: var(--tx-0);
4
  --tb-muted: var(--tx-1);
5
  --tb-faint: var(--tx-2);
6
  --tb-border: var(--bd);
7
  --tb-surface: var(--bg-2);
8
  --tb-accent: var(--ac);
9
  --tb-accent-bg: var(--ac-wash-soft);
10
  /* ... */
11
}

Every tool’s background, text, borders and accent come from the page’s theme, so they follow mainLog’s light and dark modes with no CSS in any tool. The chart colours are mapped the same way, from --tb-s1: light-dark(#2a78d6, #3987e5) through --tb-s6.

Colours are not enough for layout. A tool can give a part of its answer a role, such as answer, which the runtime renders as a data-role attribute and otherwise ignores. The page decides what an answer looks like. A host passes its own stylesheet to defineToolHost as styles, the runtime adopts it into every tool’s shadow root after its own, and the runtime’s rules sit in a cascade layer, @layer toolbench, so an ordinary, unlayered rule from the host wins whatever its specificity. mainLog’s stylesheet has this rule:

1
[data-role="answer"] .tb-fields {
2
  display: grid;
3
  grid-template-columns: repeat(auto-fit, minmax(11rem, 1fr));
4
  gap: 0.75rem 2rem;
5
  margin: 0;
6
}

On mainLog, a fields part with role: "answer" appears as a row of headline values, whichever tool returned it. The styling guide covers what a host may select, and the theme tokens reference lists every property.

Trying it#

The quickest start is the hello world: npm install, npm test and npm run dev, and the tool from the top of this post runs in a plain page. From there, the getting started guide covers a host’s side, Authoring a tool covers the four files, and the gallery shows every kind of result beside the code that returns it.

On mainLog every tool gets a page of its own, and a post embeds one by writing ::tool{id="stun-decoder"} on a line of its own, which expands to the same <tool-host> with a caption that links the tool’s page. The tools themselves live in a separate repository, toolshed, so they stay standalone functions with their own tests rather than parts of the blog.

I built toolbench for my own tools, and I’d like to know how it holds up for yours. If you build a tool with it, tell me how it went, what was easy and what the contract made awkward, in the comments below or in an issue on GitHub. Happy coding.