Where the server sits

Two listeners, two locked copies#

Think of it as mail. Alice has a letter for Bob and Carol, and each letter goes in an envelope only its addressee can open. There are three ways to get it to them:

The envelopes are the part that’s easy to miss, and they come from post 4. Keys are made in a handshake between two machines, so every pair of machines that talk directly, every leg of the call, has its own. A packet locked for one leg can’t be opened on another. That’s why making a copy is never just copying.

The packet we’re following is post 1’s: 20 ms of Alice’s voice, 80 bytes of sound and 40 of headers, 120 bytes, sent 50 times a second, which is 48 kb/s. (Post 4’s 10-byte tag makes it 130 bytes, about 52 kb/s; the counts here stay with post 1’s 120 bytes, so every kb/s figure runs about 8% low.) Here’s what happens to it in each shape, with three people on the call:

I wrote a small script, topology_cost.py, that does this counting for any size of call, and every traffic number from here on comes from it. The counting itself is a handful of lines, one function per shape:

1
def mesh(n: int) -> Cost:
2
    peers = n - 1
3
    return Cost("mesh", peers, peers, None, None, ...)
4

5
def sfu(n: int, loudest: int | None = None) -> Cost:
6
    k = n if loudest is None else min(loudest, n)
7
    heard = min(k, n - 1)  # a listener who is not among the k hears all k; a selected speaker hears k - 1
8
    out = k * (n - 1)  # each forwarded stream goes to everyone but its sender
9
    name = "SFU" if loudest is None or k == n else f"SFU, {k} loudest"
10
    return Cost(name, 1, heard, n, out, ...)
11

12
def mcu(n: int) -> Cost:
13
    return Cost("MCU", 1, 1, n, n, ...)

Trimmed to the stream counts; the ... is the per-box work, which comes later.

The four numbers in each Cost are streams: what one laptop sends, what it receives, what the server receives, what the server sends. n is the number of people. In a mesh each laptop sends and receives n - 1, one per other person, and there’s no server. The SFU receives n and sends out, each voice to everyone but its owner. k is how many voices it forwards: all of them for now (we’ll get to forwarding only the loudest), so each listener hears heard = n - 1. The MCU receives n and sends n, one mix each. Multiply by 48 kb/s and you have the traffic. For Alice’s call, n = 3:

                                  one client              the server
pattern                   up kb/s  down kb/s     in kb/s    out kb/s  out pkt/s
mesh                           96         96           -           -          -
SFU                            48         96         144         288        300
MCU                            48         48         144         144        150

The last column counts packets rather than bits, because a server spends its effort per packet: every one has to be received, unlocked or locked, and sent. To keep the counting simple, the script assumes everyone sends all the time, and leaves out the small reports from post 5.

With three people the shapes barely differ. They start to differ when the call grows, so let’s take them one at a time and see what happens to Alice’s sentence in each.

Mesh: Alice posts every copy herself#

In a mesh, Alice’s browser opens a separate connection to Bob and another to Carol, each with its own path, its own handshake and its own keys. Every 20 ms she locks that slice of “ship on Friday” twice, once for each of them, and sends it twice. That’s her 96 kb/s up.

Now imagine the call grows to ten people. Her sentence goes out nine times: 432 kb/s, nine times a two-person call, and the same from every laptop on the call, over whatever connection each person happens to have (RFC 8834 §12.1.2 describes the same cost).

Your own browser can show you this. Start the call, then add listeners one at a time, and watch Alice’s upload:

What mesh buys is privacy. There’s no machine in the middle, so nobody but the people on the call ever holds a key.

The cost is that every copy is made and sent by the laptops, and every person who joins adds one more to each of them. What if a server made the copies instead?

SFU: the post office#

With an SFU, Alice sends her sentence once, to the server, and the server sends a copy to Bob and another to Carol. Her upload stays at 48 kb/s however big the call gets.

The listeners don’t get off as lightly. Bob still receives a separate stream for each other person, and plays them together on his own laptop. And the server’s way out is where the growth goes: each person’s voice leaves it once per listener. With three people that’s 3 × 2 = 6 streams, 288 kb/s. With ten:

                                  one client              the server
pattern                   up kb/s  down kb/s     in kb/s    out kb/s  out pkt/s
mesh                          432        432           -           -          -
SFU                            48        432         480        4320       4500
MCU                            48         48         480         480        500

Ten voices, nine copies each: 90 streams, 4,500 packets every second out of one server, for one call. Twenty people would be 380. That’s growth with the square of the call: the number of people times the number of listeners each one has.

A mesh sends the same 90 copies; it just spreads them across everyone’s laptops. An SFU sends all 90 itself.

Why the post office has to open the envelope#

To forward Alice’s packet, the SFU needs to know whose voice it is, and that’s written on the outside: the RTP header, which post 4 showed is never encrypted. So you might expect it to read the label and pass the sealed envelope along untouched.

It can’t, because Alice sealed it for the SFU, not for Bob. She did her handshake with the server, and so did Bob, separately. Alice and Bob never did one with each other, so Bob holds no key that opens her packet. The SFU has to unlock each packet with Alice’s key and lock a fresh copy with Bob’s, and another with Carol’s. What it never has to do is decode the sound, turn the bytes back into audio. RFC 7667, which calls an SFU a “selective forwarding middlebox” (SFM), puts the rule plainly: “If encryption is employed, the Media Translator, SFM, and mixer need to be able to decrypt the media to perform its function” (§6).

Now be the SFU yourself. Press Forward it untouched to Bob first, then Unlock with Alice’s key, relock for Bob and Carol. The keys are made in your browser when the page loads, so the first one really does fail:

This lab runs a real call in your browser, so it needs JavaScript.

The second script, measure.py, runs a real SFU on one laptop, with real SRTP (post 4’s locking) on real packets, so we can time it. Its whole job, every 20 ms, is this loop:

python↩ WrapCopy

1
for i in range(n):
2
    plain = s_in[i].unprotect(wire[i][f])
3
    for j in range(n):
4
        if j != i:
5
            s_out[j].protect(plain)

SRTP’s words for unlocking and locking are unprotect and protect. For each person i, the SFU unlocks their packet once, with the key it shares with them. Then, for every other person j, it locks a copy with the key it shares with j. There’s no decoder in the loop and no encoder. For Alice’s call that’s 3 unlocks and 6 locks every 20 ms, and nothing else.

Forwarding only the people talking#

On a big call, most people are listening most of the time, so forwarding all ten voices to everyone is mostly forwarding silence. An SFU could forward only the loudest few, but to know who’s loud it would seem to have to decode everyone, which is exactly the work it exists to avoid.

The fix is to have each sender write down its own loudness. You saw it in Alice’s offer in post 4, as ssrc-audio-level: each packet carries one extra byte saying how loud that 20 ms was, from 0 for the loudest to 127 for silence (RFC 6464). The SFU compares those numbers across senders and forwards the top few, without listening to anything. The open-source SFU mediasoup describes its version in seven words: “No decoding of audio data is done” (docs).

Ten people	Forward everyone	Forward the 3 loudest
The SFU sends	4,320 kb/s	1,296 kb/s
One listener receives	432 kb/s	144 kb/s, or 96 for one of the three speakers
Locks every 20 ms	90	27
Sounds decoded by the server	0	0

From topology_cost.py -n 10 --loudest 3, still counting post 1’s 120-byte packet.

The byte isn’t free: in its common short form, the extension adds 8 bytes to every packet, so Alice’s packet with it and post 4’s tag is 138 bytes rather than 130.

Why skipping packets means renumbering them

Each stream Bob receives is numbered packet by packet, and post 5’s loss count reads a gap in the numbers as a lost packet. If the SFU stops forwarding Carol while she’s quiet, Bob’s laptop would count her silence as loss. So an SFU that skips packets renumbers each stream it sends, which is the rewritten number in the figure above. Post 4 showed that the header is readable but covered by the tag, so it can’t be changed without the key. That’s no obstacle here: the SFU is writing a new tag for every copy anyway, with the listener’s key, and the new number goes under it.

Can the loudness byte be hidden?

It rides in the clear, like the rest of the header, and a loudness trace can give away when someone is speaking. RFC 8834 recommends encrypting it, and RFC 9335’s “cryptex” does that, though post 4 found libwebrtc ships it switched off. Either way it isn’t hidden from the SFU, which holds the keys.

Video does the same, with sizes instead of speakers

For video, an SFU picks which version of each stream to send each receiver: senders upload a few sizes at once (simulcast, RFC 8853) or one stream built in layers, and the SFU forwards the size that fits each receiver’s connection. Still no decoding: it’s choosing packets by their headers, as with audio.

Even with the SFU doing all the copying, Bob’s laptop still receives a separate stream for each speaker, decodes each one and plays them together. What if the server did that too?

MCU: the editor#

An MCU is the editor from the mail analogy. It decodes everyone’s stream back into sound, adds the sound together, and sends each person a single stream.

But not the same stream to everyone. Bob’s mix has to leave Bob out: his own voice coming back from the server, a moment after he said it, would sound like an echo. RFC 7667 says exactly that: “each endpoint gets a mix of all other sources except its own, as this would result in an echo” (§3.6.1). A mix of everyone but the listener is called mix-minus. So Alice’s “ship on Friday” goes into Bob’s mix and Carol’s, but not into her own.

Hear it for yourself. Choose to be Bob and play your mix, then tick Leave my own voice in the mix and play it again:

So with three people the MCU makes three different mixes, and encodes and locks each one for its listener.

For the laptops this is the cheapest shape there is: one stream up, one down, one decode, whatever the size of the call. That’s the reason mixers exist at all: RFC 7667 notes they “were introduced in pre-RTP times” so that devices able to take only a single stream could join a conference (§3.6.1). The server’s traffic is modest too: one stream in and one out per person, 144 kb/s each way for Alice’s call.

The price is paid in the server’s processor. Here’s measure.py’s MCU, one 20 ms frame:

1
for i in range(n):
2
    payload = m_in[i].unprotect(wire[i][f])[12:]
3
    pcm.append(np.concatenate([fr.to_ndarray().reshape(-1) for fr in decs[i].decode(av.Packet(payload))]))
4
stacked = np.stack(pcm).astype(np.int32)
5
total = stacked.sum(axis=0)
6
for i in range(n):
7
    mix = np.clip(total - stacked[i], -32768, 32767).astype(np.int16)
8
    packets = encode_frame(encs[i], mix, f * SAMPLES)
9
    for p in packets:
10
        m_out[i].protect(rtp(f, f * SAMPLES, 0x4D430000 + i, p))

The timing lines are left out.

Read it in three steps:

1. Unlock and decode everyone. For each person, unlock the packet, drop its 12-byte header, and decode the 80 bytes of Opus back into sound: 960 numbers, one for each 1/48,000 of a second.
2. Add everyone once. total is the whole room, every voice added together, number by number.
3. Take each listener back out. total - stacked[i] is the mix-minus: the room minus person i. The sum is done in 32-bit numbers so it can’t overflow, and np.clip then squeezes it back into what a 16-bit sample can hold, because three loud voices added together can exceed it. Then the mix is encoded into Opus again (encode_frame returns a list, usually of one packet), wrapped in a fresh RTP header by rtp(...), with the MCU’s own stream ID because each mix is a new stream, and locked for person i alone.

The long second line looks worse than it is: it decodes the packet and flattens the result into one row of 960 numbers.

Where the SFU’s loop had no decoder and no encoder, this one has one of each per person, every 20 ms. The script times both on the same laptop, a thousand frames each, and the gap is about a hundred times at every size. For Alice’s call, the SFU took 7 µs a frame and the MCU about 1.2 ms. At ten people, mixing one 20 ms frame took half of those 20 ms, so one processor core could keep up with about two such calls; forwarding the same call took 77 µs, room for about 260.

Median of 1,000 frames on one Apple M3 Pro, one core, from Python. A server written in C or Go would be faster at both, so don’t plan capacity from these numbers.

The gap is the claim, and my guess is that it’s an underestimate: the MCU’s time is mostly libopus, which is already C, while the SFU’s tiny times are mostly Python calling into libsrtp, which a C server wouldn’t pay.

Two costs don’t go away with a faster processor. Opus holds back a little sound each time it encodes so it can look ahead, 6.5 ms in this build, so the MCU’s second encode adds that much delay to everyone’s audio. And Opus throws away detail to compress, so decoding and compressing again loses a little more, like a photocopy of a photocopy. (I haven’t measured that loss here.)

Now that all three are on the table, grow the call. Pick a shape, drag the number of people up, and watch where the lines pile up. Then set the SFU to forward only the 3 loudest:

Who can hear the call#

Back to the mail for a moment. The post office never reads Alice’s letter, but it does open the envelope, and whoever opens the envelope could read it. The same goes for every machine we’ve met that unlocks the audio: each laptop in a mesh, the SFU, the MCU. Each one did the connectivity checks from post 3 and a handshake from post 4, holds keys, and unlocks the sound. Being that machine is called terminating the call. A server in the middle still counts: each leg has two ends, and the SFU or MCU is one end of every leg it serves.

Here’s a packet you’ve seen before, Alice’s to her assistant in post 4, locked. Any leg looks the same, so here’s what three different kinds of machine on its path can read:

So the rule is simple: whatever terminates the call can hear it. In a mesh that’s only the people on it. With an SFU or an MCU, it’s the server too, and RFC 7667 warns that one that’s “subverted or misbehaving” has “full access” (§6).

Back to the assistant: the server is the other end#

Back to 9:17 AM, Alice and her assistant. In the mail analogy this is a letter to one pen pal: one envelope, no copies, no post office, no editor. The assistant’s server isn’t forwarding Alice to anyone or mixing her with anyone; it’s simply the other end of the call. RFC 7667’s catalogue of shapes starts with exactly this one, point to point: “A sends to B, and only B, while B sends to A, and only A” (§3.1). The server is B.

The work per conversation is one stream each way, 48 kb/s. Every 20 ms there’s one unlock and one decode of what Alice says, since the AI model needs her voice as sound, and one encode and one lock of the reply. That’s one seat at an MCU, without the mixing. measure.py times it too: 384 µs a frame on the same laptop, about fifty conversations per core at that speed.

Large voice services have written about this choice. OpenAI’s article on their voice infrastructure (OpenAI) says they considered the usual group-call design, an SFU, with the AI joining the call as one more participant. But “[m]ost sessions are 1:1”, one person talking to one model, so there’s nothing to forward.

What they built instead is a service at the edge, meaning the first of their servers a user’s packets reach, which terminates the call. They call it a transceiver. Behind it, the audio travels to the other services in “simpler internal protocols”, so that those services can “scale like ordinary services instead of acting as WebRTC peers themselves”.

Why does that matter? Because being a WebRTC peer means doing everything we’ve covered so far:

And all of that belongs to one machine for the whole call. The path from post 3 ends at that machine’s address, and the keys from post 4 live in its memory. If it’s busy, the call can’t simply move to another one. An ordinary service, the kind that’s easy to scale, works the other way: any of many identical machines can take the next piece of work, so you add machines when it’s busy and take them away when it isn’t. With a terminating edge, only the edge has to be a WebRTC peer, and the AI model behind it just receives audio, like an ordinary service.

Choosing one#

Here’s Alice’s one 20 ms slice once more, through every shape at once:

Read off the table: two or three people who care about privacy, mesh. A group call, an SFU, forwarding only the loudest few once it’s more than a handful. Devices that can only take one stream, or a bridge to the phone network, where each call is a single channel of sound, a mixer. One person and a service, terminate at the edge and let everything behind it be an ordinary service.

To compare exact numbers for a call of any size, all four shapes side by side. Its Packet choice adds what a real call carries on top of post 1’s 120 bytes: the SRTP tag, IPv6’s longer header, and header extensions such as the audio level:

The demo#

Both scripts are in where-the-server-sits/: topology_cost.py does the counting for any size of call, and measure.py runs a real MCU, a real SFU and the one-to-one case with real Opus and real SRTP, and times a frame of each.

cd mainlog-code/where-the-server-sits
python3 topology_cost.py                    # no install needed: 2, 3, 5 and 10 people
python3 topology_cost.py -n 10 --loudest 3  # an SFU forwarding the three loudest
python3 -m venv .venv && .venv/bin/pip install -r requirements.txt
.venv/bin/python measure.py                 # the timings, on your machine

Its CI checks the counting against a slow count that lists every copy of every voice, for every call size from 2 to 30, and checks that the MCU stays more than twenty times the SFU’s cost on whatever machine runs it.

So Alice’s “ship on Friday” can be copied by her own laptop, by a server that unlocks and relocks it, or by one that mixes it into everyone else’s sound, and in each case the cost lands somewhere different. Her assistant needs none of that, but its server terminates the call and has to own it, path, keys and all, for as long as the call lasts. That makes it hard to put behind an ordinary load balancer, which spreads packets across machines without knowing which call each belongs to. Next, how a packet finds the one server that owns its call, without the box in front holding any keys.