SFU, MCU, and the participant that has to decode

Three seats for a note-taker#

Picture a meeting room with a stenographer. There are three places to put them:

The first and third seats keep the voices apart. The second gets them already added together, and adding is the one step you can’t undo.

If the two kinds of server are new to you, the voice series’ Where the server sits builds them up from scratch. Here’s what matters for the agent.

Unlocking is not decoding#

Ana’s voice travels as packets, one every 20 ms. Each packet has a header, a small label on the outside saying whose stream it belongs to (its stream ID, which RTP calls the SSRC), its number in the stream, and when it was recorded. After the header comes the audio itself, compressed by Opus, the codec browser calls use for voice.

Every packet crosses the network locked. Each hop, laptop to server and server to agent, has its own key, and the server at the end of a hop holds that hop’s key. That’s SRTP: it encrypts the audio and leaves the header readable, though tamper-proof. WebRTC, the standard browser calls use, requires it on every hop: all media channels “MUST be secured via SRTP” (RFC 8827 §6.5).

So every server that ends a hop unlocks every packet, because it has to lock it again for the next hop. (Where the server sits shows why, and has a lab where you try it.) But unlocking only gives you the compressed Opus bytes. Turning those into sound, a stream of numbers measuring the sound wave, is decoding, a separate step with a real cost. A server can unlock every packet and never decode one.

That distinction is the whole article:

Remember the amber steps. They come back when we get to keys.

The forwarding server: choose by the header#

A forwarding server reads each packet’s header, decides who should get it, and sends the same audio on. It can even pick who’s speaking without hearing anyone: each sender writes its own loudness into one byte in an extension to the header, 0 for loudest and 127 for silence (RFC 6464). The open-source SFU mediasoup, which “does not transcode or mix media”, describes its version in seven words: “No decoding of audio data is done” (docs).

The simulation’s forwarder sends the agent the two loudest of Ana, Ben and Chloe. Skipping a quiet packet creates a problem first, though. Each stream the agent receives is numbered packet by packet, and a missing number reads as a lost packet. So when the forwarder skips one on purpose, it closes the gap by renumbering everything after it, but leaves a real network loss as a gap, so the agent’s decoder knows to cover for it:

1
for receiver in receivers:
2
    if receiver == sender:
3
        continue
4
    leg = (receiver, pkt.ssrc)
5
    if pkt.ssrc not in loudest:
6
        if leg in self.skipped:                  # leg already started
7
            self.skipped[leg] += 1               # close the gap we made
8
        continue
9
    offset = self.skipped.setdefault(leg, 0)     # a real loss stays a gap
10
    out[receiver] = pkt.with_(seq=pkt.seq - offset).to_bytes()  # payload unchanged

A leg is one sender’s stream on its way to one receiver, named by the receiver and the stream ID, ssrc. For each leg, skipped counts the packets the forwarder dropped on purpose, and every later packet is renumbered down by that count. A quiet packet only counts once the leg has started: before the first packet goes out, there’s no numbering yet to leave a gap in. The last line is the point: a new number, and the audio bytes untouched.

Here Chloe goes quiet for one 20 ms packet, at level 90:

Chloe seq 5200 -> 5200  ts 0     payload bytes unchanged: True  agent decodes Chloe: [300, 300, 300, 300]
Chloe seq 5201          level 90: not forwarded
Chloe seq 5202 -> 5201  ts 1920  payload bytes unchanged: True  agent decodes Chloe: [300, 300, 300, 300]

Ana and Ben were louder (levels 10 and 12), so Chloe’s quiet packet was the one skipped. The agent receives packets 5200 and 5201, with no gap where it was. The timestamp still jumps from 0 to 1920, which is 40 ms at Opus’s 48,000 samples a second, so the agent knows time passed. (The simulation’s packets carry only 4 samples each, rather than a real packet’s 960, so each line fits on the screen.)

A packet lost on the network is different, and stays a gap:

Ana   seq 5000 -> 5000
Ana   seq 5001          lost on the network
Ana   seq 5002 -> 5002

The agent sees 5000 then 5002, knows 5001 is missing, and its decoder fills the 20 ms with a plausible guess, which Opus calls packet loss concealment.

The mixer: add, and lose who said what#

A mixer has to open the audio. It decodes every input, lines them up in time (RTP’s own spec says the mixer “will make timing adjustments among the streams and generate its own timing for the combined stream”, RFC 3550 §3), and builds a mix for each listener that leaves the listener out, because hearing your own voice back is an echo. The simulation’s version:

1
others = [ssrc for ssrc in frames if ssrc != listener]    # never your own voice
2
length = len(frames[others[0]])
3
mixed = [clip(sum(frames[s][i] for s in others)) for i in range(length)]

frames holds every input’s decoded samples, keyed by stream ID. For each listener it keeps everyone else’s, then adds them up sample by sample, with clip keeping the sum inside what a sample can hold.

Put one 20 ms frame from each person in, plus the agent’s own reply, with each voice reduced to one number so you can do the sums by eye:

inputs    Ana 100, Ben 200, Chloe 300, agent 50
to Ana    SSRC 0xe5  CSRC Ben, Chloe, agent  samples [550, 550, 550, 550]
to Ben    SSRC 0xe5  CSRC Ana, Chloe, agent  samples [450, 450, 450, 450]
to Chloe  SSRC 0xe5  CSRC Ana, Ben, agent    samples [350, 350, 350, 350]
to agent  SSRC 0xe5  CSRC Ana, Ben, Chloe    samples [600, 600, 600, 600]
mixer decodes: 4, encodes: 4

Ana gets 200 + 300 + 50 = 550: everyone but herself. The agent gets 600, Ana + Ben + Chloe. Every mix comes from the mixer’s own stream ID, 0xe5, and its header carries a contributor list, the CSRC, naming who’s inside. But the 600 itself has no trace of which part was Chloe. Getting the voices back apart would take a separate model trained to untangle speech.

Where the agent sits#

Through a forwarding server, the agent is an ordinary participant. It subscribes to each person’s stream and decodes all three itself. This is the headset-per-person seat.

As a client of a mixer, the agent receives one mix of everyone but itself and decodes it once. A bot that dials into a conference bridge by phone sits here: the phone line carries one channel of sound.

Inside the mixer, the agent reads each person’s audio after the mixer has decoded it and before it’s added in, and hands its own reply back as sound. It’s the natural seat when you already run a mixer that decodes every stream.

Hear the difference. Pick a seat and play the meeting as the model receives it, then try muting one person:

If you use LiveKit

LiveKit is a forwarding server, and its docs call an agent “just another participant in the room”. One trap for a note-taker: an agent session listens to one linked participant by default, the first to join, not to everyone. A group-call agent has to be set up to listen to every person, for example with one session per participant, as LiveKit’s multi-user transcriber example does.

Why a note-taker wants each person#

For almost any agent in a group call, I’d want the per-speaker seats. Each stream belongs to one participant, which the call’s signalling, the separate channel where people join and describe their streams, says by name. With a stream per person, the model gets three things the blend can’t give it:

What each seat costs#

The simulation counts decodes for the same three people, assuming the forwarding server sends every stream to everyone, and the mixer decodes every input and sends each endpoint its own mix. (A mixer that mixes only the loudest few decodes less, and people who aren’t speaking can share a mix.)

From python -m call_topology.counts --people 3.

The whole-call row adds the agent, the server and everyone’s laptops:

• Through a forwarding server: 3 by the agent, none on the server, and each of the three people decodes the other two plus the agent’s reply, 3 × 3 = 9. 12 in all.
• Client of a mixer: 1 by the agent, 4 by the mixer (three people and the agent’s reply), and 1 mix per person, 3. 8 in all.
• Inside the mixer: none by the agent, 3 by the mixer, and 1 mix per person. 6 in all.

So the forwarding server costs the most decodes in total, but it spreads them across everyone’s laptops and the agent, and the server itself decodes nothing. The row that matters more for an agent is the one above it. Every decode sits behind a short wait for late packets, a jitter buffer, and every encode collects a full frame of sound first, so the mixer’s client pays both delays twice before the model hears anything. In the stenographer’s terms, the one in the next room hears the loudspeaker a moment after the room does.

The larger cost is usually further along. One stream per person means one speech-to-text session per person, three instead of one for the mix. Only sending a stream on while someone’s actually talking narrows that gap.

Who has to hold the keys#

Now the encryption. Every server that ends a hop holds that hop’s key (a TURN relay, which only passes packets along, holds none), which means that on an ordinary WebRTC call a forwarding server could decode your audio; it just doesn’t. RFC 7667 says it outright: a forwarder or mixer needs “to be able to decrypt the media to perform its function” (§6).

End-to-end encryption adds a second lock inside the first, with a key only the ends of the call hold: the people and machines trusted to hear it. The servers still take off the outer lock for their hop, but the sound inside stays sealed. In the stenographer’s terms, the room is soundproofed, and only someone with a key can listen in. The call’s key usually comes from a key service, part of the app, which hands it to each laptop as people join. The IETF’s standard for the inner lock is SFrame, which seals each frame, or each packet, so forwarding servers “can access the media metadata needed to make forwarding decisions without having access to the actual media”.

Go back to the amber steps in the figure. A forwarding server never does them, so it keeps working on an end-to-end encrypted call. A mixer does all of them, so it works only if it’s one of the ends. And the agent has to decode, so it has to hold the key too. The simulation seals one packet and hands it to each box (its cipher is a toy stand-in for SFrame, not real cryptography, and it leaves out each hop’s lock):

forwarding server, no key: relays it, payload bytes unchanged: True
mixer, no key:             recovers Ana's audio: False
agent with the key:        hears ('Ana', [100, 100, 100, 100])
agent with the wrong key:  wrong key: this box is not one of the ends

Now do it with real encryption, AES-GCM, one of SFrame’s cipher suites. Hand Ana’s frame to the forwarding server first, then the mixer, and see which one gets stuck; then try the agent with the right key and with a wrong one:

LiveKit’s guide to end-to-end encryption with agents describes the last case from experience: the agent “needs the shared encryption key”, and without it “The agent joins, but STT never produces transcripts, so the LLM never receives input and the agent never speaks.” (STT is speech-to-text; the LLM is the language model.)

So here’s who can hear what on an end-to-end encrypted call through a forwarding server:

Here’s the order things happen in. The key goes out first, then media flows:

Of the boxes in the diagram, only the key service, Ana and the agent can open the audio (Ben and Chloe can too), and the key service never does; the hosted service hears it only because the agent sends it there.

Don’t send the key through the server

The key has to reach the agent by a path the forwarding server can’t read. LiveKit’s guide lists two. One passes the key in the job metadata, the data attached to the request that sends the agent into the room, and that request goes through the LiveKit server, which manages it, so the server sees the key. The other has the agent read the key from your own secrets store when it starts up, which keeps it away from the server.

Two things stay visible even with the inner lock:

• Who speaks, when, and how loudly. The loudness byte is in an extension to the header, which neither lock encrypts (in SRTP, header extensions are “authenticated but not encrypted”), and SFrame’s own header carries a key ID and a counter that reveal the sender and the rate of their frames.
• Anyone with the key can speak as anyone. SFrame shares one key per sender among all the ends, so “[a]ny sender in a session can send media that will be associated with any other sender” (§7.2). Giving the agent the key means trusting it, and whoever runs it, not to put words in Ana’s mouth.

Choosing a seat#

Start with one question about the model: does it need to know who said what? For a note-taker it does, and the mixer’s-client seat is out before you draw a box.

Then check three things in your own call:

1. Find the process that calls the decoder for the agent’s input. That’s your real seat, whatever the architecture diagram says.
2. Run the counts for your real call size, python -m call_topology.counts --people N, to compare decodes per seat.
3. List everyone who can hear what the agent hears, and check it against what your users were told:
   • the mixer’s operator, in every seat that has a mixer;
   • the forwarding server’s operator, unless the call is end-to-end encrypted and the key never passes through it;
   • whoever runs the agent and any key service;
   • every speech or model API the agent calls.

The demo#

The simulation is in participant-that-decodes/: a forwarding server that reads headers only, a mixer that decodes and sends everyone but themselves, an agent, a toy end-to-end cipher, and the counts. It needs only Python 3.10 or newer; the tests need pytest.

cd mainlog-code/participant-that-decodes
python -m call_topology.demo forward --speaker Chloe  # renumbering through a forwarder
python -m call_topology.demo loss                     # a network loss stays a gap
python -m call_topology.demo mix                      # what a mixer decodes and sends
python -m call_topology.demo keys                     # who can open a sealed packet
python -m call_topology.counts --people 8             # decodes per seat
python -m pytest tests

Its “codec” is raw 16-bit samples, so decoding is just unpacking bytes. A real call uses Opus, which changes the cost of each decode but not where the decodes happen. Its CI runs the tests and checks a line from each demo quoted here.

So the note-taker’s seat decides two things at once: whether its model hears Ana, Ben and Chloe or one blend, and whether it has to be trusted with the call’s key. For a note-taker on an encrypted call, that’s one answer: through the forwarding server, decoding each person itself, holding the key as one of the ends, and with everyone it sends audio to on the list your users see.