Late is lost

Packets don’t arrive on a beat#

Alice’s assistant sends a packet every 20 ms, like a metronome. They don’t arrive like one. Every packet crosses the same links, but each one meets the queues along the way in a different state: a router’s buffer that’s empty for one packet holds a dozen others for the next. So one packet might take 31 ms and the one behind it 48.

That unevenness is jitter. RTP even measures it: every receiver keeps a running estimate of how much the gaps between arrivals differ from the gaps between send times (RFC 3550 §6.4.1), and reports it back to the sender.

The receiver can’t play packets as they land, or the sound would speed up and stall with the network. It plays each frame at a fixed time instead: when it was sent, plus the normal trip, plus a margin. The margin is the jitter buffer, and the fixed time is the frame’s playout time. A packet that turns up after its playout time is no use, and RFC 7005 says so plainly: packets delayed excessively “may be discarded if they miss their playout time.”

Here are ten packets of Alice’s answer three ways: as they were sent, as they arrived, and as two different receivers play them.

That bottom row is the whole idea. Every frame gets a slot, and a packet only has to land before its own slot. Zoom in on one frame:

Here’s what that looks like for a whole call. This post’s simulator runs a minute of Alice’s stream, 3,000 frames, over a modelled network: a 30 ms fixed delay, plus a random extra delay on each packet that averages 10 ms (that average is what the tool calls jitter), plus 3% loss, all with a 40 ms buffer. It’s a model, seeded so it gives the same minute every time, and it’s the one simulator for this whole post: we’ll come back to its later charts as we go.

Further down is what Alice hears: the first five seconds, a row per second, with a mark on every 20 ms frame that went wrong. Each one was either rebuilt from a copy (we’ll get to copies) or made up by the decoder. The chart after it is the dots from the figure above for every packet: the trip each one took, against the dashed line at 70 ms. Every dot above the line is a frame Alice didn’t get. At the top, it says how much of the minute the decoder made up, the longest gap, and the delay every frame pays, and when that’s 1% or more, which buffer or protection would fix it on the same network.

Over the whole minute, 1.7% of packets arrived after their slot. The network delivered every one of them. They were just late, which on a call is the same thing as lost.

The buffer, and what it costs#

So why not wait longer? Because the buffer’s wait isn’t spent on the late packets alone. It’s added to every frame. A 40 ms buffer makes the whole conversation 40 ms more delayed, including for the packets that arrived with time to spare.

That’s the trade a jitter buffer makes, and the simulator’s first chart puts the buffer on the x axis: the same minute, replayed at every buffer from 0 to 200 ms. On a log scale, because the interesting part is down at a few percent:

The first few milliseconds of buffer buy a lot; the last few buy almost nothing. With no buffer at all, the model makes nearly every packet late, because its extra delay is never exactly zero; a real path has some packets that do arrive right on the fixed delay, but the shape is the same. Where to stop depends on the rest of the call’s delay budget. The usual reference is ITU-T G.114, whose current edition says that below 150 ms one way, most applications “will experience essentially transparent interactivity”. The buffer is one slice of those 150 ms, and the network, the codec and the device all want theirs.

Two kinds of buffer:

The browser’s buffer is adaptive. libwebrtc’s, called NetEQ, starts at a target of 80 ms (delay_manager.cc) and then moves the target with the delays it measures, aiming at the 95th percentile of recent packets (delay_manager.h). When it holds more audio than its target, it plays slightly faster to drain; when it holds less, it stretches the audio slightly to refill (webrtcHacks walks through it). The simulator’s buffer is fixed, so it shows the trade without the adapting.

Try it in the simulator: drag the jitter buffer down to 20 ms and watch the dots above the line multiply, and the made-up frames in what Alice hears with them, then pick the Jittery path sample, where 30 ms of average jitter makes a 40 ms buffer miss over a quarter of its packets (27.3%).

When a packet is missing anyway#

However deep the buffer, some packets never come. The receiver sees the sequence number skip, knows exactly which 20 ms are missing, and has to play something in their place, because the frame’s slot comes round whether there’s a frame or not.

Silence is the crudest choice: nothing at all where 20 ms of voice should be. So decoders conceal: the Opus specification asks for it on any unreliable channel (RFC 6716 §4.4). In its speech mode the decoder extrapolates the voice’s spectral shape from the previous frame; in its music mode it repeats the last stretch of waveform at the voice’s pitch. Either way it’s a guess from what came before.

How good a guess? We can measure it. This post’s demo, fec_recovery.py, encodes a second of a synthetic vowel-like sound with libopus, drops one packet, and scores what plays in its place against what the decoder would have played had nothing been lost. The score is a signal-to-noise ratio over that 20 ms. It measures how close the stand-in is to the real frame, not how it sounds: higher is closer, 0 dB is exactly as far off as silence, and a negative score is further off than silence.

decoder libopus 1.6.1; 32 kb/s, FEC on, planning for 10% loss
50 frames of 20 ms; frame 25 is lost; packet 26 is 80 B, carries an FEC copy
  silence                      0.0 dB SNR over the lost 20 ms
  concealment (PLC)            6.5 dB SNR over the lost 20 ms
  FEC from the next packet    14.1 dB SNR over the lost 20 ms

Concealment is a real improvement on silence for one frame. It doesn’t stretch, though: each further frame is a guess built on a guess, drifting further from what was said. One concealed frame is a 20 ms patch. Six in a row is 120 ms of made-up sound. You can make both happen, and listen to them, in the lab further down.

That’s why the shape of the loss matters as much as the amount. Here are two simulated minutes, both set to 3% loss, with no protection at all, so every missing frame is concealed. In one the losses are independent; in the other they come in bursts that average three packets. They’re separate draws, so the bursty minute happened to lose fewer packets:

Less concealment overall, and a 180 ms hole, nine frames of made-up sound in a row, where the random minute never concealed more than two. (Bursts average three packets; some run much longer.) ITU-T’s model for estimating call quality treats that as worse, not better. Its burst ratio is 1 “when packet loss is random” and above 1 “when packet loss is bursty”, and the higher it goes, the worse the predicted quality at the same loss rate (G.107).

Sending it twice#

If guessing is the problem, the fix is to stop guessing: send each frame’s information again, in a later packet, so the receiver can rebuild a loss instead of making it up. That’s forward error correction, and voice over WebRTC has two flavours of it.

Opus in-band FEC. The encoder re-encodes each frame it judges important at a lower bitrate and tucks that copy into the next packet. RFC 7587 §3.3 puts it as adding “redundant information about the previous packet (N-1) to the current output packet N”. When packet 25 goes missing, the receiver waits for packet 26 and decodes 25 from the copy inside it.

Here’s the difference between one loss and a burst, packet by packet:

RED. RFC 2198 wraps whole earlier frames, at full quality, alongside each new one, as separate blocks with their own timestamps. It can carry one earlier frame or several.

The RED costs follow from carrying whole copies: webrtcHacks’ write-up of Chrome’s audio RED puts a 30 kb/s stream at 60 with one copy and 90 with two, before headers.

The demo above already showed what the FEC copy is worth: 14.1 dB for frame 25 against concealment’s 6.5. One frame could be luck, so the demo also loses every frame in turn, from frame 5 to frame 48 (it counts from 0, skips the first few while the encoder settles, and stops before the last, which has no next packet to carry its copy):

28 of 44 packets carry an FEC copy of the frame before them
concealment (PLC)            3.9 dB on average over 44 positions, median 4.7, worst -1.9
                             5.4 dB where the next packet carries a copy (28 positions)
FEC from the next packet    13.5 dB on average over 44 positions, median 17.7, worst -1.9
                            20.4 dB where the next packet carries a copy (28 positions)

Look at the first line. Only 28 of the 44 packets carried a copy at all. Opus decides per frame whether a copy is worth the bits, and in this signal it skipped the quiet troughs between the “syllables”. Where there’s no copy, the decoder falls back to concealment, which is why both methods share the same worst case, -1.9 dB: a frame where the made-up sound landed further from the real one than silence would have. Where there is one, it’s 20.4 dB against 5.4: a rebuilt frame, not a guess.

The decoding is one call. Here it is from the demo, with the part that matters being the last argument:

1
def decode(dec: ctypes.c_void_p, packet: bytes | None, fec: bool = False) -> list[int]:
2
    out = pcm_t()
3
    # The last argument asks for the FEC copy inside `packet` rather than `packet`'s own frame. No packet at all
4
    # (None) asks the decoder to conceal.
5
    n = opus.opus_decode(dec, packet, len(packet) if packet else 0, out, SAMPLES, 1 if fec else 0)
6
    assert n == SAMPLES, n
7
    return list(out)

And where frame f is the one that never arrived:

1
elif recover == "fec":
2
    out += decode(dec, packets[f + 1], fec=True)
3
elif recover == "plc":
4
    out += decode(dec, None)

FEC has to be switched on at both ends

The receiver has to say it can use the copies, with useinbandfec=1 in the call setup, and RFC 7587 defaults that to 0. The sender’s encoder also has to be told how much loss to expect (OPUS_SET_PACKET_LOSS_PERC), because the copies cost bits: the Opus docs say a higher setting buys loss resistance “at the expense of quality at a given bitrate in the absence of packet loss”. Browsers do both for you; a server that speaks WebRTC itself has to do them on purpose.

What FEC buys, and when it doesn’t#

Back to the simulator, whose protection table puts the schemes side by side on the same network, and whose what-Alice-hears chart shows the same thing frame by frame. Its default is Opus FEC. On the random minute, FEC takes concealment from 4.5% of frames to 0.7%, and the longest gap from 40 ms to a single frame. That’s most of the trouble gone, for bits taken from inside the stream’s own bitrate.

With bursts, it does much less, and the reason is in the table above: FEC reaches back one packet. In a burst of three, packets 25, 26 and 27 are all gone, and the copies of 25 and 26 went down with them. Only 27 can be rebuilt, from packet 28. On the bursty minute FEC brings concealment from 3.7% to 1.5% and leaves a 160 ms gap. It still rebuilds more than one frame in three because not everything missing came in bursts: the 1.8% of packets that were merely late are scattered one at a time, and FEC rebuilds those just as it does on the random minute.

So reach further back with RED, two copies? Here’s the part that’s easiest to get wrong: at a 40 ms buffer, two copies recover exactly as much as one. The second copy of frame 25 rides in packet 27, which leaves 40 ms after frame 25 did, and then picks up its own extra delay on the way. With a 40 ms buffer it would have to arrive with no extra delay at all to make frame 25’s slot, and it essentially never does.

A copy is only as good as the buffer that waits for it

Reaching back k packets means the copy leaves 20k ms after its frame, so the buffer has to be deeper than 20k ms, by about the jitter, before the copy can help. FEC needs more than 20 ms of buffer to do anything (at exactly 20 ms the simulator’s FEC rebuilds 1 frame in 3,000); a second RED copy needs more than 40.

Give the buffer 80 ms and RED’s second copy starts earning its bandwidth: concealment on the bursty minute falls to 0.6%, against 1.0% for FEC at the same buffer. That’s the whole trade in one sentence: against bursts you can spend bandwidth and delay, or you can spend dropouts.

Try the samples in the simulator in order: Random loss, then Bursts of three, then Bursts, deeper buffer. Watch what Alice hears for runs of made-up frames in a row, and the protection table, which sets what each scheme recovers beside what it still leaves concealed.

The simulator is a model, and its numbers are its own: it gives FEC a copy on every packet, which is the most FEC can do (the demo showed real Opus skipping some), and its jitter is random packet to packet, where real queues tend to delay neighbours together. The tool’s page lists every assumption.

Why not just resend?#

Video calls do, and it’s worth seeing why voice mostly doesn’t. A receiver that notices a gap can send a NACK, an RTCP message naming the missing sequence number (RFC 4585 §6.2.1), and the sender resends the packet (RFC 4588).

The resend costs a round trip, and that’s the whole question: does a round trip fit inside the frame’s wait? With Alice’s 40 ms buffer and a 60 ms round trip, it doesn’t, and the resent frame arrives after its slot like any other late packet. The WebRTC transport guidelines make the same call: use retransmission “when the connection RTT is within the application’s latency budget” (RFC 8854 §8), and FEC when it isn’t.

You can see that last row in libwebrtc’s own defaults: it offers nack on every video codec (codec.cc) and not on Opus (webrtc_voice_engine.cc). Audio NACK does exist if both ends negotiate it. But for a conversation, carrying the copy in advance is usually cheaper than asking for it afterwards.

What to watch in a real call#

Everything in this post shows up in the browser’s own counters. In Chrome, open chrome://webrtc-internals during a call, or call getStats() on the peer connection and read the inbound audio stream. The definitions are from the W3C webrtc-stats spec:

Here’s what that looks like for a real call. Two peer connections in one Chrome tab, one sending a generated tone to the other, after a couple of minutes, with the table filtered to the counters this post is about:

concealedSamples divided by concealmentEvents is the number I’d look at first: it’s the average length of a gap, in samples, and at 48 kHz, 960 of them is one 20 ms frame.

You don’t have to wait for a bad call to watch these move. The lab below starts a real one in this tab: two WebRTC connections talking to each other, Opus over loopback, with Alice’s exchange as the voice, so no microphone is involved. (The voices are generated with Kokoro, an open text-to-speech model; switch Sound to Hum for a steady tone instead.) Each frame passes through this page on its way into Chrome’s jitter buffer, and the page can drop it or hold it back. Everything after that is Chrome’s own, and the counters are read from getStats() once a second.

Three things worth trying, with Listen on, giving each about 15 seconds:

It needs Chrome or Edge, because it reaches into the call through an API only they ship. And it shows concealment, not FEC: the page drops frames after Chrome has received them, so the sender never learns of any loss and has no reason to send copies.

The demo#

The code is in late-is-lost/. It needs Python 3.12 or newer. PyAV brings libopus with it, and fec_recovery.py calls that same copy directly for the decoding:

cd mainlog-code/late-is-lost
python3 -m venv .venv
.venv/bin/pip install -r requirements.txt
.venv/bin/python fec_recovery.py

The sound is synthetic so every run gives the same numbers on the same libopus. The repository’s CI runs it on a different libopus build, so it checks the shape rather than the decimals: that the encoder still puts a copy in 28 of the 44 packets, and that where it does, FEC still beats concealment by more than 10 dB.

So the receiver’s job comes down to three dials: how long to wait, how to fill a gap, and how much to send twice. Every one of them spends something, delay, quality or bandwidth, and bursts make all three cost more. Next in the series, the packets have to find each other first: how two computers behind home routers get a path between them at all, with NAT, STUN, TURN and ICE.