What's inside one voice packet

From air to 80 bytes#

When Alice speaks, her microphone turns changing air pressure into a changing voltage, and the sound card measures that voltage at a fixed rate. Each measurement is a sample, stored as a number. Opus, the codec we’ll meet in a minute, works at 48,000 samples a second, and a common way to store each sample is as a 16-bit integer. That raw stream of numbers is called PCM (pulse-code modulation). If you want the longer story of what a sample rate can and can’t capture, the sample-rate post goes through it with three test tones.

Voice isn’t sent a sample at a time. The sender collects a short slice, compresses it, and sends the slice as one packet. Twenty milliseconds is the usual slice: it’s the default packet duration for Opus over RTP (RFC 7587 §6.1). Here’s the whole trip, from sound to packet:

The top row is the part worth doing the arithmetic on:

WebRTC requires every endpoint to support Opus, along with G.711, the classic telephone codec (RFC 7874 §3). Opus can encode slices from 2.5 to 60 ms, at anything from 6 to 510 kb/s (RFC 6716 §2.1.4, §2.1.1). The 80 bytes are exact here because the demo holds the bitrate constant. With real speech and a variable bitrate, frames average around that size and grow and shrink with what’s being said.

The bottom row of the table matters more than it looks

A third of every packet is headers. Halve the audio bitrate and the headers stay the same size, so the traffic drops by a third, not a half. The bitrate post works through what that means for bandwidth planning.

Alice’s packet, byte by byte#

To show real bytes rather than made-up ones, the demo for this post encodes a steady 440 Hz tone in place of Alice’s voice, with the codec a voice call would use, held at a constant bitrate so the sizes come out exact. It builds the headers around each 80-byte frame and prints one packet back, layer by layer. The addresses come from the ranges reserved for documentation.

Read it from the outside in. Each layer answers one question, and each machine on the way reads only the part it needs: a router reads IP, the receiving computer’s operating system reads UDP to find the program, and that program reads RTP.

1
def pack(self) -> bytes:
2
    first = 2 << 6  # version 2, no padding, no extension, zero CSRCs
3
    second = (int(self.marker) << 7) | self.payload_type
4
    return struct.pack("!BBHII", first, second, self.sequence, self.timestamp, self.ssrc)

1
packets = [
2
    ipv4(udp(Rtp(sequence=1000 + i, timestamp=i * SAMPLES).pack() + frame), ident=0x3A00 + i)
3
    for i, frame in enumerate(frames)
4
]

1
enc = av.CodecContext.create("libopus", "w")
2
enc.sample_rate = RATE
3
enc.layout = "mono"
4
enc.format = "s16"
5
enc.bit_rate = BITRATE
6
enc.time_base = Fraction(1, RATE)
7
enc.options = {"vbr": "off", "application": "voip", "frame_duration": str(FRAME_MS)}

80 6f 03 f2 00 00 25 80 1f 2e 3d 4c

cd mainlog-code/voice-packet
python3 -m venv .venv
.venv/bin/pip install -r requirements.txt
.venv/bin/python build_packet.py

Every packet has a deadline#

The receiver can’t play a packet the moment it arrives, because packets don’t arrive evenly: one might take 28 ms and the next 35. That unevenness is called jitter. So the receiver holds each packet briefly in a jitter buffer and plays it at a fixed time: when it was sent, plus how long packets normally take to arrive, plus a small cushion for the jitter.

That fixed time is the packet’s playout deadline, and it changes what “delivered” means:

You can’t go back and un-play a gap.

What one lost packet costs#

Back to the WebSocket. A WebSocket runs over TCP, and TCP promises that every byte arrives, in order. When a packet is lost, TCP resends it, which sounds like exactly what you’d want.

The catch is the “in order” half. TCP won’t hand the application anything that comes after a missing byte until the missing byte has been resent and has arrived. The packets behind it sit in the receiver’s buffer, perfectly fine, and the application can’t read them. That’s called head-of-line blocking, and for audio with deadlines it turns one lost packet into several late ones.

To put numbers on it, the demo’s one_lost_packet.py models one lost frame. It’s a model rather than a network test, so every run gives the same answer:

The model’s assumptions

• A frame leaves every 20 ms and takes 30 ms to arrive, so the round trip is 60 ms. No jitter.
• The jitter buffer adds 40 ms on top of the 30 ms trip, so frame n, sent at 20n ms, must arrive by 20n + 70 ms. Arriving exactly on the deadline counts as on time.
• Each frame travels in its own TCP segment (one TCP packet), and frame 5 is lost once.

TCP can find out about the loss in two ways:

200 ms is Linux’s default minimum for that timer (the kernel’s TCP_RTO_MIN); the standard itself recommends rounding it up to a full second. The heart of the model is a single rule, TCP’s in-order delivery:

1
def tcp(resend_at: int, name: str) -> Outcome:
2
    """In-order delivery: nothing after the lost frame reaches the application before the resent copy does."""
3
    repaired = resend_at + ONE_WAY_MS
4
    return Outcome(name, {n: sent(n) + ONE_WAY_MS if n < LOST else max(sent(n) + ONE_WAY_MS, repaired) for n in FRAMES})

Every frame after the lost one is delivered at whichever is later: when it actually arrived, or when the resent frame 5 did. Here’s what that does to Alice’s frames 4 to 8:

Follow frame 6. Over UDP it arrives at 150 ms, 40 ms before its deadline, and plays. Over TCP it arrives at exactly the same moment, but the application can’t have it until the resent frame 5 lands at 250 ms, and by then frame 6’s deadline passed 60 ms ago. The network delivered it on time, and TCP held it back.

Across the whole stream, the script counts:

With the one-second timeout the standard recommends, the last row would be far worse. Newer loss detection, such as RACK, which is what current Linux uses to detect loss, can resend sooner than three duplicates would allow, and its tail loss probe handles a loss at the very end of a burst without waiting for the full timeout. Neither changes the part that hurts: the frames behind the loss still wait for it.

The numbers move; the shape doesn’t

The exact counts depend on the round trip, the buffer and where the loss falls, so don’t read 80 ms as a constant. What stays the same is who pays: over TCP, the frames behind a loss pay for it.

What happens to the gap over UDP#

Over UDP, frame 5 simply never comes. The receiver sees the sequence number skip one, knows exactly what’s missing, and asks the decoder for 20 ms of something plausible in its place. That’s packet-loss concealment: the decoder extrapolates from the audio just before the gap. The Opus specification describes it (RFC 6716 §4.4) and libopus implements it. Frame 6 plays on time. How good the guess sounds, and what to do when losses come in bursts, is where the next post picks up.

So the reason voice rides UDP is control. UDP lets the receiver decide what a missing packet costs, and for live audio the right decision is almost always “cover it and keep going”.

The simulator below draws the whole comparison, frame by frame. Here’s how to read its first chart before you try it:

Try it yourself: move the lost frame, the delay, the buffer or the timeout, and watch which frames each transport still gets to the listener in time. The Bigger buffer sample is worth a look, because a big enough buffer can absorb fast retransmit completely, at the price of 80 ms more delay on every frame.

The second chart is the other half of why 20 ms is the usual slice. Every packet pays the same 40 bytes of headers whatever it carries, so smaller frames mean more packets and more header bits: at 10 ms the headers cost 32 kb/s, as much as the sound itself. Bigger frames save that, and each lost packet takes more audio with it: 60 ms instead of 20.

So is TCP ever fine for audio?#

Yes, and it’s worth being fair about it.

That last row is the WebSocket’s real job in a voice product: everything around the audio. It just shouldn’t carry the audio itself when the audio is live and the network is the open internet.

So the answer to the WebSocket question is to send the audio in small, timestamped, numbered packets over UDP, and let the receiver cover the gaps. UDP gives the receiver that choice, and it also hands it the job of dealing with every late and missing packet itself. That’s the next post: jitter buffers, concealment and forward error correction.