A voice cannot wait

For twenty years my BeeBEEP has carried text and files. BeeBEEP 5 has no calls; the new BeeBEEP will have them. Before writing one line of the call there was one question to answer: can a messenger with no server carry small packets that are worth nothing if they arrive late? A chat message can wait a second. A voice cannot: twenty milliseconds of voice that arrive after their turn are thrown away. This is the story of what was hard, what I chose and what we threw away. I write "we" where the work was done together: I am building the new BeeBEEP with Claude.

A question before the code

So the first piece of work was not audio at all. It was a measurement of the network path the call would use, with synthetic frames of the size and pace of a voice (60 bytes, fifty a second) and a simulated network that adds 20 ms each way and loses packets.

UDP gives the link's own delay whatever the loss, and loses what it loses. TCP loses nothing and pays in the tail: one retransmission holds every frame behind it. So the call travels on UDP, on a "live channel" of its own beside the normal connection, and TCP is only the fallback when UDP cannot pass.

The first run of this table was worse on TCP: 41 ms at the median with no loss. It turned out every BeeBEEP connection had Nagle's algorithm on. That was a bug of its own, fixed before going on.

What I chose

• The call is a plugin. It ships with BeeBEEP on every platform, but it stays a plugin, the kind you met in A home for plugins: the plugin interface keeps its most demanding user, and the media libraries stay out of the client and the relay. An administrator can turn calls off.
• Libraries put together, not the whole of WebRTC. Opus for the voice, Ogg for the voice notes, and from WebRTC only its audio processing library: the echo canceller (AEC3), noise suppression, gain control. We build them ourselves from pinned sources, because some Linux distributions ship versions that are too old (Opus 1.3.1, an audio processing library without AEC3).
• The voice notes moved to Opus too, on the same stack. To a BeeBEEP 5 a voice note still goes as WAV, which it can play.

Measuring a call with nobody talking

A call is hard to measure by ear, and impossible to measure the same way twice. So BeeBEEP has a synthetic microphone and a synthetic speaker. The microphone "talks": a voice-like sound with short marks in it, each a 30 ms burst the far end can recognise. The time between a mark leaving one microphone and arriving at the other speaker is the delay from mouth to ear. Two instances, two network namespaces, a simulated network between them, and a call can be run again and again in the same conditions.

The first measured call, with no echo canceller yet: from 85 to 115 ms from mouth to ear at the median, with up to 1 % loss.

The instrument sometimes lies too, and that is worth saying. With two synthetic voices making identical marks at the same rhythm, the echo canceller "found" an echo 412 ms long where there was none: no real voice is the other one's waveform half a second later. We tried marks at random times, and they collided instead. A measuring tool has to be judged like the code it measures.

Sound on a desktop

The hardest part was not the network. It was the sound server of a Linux desktop (PipeWire) and what happens between Qt and it.

• A buffer smaller than the server's quantum starves the stream. We asked for buffers of 20 ms, to keep the delay small. PipeWire's default quantum is 1024 samples, 21.3 ms. The result: the speaker played at about half of real time, about 40 % of the frames were dropped as late, and the microphone delivered 7 % fewer frames than it should.
• A pull of 170 ms that was not. The first diagnosis said the sound server pulled 170 ms of sound at once. It was Qt's own read buffer of 16 KB in between, which turned a small request into 85 ms of samples played late.
• The playout on its own clock. Until then the speaker pulled and the call filled. Now the call writes on a steady tick of 10 ms into a ring, and the speaker drains the ring at whatever pace it likes. The two clocks drift apart; the drift is absorbed by whole ticks, never by resampling.
• Each device remembers what worked. Buffers start at 20 ms and grow where a device needs more, and what a device needed is remembered for the next call. The first call on a new device pays once, the next ones start right.

The echo

Without headphones, the other person's voice comes out of the speaker and goes back into the microphone. The echo canceller learns that path and takes it away: in our tests by 33 to 35 dB, with an echo 60, 120 or 400 ms late, and it finds the delay by itself even when it is told a wrong one.

It costs something. The audio processing adds 15 ms from mouth to ear: 9 for the echo canceller, 7 for noise suppression, 1 for the high-pass filter. Every WebRTC client pays the same.

I made two choices on real sounds:

• Noise suppression stays on, at "high". My headset has a 50 Hz hum with its harmonics. The high-pass filter alone takes 15 dB off the 50 Hz and almost nothing off the rest; the suppressor at "high" takes about 24 dB off overall. Turning it off would give back 7 ms, and the hum with them.
• The gain starts at 0 dB, not at 15. With 15 dB from the first sample, a voice that starts loud is driven into the limiter until the gain control has heard enough to lower it.

And one choice was to change nothing. While both people talk at once, the echo canceller holds back the near voice until it has learned the room; with headphones, until it has decided there is no echo at all, which can take up to a minute. We could tune its insides. I decided not to: tuning an echo canceller trades one defect for another, an echo that leaks or a voice that is eaten, and the browsers ship it as it is. I will judge it by ear, in real calls.

An idea measured and thrown away

At a large quantum (8192 samples, 170 ms: a power-saving profile, or some Bluetooth headsets) the microphone hands over 170 ms of sound at once. The call encodes and sends eight or nine frames back to back, then nothing for 170 ms. The far end seemed to run dry between the bursts.

The obvious fix was to pace the sending: hand the frames to the network on a clock of 10 ms, as the playout does on the other side. We built it and measured it. It added from 20 to 60 ms of delay and bought nothing measurable. A second measurement, with the synthetic microphone given the same bursts, showed that bursts alone cost delay, not lost frames. The paced sending was not kept.

What Telegram does

Before deciding what to do instead, we read how Telegram solved the same problems, in its public code: its calls library tgcalls, its version of WebRTC, the audio of Telegram Desktop and the older engine, libtgvoip. Their choices fit their network, the mobile internet; ours have to fit ours, the LAN.

• The time on a frame is the voice's, not the network's. In WebRTC the timestamp of an audio packet counts samples, and the jitter buffer measures each packet's arrival against it; Telegram's older engine did the same with its own counter. A burst then looks like jitter, the buffer grows to cover it, and it does not run dry. Our frames carried the time they were sent, as bunched as their arrival, so the far end saw no jitter at all and stayed at its smallest size. This is what I decided to change: the far end's jitter buffer will learn from the clock of the microphone.
• Nobody paces the sending. Telegram Desktop takes every 10 ms piece the microphone has collected, one after the other, as ours does.
• The echo canceller is not tuned. Telegram uses WebRTC's audio processing with its default configuration. On iPhone the software canceller is off and Apple's voice processing does the work; on Android the older engine used the phone's own canceller when it was there, its own otherwise. It is the rule I had already chosen for phones.
• The delay told to the canceller is an estimate there too. On Linux and macOS Telegram Desktop tells it a constant 20 ms on the microphone side and lets it find the real echo. So do we.
• More delay, by choice. Telegram's calls use Opus frames of 60 ms and a jitter buffer that never goes below 50 ms: on the mobile internet robustness is worth more than 50 ms. BeeBEEP's calls start on a LAN, with frames of 20 ms and a floor of 40 ms.

So would the whole of WebRTC solve our problems? Of the four open questions it answers one, with an idea we can take without it. The others it answers the way we already do, or with more delay, at the price of a large fork to keep on five platforms. What it has and we do not is a gentler way to correct the drift inside the voice; I will listen for ours in real calls.

A budget, read correctly

The usual rule for a call is 150 ms from mouth to ear: ITU-T G.114 calls it "transparent", and up to 400 is still acceptable. With the audio processing on, four of our ten test conditions went past it: 150.8, 152.5, 159.5 and 163.3 ms, the ones with simulated jitter, 5 % loss, or TCP. The rule I have set now is 150 ms at the median on UDP with 0 and 1 % loss, where every condition stays between 111 and 131 ms, and 200 ms for the harder ones. And our simulated network adds 20 ms each way, which a real office LAN does not have.

What comes next

The first real call between two computers, which I will make with and without headphones, and then the call's window. Windows and macOS. Then the phones, on Wi-Fi: on Android the service that keeps the microphone alive with the screen off is already built. And later a walkie-talkie: a voice note heard while it is being recorded, on the LAN and over Bluetooth.