I came back from a coffee, unlocked the screen and found BeeBEEP open, just as I had left it. But my colleagues had seen me offline the whole time, as if I had switched the computer off and gone home.
The computer had never been off. It was on, connected, with BeeBEEP running. Only the screen was locked. So what had happened?
And it was worse than it looked. My computer has two network cards, one on each of two networks that cannot see each other, and BeeBEEP sits on both. That makes it a relay: a computer with a foot in each network carries the messages across, so the people on one side can talk to the people on the other. While my screen was locked, my BeeBEEP, the relay between the two networks, carried nothing. It was not only me who had vanished. It was the bridge.
A person who steps away from the desk should not disappear, and should not take the bridge away with them. That was not a small bug. It was BeeBEEP failing at the one thing a messenger has to do.
Waiting for a frame
I locked the screen again, on purpose this time, and while BeeBEEP was stuck I took a snapshot of every thread of the program. The answer was there, one wait inside the other.
The thread that runs the window was waiting for the thread that draws it. The drawing thread was waiting for the graphics driver. And the driver was waiting for the X server, the program that owns the screen on that kind of Linux desktop. And the X server had stopped answering.
On that desktop, locking the screen means switching to another virtual terminal, where the password prompt lives. The user's own terminal stays behind, inactive, and until the password is typed its X server leaves some of its graphics requests unanswered. A second program, started on that screen during the lock, hung before it could even open its first window. So it was not a setting of mine to change: the whole graphics side of that screen was asleep.
And the thread that waited for it was not only the window's. It was also the thread of Core, the part of BeeBEEP that speaks to the network, delivers the messages and answers the console. One frame that could not be drawn, and Core, with the network, the messages and the console, stood still with the window.
Why it had worked for years
BeeBEEP 5 runs everything on one thread too, and it never had this problem. In all those years its window never stalled it.
The difference is in how the window is drawn. BeeBEEP 5 uses Qt's classic widgets, which paint with the processor: the widgets never wait for the graphics card. BeeBEEP 6 uses QtQuick, which draws with the graphics card, on a thread of its own, and with QtQuick the window waits for every frame. It is faster and smoother, and it works on phones. But it gave the screen a power it never had before: to stop everything else.
Keeping Core away from the window was the idea from the start. The code even said so: a comment at the top of Core called it "meant to live on its own thread". I had set it aside, because one thread had served well for years. Now I had the reason not to wait any longer.
A locked screen is not a goodbye
There was an easier way out: pause BeeBEEP when the screen is locked, as it does when the computer goes to sleep. I said no.
A locked screen is a short absence. A lunch, a meeting, a coffee. BeeBEEP has no server: the computers pass messages to each other, and some of them, like mine, are the bridge between two networks. If every locked computer vanished, the network would lose its relays and its "delivered" every time someone went for a coffee. So the decision was this: while the screen is locked BeeBEEP stays alive. It goes on receiving, delivering and passing messages on. Only the window stops drawing.
I tried the cheaper half first: draw nothing while the session is away. It helped, and it is worth having anyway, no graphics card and no energy spent on a screen nobody looks at. But drawing nothing was not enough: a frame already being drawn at the instant of the switch stays stuck, and the switch arrives a third of a second before the system tells anyone that the session is locked. In one of my runs the window lost that race and stayed blocked for the whole lock, almost fifty-two seconds. It came back at the unlock, as it always did. Core, on the same thread, would not have been so lucky.
One door
So Core got its own thread. Moving it was the small part. Untangling it was the large one.
Every part of the interface talked to Core directly: it held it by the hand and read whatever it needed, whenever it needed it. Across two threads that is no longer allowed. So now the interface reaches Core through one door, and every screen goes through it. Through that door the interface reads a picture of Core's state, taken at the end of each of Core's turns and never changed afterwards. It sends its requests, which Core carries out on its own thread. And when a screen needs an answer, it asks, and the answer always arrives after the state it refers to, never before.
The rules are short. Core never waits for the window. The window waits for Core only where the operating system leaves no choice, at the start, at the end, when the computer goes to sleep, and each time with a set maximum time. A test checks that no file of the interface reaches past the door. Its list of exceptions is now empty.
When the window comes back
With Core free, a new question came up. What does Core do while the window is frozen?
It goes on working, and after each turn it sends the window an update. Nobody takes them, so they pile up. I measured it: a minute of freeze, with ten messages a second arriving. At the return 901 updates were waiting. The window took almost four seconds to catch up, and memory rose to 1.2 GB. With fifty messages a second it was 4501 updates, 35 seconds, 5.6 GB.
The answer: Core stops talking to someone who is not listening. Now the window acknowledges every update it applies. If an update goes unacknowledged for more than a second, Core holds: it keeps its latest state and sends nothing more. When the window acknowledges again, Core sends a single picture of the present. The lists catch up by comparing the picture with what they show, row by row, without starting over and without jumping back to the top. And what the missed events meant for you is worked out once: one notice per chat with new messages, a missed call as a missed call, at most one beep.
The same minute of freeze, at ten messages a second: the window shows its first frame 30 milliseconds after the return, and memory stays at 131 MB. At fifty a second, 39 milliseconds and 134 MB. However long the freeze, however busy the network, the return now costs about a twentieth of a second at most.
What turned up on the way
Measuring found three more bugs. Most of those four seconds were not the queue at all. Each new message moves its chat up the list, and every move rebuilt every row of the list, in normal use too, not only after a freeze. A list that keeps its order and moves only the row that moved brought the same catch-up from 3.9 seconds to a quarter of a second, before the picture even existed.
Its sibling: every time someone came online or went offline, the whole list was rebuilt and scrolled back to the top. The test with a person coming and going all the time took 16.7 seconds and 3.2 GB to settle. Now it takes 50 milliseconds.
And a quieter one, seen once in nine runs: if BeeBEEP was stopped in the instant between asking for its database and getting it, the network part could start after the shutdown, and the program crashed on its way out. Now every start carries a number, and an answer meant for an old start is thrown away.
One thing stays as it was, on purpose: the ring of a call and the beep of a message still play on the window's thread, and they wait with it. If you are away from the screen, it should not ring.
Coffee again
Then the test that counted: the same computer, the same desktop, the same screen lock, and on the other side a phone with BeeBEEP. I locked the screen for more than five minutes, while the phone and the computer kept writing to each other.
The console answered every question, one a second, without a single silence. The messages, thirty-three each way, were all delivered, the ones for the locked computer in less than a tenth of a second. The phone saw the computer online the whole time. At the unlock the window was there, up to date. The locked computer no longer vanished.
In that first test the phone was connected to the computer by cable, because there was no Wi-Fi at the desk. I ran it again without the cable, the phone on the network like anyone's, and it went the same way: the messages only took a little longer to arrive. The story I found coming back from coffee ended there: a locked screen is now just a locked screen.