The number was real. It measured the wrong thing.

August 11, 2026 18 min read 3812 words Moksh Desai
The number was real. It measured the wrong thing.
Table of Contents

macOS never asks the XM5 for its head-tracking sensor. My own tools said building it was not viable, Here is the argument I wrote against them, and the four bugs that followed.

This one runs long — around fifteen minutes, and a good deal longer than anything else I’ve put here. It’s the whole build: the measurement that told me to stop, the argument I wrote against it, and the four bugs that cost me the most time. If you’re only here for the decision, the first two sections carry it, and you can leave after that. The rest is where things broke.

There is a small gyroscope inside each Sony WF-1000XM5 earbud. It knows which way your head is turned, about twenty-five times a second. Sony uses it for spatial audio, The trick where a film’s dialogue stays anchored to your screen instead of sliding around with your skull, so it sounds like it’s coming from the room rather than from inside your head.

It works on iOS. It works on Android.

On macOS, nothing is listening.

Not because the hardware is missing. The buds happily broadcast their orientation over Bluetooth to anything that asks. macOS just never asks. The sensor is right there, publishing into a void, and the operating system has no idea it exists.

That gap bothered me for about a week before I decided to close it.

Measure first, build second

The temptation with a project like this is to start with the fun part: grab the sensor data, spin a soundfield, see if it feels cool. I’ve been burned by that before. So the first thing I built wasn’t the product at all. It was a machine for measuring whether the product could exist.

The concern was latency. Head-tracked audio is a feedback loop with your neck in it: you turn, the sound must follow, and if it follows late the illusion doesn’t just weaken, it inverts. Instead of a room that stays still, you get a soundstage that lags and swims. Bluetooth is famously bad here. A2DP – the protocol that carries decent-quality stereo to wireless headphones – buys its quality with buffering, and buffering is delay.

So Stage 0 was a rig: play a sharp click through the system, capture it with the laptop’s microphone, cross-correlate the recording against the original to find the exact offset, and repeat until the number stops moving. Ten trials per buffer size, median rather than mean so one bad correlation can’t drag the headline figure.

Crucially, the rig had to prove itself before it was allowed to judge anything. It ran the identical measurement against the MacBook’s built-in speakers, whose latency is known to be tiny. It came back with 16.3 ms, well inside its own ceiling. The harness was trustworthy. Only then did I let it look at the earbuds.

The earbuds came back at 292.7 ms.

For context, my own spec had written the acceptance bands before any measurement existed: under 150 ms good, under 200 ms workable, under 250 ms marginal. Anything beyond that was notViable. The tool printed the verdict itself, in the report it generated: notViable

Nearly three hundred milliseconds. A third of a second between turning your head and the sound catching up. By my own stated criteria, the project was over before Stage 1 began.

Overruling my own gate

Here is the part I’m proudest of, and it isn’t code.

I wrote a document arguing with the verdict, and I wrote it before proceeding, so the reasoning could be examined later rather than taken on trust or quietly forgotten if it turned out to be motivated nonsense.

The written numbers (150/200/250) were just guesses I made before testing anything. The actual goal was simply for it to feel fast and natural. When those guessed numbers conflict with how it actually feels in practice, real experience matters more than the initial guess.

And I had direct evidence. Months earlier I’d hacked together a prototype in Reaper: system audio into a virtual device, up into a 16-channel ambisonic bus, rotated by the earbuds’ own sensor, decoded back to binaural, out to the buds over Bluetooth. Ugly, slow, held together with routing spit. And the head tracking in it felt fine. Not tolerable, unnoticeable.

That prototype could not have beaten the A2DP path, because almost none of the delay belongs to the application. CoreAudio reports 234.8 ms for the device before my code touches anything. 9843 frames of device latency plus a 512-frame buffer. The acoustic measurement found 292.7 ms. The missing 57.9 ms is AAC encoding and the earbuds’ jitter buffer: delay so far down the stack that software cannot see it at all, which is exactly why I refused to trust a software-only estimate and built an acoustic rig instead.

graph LR
    A["my code
capture + convolution"] --> B["512-frame buffer
11.6 ms"]
    B -->|handed to CoreAudio| C["device latency
9843 frames, 223 ms"]
    C -->|A2DP| D["AAC encoding +
earbud jitter buffer"]
    D --> E["your ears"]

    subgraph reported["Reported by CoreAudio — 234.8 ms"]
        B
        C
    end

    subgraph invisible["Invisible to software — 57.9 ms"]
        D
    end

Where the 292.7 ms actually sits. My own share is the 512-frame buffer is about 12 ms, or 4% of the total. Everything else is either reported-but-unreachable or not reported at all. Optimising my code to literal zero would have moved the number by almost nothing: either the delay was survivable or the project was dead, and there was no third option to engineer my way into.

I also wrote down the reasons to distrust my own number. The rig’s self-reported path components summed to more than the arrival time it actually observed, which means one term was probably double-counted at around ±40 ms of slop. Good enough to tell 100 ms from 300 ms. Not good enough to decide a cliff at 250.

And then I wrote down the thing that could still sink it, because a decision document that only lists supporting evidence is marketing:

Latency detection needs a sharp spatial cue to time against. A low-order ambisonic scene decoded through a mismatched HRTF localises vaguely. The muffling and the unnoticeable latency are plausibly the same defect wearing two hats. The product uses direct binaural with two crisp point sources. Sharper localisation makes latency more detectable. Fixing the muffling may unmask the lag.

In other words: the prototype might have felt fine because it sounded bad. Make it sound good, and the delay might come screaming out of hiding.

I proceeded anyway, with that written down as the live risk.

The answer arrived at the end of Stage 1, when there was finally a real renderer to listen to. I tested it by deliberately adding delay on top of the delay that was already there. An extra 160 milliseconds was still undetectable by ear for music and video. The proxy had been wrong. Not slightly wrong, Wrong by a margin wide enough to have killed a working product.

The number was real. What it measured was not what mattered.

Getting the sensor to talk

With the gate cleared, the next problem: nobody documents this protocol.

The buds expose the tracker as a HID device, The same class of thing as a keyboard or a mouse. My first several hours went into a bug that in hindsight is almost funny. I enumerated HID devices the normal way and found nothing. The sensor isn’t published as an IOHIDDevice; it appears as an IOHIDUserDevice, which the standard enumeration doesn’t return. The fix was to stop asking for devices by class and match on vendor and product ID instead.

Then a run of failures that all looked like permission problems and were not:

  • SetReport returning 0xe00002c2. Not permissions. A missing report-ID byte at the front of the buffer.
  • The enable command accepted, and no data ever arriving. The magic value is 0x03, not the 0x01 that every instinct suggests.
  • Rotation data arriving as pure noise. Classic off-by-one: the report ID occupies byte 0, so the payload starts at byte 1, and I was reading from 0.
  • The stream working beautifully and then dying after ten seconds, every time. The buds go idle when they aren’t in your ears. They need a keepalive.

Each of these cost hours and each is one line. I kept a table of them in the spec as I went. Symptom on the left, actual cause on the right, Because the debugging time was almost entirely spent re-deriving things I’d already figured out and forgotten. That table is now the most valuable page in the repository.

What comes out the other side is a quaternion, A four-number description of an orientation in 3D. Updated 25 times a second.

Making a laptop sound like a room

Knowing where your head points is half of it. The other half is convincing your ears that a sound is coming from a specific place in space.

That’s a head-related transfer function: a measurement of how your head, shoulders and the specific folds of your ears colour a sound differently depending on where it comes from. Your brain decodes that colouring into direction. It’s why you can point at a noise behind you with your eyes closed.

I’m using the SADIE II database from the University of York, Measurements taken from a dummy head at hundreds of positions around a sphere. For each source position, convolve the audio with the impulse response measured at that position, and the sound appears to come from there. Rotate the positions opposite to your head movement, and the soundstage stays nailed to the room while you turn.

Which makes the whole system a loop that runs through the earbuds twice — audio out, orientation back:

graph LR
    A["anything playing
music, video, a call"] -->|system output| B["BlackHole
virtual device"]
    B -->|captured by the app| C["binaural renderer"]
    C -->|convolved for
each source position| D["XM5 earbuds"]
    D -->|gyroscope, 25 Hz
over Bluetooth HID| E["orientation
quaternion"]
    E -->|counter-rotate
the soundstage| C

The earbuds are both the output and the sensor. macOS routes every app’s audio into a virtual device (which is BlackHole); the renderer captures from there, convolves it against a measured head, and sends it on. The return path is the part macOS doesn’t know exists.

The first thing that broke was a spec assumption. I’d picked 48 kHz throughout, because it matches the SADIE measurements and avoids resampling. Then the measurement rig reported what the XM5 actually supports: 16 kHz and 44.1 kHz. No 48. So a resampler had to exist after all. Discovered by the measuring stage, before it could become a rewrite.

The rest of the difficulty is that convolution happens on the real-time audio thread, and that thread has exactly one rule: never block. No locks, no file access, and no memory allocation. Because malloc can take a lock, and a lock can wait, and waiting means the audio buffer isn’t filled in time, and an unfilled buffer is an audible click in someone’s ears.

So I built a trap: a hook that counts every allocation happening on a thread marked real-time, and a test suite that asserts the render path allocates zero times. That trap later caught something I never would have found by listening, but not in the way I expected. More on that shortly.

Two subtler problems, both about time:

Impulse responses can’t be swapped instantly. Change the filter abruptly between one audio buffer and the next, and you get a click on every head movement. They have to be crossfaded.

25 Hz is not enough. Sensor updates arrive every 40 ms; audio needs a new orientation every buffer, far more often. Without interpolation between updates, fast turns judder. So the renderer interpolates along the shortest arc between the last two orientations.

That interpolation hid a bug I want to describe properly, because it’s the kind that silently does nothing.

My first version aimed the interpolation at now. But the publisher timestamps each sample when it arrives, which means every read from the audio thread happens after that timestamp, so the interpolation factor was always clamped to its maximum. The renderer stepped at raw sensor rate, and the smoothing I’d written did nothing at all. Nothing crashed. No test failed. It just quietly wasn’t working.

The fix aims one report interval behind the newest sample, so the factor sweeps properly across the gap between updates. It costs 40 ms of extra tracking lag — which the Stage 1 finding had already established was affordable — and it’s safer than the alternative of predicting forward, which overshoots every time you reverse direction.

I could afford that 40 ms because I had measured what I could afford. The gate I’d overruled kept paying out.

The four bugs worth writing down

Everything above is the part that was hard because it was unknown. What follows is the part that was hard because computers are like this.

The measurement that blamed the wrong thing

My real-time allocation trap started reporting violations. Per my own spec, that meant the render core was allocating during rendering, and the remedy was drastic: rewrite the core in C.

That’s expensive advice. So before acting on it, I went looking, and the counter was telling the truth about a different thing entirely. The trap counts allocations on any thread marked real-time. But when an output device disappears, CoreAudio tears down that device’s audio thread, a thread I had marked, and its cleanup allocates.

The evidence made it unambiguous. Three reconnects produced three discrete steps in the counter — 0 → 10 → 17 → 24 — each one landing inside the window where the output device was gone, with five minutes of flat counting through healthy playback in between. A HID-only disconnect, which doesn’t restart audio, moved it not at all. If the render core were really allocating, that count would climb continuously during playback. It never did.

It had already cost me time twice in one session before I pinned it. The report now snapshots the count at both edges of a device restart and attributes the difference honestly. A measurement that lies confidently is worse than no measurement, and this one had been telling me to rewrite a component that was working perfectly.

The device that came back as a stranger

Toggling Bluetooth left audio permanently dead until I restarted the command by hand. Found by an automated soak test whose single pass condition was: audio must never go silent.

When the buds disconnect, macOS destroys the audio device’s numeric ID. When they reconnect, it mints a new ID for the same physical hardware. My audio callbacks stayed bound to the dead one and CoreAudio’s failure mode here is not an error. It simply stops calling them. Ever. No notification, no status, nothing to check.

Worse, my first attempted recovery was nonsense: I detected the loss and switched to bypass mode. But bypass changes what the callback computes, and the callback itself was dead. I’d written a fix that could not possibly work, and it read as perfectly reasonable code.

The real fix re-resolves the device by UID, the identity that survives the round trip and builds fresh callbacks. Matching on the numeric ID would eventually mistake a recycled ID for the original device and cheerfully render into a stranger’s speakers.

Same code, different identity

Everything worked as a command-line tool. I wrapped precisely the same code in a .app bundle and it broke in two different ways simultaneously.

macOS grants permissions to a bundle identity, not to code. My command-line tool had been quietly inheriting the terminal’s permissions the whole time. Every hardware test I’d ever run had passed under borrowed credentials.

Without a Bluetooth usage description in the bundle, calling into Bluetooth doesn’t get politely refused, the process is killed with SIGABRT (Signal Abort). Which also skips all cleanup, which strands the system audio output on a virtual device, which leaves the user in silence having to fix their sound settings by hand.

Without a microphone usage description, capturing audio is never authorized and the system hands the engine silent buffers instead of failing. Head tracking looks alive. The status display reads perfectly healthy. There is simply no sound.

Two of my favourite failure modes in one afternoon: one that kills the process for asking politely, and one that fakes success forever.

Silence is the worst possible failure

This one is the reason the whole design is shaped the way it is.

To do its job the app takes over your system audio output, routing everything through a virtual device so it can be processed. Which means it is holding your sound hostage, and every path out of that state has to give it back.

I had a bug where it didn’t. The app remembers which device you were using before the takeover so it can restore it. On this machine, that device is usually… the earbuds. So when the earbuds disconnect, the restore targets a device that just vanished, fails, and leaves the system pointed at the virtual device, a null sink. Silence, with nothing on screen explaining it, and no way out but the Sound settings panel.

Then I found the same failure through a different door: quitting the app didn’t restore anything either, because macOS’s terminate path doesn’t unwind the app the way a normal exit does, so the cleanup never ran. The process just died holding your audio.

Both are now fixed. A fallback to built-in output when the remembered device is gone, and a terminate handler that runs the same shutdown every quit goes through. There’s a third safeguard too: if the app is force-killed and can’t clean up at all, it leaves a marker on disk, and the next launch notices and repairs the routing.

And one deliberate non-fix. When the buds drop out briefly, the app keeps holding your output for a full minute before giving it back. That looks wrong until you think about who’s wearing the headphones: restoring your speakers mid-dropout would blast your music into the room. Silence is the safe failure for the few seconds a reconnect takes. A full minute of it means you’ve walked away, and then you should get your device back.

Every path out of the takeover, including the ones I don’t control:

stateDiagram-v2
    direction TB

    [*] --> Idle
    Idle --> Rendering: "buds present,
switch on"
    
    Rendering --> Paused: "buds disconnect"
    Paused --> Rendering: "buds return
within 60s"
    
    Paused --> Idle: "60s elapsed,
restore device"
    Rendering --> Idle: "switch off or quit,
restore device"
    
    Rendering --> Killed: "force quit
or crash"
    Killed --> Repairing: "next launch
reads marker"
    Repairing --> Idle: "routing
restored"

    note right of Paused
        Still holding the virtual device.
        Restoring now would blast the
        speakers mid-dropout.
    end note

Only one state can leave you silent, and it is deliberately time-limited. The Killed path is the one that matters most: it is the only exit the app cannot handle while running, so it leaves a marker on disk and the next launch cleans up after the last one.

The part that isn’t the audio

The last stretch was turning a command-line tool into something with a switch in the menu bar. I expected this to be the easy part. It was not, but it was interesting in a completely different way.

The best lesson came from a status message. With the earbuds off, the app said:Required devices missing. Run xm5-latency devices to check.

That’s a perfectly good sentence in a terminal. In a menu-bar app it’s absurd twice over: it tells a person with no terminal open to go run a command, and it reports a fault when nothing is wrong. The earbuds being in their case is the single most common state this app will ever be in. It’s a state, not an error, and it now says “Waiting for the buds.”

The other lesson was about knowing when to stop. I tried to add a light/dark toggle for the app’s window, and it failed three times in a row. Each fix cured one symptom and produced another: the theme could be set but never unset; then it could be unset but repainted a beat late; then it repainted on time but came up wrong. Three failures is a signal. It means you don’t have a bug, you have the wrong design.

macOS and SwiftUI handle unset theme preferences differently, which caused issues whenever one had to guess. I fixed it by explicitly setting a concrete choice, light or dark. So a preference is never missing.

What it took

  • 405 automated tests, none of which need the hardware, because every decision the app makes is a pure function tested on its own.
  • One measuring rig, built before the product, that proved itself against a known-good reference before it was allowed to judge anything.
  • A table of protocol gotchas that is worth more than any individual file of source code.
  • A written argument with my own tooling, made in advance, that turned out to be right.

A note on how it was built, since the commit history makes it obvious anyway: I did this with the help of an AI agent (Claude Specifically), working from written plans, stage by stage, test-first. That worked well for the things it’s good at, holding a large design in mind, refusing to skip verification, writing the tests before the code. It was worth much less for the things that actually decided this project. It couldn’t hear the muffling. It couldn’t tell me the tracking felt fine in a prototype from months ago. It couldn’t judge whether 292.7 ms was survivable, and when I told it the number was survivable it recorded the reasoning and the risk rather than agreeing with me. Every hardware verification in this project ran through my ears, and every one of the four bugs above surfaced there first.

The interesting part isn’t that an agent wrote a lot of the code. It’s that the discipline it enforced, measure before building, write the reasoning down before deciding, don’t call it done until it’s verified, is the discipline that made overruling a “not viable” verdict a defensible engineering decision instead of wishful thinking.

The thing macOS wouldn’t do

There’s a switch in my menu bar now. I turn it on, and audio from anything on my Mac gets processed through a measured model of a human head and delivered to my earbuds. I turn my head, and the sound stays exactly where it was — the dialogue stays in the laptop, the music stays in the room.

My tools said 292.7 milliseconds and notViable. Both halves of that were accurate measurements. Only one of them was about the thing I actually cared about.

The number was never the criterion. Whether it sounds right was the criterion, and the only instrument that could measure that was already attached to my head.

The code is on GitHub: okku007/xm5-head-tracking-macos