← Build log

The Bug That Made Whisper Drop the Word "Auto"

2026-06-24

A heuristic that’s right for one 50ms window can be wrong for the whole recording — statelessness in a real-time audio pipeline isn’t simplicity, it’s a bug generator. That sentence is the entire lesson of this post, and it cost me an evening of staring at WAV files to earn it. The symptom: plug a Focusrite Scarlett 2i2 into VoiceCatcher, say “auto mode,” and Whisper hands back “mode.” The word “auto” just evaporated. So did the first word or two of most clips. The audio was captured perfectly. The transcription still came out mangled. Here’s the chase, the wrong turn I took, and the fix.

TLDR

What You’ll Gain:

  • Why a multi-channel USB audio interface (the Scarlett 2i2, 4 channels, one mic plugged in) can silently corrupt transcription even when the recorded audio is pristine
  • The actual root cause: a per-chunk channel-averaging decision that re-litigates itself every ~50ms and amplitude-modulates your signal at exactly the word boundaries
  • How to tell a capture-truncation bug from a mixing bug using a 100ms RMS profile of the recorded WAV (with the real numbers I saw)
  • The before/after of the divisor logic — a four-line change in where a decision is made, not what it decides
  • Bottom Line Value: If you mix multi-channel audio down to mono for transcription, do not make the “which channels are live” decision independently per buffer — latch it across the recording, or it’ll eat your soft words.

Date-stamped 2026-06-24, macOS 26.5.1, Tauri 2, cpal capture, whisper-rs transcription, Scarlett 2i2 4th Gen.

The bug report that didn’t make sense

Here’s the report, paraphrased from me, to me, at my desk: “I say ‘auto mode’ and it transcribes ‘mode.’ First couple words of every clip come through wrong or missing. It is NOT a gain problem — same mic, same gain works fine in the legacy app.”

That last clause is the one I want to flag now, because I’m about to admit I ignored my own correct instinct. I knew it wasn’t gain. I’d used this exact Scarlett, at this exact gain, into the legacy Electron version of VoiceCatcher for months without a dropped word. And yet the first thing I did — the very first thing — was go chasing the gain theory anyway, because the recordings looked quiet and “quiet audio confuses Whisper” is the kind of explanation your brain reaches for when it wants the problem to be simple.

It was not simple. It was never gain. I was right the first time and then talked myself out of it for forty minutes. Embarrassing, documented, moving on.

Why does Whisper drop the first word of my recording?

The honest answer for my bug: because the recorded signal was getting quietly turned down at word boundaries before Whisper ever saw it, and Whisper treats “too quiet to be confident” as “not there.” But before you trust that, you have to rule out the much more common cause — capture-start truncation, where the recording stream opens a beat late and clips the front of the audio. If the first 100ms of your WAV is silence, you’ve got a truncation bug, and the fix is somewhere in your stream-start handshake, not your mixer.

So I profiled it. That’s the step I’d skip if I could only give you one piece of advice and have it stick.

How do you tell a capture bug from a mixing bug?

You RMS-profile the recorded WAV in fixed windows and look at where the energy is. I sliced each clip into 100ms windows and printed peak and RMS per window. Here’s what came back, and here’s why it pointed away from where I was looking:

  • Energy at 0ms. The very first 100ms window already had speech-level RMS. The recording did not open with silence. Capture-start truncation: ruled out. The front of the word was in the file.
  • Correct file duration. No samples missing off the end either. The WAV was the right length for the clip.
  • Quiet overall. Peak around -34 dB, RMS hovering ~460–680 out of a possible 32767. That’s the number that suckered me toward the gain theory. The whole file was low.
  • Suspicious dips. And here’s the tell: between and at word onsets, the RMS didn’t just dip a little — it cratered to 75–121. Roughly a 4-to-6x drop, right at the boundaries where words start and stop.

A uniformly quiet file is a gain story. A file that’s quiet and lurches down by 4x precisely at word onsets is not a gain story — it’s a modulation story. Something in the pipeline was turning the signal down, on and off, in time with the speech. Gain doesn’t do that. A mixer with a bad memory does.

That dip-to-121 pattern is the fingerprint. Once I saw it, the gain theory was dead and I knew exactly which stage to open: the channel mixer.

Why does a 4-channel audio interface break transcription?

Because of a fix I’d made earlier — a good fix, applied at the wrong scope.

The Scarlett 2i2 4th Gen presents as a 4-channel device even with a single mic plugged into input 1. So cpal hands the pipeline four interleaved channels, three of which are dead silence. If you naively average all four channels to mono, you divide your one real signal by 4 — an instant -12 dB haircut just for owning a nicer interface. Earlier in the project I’d fixed exactly that (catalogued as AUD-012): the mix_to_mono function was taught to average only the channels that carry signal, defined as RMS at or above a floor:

const CHANNEL_ACTIVE_RMS_FLOOR: f32 = 0.0005;

A channel above the floor counts toward the divisor. Channels below it are treated as dead and ignored. One live mic of four → divide by 1, full level. Correct. Tested. Shipped.

The bug wasn’t what that function decided. It was how often.

VoiceCatcher’s real-time pipeline drains the capture ring buffer in small chunks — source_rate / 20, so roughly 50ms of audio per chunk (pipeline.rs). And the old mix_to_mono was stateless: every single chunk re-ran the active-channel test from scratch, with zero memory of the chunk before it. Fine when the mic is mid-vowel and loud. A disaster the instant the mic goes quiet.

Picture the /aw/ at the front of “auto.” It’s a soft onset — low energy for those first tens of milliseconds. In the 50ms chunk that happens to contain it, the one live channel’s RMS dips below 0.0005. The stateless function looks at that chunk, sees zero channels clearing the floor, and hits its fallback: when nothing’s active, average all four channels to preserve silence. So that chunk — the chunk holding the word “auto” — gets divided by 4. The next chunk, where the vowel’s gone loud, divides by 1. Then a quiet inter-word gap divides by 4 again.

The signal came out amplitude-modulated, ducking down by ~4x at every soft onset and every pause — which is to say, at every word boundary. Whisper got a recording where the quiet words had been turned down relative to the loud ones, and it did the reasonable thing: it dropped the ones it couldn’t hear. “Auto,” soft and leading, never had a chance.

The mixer wasn’t broken. It was indecisive. It re-took a vote on reality every 50ms, and at the worst possible moments the vote came back wrong. Think of a motion-sensor light that you have to keep waving at — stand still for half a second and it plunges you into the dark, right as you’re trying to read.

And the reason legacy never had this? Legacy never did per-chunk active-channel gating at all. It mixed naively and ate the -12 dB. Worse on paper, but consistent — no modulation, no dropped words. My clever fix introduced a subtler bug than the one it replaced.

The fix: latch the decision, don’t re-litigate it

The decision “which channels carry the mic” is true for the whole recording. A mic doesn’t hop from input 1 to input 3 mid-sentence. So the active-channel set should be decided once and held — not recomputed every 50ms.

I built a stateful ChannelMixer (in resampler.rs) with a sticky active set. The first time any channel clears the floor in any chunk, that channel is latched into the active set and stays there for the rest of the recording. Quiet chunks no longer collapse the divisor, because the channel that was loud two chunks ago is still in the set. The pipeline now owns one ChannelMixer per recording and calls mixer.mix() instead of the stateless mix_to_mono.

Here’s the before/after, and notice how small the actual change in behavior is:

Before (stateless, per-chunk):

each ~50ms chunk:
  active = channels with RMS >= 0.0005  (recomputed from THIS chunk only)
  if none active: divide all channels by channel_count   <-- the trap
  else:           divide active channels by active_count

After (sticky, per-recording):

sticky_active = {}   // persists across all chunks of the recording

each ~50ms chunk:
  for each channel active in THIS chunk:
    sticky_active |= channel        // latch it, permanently, for this recording
  if sticky_active is non-empty:
    divide sticky channels by |sticky_active|   // quiet chunk still divides by 1
  else:
    fall back to per-chunk decision  // only until the first live channel is seen

A quiet chunk after a loud one now divides by 1, not 4. The modulation is gone. The signal that reaches Whisper is the signal that was in the mic.

The safety argument is the part I want skeptics to chew on:

  • A single mic on a multi-channel interface latches one channel and mixes at full level through every quiet chunk. This is the case that was broken; it’s now fixed.
  • Genuine stereo or multi-mic input latches all the live channels on the first loud chunk and averages them — bit-for-bit identical to what the stateless path did. Latching can’t over-include a channel that never carries signal, so a true stereo recording is untouched.
  • A recording that opens with silence has an empty sticky set, so each chunk falls back to the old per-chunk decision until the first real channel appears — then it latches. No regression for “start recording, then talk.”

The stateless mix_to_mono didn’t get deleted. It’s retained for offline mixing and tests, where you hand it a complete buffer and there’s no stream to keep state across. Stateless is correct when you have the whole thing at once. It’s only a bug when you feed it a stream one chunk at a time and ask it to remember nothing.

Did the fix actually work? (Yes — verified live and in tests)

Two regression tests pin the behavior so this can’t silently come back:

  • channel_mixer_sticky_set_holds_through_quiet_chunk — feeds a loud chunk then a quiet one and asserts the quiet chunk keeps full level instead of collapsing ~4x. This is the exact failure, frozen into a test.
  • channel_mixer_genuine_stereo_averages_both — feeds real two-channel content and asserts both channels still get averaged, proving the sticky set doesn’t break legitimate stereo.

Full suite: 197 Rust tests pass, clippy clean. And the part that actually matters — live verification on the Scarlett 2i2: “auto mode” now transcribes as “auto mode,” in both Auto mode and Manual mode. The word came back. Catalogued as AUD-018 in ERRORS.md, which is where the next person who hits a multi-channel transcription oddity should look first.

The Human Reality

The genuinely humbling part wasn’t the bug. It was that I’d written the bug, three weeks earlier, as a fix I was proud of. AUD-012 was a real win — multi-channel mic dilution is a nasty, mic-dependent problem and that fix was correct. I just didn’t ask the one question that mattered: does this function ever run on a partial view of the audio? It did. It ran on 50ms slices. And a decision that’s obviously correct on a whole recording is a coin flip on a 50ms slice of one.

The other humbling part: the user (me) said “it’s not gain” in the first sentence of the bug report, and was right, and I spent the next forty minutes acting like he might be wrong. Trust the person who’s used the legacy build for months. They’ve already run the A/B test you’re about to waste time re-running.

What This Means for Us

If you’re building any real-time audio pipeline — transcription, effects, VAD, anything that drains a buffer in chunks — go audit every heuristic in your hot path and ask: is this decision a property of this chunk, or a property of the recording? Channel selection is a property of the recording. Gain normalization, mostly the recording. Silence detection, the chunk. Mix those up and you get my bug: a per-chunk decision that’s individually defensible and globally catastrophic, modulating your signal at the exact frequency of human speech.

The deeper takeaway, and the one I’ll be repeating to myself: statelessness is not free. In a batch function it’s a virtue. In a streaming function it’s a decision to forget context that the stream actually has — and forgetting, in audio, sounds like dropped words.

Next steps you can take today:

  • If your transcription drops soft or leading words but the WAV sounds fine: RMS-profile the recorded file in 100ms windows before you touch gain. Energy at 0ms rules out capture truncation; dips that track word boundaries point at your mixer, not your mic.
  • If you mix multi-channel to mono for ASR: make the active-channel decision once per recording and latch it. Never recompute “which channels are live” per buffer — a soft onset will dip a live channel below your floor and collapse your divisor at the worst moment.
  • If you’re on a multi-channel interface (any Scarlett, most USB interfaces): know that one mic can present as 4 channels, and that naive averaging costs you -12 dB while naive per-chunk gating costs you whole words. The fix is in the middle: latch the live set, average only that, hold it steady.
  • Keep your legacy build around as an oracle. “It worked in the old version with the same hardware” is the most valuable sentence in any bug report. It already tells you the problem is in what changed.