whisper-rs in Production: Kill the Subprocess, Cache the Model, Delete the Temp Files

The specific architecture move that took transcription from “spawn a CLI, write a WAV, parse JSON” to “in-process, persistent model, direct memory results” — and the gotchas around Metal GPU on Apple Silicon.

Published: May 2026 Tags: #SoloBuilder #Rust #Whisper #LLM #LocalFirst #AudioProcessing

---

The “So What?” - Why You Should Care

Bottom Line Value: A practical blueprint for replacing whisper.cpp CLI subprocess transcription with in-process whisper-rs. Specific module structure, model caching pattern, Metal GPU enable, and how to handle the model-download UX.

What You’ll Gain:

• The 4-module structure that separates model loading from inference
• The pattern for keeping the model loaded across transcriptions (the single biggest win)
• Metal GPU acceleration on Apple Silicon (no temp files, no JSON parsing, no subprocess)
• The model-download UX that doesn’t break first launch
• The exact memory + CPU savings vs. subprocess approach
• ROI: ~2 weeks of work eliminates subprocess fragility, cuts per-transcription latency by 30–50%, and removes a 200ms cold-start tax on every recording

Who This Helps: Solo builders running whisper-cli (or any LLM CLI) as a subprocess from inside their app. Anyone who’s tired of writing temp WAVs and parsing JSON output for an inference path that could be in-process.

Why Bother: Subprocess LLMs are slow (cold-start on every call), fragile (any crash takes down inference), and impossible to instrument cleanly. In-process LLMs are none of those. With Metal GPU, they’re also dramatically faster on Apple Silicon.

---

TLDR: The Move in 5 Steps

1. Add whisper-rs with Metal feature flag (5 min)
2. Split transcription into 4 modules — model, engine, types, error (planning: 1 hour)
3. Implement model loading with persistent caching — load once, transcribe many (3–4 hours)
4. Wire inference + result extraction — no temp WAVs (2–3 hours)
5. Add model-download UX — first-launch download for whichever model you bundle (4–6 hours)

Result: 30–50% faster per-transcription, zero temp files, zero subprocess fragility, Metal GPU on Apple Silicon. ~2 weeks of work to replace the subprocess approach.

---

The Moment You’re Probably In

Your app does voice transcription. The path looks like this:

1. Record audio → write recording.wav to a temp dir
2. Spawn whisper-cli -m ggml-base.en.bin -f recording.wav --output-json
3. Wait for the subprocess to finish
4. Read recording.json
5. Parse it
6. Delete temp files

Every transcription does this, every time. And:

• The model loads from disk on every invocation — adds 200–500ms cold start
• You can’t see what the model is doing while it’s running
• Subprocess crashes leave you with half a JSON file
• Temp file management is a constant background hazard
• macOS Sequoia broke your subprocess once with a code-signing change
• Adding streaming text? Nope — subprocess output is “all at the end”

You’ve thought “I should just call whisper directly from Rust.” But:

• whisper.cpp is C++, the FFI feels scary
• You’re not sure if whisper-rs is mature enough
• The cold start “isn’t that bad” (it is)

The fix is whisper-rs and a 4-module pattern. Here’s how.

---

Step 1: Add whisper-rs (5 minutes)

[dependencies]
whisper-rs = { version = "0.13", features = ["metal"] }

The metal feature flag matters. Without it, you’re on CPU even on Apple Silicon. With it, you get GPU-accelerated inference on M-series Macs (about 3–5× faster for base.en).

For Windows/Linux/Intel Mac users, drop features = ["metal"] and you’ll fall back to CPU (or CUDA if available — see the whisper-rs README for the GPU feature matrix).

---

Step 2: The 4-Module Structure (1 hour to plan)

Don’t put everything in one file. Separate concerns:

src-tauri/src/transcription/
├── mod.rs       # public API + re-exports
├── types.rs     # TranscriptionConfig, TranscriptionResult, TranscriptionSegment
├── error.rs     # typed TranscriptionError variants
├── model.rs     # WhisperContext loading + caching
└── engine.rs    # inference orchestration

Module roles:

Module	Job
model.rs	Load WhisperContext from disk. Cache it across calls. Handle “no model present” with download flow.
engine.rs	Take audio samples → return TranscriptionResult. The hot path.
types.rs	Public types crossing module/IPC boundaries.
error.rs	thiserror-derived error enum. Never bare unwrap().

The clean split makes model loading testable independent of inference, and lets you add new inference paths (streaming, batch, etc.) without touching the model code.

---

Step 3: Model Loading + Caching (3–4 hours)

The single biggest win is load the model once, transcribe many times. Subprocess whisper-cli reloads on every invocation. Don’t.

use whisper_rs::{WhisperContext, WhisperContextParameters};
use std::sync::{Arc, OnceLock};
use std::path::PathBuf;

pub struct WhisperModel {
    context: Arc<WhisperContext>,
    path: PathBuf,
}

static MODEL: OnceLock<WhisperModel> = OnceLock::new();

pub fn get_or_load_model(model_path: &PathBuf) -> Result<&'static WhisperModel, TranscriptionError> {
    MODEL.get_or_try_init(|| {
        tracing::info!(path = %model_path.display(), "loading whisper model");
        let params = WhisperContextParameters::default();
        let context = WhisperContext::new_with_params(
            model_path.to_str().ok_or(TranscriptionError::InvalidModelPath)?,
            params,
        )?;
        Ok(WhisperModel {
            context: Arc::new(context),
            path: model_path.clone(),
        })
    })
}

Three things to know:

1. OnceLock is the right primitive. The model is initialized exactly once for the process lifetime. Subsequent calls return the cached reference.
2. Arc<WhisperContext> is shareable across threads — important because whisper inference uses multiple internal threads.
3. Loading takes 1–3 seconds for base.en on first call, then ~0ms for every subsequent call. That’s the whole win.

If you want to support changing models at runtime, replace OnceLock with a Mutex<Option<WhisperModel>>. Slightly more bookkeeping; same idea.

---

Step 4: Inference Without Temp Files (2–3 hours)

The subprocess approach writes a WAV, runs whisper, parses JSON. Skip all of that — pass samples directly:

use whisper_rs::{FullParams, SamplingStrategy};

pub fn transcribe(
    model: &WhisperModel,
    samples: &[f32],
    language: Option<&str>,
) -> Result<TranscriptionResult, TranscriptionError> {
    let mut state = model.context.create_state()?;

    let mut params = FullParams::new(SamplingStrategy::Greedy { best_of: 1 });
    if let Some(lang) = language {
        params.set_language(Some(lang));
    }
    params.set_print_progress(false);
    params.set_print_special(false);
    params.set_print_realtime(false);
    params.set_print_timestamps(false);

    state.full(params, samples)?;

    let num_segments = state.full_n_segments()?;
    let mut segments = Vec::with_capacity(num_segments as usize);
    let mut full_text = String::new();

    for i in 0..num_segments {
        let text = state.full_get_segment_text(i)?;
        let t0 = state.full_get_segment_t0(i)? as f64 / 100.0; // centiseconds → seconds
        let t1 = state.full_get_segment_t1(i)? as f64 / 100.0;

        segments.push(TranscriptionSegment {
            text: text.clone(),
            start: t0,
            end: t1,
            index: i as usize,
        });
        full_text.push_str(&text);
    }

    Ok(TranscriptionResult {
        text: full_text,
        segments,
    })
}

The wins:

• No temp files. Samples → result, in memory.
• No JSON parsing. Segments come out as native Rust structs.
• You see what’s happening. Use tracing to log timing, segment counts, confidence — instrument freely.
• Metal GPU is automatic if you compiled with the feature flag and you’re on Apple Silicon. No code change.

---

Step 5: The Model-Download UX (4–6 hours)

Here’s the gotcha that bites every first-launch user: you don’t want to bundle a 1 GB model into your app installer. But you also can’t transcribe without one. The flow:

First launch
├── No model in app-data dir
├── Show "Downloading speech model (141 MB)" UI
├── Download ggml-base.en.bin from Hugging Face (or your CDN)
├── Verify SHA-256 against known-good hash
├── Move atomically into app-data dir
└── Ready to transcribe

pub async fn ensure_model_downloaded(
    app_data_dir: &PathBuf,
    progress_tx: tokio::sync::mpsc::Sender<DownloadProgress>,
) -> Result<PathBuf, TranscriptionError> {
    let model_path = app_data_dir.join("ggml-base.en.bin");
    if model_path.exists() && verify_sha256(&model_path).await? {
        return Ok(model_path);
    }

    let tmp_path = model_path.with_extension("bin.tmp");
    let url = "https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-base.en.bin";

    download_with_progress(url, &tmp_path, progress_tx).await?;
    verify_sha256(&tmp_path).await
        .map_err(|_| TranscriptionError::ModelHashMismatch)?;

    std::fs::rename(&tmp_path, &model_path)?;
    Ok(model_path)
}

Three things to handle:

1. Atomic rename (tmp → final) so a crash mid-download doesn’t leave a corrupt partial file the next launch interprets as complete
2. SHA-256 verification — Hugging Face URLs are stable, but download corruption is real. Match against a known-good hash you ship in your binary
3. Progress reporting to the UI — the user is staring at “Downloading…” for 30 seconds; show them the bar

Optional bundling: if you want to bundle a smaller tiny.en model (75 MB) for instant first-transcribe, and download base.en in the background after first launch, that’s a clean UX move. Adds ~75 MB to installer size.

---

What This Doesn’t Fix

• Cold start for the first transcription is still 1–3 seconds (model load). Subsequent transcriptions are 50–200ms before inference starts. If 1–3 seconds matters, pre-warm the model in your app’s setup().
• Memory usage goes up while model is loaded (~300 MB for base.en). Subprocess approach offloaded this to the subprocess. Tradeoff: in-process is faster + simpler + more memory-resident.
• Metal GPU works on macOS only. Windows/Linux/Intel Mac will use CPU. That’s fine for base.en (still real-time-ish) but medium.en and large-v3 need GPU to be usable in real time.
• Model versioning is your problem. If you ship a v2 that uses a different whisper.cpp version with incompatible model files, you’ll need to re-download. Plan for it.
• No live streaming of text to the UI yet — that’s post #142. The pattern above gives you final results only.

---

Try It This Week

Saturday afternoon (3 hours):

• Add whisper-rs with the metal feature
• Create the 4-module skeleton
• Manually download a base.en.bin to a hardcoded path so you can iterate without writing the download flow yet

Sunday morning (3 hours):

• Write model.rs with OnceLock caching
• Write engine.rs with the basic transcribe function
• Call it from a unit test against a known-text WAV — assert the output contains expected words

Sunday afternoon (2 hours):

• Benchmark: 1 minute of audio. Note the cold-start time and the warm-start time. You should see ~1.5s vs ~150ms for the model overhead.
• Compare against your old subprocess approach. The win should be visible.

Monday (4 hours):

• Write the download UX
• Wire it into the app’s setup flow
• Test with deleted-model scenario

Tuesday–Friday (1–2 hours/day):

• Polish: progress UI, error messages, retry logic
• Validate against your existing test corpus

That’s ~2 weeks part-time to ship in-process transcription.

---

The Takeaway

Subprocess LLMs are a pattern from 2023 — when the in-process Rust bindings weren’t mature enough and the subprocess felt safer. They’re mature now.

whisper-rs with persistent model caching gives you:

• 30–50% faster per-transcription
• Zero temp files
• Zero subprocess fragility
• Metal GPU on Apple Silicon (free 3–5× speedup)
• Real-time instrumentation
• A clean foundation for streaming (next post)

Two weeks of focused work. The pattern transfers to any LLM you’re currently subprocessing — llama.cpp bindings, candle, mistral.rs, etc. The architecture is the same: typed Rust modules, persistent model state, samples in, structured results out.

If you’re currently shelling out to a CLI for inference: this is the path off. The “in-process feels scary” instinct was correct in 2023 and wrong in 2026.

---

Coming up next: post #139 — When NOT to rewrite. The counterintuitive companion to the rewrite decision. Specifically: why we kept 1,722 lines of TypeScript in our Rust port, and the engineering principle that protects you from over-rewriting.