← Build log

Building Sophisticated Transcript Formatting: A Deliberate A/B Testing Approach

December 2024

Sometimes the best engineering decisions come from admitting you don’t know which approach is better—and building the infrastructure to find out scientifically.

The Problem: One Size Doesn’t Fit All

VoiceCatcher’s paragraph break detection was working well for clear speakers with natural pauses. Our simple approach: wait for a 600ms+ pause, validate it’s at a sentence boundary, insert a paragraph break. Clean, fast, predictable.

But users have different speaking patterns. Some rush through thoughts without pauses. Others have complex discourse with topic shifts and transitions that our simple threshold missed entirely.

The question: Should we build a more sophisticated system, or stick with simplicity?

The Meta-Challenge: Designing the Design Process

Here’s where it gets interesting. Instead of just coding up a “better” solution, I worked with ChatGPT to design the engineering process itself.

The prompt engineering session went like this:

“I want to create a more sophisticated paragraph detection system, but I don’t want to replace what we have until we know it’s actually better. How should we approach this?”

The AI helped me think through:

  • Complete system separation (no shared logic)
  • A/B testing infrastructure from day one
  • Clear naming conventions that describe the approaches
  • Comprehensive logging for debugging both systems
  • Success metrics that matter to users

This wasn’t about AI writing code—it was about AI helping design a better engineering process.

Simple vs Contextual: Two Philosophies

The result: two completely separate approaches with crystal-clear differences.

Simple Pause Detection

Philosophy: Trust the speaker’s natural pauses

  • Single threshold: 600ms+ pause = paragraph break
  • Binary decision making
  • User configurable threshold
  • ~1ms processing time
// The beauty of simplicity
if (pauseDuration >= this.PARAGRAPH_PAUSE_THRESHOLD) {
  if (this.isNaturalParagraphBoundary(prevWord, currentWord, pauseDuration)) {
    breaks.push(position);
  }
}

Contextual Scoring

Philosophy: Consider everything we know about language

  • 6-factor scoring system (0-10 scale)
  • Pause duration, sentence boundaries, discourse markers
  • Topic transitions, paragraph length context
  • Negative scoring for poor break locations
// Sophisticated but transparent
let score = 0;
score += this.scorePauseDuration(pauseDuration); // 0-5 points
score += this.scoreSentenceBoundary(wordBefore, wordAfter); // 0-4 points
score += this.scoreDiscourseMarkers(wordAfter); // 0-2 points
score += this.scoreTopicTransitions(wordAfter); // 0-1 points
score += this.scoreParagraphContext(position); // 0-1 points
score -= this.penalizeContinuations(wordAfter); // Negative factors

return score >= 3.0; // Configurable threshold

A/B Testing Infrastructure: Built for Learning

The key insight: Build comparison into the architecture from day one.

public compareFormattingApproaches(transcript: string, timingData: TimingData): {
  simplePause: FormattingResult;
  contextualScoring: FormattingResult;
  comparison: string;
} {
  const simplePauseResult = this.formatWithSimplePauseDetection(transcript, timingData);
  const contextualResult = this.formatWithContextualScoring(transcript, timingData);

  // Side-by-side analysis with detailed metrics
  return { simplePauseResult, contextualResult, comparison };
}

Testing tools:

# Compare on specific test case
node scripts/compare-paragraph-methods.cjs test-medium-pauses

# Compare across all test cases
node scripts/compare-paragraph-methods.cjs --all

The Deliberate Engineering Process

Phase 1: Complete Separation

  • Two independent classes: NaturalFormatter and ContextualParagraphFormatter
  • No shared logic between approaches
  • Clean interfaces that don’t leak implementation details

Phase 2: Clear Naming

  • “Simple Pause Detection” vs “Contextual Scoring”
  • Method names that describe behavior: formatWithSimplePauseDetection()
  • Consistent logging prefixes: ⏱️ [SIMPLE] vs 🧠 [CONTEXTUAL]

Phase 3: Comprehensive Logging

console.log(`🎯 [SCORE] Position ${position}: ${score.toFixed(1)}/10 - ${reasons.join(', ')}`);

Every scoring decision is visible. Every threshold check is logged. When the contextual system makes a choice, you can see exactly why.

Phase 4: Production Safety

  • Simple Pause Detection remains the default
  • Contextual Scoring available for comparison only
  • No user-facing changes until we have data

Early Results: Conservative by Design

Testing on our audio test suite revealed something important: both systems are appropriately conservative.

PARAGRAPH FORMATTING COMPARISON:
===============================
Simple Pause Detection: 1 paragraphs
Contextual Scoring: 1 paragraphs

METHOD DESCRIPTIONS:
Simple Pause: Uses 600ms threshold only
Contextual: Uses multi-factor scoring (pause + grammar + discourse + context)

The test files we have don’t contain the complex discourse patterns where contextual scoring would shine. Both systems correctly preserved single-paragraph structure.

This is exactly what we want to see. Neither system is over-eager to break paragraphs. The sophistication is there when needed, but doesn’t activate unnecessarily.

The Meta-Lesson: Process Design Matters

The most valuable part of this project wasn’t the contextual scoring algorithm—it was designing an engineering process that:

  1. Preserves what works (Simple Pause Detection stays unchanged)
  2. Enables experimentation (Complete A/B testing infrastructure)
  3. Provides visibility (Detailed logging and comparison tools)
  4. Supports decisions (Clear metrics and success criteria)

Working with AI to design the process led to better engineering than working with AI to write the code.

Production Strategy: Data-Driven Decisions

We now have the infrastructure to answer the question scientifically:

Phase 1: Gather data on edge cases where Simple Pause Detection struggles Phase 2: Test Contextual Scoring on those specific scenarios
Phase 3: Compare not just accuracy, but user satisfaction and performance Phase 4: Make the switch only if contextual scoring proves meaningfully better

The Bigger Picture: Engineering for Uncertainty

This approach generalizes beyond paragraph detection:

  • When you’re not sure which approach is better, build both
  • Design comparison into the architecture from day one
  • Use AI to design better engineering processes, not just write code
  • Default to conservative, proven approaches while experimenting
  • Make decisions based on data, not sophistication

The sophisticated approach might be better. Or the simple approach might be optimal for our users.

The point is: now we can find out.

VoiceCatcher’s contextual paragraph formatting represents a new approach to handling uncertainty in product development—build the infrastructure to learn, rather than making assumptions about what users need.