← Build log

The AI Persistence Problem: When Your Assistant Won't Let Go of Legacy Code

“Remove all the regex code. We’re going pure NLP.”

Seems clear, right? Except three commits later, I’m still finding regex patterns lurking in tests, utility functions, and configuration files. It’s like asking someone to clean out the garage and finding they’ve just moved everything to different corners.

This is the AI persistence problem: when your assistant simply cannot let go of legacy patterns, even with explicit instructions to eliminate them entirely.

What You’ll Gain:

  • Recognition patterns for incomplete legacy code removal
  • Specific prompts that force complete architectural cleanup
  • Strategies for achieving true A/B testing separation
  • Framework for detecting hidden legacy dependencies
  • Bottom Line Value: Actually complete your refactoring instead of maintaining hybrid systems that increase complexity

The Setup: Clean Separation for A/B Testing

The goal was simple: completely separate our regex-based text processing from the new NLP approach so we could A/B test them properly. Clean architectural boundaries, isolated systems, measurable comparison.

What I got instead was a Frankenstein hybrid where:

  • The main code used NLP
  • Tests still expected regex outputs
  • Utility functions contained regex fallbacks
  • Configuration files referenced regex settings
  • Error handling assumed regex behavior

It’s not A/B testing when both systems are tangled together. It’s just technical debt with extra steps.

The Problem: AI’s Selective Hearing

Here’s the pattern I’ve observed: AI has trouble with complete system replacement. It can implement new features brilliantly, but removing old systems entirely? That’s apparently much harder.

The symptoms are always the same:

  1. Implements the new system correctly in main code paths
  2. Leaves legacy code in secondary systems (tests, utilities, configs)
  3. Creates hybrid dependencies where new code calls old patterns
  4. Maintains fallback logic “just in case” without being asked
  5. Preserves old interfaces even when they’re no longer needed

When I say “remove all regex processing,” AI hears “implement NLP but keep regex as backup.” That’s not the same thing.

The Real Example: Regex Zombie Code

Even after multiple explicit requests to eliminate regex patterns, I kept finding things like:

// In tests - still expecting regex behavior
expect(result.improvements).toContain('NLP processing');

// In utilities - regex fallback logic
const processText = (text) => {
  try {
    return nlpProcessor.enhance(text);
  } catch (error) {
    return regexProcessor.enhance(text); // Why is this here?!
  }
};

// In configuration - regex-specific settings
const settings = {
  enableNLP: true,
  regexPatterns: [...], // Still being maintained
  regexFallback: true   // Never asked for this
};

This isn’t architectural migration - it’s feature accumulation. The codebase grew instead of transforming.

Why This Matters: The A/B Testing Problem

For solo builders trying to make data-driven decisions, hybrid systems kill your ability to measure properly. You can’t compare “NLP with regex fallbacks” against “pure regex” - you’re not testing what you think you’re testing.

The business impact:

  • Impossible to measure actual NLP vs regex performance
  • Technical debt compounds as you maintain two systems
  • Decision-making becomes unclear without clean data
  • Future changes affect both systems unpredictably

The Framework: Forcing Complete Removal

Here’s what I’ve learned about getting AI to actually eliminate legacy systems:

1. Explicit Enumeration

Don’t say “remove regex” - list what needs to go:

“Remove ALL regex patterns from: main processing code, test files, utility functions, configuration files, error handling, and fallback logic. If it mentions regex, delete it.”

2. Search and Destroy Prompts

Make AI hunt down legacy references:

“Search the entire codebase for any remaining references to regex, pattern matching, or string replacement. Show me every occurrence and remove them.”

3. Architectural Boundary Enforcement

Be explicit about system separation:

“The NLP system should have zero dependencies on regex code. If NLP fails, the app should fail - no fallbacks to old systems.”

4. Test Update Requirements

Force test suite alignment:

“Update ALL tests to expect NLP behavior only. Remove any tests that validate regex patterns or outputs.”

The Human Reality

I’ll admit, there were moments where AI’s “helpful” fallback logic seemed reasonable. “What if NLP fails? Isn’t it good to have backup processing?”

But that’s exactly the trap. Backup systems prevent you from fully committing to new approaches. They create comfort zones that reduce pressure to make the new system robust.

The breakthrough came when I realized: if I’m not confident enough in NLP to remove regex entirely, then I’m not ready to ship NLP at all. Hybrid systems are confidence hedges disguised as technical decisions.

Detection Patterns: Finding Legacy Zombies

Watch for these persistence patterns:

  • Try/catch blocks wrapping new systems with old system fallbacks
  • Configuration options for systems you thought you’d removed
  • Test assertions expecting outputs from deprecated systems
  • Import statements for libraries you’re replacing
  • Error messages referencing old system components
  • Documentation describing deprecated workflows

The Search and Destroy Approach

When AI won’t let go, get explicit:

# Grep for legacy patterns
grep -r "regex\|pattern\|replace" src/ tests/ config/
grep -r "OLD_SYSTEM_NAME" .
find . -name "*.test.*" -exec grep -l "legacy_pattern" {} \;

Then demand removal of each occurrence individually.

What This Means for Us

This experience taught me that architectural cleanliness requires aggressive enforcement with AI. It will default to additive solutions unless you specifically demand subtractive ones.

The deeper insight: AI optimizes for safety through redundancy, but we need to optimize for clarity through elimination. Sometimes the bravest architectural decision is removing the safety net entirely.

As solo builders, we can’t afford to maintain multiple systems indefinitely. Every line of legacy code is future maintenance debt. Better to commit fully to new approaches and deal with their limitations directly than to hedge with complex hybrid systems.

The Copy-Paste Solution

Here are the prompts that finally worked:

1. "Search the entire codebase for ANY remaining references to [OLD_SYSTEM]. List every file and line number."

2. "Remove every occurrence you found. Do not replace with fallback logic - delete entirely."

3. "Update all tests to expect [NEW_SYSTEM] behavior only. Remove any tests validating [OLD_SYSTEM] outputs."

4. "Scan configuration files and remove any settings related to [OLD_SYSTEM]."

5. "Verify that [NEW_SYSTEM] has zero dependencies on [OLD_SYSTEM] code."

The key is specificity and enumeration. AI needs explicit instructions for destruction - it’s much better at creation.

Next week: How to build A/B testing capabilities into AI-assisted development so you can actually measure whether your architectural decisions are improvements or just different complexity.