← Build log

How Questioning My AI Assistant Led to Better Product Decisions (And Why You Should Challenge Every AI Answer)

December 19, 2024

I was building VoiceCatcher’s paragraph break detection feature when my AI assistant confidently suggested 1.5 seconds as the “most common natural break length.” It sounded reasonable. It felt like something that could be true. And for a split second, I almost implemented it without question.

Then I did something that saved my product from a potentially poor user experience: I asked for the research behind that number.

TLDR:

  • What You’ll Gain: A framework for questioning AI recommendations and finding evidence-based solutions
  • Time Saved: Hours of user testing and feature rewrites by getting it right the first time
  • Problems Solved: Preventing AI hallucinations from becoming product decisions
  • Implementation: 5-minute questioning technique you can use immediately
  • Bottom Line Value: Transform AI from a liability into a research powerhouse that actually improves your product decisions

The Problem: AI Confidence vs. AI Accuracy

Here’s the thing about AI assistants that makes them both incredibly useful and potentially dangerous for technical product managers: they sound confident even when they’re completely wrong.

When I asked about natural pause durations for speech processing, my AI confidently threw out “1.5 seconds” like it was citing some fundamental law of linguistics. No hesitation, no qualifying language, just pure algorithmic confidence.

But confidence isn’t accuracy. And as TPMs, we know that shipping the wrong feature confidently is worse than shipping no feature at all.

The stakes here weren’t trivial. VoiceCatcher is a privacy-focused voice-to-text app, and getting paragraph breaks wrong would make transcriptions feel unnatural and choppy. Users would notice immediately. They’d assume the AI transcription was broken, when really it was my AI product advisor that was broken.

The 5-Minute Questioning Framework

When your AI gives you a specific recommendation, especially one involving numbers, user behavior, or “best practices,” here’s how to validate it before it becomes a product decision:

1. Ask for Sources (30 seconds)

“What research do you have to back that up?” “Can you find studies that support this recommendation?” “Where does this number come from?“

2. Request Alternative Perspectives (1 minute)

“What are other common approaches to this problem?” “Are there industry standards or competing methodologies?” “What would happen if we went with a different approach?“

3. Demand Specificity (2 minutes)

“Can you search for actual research on [your topic]?” “Find me studies with real data, not general principles.” “What do the actual numbers show in controlled studies?“

4. Cross-Reference Claims (1.5 minutes)

“Do multiple sources agree on this?” “Are there conflicting findings I should know about?” “What’s the confidence level of this recommendation?”

Total time investment: 5 minutes. Potential time saved: Hours or weeks of building the wrong thing.

What Happened When I Actually Questioned the AI

When I pushed back on that 1.5-second claim, something magical happened. My AI assistant admitted it had no research backing that number – it was essentially an educated guess dressed up as expertise.

Then it went to work.

Within minutes, it had found multiple peer-reviewed studies on speech pause duration. The real research showed something completely different:

  • Comma pauses in natural speech: 0.38-0.78 seconds
  • Period pauses: 0.81-1.43 seconds
  • Optimal naturalness perception: 0.6 seconds (across multiple language groups)

The research was solid. Multiple studies. Cross-cultural validation. Actual data from real speakers and listeners.

Not only was 1.5 seconds wrong – it would have made VoiceCatcher sound robotic and unnatural. Users would have felt those pauses dragging on forever.

The Business Impact of Questioning AI

This 5-minute validation process saved VoiceCatcher from:

  • User experience disaster: Transcriptions that feel unnatural and choppy
  • Development rework: Building a feature based on bad assumptions, then having to rebuild it
  • User trust erosion: People assuming the AI transcription is broken when it’s just poorly tuned
  • Competitive disadvantage: Missing the evidence-based sweet spot that makes speech feel natural

Instead, we now have:

  • Evidence-based defaults: 0.6 seconds backed by actual research
  • Cross-cultural compatibility: Settings that work for native and non-native English speakers
  • Immediate user value: Transcriptions that feel natural out of the box
  • Confident product decisions: I know exactly why we chose these settings

ROI on 5 minutes of questioning: Potentially weeks of development time and user frustration.

The Human Reality: When AI Becomes Your Research Partner

Here’s what I learned from this experience: AI assistants are incredible research partners when you treat them like research partners, not oracles.

When I first got that 1.5-second answer, I felt relieved. Someone (something?) had given me a number I could implement. Decision made, move on to the next feature.

But that relief was dangerous. It was the feeling of avoiding the hard work of validation – the same trap that leads to shipping features based on assumptions instead of evidence.

When I started questioning the AI, something shifted. Instead of just accepting answers, I was directing the research process. The AI went from giving me potentially wrong answers to finding me peer-reviewed studies with actual data.

The breakthrough moment wasn’t when AI gave me the “right” answer. It was when I realized I could make AI do the research legwork while I focused on interpreting the results and making product decisions.

What This Means for Solo Builder TPMs

As technical product managers building solo, we’re constantly tempted to treat AI as a replacement for domain expertise we don’t have. That’s a mistake.

AI isn’t a substitute for expertise – it’s a research acceleration tool. But only if you know how to question it properly.

The 5-minute framework I outlined above turns every AI interaction into a mini-research project. You’re not just getting answers; you’re getting evidence-based answers with known confidence levels.

This is especially critical for solo builders because we don’t have teams of researchers and domain experts to fact-check our decisions. We have to build that validation process into our workflow.

And here’s the thing: questioning AI actually makes it more valuable, not less. When you push for sources and evidence, AI transforms from a potentially unreliable advisor into a research powerhouse that can find and synthesize information faster than any human could.

The Bigger Mission

This experience reinforced something important about building products in the AI era: the humans who learn to collaborate effectively with AI – not just accept its output – will build better products.

We’re not trying to replace human judgment with AI judgment. We’re trying to augment human judgment with AI research capabilities.

For VoiceCatcher, this means we can make evidence-based product decisions about complex technical problems (like speech processing) without hiring domain experts. But only because we’ve built a process for validating AI recommendations before they become product features.

That’s the competitive advantage for solo builders: not just using AI, but using AI correctly. And “correctly” means questioning everything, demanding evidence, and treating AI like the research tool it is rather than the expert it pretends to be.

Now if you’ll excuse me, I need to go test whether 0.6-second paragraph breaks actually feel natural in my own voice. Because even with peer-reviewed research, there’s no substitute for eating your own dog food.

Actionable Prompts You Can Use:

  1. Research Validation: “What research supports [specific AI recommendation]? Find me studies with actual data.”

  2. Source Checking: “Can you search for peer-reviewed studies on [topic] and summarize the key findings?”

  3. Alternative Exploration: “What are 3 different approaches to [problem] and what does the research say about each?”

  4. Confidence Assessment: “How confident should I be in this recommendation? What are the limitations or uncertainties?”

Implementation Timeline: Start questioning your next AI recommendation (5 minutes)

Skill Prerequisites: Ability to read research abstracts and basic statistical literacy

Decision Tree: Question AI when it gives you specific numbers, claims about user behavior, or recommendations that will directly impact user experience.