Product Experiments

The cheapest way to be wrong is before you build anything. Experiments turn opinions into evidence and hunches into decisions.

An experiment is a structured way to test whether what you believe about your product is actually true. You start with a hypothesis (“We believe that if we do X, then Y will happen”), define what you’ll measure, run the test, and decide what to do based on the results. The goal is to generate evidence, not to confirm what you already think.

Experiments vary widely in cost and fidelity. Interviews and surveys are cheap but subjective. Landing pages and fake door tests measure actual behavior. Concierge tests and Wizard of Oz prototypes simulate the product manually. MVPs are the smallest thing you can build and ship to test a hypothesis in a real environment. The right experiment depends on what assumption you’re testing and how much evidence you need. Assumption mapping helps prioritize which assumptions to test first: start with the riskiest ones, use the cheapest test that generates credible evidence.

Good experiment design requires three things: a clear hypothesis, a measurable expected outcome, and a decision rule for what you’ll do if you’re right or wrong. Teams that skip the decision rule end up running experiments that don’t change anything. The 3X Model suggests that early-stage products (Explore phase) should run more experiments with lower fidelity, while established products (Expand/Extract) can afford more rigorous, higher-cost validation.

Traps

• Too many experiments running at once dilutes focus and makes it hard to attribute results
• Too few kinds of experiments creates blind spots; vary across desirability, viability, and feasibility
• Not enough rigor turns experiments into anecdotes instead of evidence
• Lack of controls makes it impossible to know whether results are meaningful
• Failure to process learnings means experiments generate data nobody acts on

Resources

• David J. Bland and Alex Osterwalder, “Testing Business Ideas” (Wiley, 2019) — a field guide to experiment types organized by category
• Assumption Mapping — how to identify which assumptions to test
• 3X Model — how experiment strategy changes across product phases