3 Types of Design Experiment (and when to use them)

You’ve done direct research with customers, you’ve extracted the insights, priortised a problem, ideated on solutions, identified the leap-of-faith assumptions and now you are ready to start testing them to reduce the risk in your solution. But where do you start?

Testing assumptions effectively requires a structured approach. Different experiment types align with specific product risks, enabling you to validate assumptions systematically. However, the complexity, and therefore time and cost, of tests can vary widely.

Early-stage tests should be quick and inexpensive, allowing for rapid iteration. As you gain confidence in your idea, the complexity and investment in tests can increase. By categorising experiment types based on the stage of product development—exploratory, evaluative, and verification—you can create a clear roadmap for de-risking your product. Let’s dive into each category and outline how to build an effective experimentation flow.

1. Exploratory Tests: Understanding Early-Stage Ideas

Exploratory testing focuses on uncovering insights at the idea stage when the goal is to evaluate desirability and basic viability before investing significant resources.

Desirability Tests

These tests evaluate whether potential customers find your idea compelling. They are split into two approaches: reported behaviour and real behaviour.

• Reported Behaviour: These methods focus on how potential customers express their interest in a concept or feature.

  • Single-question surveys ask users simple, pointed questions to gauge preferences or intentions.

  • Buy-a-Feature exercises simulate resource constraints, asking participants to prioritise features they would "pay for" with limited budgets.

  • Press releases or brochures describe a hypothetical product, simulating how it might be marketed and testing whether customers find the offering compelling.

• Real Behaviour: Instead of self-reported interest, these methods test whether potential customers actively engage with an idea.

  • Ad campaigns (Google Ads, Facebook, etc.) can be used to verify interest in a topic or area.

  • Fake-door tests present users with an option to interact with a feature that doesn’t yet exist, measuring clicks to gauge interest.

  • Landing pages collect user data (sign-ups or pre-orders) to quantify demand for a concept.

Often the biggest risk with a new idea is that we all believe that a customer will want what we are building. Evaluating whether this is true is critical to understand whether it is worth investing in further development.

Usability Tests

Usability tests at this stage focus on uncovering whether users can intuitively interact with early design concepts. While you won’t be able to validate full usability at this stage, you can identify navigational or conceptual misunderstandings.

• Paper Prototypes: Sketches of interfaces or workflows allow users to provide immediate feedback on navigation, layout, or comprehension.

• Low-fidelity prototypes: Tools like wireframes let you explore user flows without the need for polished visuals.

• Card Sorting: Ideal for testing information architecture, this technique helps you understand how users group or prioritise content.

2. Evaluative Tests: Building Confidence as Designs Take Shape

Once you’ve passed the exploratory stage, evaluative tests validate riskiest assumptions using higher-fidelity prototypes and real-world scenarios.

Desirability/Viability Tests

To really validate if customers want the solution you are creating you need to see if they will put their money where their mouth is. Platitudes are easy to give, but real buying intent is much harder to obtain. If you can get a good signal here you know that you are solving a real problem.

• Letter of Intent: Customers commit in writing to buying or using a product when it’s ready, signaling genuine interest.

• Mock Sale: Create a realistic sales scenario where users are asked to "purchase" the product, providing insights into willingness to pay.

• Pre-Sale Campaigns: Go one step further from the mock sale and actually charge customers for the solution, even though it doesn’t exist yet.

• Crowdfunding: Test viability and demand while securing resources, as customers fund the development of your product (e.g., Kickstarter or Indiegogo).

Usability/Feasibility Tests

This phase includes ever increasingly complex prototypes and early MVPs to evaluate whether users can successfully interact with your solution and whether the solution can be feasibly implemented.

• Digital Prototypes: High-fidelity, interactive prototypes to simulate the final experience.

• MVPs: A stripped-down version of your product including only core features to test usability and initial market fit.

• Wizard of Oz Tests: Users interact with what appears to be a fully functional system, but operations are manually handled behind the scenes.

• Concierge Tests: Provide a personalised experience to users to test workflows or demand (e.g., manually delivering a service to mimic the product experience).

Usability testing happens right through the design and delivery phases because we need to think of development as design. We will continue to iterate on the product right up until we have satisfied customers.

3. Verification Tests: Monitoring and Validating in Real Life

Once your product or feature is live, verification tests ensure that it delivers value, meets user needs, and performs as expected over time.

Key Verification Techniques

• Usage Analytics: Regularly analyse user behaviour to identify friction points or underutilised features.

• A/B Testing: Test variations of features or interfaces to optimise for user preferences or business goals.

• User Feedback Loops: Continuously collect and act on feedback through surveys, reviews, or customer support interactions.

• Performance Monitoring: Ensure features function smoothly under real-world conditions, maintaining speed, uptime, and reliability.

Putting it All Together

The experiments you choose should align with your confidence in the idea and the complexity of the test. Let’s explore a practical flow for launching a task management app:

1. Exploratory Testing (Desirability and Basic Usability)
   Week 1: Create a landing page with a description of your task management app. Use an ad campaign to drive traffic, measuring sign-ups or interest forms.
   Week 2: Begin solution interviews with prospects, running card sorting exercises and paper prototype tests.

2. Evaluative Testing (Viability and Usability)
   Week 3: Launch a pre-sale campaign offering early-bird pricing, ensuring there’s enough demand to justify further investment.

   Weeks 3-4: Run more detailed prototype testing to gather feedback on core functionality.

   Week 6: Launch MVP with analytics built-in.

3. Verification Testing (Post-Launch Monitoring)
   Week 8 and Beyond: Continue improving the product and analyse usage metrics (e.g., daily active users, task completion rates).
   Collect ongoing user feedback through NPS surveys and iterative usability tests for feature refinement.

Conclusion

Design experiments are essential for reducing uncertainty and making informed decisions throughout the product lifecycle. Exploratory tests help refine raw ideas, evaluative tests validate evolving designs, and verification tests ensure long-term success.

By aligning test complexity with your confidence levels, you can create an efficient and systematic approach to product development. Whether you’re crafting landing pages, running Wizard of Oz tests, or analysing post-launch metrics, choosing the right tests at the right time can make all the difference in building products that users love.