Entrepreneurs: Test Sequencing and Design Matters More Than You Think

by Mike Joslin and Brady Broadbent

Today’s entrepreneurs are increasingly using hypothesis-driven lean testing methodologies to launch early, learn fast, and iterate quickly, and as a result an entire industry is rapidly developing around the movement. New web-based products and services such as Unbounce.com and Usertesting.com continue to pop up and, for both competitive and technological reasons, the costs of conducting such tests are falling dramatically. While this has provided entrepreneurs with an amazing new opportunity, it has also created new problems.

Given negligible costs and significantly lower risks, many entrepreneurs are now falling into what we consider the “testing trap.” In a rush to test and learn quickly, entrepreneurs are jumping immediately into testing before putting enough thought into which tests they will conduct and in which order they will conduct them. By not following a logical testing sequence, entrepreneurs are left with random statistics and feedback that do not actually help refine and improve the business. In an effort to help entrepreneurs avoid this “testing trap,” we recommend using the following approach:

1. Use qualitative testing to help you develop an initial set of hypotheses

It’s tough to design effective tests when you don’t know the right questions to ask. The first step for every entrepreneur is to get out and talk to (or observe) potential consumers to better understand existing behaviors, identify potential pain points, and evaluate current solutions to those pains. In this phase, entrepreneurs should avoid discussing the business concept until the very end, if at all.

This process helps entrepreneurs unearth previously unforeseen needs, pains, and potential competitors and helps identify the biggest risks or benefits of the business concept. After completing this phase, we recommend creating a list of all hypotheses (Excel works great) that must prove true for the business to be successful.

Appropriate Tests: Informational Interviews, Focus Groups, Ethnographic Research

Research Focus: Current consumer behavior, pain points, decision making processes, existing solutions/competitors, etc.

Output: Exhaustive list of all hypotheses that must be proven for the business to be successful.

2. Use quantitative testing to determine which hypotheses to focus on, and in what order

Many entrepreneurs make the mistake of using quantitative research (typically statistically insignificant given a low number of respondents) as validation of demand for a new business. Instead, quantitative research should be used to identify critical hypotheses that require further testing. If certain hypotheses prove to be unimportant or overwhelmingly positive, it is likely not worth your time or money to conduct further testing. But if you find that certain important hypotheses prove worrisome, entrepreneurs should create further quantitative or field tests prior to investing in building out the product or service. For example, we recently fielded a survey on a new professional networking tool and found that while there was a willingness of users to sign up, there was a significant reluctance to share one’s professional contacts with others following sign up. Given this is a very important aspect of the business model, we are now designing an ad-hoc field test to measure how often a user actually shares their contacts.

After conducting quantitative research, entrepreneurs should rank each hypothesis on importance to the business model (e.g., 1-5) and range of uncertainty (e.g., high to low). Next, field tests should be designed to address the most important, most uncertain hypotheses and sequenced in that order.

Appropriate Tests: Surveys, Conjoint Analysis, Concept Testing, etc.

Research Focus: Likelihood to [x or y], feature importance, consumer tradeoffs, etc.

Output: Ranked list of hypotheses based on importance and uncertainty. Prioritized schedule of field tests for most important, uncertain hypotheses.

3. Use field testing to prove or disprove the most important, uncertain hypotheses

In our opinion, “lean” field tests are the only way to truly validate or disprove your business model. However, many entrepreneurs make the mistake that “real-world” testing requires building out a working prototype and attracting early adopters. In some cases this may be true, but for most the ideal way to test critical hypotheses is to test a smaller amount of related hypotheses together using an ad-hoc test. For example, our team developed a “good enough” landing page with sign up functionality to test demand-related hypotheses and will be leveraging LinkedIn soon to test sharing-related hypotheses. By focusing on only a few hypotheses at time through ad-hoc tests, tests can be designed in a manner that isolates variables and shows a clearer cause and effect.

Appropriate Tests: Ad hoc tests as well as Minimum Viable Product (MVP) tests such as landing page, usability, and other tests.

Research Focus: Actual user behavior, quantitative business metrics (e.g., conversion rate), customer feedback.

Output: List of proven/disproven hypotheses with specific actions to address disproven hypotheses (e.g., pivot to new business model/close up shop) and modify concept to exploit positive aspects.

By putting in a bit more effort at the beginning using our recommended approach, we believe entrepreneurs can successfully avoid the “testing trap.” In the long term, entrepreneurs will save both time and money while also producing more useful, actionable test results that will help them refine their businesses.