Smoke (and other) tests: smoke in your face? Pitfalls of lean start-up testing

by Lauren Miller

The lean start-up approach has the advantage of using genuine customer feedback as a means of improving the entrepreneur’s product specifically for those perceived as the niche consumer, but numerous stories from entrepreneurs and case protagonists about acting on false positives and negatives has caused me to wonder: can smoke and other lean start-up tests just end up being smoke in the entrepreneur’s face?” I question whether an entrepreneur should take the results of his or her tests as “gospel” since pitfalls in the experiment design and interpretation of results can be numerous. I will focus on a few of these potential pitfalls.

Focus groups
Steven Carpenter shared in the Cake Financial case that he has “learned that focus groups… can be unreliable because people can’t always say what they want until they can actually see it and play around with it.” This highlights the fact that customers often can’t or won’t tell you what they want or need and are often limited in what they can even think to ask for. It further underscores the important lessons revealed in Dropbox and Aardvark. Be careful in design of focus groups to utilize the most valuable feedback, as it is not all created equal. Also, make sure to listen to your customers but don’t simply obey them.

Usability tests
Martin Kessner’s study reveals the necessity of running usability tests iteratively. In his study, six usability teams could not find a single common usability problem when independently testing the same product, and no team found 50% or more of the total problems found by all. This should be a sign of caution to the entrepreneur. Though there are a plethora of studies and start-up case examples that document the improvement achieved from usability testing, Kessner’s work could show that in testing products or usability, the iterations make the test more reliable and decrease the likelihood of wide variations in responses. My takeaway: the repetition of usability tests allows an entrepreneur to better qualify the feedback he or she receives and in turn use it as a better determinant of when and how to pivot in hopes of achieving product-market fit.

Feature lists
I am working on an education software start-up, and we currently have a survey on our site that asks teachers which features they would like. Results have been great, as teachers seem to want all of the features we plan to create. Nevertheless, I have recently begun to question the reliability of their responses. I hypothesize that if we added fifteen more features, teachers would likely indicate that they want all of those as well, but the famous Columbia University marketing study implies that products with too many features often overwhelm customers. So how much should an entrepreneur trust customers when they say they want everything? Cindy Alvarez, formerly of KISSmetrics, says that customers want everything because “wanting” is free. Testing how much value features provide by adding a cost to additional features may yield more accurate results. If entrepreneurship is an art and a science, utilizing results of tests like this is the science and the decision of whether to follow them and how is the art.

I advocate that in the lean start-up model, focus should be placed as highly on test design and accuracy in feedback interpretation as it is on running tests. This way, entrepreneurs will avoid smoke-hazed and unreliable results that end up leading to pivots that could destroy their ability to connect with and address the needs of their potential consumer base. However, I’m interested in your thoughts. What other reliability traps have you seen entrepreneurs fall into when using various mechanisms to get customer feedback?