Smoke (and other) tests: smoke in your face? Pitfalls of lean start-up testing
by Lauren Miller The lean start-up approach has the advantage of using genuine customer feedback as a means of improving the entrepreneur’s product specifically for those perceived as the niche consumer, but numerous stories from entrepreneurs and case protagonists about acting on false positives and negatives has caused me to wonder: can smoke and other lean start-up tests just end up being smoke in the entrepreneur’s face? ” I question whether an entrepreneur should take the results of his or her tests as “gospel” since pitfalls in the experiment design and interpretation of results can be numerous. I will focus on a few of these potential pitfalls. Focus groups Steven Carpenter shared in the Cake Financial case that he has “learned that focus groups… can be unreliable because people can’t always say what they want until they can actually see it and play around with it.” This highlights the fact that customers often can’t or won’t tell you what they want or need and are often li...