The many faces of feedback

We’ve seen a lot of mechanisms for getting customer feedback, e.g., focus groups, one-on-one interviews, usability tests, surveys, website feedback forms, website blogs, smoke tests, etc. What have you learned about when different mechanisms are most reliable?

By Cara Hoy

Like other aspects of the lean startup methodology, effectively gathering customer feedback is a game of tradeoffs. The way that I see it, there are four key dimensions to customer feedback that need to be traded off; breadth vs depth, and quantitative vs. qualitative learnings.  Different mechanisms serve different purposes, but it is important to maintain a balance across these dimensions as they are highly complementary. In determining what the most appropriate mechanism might be for an occasion, it all begins with a simple question –what exactly do you want to know about the customer? The answer is dictated by what you already know about the customer, and where you are in the life cycle of understanding, developing, and delivering a solution with a compelling customer value proposition.

One category of answers to the golden question is highly ambiguous.  It could be that we don’t know who we are targeting yet, or that we don’t have a strong sense of what the problem is exactly that we are trying to solve.  We need the customer to show or tell us what we should be asking; it is a process of discovery and exploration.  This seems to be most common in two situations –when we are in the early stages of a startup, or a completely novel green field market is being explored.  Here, depth is more valuable than breadth, and there is a lot of value that can be derived from qualitative, highly interactive feedback.  Open ended interactions allow us to explore different topics, then hone in on areas we feel are more important in real-time. In situations like this, some of the more effective feedback mechanisms include one on one interviews, and focus groups. Website feedback forms where customers can respond in freeform also fit into this category, as do usability tests in the early stages of product development. 

Another category of answers is highly concrete.  This seems to occur most often when a startup is further along in its development and refining a product or go to market strategy, or when our entrepreneurial idea is a better version of a well understood solution.  When there is a highly specific hypothesis, with an obvious potential range of answers, then quantitative feedback via mechanisms like multiple choice surveys can be more reliable.  There is less concern that an unanticipated potential answer is being overlooked or omitted.

Another factor to consider is how different feedback mechanisms naturally attract different groups of users.  Oftentimes it seems that early adopters are the most active feedback providers; as passionate users/potential users, they will respond to passive outreach features like feedback forms, and will seek out website blogs.  They will be more likely to volunteer to invest in time-consuming feedback activities like usability tests and interviews.  On the other hand, mainstream adopters are more difficult to engage with, as they oftentimes simply want to consume in one direction.  To collect feedback from these users, you must either make feedback provision extremely quick and easy (eg. a simple short survey), or expend resources to incent them to participate in providing deeper feedback.