Learning About Learning: Lessons about early customer feedback via interviews and focus groups

by Genevieve Sheehan, Luke Langford, Allison Greenwell

“Great. So why haven’t you build it?” Maybe Eric Ries’s reaction shouldn’t have surprised us. But we felt thrown. Hey, we’d been progressing methodically through product development. We had understood user needs, structured mock-ups for our proposed user interface, and tested those ideas with a range of users. We now knew the product should be built – we just hadn’t done it yet.

The previous weeks hadn’t been a waste, though. We’d moved slower than Eric-Ries-warp-speed. But in the process, we determined the type of product we should build, built a belief that people would use it – and gained some learnings about customer feedback and user testing.

Being the target user makes everything faster and easier – so long as you recognize and test your biases. Product developers are often criticized for making a product they think people want, or a product they themselves want, without listening to potential customers. But being the target user can simplify the process of understanding generalized user needs. We found that we had already experienced many of the “pain points” that users described, and we were able to probe for those and other potential issues with the current system, as well as quickly understand the problems other users were having. We consciously remained open to other points of view: we debriefed ourselves, then had a series of open-ended conversations about others’ use patterns where we discovered that not everyone was exactly like us. But we were able to move faster through the upfront understanding of the current problem, and we already had a handle on the basic ways users today would seek to use a new product.

Being the target user also makes it easier to find other potential customers – because you already know them, and they’re pretty similar to you! We had the advantage of a friendly, captive group of HBS MBAs all around us. They already trusted us (because we were like them) and were willing to give us their time and share experiences and opinions. Since they knew we’d felt their pain, too, they believed that we had a vested interest in changing the system. We actively recruited relatively random students, so that the feedback wouldn’t be too like our own – tech-focused future product managers who critique user interfaces for breakfast – but we had the userbase. We didn’t face the problem of finding, recruiting, and building trust with an entirely different set of users.

Structure interviews and focus groups to provide actionable answers. Focus group best practices:

  • Focus on “why.” It’s not interesting that a participant would not use your product; it’s much more illuminating that he wouldn’t use it because it lacks a key feature or because he doesn’t like a certain component. Similarly, participants who say that they would use your product are less helpful than those who would use it because it mirrors their thought process, because it would save them time, or because they like the interface. 
  • Structure focus groups around a series of yes/no questions (Did you do X? Would you use Y?) and force participants to choose each time. Then probe their rationales. Returning to the yes/no question each time allows the moderator to push towards “why.” 
  • Collect written feedback from participants; have them mark it down as you go along. This lets you “see” participants’ thoughts, especially those who contribute less freely to the group discussion. 
  • Use a moderator and a note-taker. The moderator should be free to react to and guide the discussion. Plus, two listeners in the room will generate better understanding of the range of users’ reactions. 
  • Forced ranking and points allocation elicit extra, useful information. Most focus groups result in participants saying “yes, yes, yes.” To know how to prioritize, ask participants to allocate 100 points among the proposed features. The results are not always apparent from the discussion. 

But know focus groups won’t make the business decision for you. Focus groups only prove so much. Participants have an affirmation bias – particularly since they aren’t being asked to actually adopt the product – and will frequently say that they like everything. Some structure will help you sort through this. But regardless of how positive the focus group results are, they cannot force you to move to the next phase of product development, nor can they assure that that next phase will be the right way – even if every participant loves the mock-up or minimum viable product you’ve created.

There’s another aspect of using focus groups to inform business decisions: focusing requires you to be the lens. Not every lens is the same. Therefore, multiple observers may walk away from focus groups having heard or learned very different things. In limited form, this can be helpful – your colleagues supplement your findings and you build a richer picture of user reactions. But in extremis, it may be more problematic. If you and your cofounder believe very different things about the product as a result of the focus group feedback you heard, who is right? Do you want to work with people who hear very different things from the same conversation, or does this phenomenon just problematize the single-minded focus you need to build a product?

It’s easy to “over focus” – when is enough information enough? (Or: Don’t get caught in Eric Ries’s trap). So we’re back to today. Eric Ries was pushing us to build a minimum viable product; we’d just completed a promising set of focus groups with positive reactions; and the commonest question when we shared our findings with first-year students was “Can I get the prototype?” Did we totally miss the Lean Startup lesson of build-to-learn?

Not so fast. A rough prototype is our next step. But we had only just collected enough user feedback to be sure we would build the right prototype at all. Our n=20 focus groups allowed us to articulate use cases and understand the in-demand features we’d envisioned for our potential product. And we had enough information to prioritize the build according to what features users said they wanted most. The problem is: once we had just enough information to make a decision, we could see that we might have made that decision with slightly less information. But until then we couldn’t be sure. Once you hit that feedback wall – where you aren’t going to change your mind or product direction based on an incremental input – it’s time to stop asking and start doing.

When we do build a product, we’re ready for reactions to differ a lot from what we heard on mock-ups. Seeing a piece of paper is very different from using a tool. And we’ll need multiple iterations of testing and refining: what students will forgive in a prototyped student contribution, they may not accept from the administration. So we’ll keep testing. And keep asking why. And finally – with user feedback – we hope to have a product that solves a real problem.


Popular posts from this blog

Quiz Time 129

TCS IT Wiz 2013 Bhubaneswar Prelims

The 5 hour start-up: BrownBagBrain