Knowing Who You Are Gathering Insights From

By Aileen Wu

One of my favorite activities related to product development is gathering consumer insights. I enjoy not only the process of observing people and interpreting their feelings and motivations, but also sitting at the front lines of where your creation meets the test of the outside world. Throughout Launching Tech Ventures and entrepreneurial endeavors, I have seen and used many techniques to gather insights from smoke tests to surveys to usability tests. In order to get reliable and valid customer data, it is important to think about not only what test to use, but also who to perform the tests on.

Dharmesh Shah highlights the dangers of acting on survey data in a great article called “Early Evidence Is Often Too Early And Not Really Evidence.” He warns that even though 80% of your survey respondents might care tremendously about keeping an extra feature of your product, they may not be represent of your average user. Early on, you may have actually only gathered the responses of the few users of your entire user population who use this feature regularly.  This example brings to light the importance of collecting data from a randomized group of users.

When it comes to usability tests, it can be tricky to determine what type of user to watch. I went to great talk by UI designer, Juhan Sohin, who claimed it was important to design for the expert user, not the beginner. He cited the example of the wheel button on the first iPods. First time users fumbled when first introduced to the wheel because it was like nothing they had ever seen on a consumer electronics device before. However, after they discovered how the wheel worked, they embraced its clever design and ease of use.

When determining who to observe in usability tests, it is important to think about what the dominant use cases will be. If you are developing a product that users will only interact with occasionally, such as Comcast’s bill payer, it may be more effective to optimize website design for a beginner user. However, if you are developing a product like RentJuice that your target population, in this case real estate brokers, might be using for several hours each day, it may be more advantageous to observe an expert user (someone who has spent several hours using your website already).   

At the end of the day, it is important for you to take extra steps to know exactly who you are gathering insights from. You can use qualifying questions as well as randomization to make sure you are pulling data from a relatively unbiased group of users. Furthermore, it is important to think what type of user you want to help you make decisions to drive product design forward.