Just Ship It! The Limitations of Guessing and Testing

By Kelvin Kwong

With the advancement of internet development over the last decade, I believe that product management has increasingly shifted towards a ‘Bias to Ship’ mentality. This approach has proliferated as the cost to tweak and re-ship has dropped precipitously. I experienced this firsthand this past summer as a PM at a large tech company when I was instructed to ship something that was only 80% done. And that absolutely made sense in that context. The amount that I learned in the next two days as several hundred thousand users interacted with my product revealed many more issues than I could have ever identified staring at it for weeks.

However, I’ve been wondering whether this is actually the optimal way to build and if many product managers out there today have been trained to focus too much on this method of testing. To remember why we guess and test, I thought back to my third grade math class when I was first introduced to this concept. If a math problem was too complex to solve, you could just try plugging in different numbers until your result was closer and closer to the right answer. But that method falls apart when we later learn about quadratic equations and there might be more than one answer that works. And I realized that that is the exact issue. Guessing and testing will get us to a local maximum but not necessarily the absolute maximum.

And so when we hear the so creatively-inclined tell us that “testing kills design,” I see the underlying logic. By jumping in and immediately testing ideas, we will be drawn towards one of the local maxima and possibly be doomed for mediocrity. I once heard Harry West, CEO of Continuum, say, “People generally don’t like new things.” In other words, people struggle with change. And so to prioritize ideas or features by engagement and traffic is really just a proxy to measure which of the options is the most similar to what is out there today. If your hope is to improve the world incrementally, then I think you found your method. But if you hope to create something that might alter human behavior or the way that we see the world, then you need to broaden your methodology to start searching for absolute maxima.

This highlights the importance of starting with conviction on a consumer insight – in other words, thoughtfully framing the pain point to be resolved. While the idea of ethnographic research has shown up in countless blogs recently, my view is that it is still greatly under-utilized. In our mainstream tech blogs today, we seem to continue to romanticize the product side of design and not the human side of design. Instead of having masses flock towards Code Academy to learn Ruby, I believe that budding PM’s should be studying Kahneman’s work in social psychology and Thaler’s work in behavioral economics. Understanding how we as humans perceive things and how we consume the world is the work on the front-end that I believe is often far-overlooked.

Certainly, running numerous tests on your idea and running them quickly and cheaply will yield drastic improvements. All I am saying is that if you started on the wrong part of the curve to begin with, your best result may just be a local maximum. A much better result might have been nearby. It was just missed because of your frantic search for any type of maximum at all.