Lean Principles in Video Games – Can We Test for Fun?

By Jonathan Lai

Over the last two years, HBS has relentlessly preached the virtues of lean testing in many different contexts.  Not sure what product to build?  No sweat – just build a bunch of simple prototypes (use a mechanical turk if needed), put them in front of your customers, and see what they get excited about.

Throughout it all, I’ve often thought back to my own industry of choice – traditional video games – and wondered if lean testing techniques would work there as well.  Traditional, big-budget video game design bears more resemblance to Hollywood movie production than to software development.  The games industry revolves around a small cadre of well-known studios and celebrity designers, who use their intuition and gut feeling for “what gamers want” to design blockbuster games, many of which cost over $100 million in budget and employ hundreds of people for several years.  While there may be some pre-release screenings and marketing trailers, most blockbuster games are produced under a veil of absolute secrecy.  Unless you are a critic or industry pundit, release day is the first time that most fans will be able to get their hands on a game.  Indeed, the secrecy and hype surrounding games is a core part of the business model – publishers often sell a preordered “collector’s edition” of a game for a huge mark-up, game websites prominently feature countdown timers that fans follow with religious zeal, and many retail outlets such as GameStop host launch parties rivaling the fervor of New Year’s Eve.

In such an intuition-driven development environment that places a premium on secrecy and hype, can lean testing techniques play a role?  It would certainly appear laughable and perhaps even offensive to suggest that celebrity game designers such as Peter Molyneux (Fable, Dungeon Keeper) or Will Wright (The Sims, SimCity) test their game concepts via a minimally viable product in small focus groups.  Yet, perhaps this melding of creative intuition and lean testing is just what the games industry needs to solve its problem of being a risky, hits-driven business.

The problem is this – even the classic recipe for success: “great teams with big ideas” fails spectacularly in gaming.  Take the sad history of the recent MMOG Tabula Rasa.  Tabula Rasa was widely anticipated as the World of Warcraft (WoW) killer – the studio consisted of a crack team of designers from Ultima Online (one of the first and longest-running MMORPGs) and was led by Richard Garriot, a well-known celebrity and author of the original 1980 Ultima RPG that spawned a 20-year franchise that still lives today.  In addition, the target genre was science-fiction – a rich and massive segment that was expected to penetrate a largely untapped market of gamers.  Throw in a first-person-shooter mode for twitch-gamers, cutting-edge graphics that took advantage of new computing hardware, a massive $100 million budget, and it seemed you had as close to a “sure thing” as possible.

When Tabula Rasa launched in Nov 2007, it received rave reviews.  GameSpy rated it 4/5, Eurogamer 8/10, and GameSpot 7.5/10.  Subscriber levels climbed from 50k to 150k…  Everything seemed great.  Then mysteriously, as if a spigot had been turned off, things started unraveling.  Tabula Rasa lost subscribers left and right, and after several quarters of attrition, officially shut down operations in Feb 2009.  Richard Garriot has since moved on to space flight, among other things.

What can we do in light of flops from even the most well-intentioned and experienced teams?  Lean testing would suggest that perhaps we can build light-weight demos before sinking vast sums of money into full-blown development.  And even if this is not possible (a game being developed under secrecy), we can still apply another powerful lean tool – the agile development process.  Don’t try to waterfall your way to a perfectly polished game.  Just get it to where it’s good enough.  Release early, release often.  Listen carefully to users, incorporate their feedback into the next sprint, and iterate to hell.

Yet, faced with this very obvious solution, game designers often balk and object – “you can’t test for fun.”  This is the crux of the issue – you can get monetization, UX design, and a lot of other important things right through testing, but can you make a product fun through tests?  The best game designers would claim no, and I think they have a point.  The metrics-driven approach of the newest generation of social and mobile game developers have often back-fired.  Zynga, considered to be the best at lean testing, tracking, and tweaking of its games, has struggled to hold on to its users over the last few years.  Zynga started Jan 2013 with 45.6M daily active users, down 16% from 54M DAU in Dec 2011.

As of now, I don’t know the answer to this question.  The truth might lie somewhere in the grey area between total free-wheeling creative intuition and robotic, metrics-driven, lean testing.  Then again, the best movies and games are the ones that touch us at the core – emotionally and viscerally.  Perhaps we simply can’t test for fun, because in doing so, we’d be trying to test for what it means to be human.