The first session of GDC China's Social Games Summit in Shanghai was presented by Andrew Mo, director of product for Playfish's Beijing studio. His topic, intuition versus metrics, taps right into a hot area of debate in social game development.
As Mo noted, social games is "really is a marriage between traditional games and web development", and "there are some complaints that the social game space is more and more becoming a numbers-driven game, about optimization, number-crunching, and tweaking, without having the same soul and spirit that game developers are very passionate about."?
Crucially, said Mo, "now that we have grown to a much larger audience... We find that having a grasp of the numbers and understand our customers' behavior is critical to succeed."
However, "one of the biggest reasons for failure is using the wrong tool at the wrong time."
Looking Outside Games: Two Tragic Examples
Mo presented two examples from the non-games business world; before the BP oil spill, the company was highly focused on improving safety metric. "Initiatives [were] underway and money was being spent... but there were clear signs of problems" even before the spill. "Metrics may not have been the best tool to use in this example to make sure you are running a safe but very complex operation."?
On the other hand, Lehman Brothers "was exposed to a lot of risk that the data said it shouldn't be" because the CEO of this company valued intuition, and empowered his management to act on it.
Building a Balanced Culture
When formulating a strategy, said Mo, "the team also matters. Make sure there is a balance between creative and data. In terms of leadership, in [traditional] game development we look for a balance between the producer and game designer," who are both creative-led. Meanwhile, web development focuses on a product manager and a designer -- one led by metrics, the other creative. Social games is the same split, with a producer/designer and a product manager.
And while you should have user testing as an integral part of development, said Mo, "it's really important to separate your user researcher" from the game design team, and only feed in data as required.
The creative versus metrics balance of power is key, he said. "For us, we try to manage that by having the producer and the PM really sitting at the same level and no one person really reporting into the other, and the decisions really being mutually agreed-upon decisions."
Intuition Best Practices
In Mo's view, highly complex problems can't be solved with data; highly simple problems, on the other hand, should be tested.
He recommended the book Blink by Malcolm Gladwell -- which explains "how do you think without thinking."
When you develop in a purely metrics driven fashion, there comes a point where you reach a point where you can't improve. "Whatever moves you make and whatever tweaks you make to the product will see your numbers declining... you would typically optimize yourself to a point where you can't maximize yourself any longer... But if you take off your blinders and look at the big picture you would probably find a better point."
For example, in Playfish's Pet Society, "one of our designers came up with the idea that it's a pets game -- why don't we add fishing in our game? And when they introduced fishing they really saw the DAU really take off."
When it comes to intuition, said Mo, "if you haven't seen or you haven't experienced a lot of the genre you can't lead. Typically, we want to lean toward intuition for complex problems and not details. And for long term planning -- how do we get toward the next big bump in my business." Intuition is useful for "creating something brand new." But when it comes to making intuition-based decisions, "try to back it up with data wherever you can."
On the other hand, intuition can lead you astray -- the book to read here is Predictably Irrational by Dan Ariely, said Mo. It will teach you "why intuition is wrong."
For example, in the social games space, "just that term 'microtransactions' is misleading. When you think about it, most people would think about making an item extremely low cost. But surprisingly if you actually look at the numbers, the price ceiling you can have for items in your game could be very very high."
And how do you determine that ceiling? "The best way to find that out is to have a controlled environment where you can track these things." ?
Good and Bad: Metrics
When it comes to metrics, the good is that "what gets measured usually gets done. If you give your team a weekly sales target... Usually you'll find that week on week they will improve that metric, because that's the thing they're focusing on day-in, day-out."
"Good metrics will drive good behavior. If you're leading with a set of numbers, those will get done," said Mo. "And good metrics will help you make good decisions. They are very good for highly measurable environments with short feedback loops and not so much noise and false data clouding the mix."
However, these can be weaknesses, too. "What gets measured gets done. If your game is just taking off and has a few thousand DAU... if you are telling a team to focus on its sales metrics, that's probably not the way to go. You're giving away a lot of focus and growth behind the title."
Said Mo, bad metrics will actually enforce bad behavior, especially when the metrics are founded on bad data. And in noisy environments... "you'll find yourself spending more time preparing the metrics than taking action" It's also a concern that "metrics age really quickly... It's important to stay on top of these things."
So, the key is that you "align them with your goals -- if you are worried about retention, put DAU rather than sales," said Mo.
It's also important to make sure your metrics are "good enough versus perfect... Keep it simple, a few good ones. If you are tracking too many metrics... You won't know which one to choose or take action on. And don't be afraid to revise, revise, revise."?
"Trust your gut but prove it with your data. Use the right one at the right time, but don't hide behind either," he concluded.
Audience Q&A
The first audience question asked Mo to give another example regarding data, and he provided one from Playfish Beijing's My Empire.
"There was a period in time when we saw some pretty rapid declines in DAU that, for all intents and purposes, we were not fixing with metrics... A lot of numbers were going around," said Mo.
"But one day out of brainstorming, we realized that [since] this game looks like Age of Empires, so why not build in a war feature? We spent some time building this war feature and we took [the game] in this war management direction... It's one of the things we did that helped DAU... And also monetization... Because people were buying soldiers and generals."
Another questioner asked Mo how to weed out bad metrics. Mo seemed to think the most important thing was building a robust system. "Rarely do we [developers in general] place any emphasis that we QA our stats system... We shove it into the back end of development. We are at fault for doing that, but we have only recently stepped up on these efforts," Mo replied.
"Making sure you have the PMs involved early on to know what numbers they want... And make sure you test it early on... Make sure the numbers are reflective of behavior. Most likely there are going to be bugs... tracking your data." ?
And while Playfish has its own proprietary tracking, a developer at a smaller studio asked Mo what he would recommend for companies that can't afford to make that spend. "There are already quite a few rudimentary things you can track using Google Analytics... We typically can't bring our own stats backend into new markets from day one so we use Google Analytics quite robustly until another solution comes along," said Mo.
Source: http://feedproxy.google.com/~r/WorldsInMotion/~3/MoaL-m-DqY0/gdc_china_playfish_on_metrics.php
No comments:
Post a Comment