May 16, 2009

Designing a Black Box (Part 5)

This is the fifth and final post in a series about data mining games to discover why players are quitting.
Read Part 1 here
Read Part 2 here
Read Part 3 here
Read Part 4 here

Which players matter more?

After working through the previous 4 steps, we should have a list of some very specific player demographics, based on both the type of players that are unhappy and the reasons they are unhappy.

For example, here are some hypothetical examples of groups that our unhappy players could fall into:
  • Social players who find it hard to find a casual guild
  • Expert raiders who are out of content
  • Beginners who aren't prepared well enough by the tutorial
  • PvP players who can't find anybody to fight
  • PvE players who are sick of being killed
  • Latecomers who can't play with their high level friends
All of these problems can be solved, but not all of them can be solved for every game, and certainly not all at once. We'll have to decide which of these player/problem combinations are the most in need of our limited development time, and also in which cases our efforts will be the most effective.

This can be a very difficult question to answer. Some games fail to define their key audience well, which means the argument about which leaks to plug will likely be very hard to resolve. Other games get too greedy, reaching out to a new demographic at the cost of angering their loyal fanbase. Star Wars Galaxies is a classic example of a game that tried to reach a new demographic and failed, losing many of its old players without gaining many new ones.

There are a few obvious factors in prioritizing which problems to address:
  • Which player types are the most important to retain?
  • Which problems are causing the most overall players to leave?
  • Which problems are causing the most of the key audience to leave?
  • How early early in the game are players encountering those problems?
  • How easily/cheaply/effectively can those problems be fixed?
Most designers probably think of these factors, but there are a couple very important factor that most people neglect:
  • How much of the damage has already been done?
  • How many of these players are we acquiring?
If our game launched 5 months ago with a horrible PvP experience, that fact has probably been announced loudly by reviews, fan sites, and angry ex-players. If we then decide to spend tons of time and money to improve the PvP experience, it's unlikely any players who enjoy PvP will be left to notice. We can try to entice them back, but perhaps it's been decided for us that our game is no longer a PvP game, even if we intended it to be. If PvP was all our game had going for it, we're probably already doomed, but if there are another group of players who are enjoying the game, it's probably wiser to improve what our game is already doing somewhat well.

Similarly, if only a very small percentage of our new players tend to be PvP players, and they all quit, fixing PvP may be addressing a demand that is not actually there. In order to be able to retain players, we first have to acquire them. Happy Farming Town Online will never draw many PvP players in the first place, so even 100% retention of PvP players won't ever really do us very much good.

Once a game is launched, its players define it. If we thought we were making a game for one group of people, and that group snubs it but another group likes it, we'd better get to work making the game suit the players who actually care about it. We can worry about getting other players once we're sure that we're taking care of the ones we've got.

Reach out to the right players

Once we've decided to fix a problem that a particular group of players is having, we'll have to actually design and implement a fix to that problem, not to mention make sure the problem is actually fixed. Once that's done, we just have to let those players know that those problems are fixed.

Since we used our black box to divide players into groups and problems in the first place, it should be possible to write in the functionality to send mass emails to groups of players, without even having to know which players fall into each group. By using a data driven system in this way, we can help specific players without even having to know which specific players those are. This should both stop players and developers from abusing the system, as well as avoid the majority of privacy concerns.

In a subscription-based game, communicating with players by email can be a dangerous thing. I heard a funny story from a friend about how a company sent a big update email out to all its players, many of whom had apparently forgotten they were still paying for it. Upon receiving the update email, a huge chunk of lapsed players promptly unsubscribed.

In my opinion this shouldn't be a deterrent. Hopefully a good black box system would prevent so many players from entering that limbo state to begin with, and even the data of which players were driven to quit would help us make the game better in the long run.

Once we've rolled out a change and an announcement tailored to a specific player group, we'll probably also want to track data for how well it worked. How many players came back? How long did they stay? If they've lapsed again is it a new problem or the same problem?

Don't rule out temporary fixes!

Fixing the game's problems is our ultimate goal, but sometimes is makes sense to temporarily compensate for those problem until they can be fixed.

If we find that the leveling curve is too slow in a particular area, or a level range seems to be a complete disaster, we can give every player in that zone or level range some bonus XP or free levels to get them past the rough patches. If a group of players who tried out PvP servers is having a miserable time, we can offer them free transfers to PvE servers, or give them some free PvP gear tokens to get them started off on the right foot.

In drastic cases, it may even be wise to disable a feature entirely rather than let it drive players away. We can also offer some playtime free or at a discount. If there are some problems that will take a long time to fix, even just announcing our intention to do so is something players really appreciate.

Advanced uses of black boxes

Even once we manage to plug all of our big player leaks, there are plenty of tasks that this type of data mining can be useful for:
  • How can we turn all the average players into happy players? This will follow a very similar process as making unhappy players average.
  • Which kind of marketing is successful with different kinds of players? We can experiment with different kinds of marketing in different time periods, and see how that affects our player acquisition and retention.
  • What is the best price point for our game or subscription? We can test out different prices and fees on subgroups of players to find out where the sweet spot is. It's possible that we'd keep many more players, and make much more money, by making our game cheaper.


All this may sound like a lot of extra work when we could just post a survey on our forums or make people fill out a form when they unsubscribe from the game. It really is worth it, though. Systematically identifying problems is a huge advantage over interviewing players, because players are not game designers and often don't consciously recognize the true causes of fun or frustration. For that matter, even game designers often bark up the wrong tree and rationalize or misinterpret bugs.

Targeting announcements and incentives to a particular group is incredibly powerful when done correctly, because it shows players that we've responded to their specific needs, even if they were unspoken. To players, this feels like we're reading their minds, or that we have the same priorities they do.

This sort of trust is very powerful when it comes to making longterm fans of a game or a company, and I believe it's more integral to success than most developers will ever realize.


Nels Anderson said...

Systematically identifying problems is a huge advantage over interviewing players, because players are not game designers and often don't consciously recognize the true causes of fun or frustration.Another benefit is that systematic analysis can give a far more accurate picture of how the players in aggregate are playing the game.

The folks more likely to visit forums, fill out surveys, etc. are rarely representative of the average player.

Some games get into a lot of trouble trying to satisfy their most vocal players, rather than solving the problems average players have. Having access to a more accurate data can help prevent "fixing" a problem for a small amount of people that ends up creating a problem for a larger amount of people.

Mike Darga said...

Excellent point Nels, I think a black box can be very useful for avoiding that type of overzealousness.

Chris said...

I'd never really thought much of a black box being utilized before this series of articles. It's actually been a fascinating read and something that will most definitely be in the back of my head in the future.

Mike Darga said...

Thanks Chris, it's good to get the ideas out there so they can go lumber around and maybe even make themselves useful.