April 22, 2009

Designing a Black Box (Part 3)

This is the third post in a series about data mining games to discover why players are quitting.
Read Part 1 here
Read Part 2 here

It's never too soon to find problems

If we're developing a black box for a game that's been out for a year, and has a stockpile of data already, it's pretty obvious we can just jump right in and start analyzing. People seem a little more hesitant to start gathering data on a game that hasn't even been released yet, but that's exactly what we should be doing. The great thing about basing our black box on the amount of time since players have logged out is that we don't need any paying customers to start getting data.

This system will start working as soon as we start letting normal players play our game under normal circumstances, which probably happens in alpha or beta. To clarify, I consider "normal players" to mean non-developers who are hopefully from our intended audience. I consider "normal circumstances" to mean that players can play as much as they want, as often as they want, with no level caps or dev commands.

Once these conditions are met, we can start looking at the amount of time between play sessions right away, and start finding our largest player leaks before the game has even launched. If a player doesn't want to play the game for free, we can bet they won't pay to do so.

Once the game has gone live, and a billing cycle or two has elapsed, we can begin to examine our assumptions about the correlation between time since logout and actual cancelled subscriptions. Perhaps we'll see that players tend to log in to the game as infrequently as once every two weeks and still remain active in the game for a long time, or we might see that an absence of even four days signifies a player who is thinking of quitting. In the short term it's ok to just take our best guess.

The more data, the merrier

We can refine these numbers over time, and begin to build sophisticated profiles and criteria based on level of involvement, character advancement, and even playstyle. We might find that danger point for all players might be an average of a week without logging in, but more likely the number will be different in different contexts.

Perhaps any player who has played less than 30 hours or is below level 20 is unlikely to return if they don't play every day. Perhaps max level players in casual guilds can log in once a week for raid night and once a week for a crafting session and still remain enthusiastic about the game.

We can become more and more ingenious with our detective work over time, tracking demographics, guild dynamics, important players and guilds who have a kind of social gravity among other players. We can compensate for local holidays, cross reference the patch notes of our game and of our competition, and even analyse crazy things such as how the color palette of different zones impacts the duration of play sessions. Let's not go crazy quite yet, though.

Solve the biggest problems first

Before we get to any finicky analysis, demographics, playstyle, and all kinds of other subtleties, let's just make sure our game will be around long enough for us to do so. When solving problems in game design (or any discipline really), there are always some "low-hanging fruit" problems which can be diagnosed and corrected much more easily than others.
First of all, what percentage of our total playerbase is unhappy? If it's only 5 percent, then we're doing a pretty good job already, and will probably just need a few tweaks. If it's 40 percent, things aren't looking good. If the number is growing every month, we're making things worse or our game is out of content. If the number is going down, our endgame is more enjoyable than our leveling, or our fixes are making people happier.

Let's take all of our available data, from every player who has ever quit or shown signs of quitting, and graph it. Let's look at the broad generalities. For each unhappy player, let's look at some statistics about whichever character they played the most in the last month:
  • How many hours the player has logged on that character
  • What level the character is
  • What class the character is
  • How often the character dies
  • What zone each death was in
Wait a minute, we've got this beautiful mountain of data, and we're standing back a mile away to see how it smells? Where's the digging and sifting and rolling in it?!
Don't worry, we'll get there. Big problems before small problems. Just looking at those four data sets, and combinations thereof, can give us a quite a bit of information about our most obvious problems. It will also suggest exactly where we should start combing through data for more clues.

How bad is bad?

Using wow as an example, let's imagine some potential data points and what they would tell us. If the unhappy players seem to be distributed evenly in the level range, that's more or less a good sign, and probably indicates that our game just isn't cut out for some people. If lots of players play to the level cap and then quit, we're likely in need of some better endgame content, but at least the early game is fun and we know where to focus our efforts.
If there's a big spike in unhappy players at level 10, this is a medium to terrible problem. Players might be feeling lost after the starter zone, the tutorial might not be preparing them well for the world, or maybe there is a problem with high level players griefing lowbies. These problems are all solvable and pretty small. However, players may just be deciding that the game is horrible early on, and fixing that level 10 spike might just turn it into a level 15 spike. We'll be able to get some idea by seeing what percent of our players make it to max level, but we'll also just have to wait and see.

Hopefully any low level problems are just potholes, and fixing them will pave the way for players to enjoy the game up to the level cap, but there's no way to know for sure until we start fixing things. Unhappy players at a low level is a very dangerous problem, and is an even bigger emergency than unhappy high level players, because new players will be entering the game, growing frustrated, and quitting before you can even address the problem. This is one reason that it's hugely important to get the jump on solving these problems before the game has gone live.
What else can we tell from a quick glance at these data? Let's take a look at how often players die, combined with some of the other information:
  • If players are dying very often in every situation, the player health and damage table may be tuned too low.
  • If players are dying very often in a certain class, that class may be a bit in need of a buff.
  • If players are dying very often at a certain low level, the tutorial may need some work.
  • If players are dying very often at a certain high level, the itemization for that level range may be too sparse or ineffective.
  • If players are dying very often at a certain low level in a certain low level zone, that zone is too hard or high level players are griefing them
  • If players are dying very often at a certain mid level in a certain mid level zone, that zone may be too hard, a focus of PvP, or contain a nasty roving boss mob.
  • If players are dying very often at a certain low level in a certain zone that is higher level than they are, the world may be laid out badly or in a confusing manner, so that low level players can't tell where they're supposed to go.
The ability to visualize and cross reference data can tell you quite a bit about your game that you might not have realized before, or at least give you some leads to start working on. Take a moment to glance over the lists in Part 2 again, and see if you can't pick out a few more sets of data that would be useful to examine.

Next, we'll begin building profiles of different kinds of players, and looking at how different patterns may emerge for each different player type.
Continue to Part 4


Olive Tree Guitar Ensemble said...

Hi, it's a very great blog.
I could tell how much efforts you've taken on it.
Keep doing!

Chris said...

I've been reading this blog for the past several months. Just wanted to say that I absolutely love it and have a live bookmark in my browser at work. Keep updating...this is easily a hidden game.

Mike Darga said...

Thanks very much you two, I'm glad you like it. Please help me spread the word!

Nels Anderson said...

Excellent Mike. My next blog post, which you may have noticed me hinting at, is going to be about the importance of data collection to verify design hypothesis. I imagine you won't be opposed to me linking to this series as an example of how to do it right? ;)

Mike Darga said...

Link away good sir. I look forward to reading it. I've got an older post that touches on that subject a bit as well, if you're interested:

Think More Like a Scientist

Ali I said...

Lots of great points about Data and how to use it properly. I've said it before, but this same topic helps in a lot more than games.

Data driven is great. Let the user tell you everything instead of you telling the user that is should be this way and thinking everything is honky dorry.

Mike Darga said...

Exactly, but with the added bonus that you don't have to rely on the user to understand or accurately convey how they feel and why.