It's never too soon to find problemsIf we're developing a black box for a game that's been out for a year, and has a stockpile of data already, it's pretty obvious we can just jump right in and start analyzing. People seem a little more hesitant to start gathering data on a game that hasn't even been released yet, but that's exactly what we should be doing. The great thing about basing our black box on the amount of time since players have logged out is that we don't need any paying customers to start getting data.
This system will start working as soon as we start letting normal players play our game under normal circumstances, which probably happens in alpha or beta. To clarify, I consider "normal players" to mean non-developers who are hopefully from our intended audience. I consider "normal circumstances" to mean that players can play as much as they want, as often as they want, with no level caps or dev commands.
Once these conditions are met, we can start looking at the amount of time between play sessions right away, and start finding our largest player leaks before the game has even launched. If a player doesn't want to play the game for free, we can bet they won't pay to do so.
Once the game has gone live, and a billing cycle or two has elapsed, we can begin to examine our assumptions about the correlation between time since logout and actual cancelled subscriptions. Perhaps we'll see that players tend to log in to the game as infrequently as once every two weeks and still remain active in the game for a long time, or we might see that an absence of even four days signifies a player who is thinking of quitting. In the short term it's ok to just take our best guess.
The more data, the merrierWe can refine these numbers over time, and begin to build sophisticated profiles and criteria based on level of involvement, character advancement, and even playstyle. We might find that danger point for all players might be an average of a week without logging in, but more likely the number will be different in different contexts.
Perhaps any player who has played less than 30 hours or is below level 20 is unlikely to return if they don't play every day. Perhaps max level players in casual guilds can log in once a week for raid night and once a week for a crafting session and still remain enthusiastic about the game.
We can become more and more ingenious with our detective work over time, tracking demographics, guild dynamics, important players and guilds who have a kind of social gravity among other players. We can compensate for local holidays, cross reference the patch notes of our game and of our competition, and even analyse crazy things such as how the color palette of different zones impacts the duration of play sessions. Let's not go crazy quite yet, though.
Solve the biggest problems firstBefore we get to any finicky analysis, demographics, playstyle, and all kinds of other subtleties, let's just make sure our game will be around long enough for us to do so. When solving problems in game design (or any discipline really), there are always some "low-hanging fruit" problems which can be diagnosed and corrected much more easily than others.
First of all, what percentage of our total playerbase is unhappy? If it's only 5 percent, then we're doing a pretty good job already, and will probably just need a few tweaks. If it's 40 percent, things aren't looking good. If the number is growing every month, we're making things worse or our game is out of content. If the number is going down, our endgame is more enjoyable than our leveling, or our fixes are making people happier.
Let's take all of our available data, from every player who has ever quit or shown signs of quitting, and graph it. Let's look at the broad generalities. For each unhappy player, let's look at some statistics about whichever character they played the most in the last month:
- How many hours the player has logged on that character
- What level the character is
- What class the character is
- How often the character dies
- What zone each death was in
Don't worry, we'll get there. Big problems before small problems. Just looking at those four data sets, and combinations thereof, can give us a quite a bit of information about our most obvious problems. It will also suggest exactly where we should start combing through data for more clues.
How bad is bad?Using wow as an example, let's imagine some potential data points and what they would tell us. If the unhappy players seem to be distributed evenly in the level range, that's more or less a good sign, and probably indicates that our game just isn't cut out for some people. If lots of players play to the level cap and then quit, we're likely in need of some better endgame content, but at least the early game is fun and we know where to focus our efforts.
If there's a big spike in unhappy players at level 10, this is a medium to terrible problem. Players might be feeling lost after the starter zone, the tutorial might not be preparing them well for the world, or maybe there is a problem with high level players griefing lowbies. These problems are all solvable and pretty small. However, players may just be deciding that the game is horrible early on, and fixing that level 10 spike might just turn it into a level 15 spike. We'll be able to get some idea by seeing what percent of our players make it to max level, but we'll also just have to wait and see.
Hopefully any low level problems are just potholes, and fixing them will pave the way for players to enjoy the game up to the level cap, but there's no way to know for sure until we start fixing things. Unhappy players at a low level is a very dangerous problem, and is an even bigger emergency than unhappy high level players, because new players will be entering the game, growing frustrated, and quitting before you can even address the problem. This is one reason that it's hugely important to get the jump on solving these problems before the game has gone live.
What else can we tell from a quick glance at these data? Let's take a look at how often players die, combined with some of the other information:
- If players are dying very often in every situation, the player health and damage table may be tuned too low.
- If players are dying very often in a certain class, that class may be a bit in need of a buff.
- If players are dying very often at a certain low level, the tutorial may need some work.
- If players are dying very often at a certain high level, the itemization for that level range may be too sparse or ineffective.
- If players are dying very often at a certain low level in a certain low level zone, that zone is too hard or high level players are griefing them
- If players are dying very often at a certain mid level in a certain mid level zone, that zone may be too hard, a focus of PvP, or contain a nasty roving boss mob.
- If players are dying very often at a certain low level in a certain zone that is higher level than they are, the world may be laid out badly or in a confusing manner, so that low level players can't tell where they're supposed to go.
Next, we'll begin building profiles of different kinds of players, and looking at how different patterns may emerge for each different player type.