April 22, 2009

Designing a Black Box (Part 3)

This is the third post in a series about data mining games to discover why players are quitting.
Read Part 1 here
Read Part 2 here

It's never too soon to find problems

If we're developing a black box for a game that's been out for a year, and has a stockpile of data already, it's pretty obvious we can just jump right in and start analyzing. People seem a little more hesitant to start gathering data on a game that hasn't even been released yet, but that's exactly what we should be doing. The great thing about basing our black box on the amount of time since players have logged out is that we don't need any paying customers to start getting data.

This system will start working as soon as we start letting normal players play our game under normal circumstances, which probably happens in alpha or beta. To clarify, I consider "normal players" to mean non-developers who are hopefully from our intended audience. I consider "normal circumstances" to mean that players can play as much as they want, as often as they want, with no level caps or dev commands.

Once these conditions are met, we can start looking at the amount of time between play sessions right away, and start finding our largest player leaks before the game has even launched. If a player doesn't want to play the game for free, we can bet they won't pay to do so.

Once the game has gone live, and a billing cycle or two has elapsed, we can begin to examine our assumptions about the correlation between time since logout and actual cancelled subscriptions. Perhaps we'll see that players tend to log in to the game as infrequently as once every two weeks and still remain active in the game for a long time, or we might see that an absence of even four days signifies a player who is thinking of quitting. In the short term it's ok to just take our best guess.


The more data, the merrier

We can refine these numbers over time, and begin to build sophisticated profiles and criteria based on level of involvement, character advancement, and even playstyle. We might find that danger point for all players might be an average of a week without logging in, but more likely the number will be different in different contexts.

Perhaps any player who has played less than 30 hours or is below level 20 is unlikely to return if they don't play every day. Perhaps max level players in casual guilds can log in once a week for raid night and once a week for a crafting session and still remain enthusiastic about the game.

We can become more and more ingenious with our detective work over time, tracking demographics, guild dynamics, important players and guilds who have a kind of social gravity among other players. We can compensate for local holidays, cross reference the patch notes of our game and of our competition, and even analyse crazy things such as how the color palette of different zones impacts the duration of play sessions. Let's not go crazy quite yet, though.

Solve the biggest problems first

Before we get to any finicky analysis, demographics, playstyle, and all kinds of other subtleties, let's just make sure our game will be around long enough for us to do so. When solving problems in game design (or any discipline really), there are always some "low-hanging fruit" problems which can be diagnosed and corrected much more easily than others.
First of all, what percentage of our total playerbase is unhappy? If it's only 5 percent, then we're doing a pretty good job already, and will probably just need a few tweaks. If it's 40 percent, things aren't looking good. If the number is growing every month, we're making things worse or our game is out of content. If the number is going down, our endgame is more enjoyable than our leveling, or our fixes are making people happier.


Let's take all of our available data, from every player who has ever quit or shown signs of quitting, and graph it. Let's look at the broad generalities. For each unhappy player, let's look at some statistics about whichever character they played the most in the last month:
  • How many hours the player has logged on that character
  • What level the character is
  • What class the character is
  • How often the character dies
  • What zone each death was in
Wait a minute, we've got this beautiful mountain of data, and we're standing back a mile away to see how it smells? Where's the digging and sifting and rolling in it?!
Don't worry, we'll get there. Big problems before small problems. Just looking at those four data sets, and combinations thereof, can give us a quite a bit of information about our most obvious problems. It will also suggest exactly where we should start combing through data for more clues.

How bad is bad?

Using wow as an example, let's imagine some potential data points and what they would tell us. If the unhappy players seem to be distributed evenly in the level range, that's more or less a good sign, and probably indicates that our game just isn't cut out for some people. If lots of players play to the level cap and then quit, we're likely in need of some better endgame content, but at least the early game is fun and we know where to focus our efforts.
If there's a big spike in unhappy players at level 10, this is a medium to terrible problem. Players might be feeling lost after the starter zone, the tutorial might not be preparing them well for the world, or maybe there is a problem with high level players griefing lowbies. These problems are all solvable and pretty small. However, players may just be deciding that the game is horrible early on, and fixing that level 10 spike might just turn it into a level 15 spike. We'll be able to get some idea by seeing what percent of our players make it to max level, but we'll also just have to wait and see.


Hopefully any low level problems are just potholes, and fixing them will pave the way for players to enjoy the game up to the level cap, but there's no way to know for sure until we start fixing things. Unhappy players at a low level is a very dangerous problem, and is an even bigger emergency than unhappy high level players, because new players will be entering the game, growing frustrated, and quitting before you can even address the problem. This is one reason that it's hugely important to get the jump on solving these problems before the game has gone live.
What else can we tell from a quick glance at these data? Let's take a look at how often players die, combined with some of the other information:
  • If players are dying very often in every situation, the player health and damage table may be tuned too low.
  • If players are dying very often in a certain class, that class may be a bit in need of a buff.
  • If players are dying very often at a certain low level, the tutorial may need some work.
  • If players are dying very often at a certain high level, the itemization for that level range may be too sparse or ineffective.
  • If players are dying very often at a certain low level in a certain low level zone, that zone is too hard or high level players are griefing them
  • If players are dying very often at a certain mid level in a certain mid level zone, that zone may be too hard, a focus of PvP, or contain a nasty roving boss mob.
  • If players are dying very often at a certain low level in a certain zone that is higher level than they are, the world may be laid out badly or in a confusing manner, so that low level players can't tell where they're supposed to go.
The ability to visualize and cross reference data can tell you quite a bit about your game that you might not have realized before, or at least give you some leads to start working on. Take a moment to glance over the lists in Part 2 again, and see if you can't pick out a few more sets of data that would be useful to examine.

Next, we'll begin building profiles of different kinds of players, and looking at how different patterns may emerge for each different player type.
Continue to Part 4

April 18, 2009

Speaking of Videogame Data Mining

In the course of various projects I've been working on lately, I stumbled across two great blogs focused on MMO data mining that I'd never seen before:

PARC's PlayOn Blog - This site is mainly focused on WoW, but also looks at other games from time to time. Lots and lots of juicy data here, and they also have a few posts on character abandonment, which is interesting in the context of the black box series I'm working on now.

Armory Data Mining - This site is focused entirely on data gathered from character profiles on WoW's Armory. There are some very interesting analyses here of battleground compositions and class talent distributions.

Both of these sites are very interesting because they're run by researchers who aren't part of the dev team and therefore don't have special access to any data. If sites like this can exist based on only public data (as Armory Data Mining does) or data mining from in the game and surveys (as PlayOn does), imagine what a developer with free access to all information and some professional engineers could achieve.

April 15, 2009

Designing a Black Box (Part 2)

This is the second post in a series about data mining games to discover why players are quitting.
Read Part 1 here

Time to make a big list

Ok, we know what a black box is for and why we need one, but which information should it actually mine and process?

This brings us to some actual work. Most people, including many game developers, don't actually know what game designers do. When people say they want to be game designers, they're usually thinking of what creative directors or executive producers do. Actual game design is filled with unglamorous tasks such as generating thorough lists of things: possible problems, necessary assets, edgecases, tuning hooks, etc. (In fact, making lists is such a fundamental skill for designers that "generate a big list of things you'd need to do x" is a common question on screening tests for design applicants.)

Hopefully, you're already collecting as much data as possible about your players and how they are playing the game. Using WoW as an example, here's the data I'd want my black box to have access to:

Character Data

Basic information about character demographics, advancement, and investment.

  • Number of current active characters (over level 20, played more than 10 hours in the last month)
  • Number of total characters
  • Class breakdown of active characters
  • Total /played time per character, active character, and account
  • Percentage of time played by active character and by class per day/week/month
  • Level, server type, spec, stats, cash, honor, reputation, arena points, etc for each character
  • Comparison of all stats to an average player at their level, stats compared to average player at their level, as well as average player of their class at their level
  • Guilded status, as well as guild ranking/stats/size if applicable
  • Time since guilded, if applicable
  • Time since unguilded, if applicable, and whether quit or kicked.
  • Time since each active character has been logged onto
  • Time since any character on the account has been logged onto
  • Average play session length per character and per account
  • Play session lengths per character and per account in the last month
  • Time spent crafting per day/week/month
  • Time spent in PvE instances per day/week/month
  • Time spent in PvP instances per day/week/month
  • Time spent in combat in the world per day/week/month
  • Time spent online but not in combat, crafting, or in instances per day/week/month
  • Character map coordinates at login and logout
  • Combat and chat log data for 10 minutes before each recent logout.
  • All player deaths, the time between them and their cause (which player/mob)
  • Kill-death ratio per hour/level/day/month/lifetime, by player and by mob
  • Cause and position of recent deaths where the player chose to take the death penalty
  • Time at logout since last level
  • Time at logout since last large cash expenditure, and what it was
  • Time at logout since last pvp fight
  • Time at logout since last pvp death
  • Time at logout since last quest accepted
  • Time at logout since last quest completed
  • Most recently failed quests
  • Time from acceptance to completion, for most recently accepted quest
  • Time from acceptance to completion, for most recently completed quest
  • Current number and names of accepted but uncompleted quests
  • Percentage of quests completed for current zone
  • Amount of rested xp at login and logout
  • Ratio of failed attempts versus successful attempts to clear each instance/raid boss
  • Time spent in PvE instances since the last successful boss takedown
  • Number of deaths in PvE instances since the last successful boss takedown
  • Time at logout since last loot drop of rarity uncommon/rare/epic
  • Time spent in PvE instances since the last uncommon/rare/epic item successfully looted.
  • Ratio of items rolled on versus rolls won
  • Percentage of current tier items attained, if applicable

Battleground Data

PvP data for Battlegrounds. We could also include a section about about world PvP, but since it's not emphasized in WoW we won't need any data aside from those covered in the first section.
  • Total time spent in BGs for character lifetime
  • Total time spent in BGs per character level
  • Total honor/PvP gear tokens/enemy kills over time
  • Total rounds played in each Battleground map per day/week/month
  • Number/percentage of abandoned Battlegrounds per BG map
  • Average ranking on each scoreboard (kills/dps/healing/captures/etc) per day/week/month
  • Average difference between each score and the average teammate score/average enemy score, for all players
  • Average difference between each score and the average teammate score/average enemy score, for players of the same class
  • Win/loss ratio per day/week/month
  • Average duration of BGs per day/week/month
  • Average duration of BGs won per day/week/month
  • Average duration of BGs lost per day/week/month
  • Average duration of BGs abandoned per day/week/month
  • Percentage of available PvP gear that player has attained for their bracket

Arena Data

How well is the player doing in ranked PvP, if they participate?
  • Arena points over time
  • Arena Rankings over time for 2s/3s/5s
  • Arena Rankings over time for 2s/3s/5s
  • Average duration of matches won/lost/both per ladder
  • Average life span of character in matches won/lost/both per ladder
  • Average difference in ranking between their teams and opponents per ladder
  • Percentage class compositions of enemy teams for matches won/lost/both per ladder
  • Average matches played per week
  • Matches played per week, over time.
  • Percentage of current tier arena items attained

Misc Data

Data pertaining to the general state of the game or other games that may be drawing players away.
  • Server performance the minute/hour/day of logout
  • Client's framerate the minute/hour/day of logout
  • Client/Server crashes the minute/hour/day of logout
  • GM tickets submitted the day/week/month of logout, and how long they took to be resolved
  • Most recent patch notes at the time of last logout
  • Competiting companies' new game releases or major patches the week/month of last logout
Ok, that covers the tedious stuff. Bear with me. Once you've done this small amount of work up front, you can spent the bulk of your time actually analyzing the data, with a little bit of maintenance here and there as you add new features.
The data that I've listed here is the kind of data I suspect would reveal why a player is quitting the game. Of course if you've left some important data off the list you won't have a very effective black box, so when in doubt, mine more data than you think you need to.

Next, we'll take all the data we've gathered up and start sifting through it for clues.
Continue to Part 3

April 13, 2009

Designing a Black Box (Part 1)

When a plane crashes, the best way for investigators to discover why it happened is to find the black box, which contains all known data about the position and condition of the plane immediately before the crash.

Games also need black boxes, to find out why players are quitting. Using this information to identify problems and improve the game can help make sure that subsequent players don't quit for the same reasons, and even help bring lapsed players back.

I'll be using a level-based subscription MMO such as WoW as an example for this series, but every online game should have a black box. Differences in genres and gameplay should only make slight changes in the type of data gathered and analyzed, and I'll leave modifications as an exercise for the reader.

Why your game needs a black box

It's easy to see how a game that charges subscriber fees needs a black box, but any game with an online multiplayer element benefits greatly from a large, happy player base. The best way to have lots of players is to never lose any, even if you only gain them slowly.To keep from losing players, you'll have to identify problems early and correct them before it's too late. The best way to find and correct subtle problems is through data mining and analysis.

Many of the problems that a black box can reveal should really be obvious to any good designer in advance, so having a black box is not a substitute for thorough design beforehand or anticipating consequences. They are particularly useful, though, for revealing subtle trends and problems with systems that look good on paper or during limited playtesting, but start to break down after hundreds of hours of gameplay.

Identifying at-risk players

The first thing we'll need to do is figure out which players are likely to quit soon. Since subscription-based MMOs charge in advance for playtime, players often check out or decide to quit a long time before their cancellation will actually show up on the books. This is a good thing, because our black box system will allow us to see why they're unhappy and to hopefully correct those problems before they quit. There's no need to wait for these players to officially quit before figuring out what went wrong, so we'll try to identify which players seem most likely to be quitting soon before it actually happens.

Depending on the type of game you're running, you could simply define a variable for how long it's acceptible to go without logging into the game before a player enters the at-risk category. You will also probably want to check for a minimum playsession time, because a player who logs in every 2 days but only stays for 10 minutes probably isn't very happy either.

If your game has classes or other meaningful differentiators between characters, and you're worried about certain character types becoming abandoned more than others, a variant of this type of system could be used to see why particular characters are being abandoned. For this example, I'll just focus on players who are abandoning the game entirely, but it should be easy to see how the system could be modified to examine character abandonment as well.

There are likely to be several factors that can help predict character abandonment, but for this example I'll stick to time since logout as the main signifier of at-risk players.

Next, we'll examine the kind of data the black box will need to mine, and then start to look at what these data will be able to say about why players are quitting.
Continue to Part 2

April 1, 2009

Game Mechanics as Fun as Cheating

What is it about cheating at games that appeals to people so much? Having more information at your disposal than the competition? The feeling of complete confidence and indestructibility? The satisfaction of ruining somebody else's plans?

In many cases I think the pleasures of cheating are similar, or even identical to, the pleasures of being very good at the game. The fun of cheating is a surrogate for the fun of being skilled, presumably for players who would not be able to enjoy those feelings unaided.

I wrote yesterday about including cheat-like mechanics in your games as a method of neutralizing cheating players. Writing that post and reading some more about cheaters also reminded me of how those features that impersonate cheats can give players a rush of satisfaction in the same way that cheating would.

NetBat HUD (Battlefield 2142)

Battlefield 2142, much like Natural Selection, includes an elaborate HUD system which players can use to mark enemies, revealing them to their friends through walls or any other obstacle.

BF's is quite a bit more elaborate than NS's, and also ties into the map and minimap heavily. Several different abilities and unlocks provide players with different ways to get more information that they would normally have. These features do a lot to encourage teamwork, but they're also just very satisfying.

If a team activates a UAV scan, that team's players can temporarily see nearby enemies as red dots on their minimaps:


One of the central themes in 2142 is that of information as power, and the designers have found many different ways to give players the powerful feeling of temporary omniscience.

Flashbang (Various Shooters)

Some of my favorite weapons in online shooters are Flashbangs and Concussion Grenades. A perfectly thrown flashbang blinds its target and gives its owner a window of huge advantage over their enemy. A badly thrown flashbang can completely backfire, blinding the player who threw it and getting them killed.

In games with flashbangs, players learn to recognize the sound of an incoming flashbang and dive for cover. It's a great moment of terror that I always love.

Ubercharge (Team Fortress 2)

When I heard that Team Fortress 2 would be introducing a form of invulnerability to the game, I was really worried. It's exactly the kind of feature that can cancel out skill and become way too powerful. Ultimately, Ubers achieved Valve's stated goal of being a great stalemate breaker, and now that they've added more counters to them, they feel like just another interesting game mechanic.

Rushing into a fight as a near-unstoppable killing machine can be incredibly fun. It allows less-skilled players to have a moment of glory, it brings rounds to a close sooner, and it more or less completely won me over.


Don't forget to provide counters

The reason I like all of these examples is that each of them can be counteracted to some degree:
  • In 2142, players can disable the enemy's UAV emitter, choose to deploy their own UAV simultaneously, or to shoot down the enemy's UAV as it circles above.
  • In TF2, Ubered enemies can now be separated with a Compression Blast or stunned with the Sandman's baseball.
  • Flashbangs in shooters can often be seen coming, looked away from, or otherwise avoided. If the player who is assumed to be blind can actually see perfectly, the tables are turned on the flashbang's owner, who is now likely to be playing carelessly. Blinded players still also have a chance to get a lucky kill, for extra bragging rights.
I really believe in counters for almost everything in games, even if they're very hard to take advantage of. For example, if WoW's Paladin bubble had a .5 second cast time that could be interrupted, or only absorbed a set amount of damage before disappearing, it would also be on this list. Powerful abilities that can't be prevented or countered tend to be frustrating at best.

That may be a particular bias of mine, though, as I believe that all game mechanics should specifically encourage gameplay, and helplessness doesn't make for enjoyable gameplay.