May 3, 2009

Designing a Black Box (Part 4)

This is the fourth post in a series about data mining games to discover why players are quitting.
Read Part 1 here
Read Part 2 here
Read Part 3 here

Correlation Versus Causality

By this time, we've hopefully made some large generalizations about what makes our players unhappy and begun to investigate them. During this process, we'll have to be careful not to throw out the baby with the bathwater: When a player has a bad play experience and logs off for 2 weeks, how do we know which of the experiences they had were the bad ones that caused them to log off, and which ones were neutral or even enjoyable?

Every experiment needs a control group. We need to see how happy players are playing the game, so it becomes obvious how players who are unhappy are different. There are some events that correlate with a player logging off but don't cause them to log off, such as returning to a city, or checking the auction house. It's obviously a mistake to decide that entering cities or putting objects up for sale makes players angry or causes them to quit.

We are also likely to find many events that cause players to log off, but are not a factor in how long the player stays logged off. For example, when given the choice, most players will log off at a natural "stopping point" such as completing an instance, finishing a quest chain, or hitting the next level. Players who are unhappy are likely to log off at these moments, but so are players who love the game and are having a great time. This commonality makes it clear to us that players who are leaving the game aren't doing so because they hate completing goals.

When we see both unhappy players and the happy player control group tend to hit a new level, purchase their new powers, and then log off, we can recognize that those events are not causal to staying logged off for a long time. However, once our players are split up into happy and unhappy piles, we may notice that the unhappy players took much longer than the happy players to reach that next level.

This could turn out to be because the unhappy players have a higher rate of deaths per hour, or that they fail missions more frequently, or that they don't have enough friends in the game to help them, etc. The control group allows us to make comparisons that lead us down a string of clues and help us determine what's going wrong.

Identifying happy players

In order to create a data set for our control group, we'll need to determine which players are happy. Since our black box uses time spent logged out as an indicator of unhappiness, let's consider happy players to be those that stay logged out for the lowest amount of time.

However, we should also keep in mind that it's definitely possible to play a game every day and get burned out enough to quit. We may find that hardcore players who are unhappy may still play more often that the happiest casual players. In this case we might want to look at logout times between sessions over a timeline, and place players who log in frequently but less often than they used to in the unhappy category.

Other symptoms of unhappiness

This black box is based on the amount of time players spend logged off, but there are many different ways that players may indicate their unhappiness with the game. We may need to create more than one black box, or just base ours on a different criterion for happiness. Every game and playerbase is different, so there are no rules to follow every time. If we understand psychology and our audience well, we'll hopefully always have the means available to diagnose and solve problems.

For example, if our game is based on microtransactions, a decrease in buying things could be an indicator of unhappiness. If our game includes a large social element, it's possible that unhappy players might abandon all other features and start using the game as a glorified chat room. Such a player would be held to the game by their social connections alone, and be very likely to quit if any of their friends did. This type of player could be identified by an increased ratio between standing in town chatting versus other gameplay activities.

Player archetypes

While there are some problems with a game that generally all players will dislike (some of which we covered in Part 3), any playerbase is likely to be made up of a diverse group of players who value different things and have fun in different ways. Because of this, it's helpful to split players into subgroups by playstyle or personality.

There have been many attempts at defining some generic player archetypes, but defining custom archetypes suited to a partular game works better in my opinion. When I worked on The Sims, we talked about our players in terms of Storytellers, Builders, Moviemakers, and Powergamers. Not every player fit perfectly into one of those archetypes, and most players fit into more than one, but they were a useful way to discuss new features and their intended audience. Each of our developers tended to favor one of those groups as well, and after awhile we started to consciously advocate for our constituencies, not unlike senators.

Just as we've defined happy players and unhappy players, we can also define data sets for each player archetype, and split those data sets further by hardcore and casual, soloers and groupers, beginners and experts, or any other divisions that are useful to us. Once we've done that we'll have many sets of data that describe our players in some detail and give us a much higher resolution when analyzing those data. Players (and their corresponding data) can move around between data sets over time, as their temperament, level of involvement, and happiness change.

The reason it's important to make these distinctions is because the same events may take on different significance to different players. A player who loves PvP may take a string of crushing defeats as a challenge to play much more often, while a casual socializer may quit the game in disgust. Hardcore raiders who begin standing around all day in town talking may be bored of the game's content, but socializers and crafters may make that their main activity and remain quite happy.

The other thing that tracking data by archetypes does is illustrate which kinds of players like the game and which do not. Perhaps casual socializers aren't very interested in the game. If the game is an MMOFPS PvP fragfest, maybe that's ok. If the game is Happy Farming Town Online, we'd better make sure those players are happy, because they're the only players we've got. Ultimately no game can make every player happy, and personally I prefer to make games that are clearly for some groups of players, and clearly not for other groups of players. The more specific the groups we're catering to, the better we can do so.

Finally, some results

At the end of this step, we should have some detailed hypotheses about each of the player groups in our game, how happy they are, and what sort of things we suspect make them unhappy. In other words, we're now on step 2 of 4 of the scientific method. Steps 3 and 4 are outside of the scope of this series, but we'd have to test our hypotheses with some experiments, and verify our fix by waiting for some data to be gathered that corroborates our hypothesis of what was wrong and that we fixed it correctly.

In the final installment, we'll look at how to present the right fixes to the right players, and finally use all this data and analysis to bring back some lapsed players.
Continue to Part 5

No comments: