May 24, 2009

Learn Your Lesson and Let it Go

Like everyone, game designers make lots of mistakes. We might lose players because of bad design decisions, or water down our game by trying to appeal to too many kinds of players at once. We might make a bad impression on someone who can influence our career, or vouch for someone who turns out to be untrustworthy.

We might work on a game that gets horrible reviews, design a feature that ends up being the worst part of the game, or pour years of our lives into a game that gets cancelled or a company that goes bankrupt. Or, it might be something as small as introducing a bug that ends up breaking the build.

In some cases, we can recognize these problems and smooth things over, but sometimes the damage is irreparable. In cases like these, all we can do is quickly learn what lessons can be learned, and be on the lookout for our next chance to apply them.

It's really ok to move on

When your balloon escapes, you don't start climbing fire escapes to chase it; you buy a new one and tie it to your wrist. When you drop an ice cream cone, you hopefully don't eat it off of the ground; you learn not to eat and dance at the same time.

One of the hallmarks of a veteran designer is spending less time and energy lamenting mistakes, and more time and energy on preventing those problem from recurring.

People who chase missed opportunities rarely succeed at recapturing them. Without a time machine, we'll never get a chance to catch that winning touchdown pass we missed, and in real life barging into weddings usually doesn't get us a second chance with the one who got away.

What we can do is analyze what went wrong, build a pattern in our mind that will allow us to recognize our next opportunity to make a similar choice, and move on with our lives. This is the essence of learning, and it's what we ask our players to do all the time in our games. Even in games that let players rewind, those who can't learn from their mistakes will be trapped forever, in an endless failure loop.

Luckily, this kind of learning is one of the things that humans are best at. It's the essence of science, interpersonal communication, and especially playing games. In fact, some people think that this kind of learning is the entire reason that games exist. Making games is really just a very elaborate metagame itself, so it's no great stretch that we should ask ourselves to learn from our mistakes.

Player retention vs re-acquisition

Here's a good example of how fixing problems can't change the past:

Now that we've used a black box to fix the problems a game has, we can go out and get all those old players back, right? Well, maybe, but not necessarily. If it takes 10 times as much effort to get a player as it does to keep a player, it might take 50 times as much effort to get an old player back, once they've decided our game is crap. Some players may have had a bad enough experience that they'd never come back if we paid them.

What'd we waste all the time fixing the problem for then, if it can't actually get our players back? Improving our player retention gives us another chance to succeed with new players. We need to work on making sure the players have are sticking around, because retention is more important than acquisition. If our player acquisition is not good, then now we'll need to work on fixing that too, but only after we've made sure that our retention is high enough.

Maybe after a while, good word of mouth might spread to our old players, and then we'll have another chance to bring them back, but maybe those players are lost to us forever and we'll have to focus on making our new players happy. If drastic amounts of players have left the game, and the new players coming in are very different from the old ones, we may find ourselves with a whole new set of problems to solve.

Worse consequences, stronger lessons

Those players we lost may actually be a better lesson to use than if we had managed to get them back. In a way we spent or consumed those players. They taught us how to fix the game, but now they may be contaminated or otherwise invalid to us. Next time we'll be sure to be more careful about those particular kinds of problems.

Maybe the game lost so many players that it ends up dying, but in that case we'll be sure not to make those same mistakes again on our next game. If we do, then maybe survival of the fittest should apply, and we shouldn't be making games in the first place.

This learning process is very important to being a good game designer, and is one of the main assets that experienced designers possess over the most talented new designers. Properly analyzing mistakes and recognizing opportunities to apply that knowledge is a skill of its own.

Becoming better at learning from our mistakes and knowing when to apply those lessons is the best shortcut there is to becoming much better designers, much more quickly.

May 16, 2009

Designing a Black Box (Part 5)

This is the fifth and final post in a series about data mining games to discover why players are quitting.
Read Part 1 here
Read Part 2 here
Read Part 3 here
Read Part 4 here

Which players matter more?

After working through the previous 4 steps, we should have a list of some very specific player demographics, based on both the type of players that are unhappy and the reasons they are unhappy.

For example, here are some hypothetical examples of groups that our unhappy players could fall into:
  • Social players who find it hard to find a casual guild
  • Expert raiders who are out of content
  • Beginners who aren't prepared well enough by the tutorial
  • PvP players who can't find anybody to fight
  • PvE players who are sick of being killed
  • Latecomers who can't play with their high level friends
All of these problems can be solved, but not all of them can be solved for every game, and certainly not all at once. We'll have to decide which of these player/problem combinations are the most in need of our limited development time, and also in which cases our efforts will be the most effective.

This can be a very difficult question to answer. Some games fail to define their key audience well, which means the argument about which leaks to plug will likely be very hard to resolve. Other games get too greedy, reaching out to a new demographic at the cost of angering their loyal fanbase. Star Wars Galaxies is a classic example of a game that tried to reach a new demographic and failed, losing many of its old players without gaining many new ones.

There are a few obvious factors in prioritizing which problems to address:
  • Which player types are the most important to retain?
  • Which problems are causing the most overall players to leave?
  • Which problems are causing the most of the key audience to leave?
  • How early early in the game are players encountering those problems?
  • How easily/cheaply/effectively can those problems be fixed?
Most designers probably think of these factors, but there are a couple very important factor that most people neglect:
  • How much of the damage has already been done?
  • How many of these players are we acquiring?
If our game launched 5 months ago with a horrible PvP experience, that fact has probably been announced loudly by reviews, fan sites, and angry ex-players. If we then decide to spend tons of time and money to improve the PvP experience, it's unlikely any players who enjoy PvP will be left to notice. We can try to entice them back, but perhaps it's been decided for us that our game is no longer a PvP game, even if we intended it to be. If PvP was all our game had going for it, we're probably already doomed, but if there are another group of players who are enjoying the game, it's probably wiser to improve what our game is already doing somewhat well.

Similarly, if only a very small percentage of our new players tend to be PvP players, and they all quit, fixing PvP may be addressing a demand that is not actually there. In order to be able to retain players, we first have to acquire them. Happy Farming Town Online will never draw many PvP players in the first place, so even 100% retention of PvP players won't ever really do us very much good.

Once a game is launched, its players define it. If we thought we were making a game for one group of people, and that group snubs it but another group likes it, we'd better get to work making the game suit the players who actually care about it. We can worry about getting other players once we're sure that we're taking care of the ones we've got.

Reach out to the right players

Once we've decided to fix a problem that a particular group of players is having, we'll have to actually design and implement a fix to that problem, not to mention make sure the problem is actually fixed. Once that's done, we just have to let those players know that those problems are fixed.

Since we used our black box to divide players into groups and problems in the first place, it should be possible to write in the functionality to send mass emails to groups of players, without even having to know which players fall into each group. By using a data driven system in this way, we can help specific players without even having to know which specific players those are. This should both stop players and developers from abusing the system, as well as avoid the majority of privacy concerns.

In a subscription-based game, communicating with players by email can be a dangerous thing. I heard a funny story from a friend about how a company sent a big update email out to all its players, many of whom had apparently forgotten they were still paying for it. Upon receiving the update email, a huge chunk of lapsed players promptly unsubscribed.

In my opinion this shouldn't be a deterrent. Hopefully a good black box system would prevent so many players from entering that limbo state to begin with, and even the data of which players were driven to quit would help us make the game better in the long run.

Once we've rolled out a change and an announcement tailored to a specific player group, we'll probably also want to track data for how well it worked. How many players came back? How long did they stay? If they've lapsed again is it a new problem or the same problem?

Don't rule out temporary fixes!

Fixing the game's problems is our ultimate goal, but sometimes is makes sense to temporarily compensate for those problem until they can be fixed.

If we find that the leveling curve is too slow in a particular area, or a level range seems to be a complete disaster, we can give every player in that zone or level range some bonus XP or free levels to get them past the rough patches. If a group of players who tried out PvP servers is having a miserable time, we can offer them free transfers to PvE servers, or give them some free PvP gear tokens to get them started off on the right foot.

In drastic cases, it may even be wise to disable a feature entirely rather than let it drive players away. We can also offer some playtime free or at a discount. If there are some problems that will take a long time to fix, even just announcing our intention to do so is something players really appreciate.

Advanced uses of black boxes

Even once we manage to plug all of our big player leaks, there are plenty of tasks that this type of data mining can be useful for:
  • How can we turn all the average players into happy players? This will follow a very similar process as making unhappy players average.
  • Which kind of marketing is successful with different kinds of players? We can experiment with different kinds of marketing in different time periods, and see how that affects our player acquisition and retention.
  • What is the best price point for our game or subscription? We can test out different prices and fees on subgroups of players to find out where the sweet spot is. It's possible that we'd keep many more players, and make much more money, by making our game cheaper.


All this may sound like a lot of extra work when we could just post a survey on our forums or make people fill out a form when they unsubscribe from the game. It really is worth it, though. Systematically identifying problems is a huge advantage over interviewing players, because players are not game designers and often don't consciously recognize the true causes of fun or frustration. For that matter, even game designers often bark up the wrong tree and rationalize or misinterpret bugs.

Targeting announcements and incentives to a particular group is incredibly powerful when done correctly, because it shows players that we've responded to their specific needs, even if they were unspoken. To players, this feels like we're reading their minds, or that we have the same priorities they do.

This sort of trust is very powerful when it comes to making longterm fans of a game or a company, and I believe it's more integral to success than most developers will ever realize.

May 8, 2009

The Tortoise and the Hare

As you've probably noticed, I've been thinking a lot about player retention this year.

Back in February, I took some subscriber numbers from EVE Online and Warhammer Online and made some projections as to what their subscriber numbers were at that time. This involved some educated guessing, and some blind speculation.

EVE seemed to be a game with low player acquisition and high player retention, while WAR so far had shown high acquisition, but low retention. I ended up hypothesizing that EVE was in the process of passing WAR in subscription numbers, and that more or less everyone would be completely shocked when they realized this:

EVE is old and gruff and complex, and a little bit crazy. WAR is new and loud and very well marketed. It'd be the game industry equivalent of Tom Waits outselling 30 Seconds to Mars.

So who's winning?

Today EVE is six years old and WAR is six months old, and they've both announced subscriber numbers of "over 300k." I was a few months off on the timing, and I was apparently dead wrong that anybody besides me would find this event to be so significant.

In January, Warhammer's numbers were also announced as "over 300k," so it's hard to guess if those numbers have stabilized, are still falling, or are increasing again. I believe that 350k is a round enough number that a company would announce "over 350k" if they had broken that threshold, so I assume both games are fluctuating somewhere between 300k and 350k.

EVE's current numbers are significantly lower than my estimate (I extrapolated that they were at 323k back in February). EVE officially had 236k subscribers on June 2, 2008, and 244k at the start of 2009, which is a very modest growth of only 8k players in 7 months. However, their growth so far this year has been much faster:

We started out the year with around 244,000 subscribers and in five short months we've had a 22% growth in subscribers. In the past couple days we surpassed the impressive milestone of 300,000 active subscribers.

Why does this matter?

What I find so interesting about this situation is that we have two games, both somewhat niche and aiming for fans of PvP, at approximately the same number of subscribers. One is good at attracting players, and seems to be getting better at retaining them. One is good at retaining players, and getting better all the time at attracting them. I'm extremely interested to see what happens from here.

It actually seems that both games are shoring up their weaknesses. EVE has seen no shortage of scandal this year, which I'm sure has been great for publicity and player acquisition. WAR seems to no longer be losing players, and has taken strides to bring back lapsed players.

These two games, occupying a similar niche, have arrived at almost exactly the same subscription numbers, at the same time, by very different means. This is about as close as the market can provide to controlled conditions. We'll be able to see how each of the games adapt and progress from here, and how well each approach works over time.

Perhaps both games will level off around 400k, and we'll see that 800k players (plus however many Darkfall has) is the size of the current PvP market. After all, a million players is quite a bit more than most people ever believed PvP games could secure. Perhaps after a certain market cap is reached, the PvP games will stop growing organically and need to rely more on enticing players from each others' audiences.

Maybe we'll see that EVE really does have another ten years of growth left in it, as it continues to become more accessible and while keeping up its high levels or retention. Maybe WAR has distilled its true target audience, and that same 300k players will be playing the game until they unplug the servers.

Right now, these two games combined have about 5% of WoW's subscribers (600k versus 13MM). If those ratios stay constant as WoW continues to grow (and I do believe it's reasonable that 5% of WoW's players end up becoming fans of PvP), games like these can continue to grow at a healthy rate without ever having to directly compete with WoW. If WoW was a successful sushi restaurant, PvP games would be the fat, happy cats who live in the alley behind it.

May 5, 2009

Still Speaking of Data

All my posts on data-driven design lately have been bringing a lot of data-loving new readers to the blog, who in turn have been pointing me toward no end of interesting stuff to read and think about. It's bordering on overwhelming, in fact.

Nels Anderson has a great series of posts going which touches on data. In it, he links to a surprisingly interesting talk by Malcolm Gladwell on Spaghetti Sauce and making your customers happy by not listening to them.

He also mentions a GDC talk by Valve's Mike Ambinder, which goes into some detail about just how many different kinds of user testing Valve actually does.

Speaking of Valve, Left 4 Dead Stats have gone live, and there also seem to be some surprising changes to TF2 coming up soon. According to Robin Walker, you can try just about anything as long as you've got good enough communication with your players:

Shacknews: Any plans for consumable items? And will these items be tied to each class, or carried over to all of your classes?

Robin Walker: The short answer is: possibly. Part of the reason we want to do this kind of design work in TF2 is because it's a product in which players send us a ton of great feedback. This means we'll quickly find out if we're screwing up, which is an amazingly useful thing when we want to try new things.

Shacknews: Should we expect to see more item slots--chest, feet, elbows--added in the future?

Robin Walker: See the "possibly" answer above.

Shacknews: Will players be able to trade these items?

Robin Walker: This is a good example of the kind of decision we'll be listening to the players around. If it's something they all want to do, then yep, you can expect us to do it.

For those of you who have had your fill of data for awhile, don't worry. I plan to wrap up the black box series with the next installment, and then we'll find something new to fixate on for awhile.

Since I'm breaking my first cardinal rule (don't just post a bunch of links to things), I may as well break my second cardinal rule (don't post to apologize for a lack of posting), just this once. Sorry about the two week lag there.

Generally I prefer to write more substantial posts less often rather than lots of short posts, but hopefully I'll settle into a bit of a better rhythm soon. Apparently getting too busy to post just as your blog receives a huge influx of readers is a tale as old as time. In the meantime, please subscribe!

May 3, 2009

Designing a Black Box (Part 4)

This is the fourth post in a series about data mining games to discover why players are quitting.
Read Part 1 here
Read Part 2 here
Read Part 3 here

Correlation Versus Causality

By this time, we've hopefully made some large generalizations about what makes our players unhappy and begun to investigate them. During this process, we'll have to be careful not to throw out the baby with the bathwater: When a player has a bad play experience and logs off for 2 weeks, how do we know which of the experiences they had were the bad ones that caused them to log off, and which ones were neutral or even enjoyable?

Every experiment needs a control group. We need to see how happy players are playing the game, so it becomes obvious how players who are unhappy are different. There are some events that correlate with a player logging off but don't cause them to log off, such as returning to a city, or checking the auction house. It's obviously a mistake to decide that entering cities or putting objects up for sale makes players angry or causes them to quit.

We are also likely to find many events that cause players to log off, but are not a factor in how long the player stays logged off. For example, when given the choice, most players will log off at a natural "stopping point" such as completing an instance, finishing a quest chain, or hitting the next level. Players who are unhappy are likely to log off at these moments, but so are players who love the game and are having a great time. This commonality makes it clear to us that players who are leaving the game aren't doing so because they hate completing goals.

When we see both unhappy players and the happy player control group tend to hit a new level, purchase their new powers, and then log off, we can recognize that those events are not causal to staying logged off for a long time. However, once our players are split up into happy and unhappy piles, we may notice that the unhappy players took much longer than the happy players to reach that next level.

This could turn out to be because the unhappy players have a higher rate of deaths per hour, or that they fail missions more frequently, or that they don't have enough friends in the game to help them, etc. The control group allows us to make comparisons that lead us down a string of clues and help us determine what's going wrong.

Identifying happy players

In order to create a data set for our control group, we'll need to determine which players are happy. Since our black box uses time spent logged out as an indicator of unhappiness, let's consider happy players to be those that stay logged out for the lowest amount of time.

However, we should also keep in mind that it's definitely possible to play a game every day and get burned out enough to quit. We may find that hardcore players who are unhappy may still play more often that the happiest casual players. In this case we might want to look at logout times between sessions over a timeline, and place players who log in frequently but less often than they used to in the unhappy category.

Other symptoms of unhappiness

This black box is based on the amount of time players spend logged off, but there are many different ways that players may indicate their unhappiness with the game. We may need to create more than one black box, or just base ours on a different criterion for happiness. Every game and playerbase is different, so there are no rules to follow every time. If we understand psychology and our audience well, we'll hopefully always have the means available to diagnose and solve problems.

For example, if our game is based on microtransactions, a decrease in buying things could be an indicator of unhappiness. If our game includes a large social element, it's possible that unhappy players might abandon all other features and start using the game as a glorified chat room. Such a player would be held to the game by their social connections alone, and be very likely to quit if any of their friends did. This type of player could be identified by an increased ratio between standing in town chatting versus other gameplay activities.

Player archetypes

While there are some problems with a game that generally all players will dislike (some of which we covered in Part 3), any playerbase is likely to be made up of a diverse group of players who value different things and have fun in different ways. Because of this, it's helpful to split players into subgroups by playstyle or personality.

There have been many attempts at defining some generic player archetypes, but defining custom archetypes suited to a partular game works better in my opinion. When I worked on The Sims, we talked about our players in terms of Storytellers, Builders, Moviemakers, and Powergamers. Not every player fit perfectly into one of those archetypes, and most players fit into more than one, but they were a useful way to discuss new features and their intended audience. Each of our developers tended to favor one of those groups as well, and after awhile we started to consciously advocate for our constituencies, not unlike senators.

Just as we've defined happy players and unhappy players, we can also define data sets for each player archetype, and split those data sets further by hardcore and casual, soloers and groupers, beginners and experts, or any other divisions that are useful to us. Once we've done that we'll have many sets of data that describe our players in some detail and give us a much higher resolution when analyzing those data. Players (and their corresponding data) can move around between data sets over time, as their temperament, level of involvement, and happiness change.

The reason it's important to make these distinctions is because the same events may take on different significance to different players. A player who loves PvP may take a string of crushing defeats as a challenge to play much more often, while a casual socializer may quit the game in disgust. Hardcore raiders who begin standing around all day in town talking may be bored of the game's content, but socializers and crafters may make that their main activity and remain quite happy.

The other thing that tracking data by archetypes does is illustrate which kinds of players like the game and which do not. Perhaps casual socializers aren't very interested in the game. If the game is an MMOFPS PvP fragfest, maybe that's ok. If the game is Happy Farming Town Online, we'd better make sure those players are happy, because they're the only players we've got. Ultimately no game can make every player happy, and personally I prefer to make games that are clearly for some groups of players, and clearly not for other groups of players. The more specific the groups we're catering to, the better we can do so.

Finally, some results

At the end of this step, we should have some detailed hypotheses about each of the player groups in our game, how happy they are, and what sort of things we suspect make them unhappy. In other words, we're now on step 2 of 4 of the scientific method. Steps 3 and 4 are outside of the scope of this series, but we'd have to test our hypotheses with some experiments, and verify our fix by waiting for some data to be gathered that corroborates our hypothesis of what was wrong and that we fixed it correctly.

In the final installment, we'll look at how to present the right fixes to the right players, and finally use all this data and analysis to bring back some lapsed players.
Continue to Part 5