April 13, 2009

Designing a Black Box (Part 1)

When a plane crashes, the best way for investigators to discover why it happened is to find the black box, which contains all known data about the position and condition of the plane immediately before the crash.

Games also need black boxes, to find out why players are quitting. Using this information to identify problems and improve the game can help make sure that subsequent players don't quit for the same reasons, and even help bring lapsed players back.

I'll be using a level-based subscription MMO such as WoW as an example for this series, but every online game should have a black box. Differences in genres and gameplay should only make slight changes in the type of data gathered and analyzed, and I'll leave modifications as an exercise for the reader.

Why your game needs a black box

It's easy to see how a game that charges subscriber fees needs a black box, but any game with an online multiplayer element benefits greatly from a large, happy player base. The best way to have lots of players is to never lose any, even if you only gain them slowly.To keep from losing players, you'll have to identify problems early and correct them before it's too late. The best way to find and correct subtle problems is through data mining and analysis.

Many of the problems that a black box can reveal should really be obvious to any good designer in advance, so having a black box is not a substitute for thorough design beforehand or anticipating consequences. They are particularly useful, though, for revealing subtle trends and problems with systems that look good on paper or during limited playtesting, but start to break down after hundreds of hours of gameplay.

Identifying at-risk players

The first thing we'll need to do is figure out which players are likely to quit soon. Since subscription-based MMOs charge in advance for playtime, players often check out or decide to quit a long time before their cancellation will actually show up on the books. This is a good thing, because our black box system will allow us to see why they're unhappy and to hopefully correct those problems before they quit. There's no need to wait for these players to officially quit before figuring out what went wrong, so we'll try to identify which players seem most likely to be quitting soon before it actually happens.

Depending on the type of game you're running, you could simply define a variable for how long it's acceptible to go without logging into the game before a player enters the at-risk category. You will also probably want to check for a minimum playsession time, because a player who logs in every 2 days but only stays for 10 minutes probably isn't very happy either.

If your game has classes or other meaningful differentiators between characters, and you're worried about certain character types becoming abandoned more than others, a variant of this type of system could be used to see why particular characters are being abandoned. For this example, I'll just focus on players who are abandoning the game entirely, but it should be easy to see how the system could be modified to examine character abandonment as well.

There are likely to be several factors that can help predict character abandonment, but for this example I'll stick to time since logout as the main signifier of at-risk players.

Next, we'll examine the kind of data the black box will need to mine, and then start to look at what these data will be able to say about why players are quitting.
Continue to Part 2


Ysharros said...

Hurry up with part 2 already -- this is very interesting stuff! I was going to comment on privacy concerns and the like (the data can't be impersonal if it's going to be used to prevent me from unsubbing?), but I'll wait till the next installment. Your point about not logging in often and/or not for long hit very close to home -- that's how my "I'm leaving but not really gone yet" phase in MMOs tends to start. Also, logging on and not really knowing what to do (aka not really wanting to do), but that would be hard to measure?

Also... I only just realised you're working on CO (via the LinkedIn stuff). Now I'm going to have to get all smarmy and ass-kissy to ensure a place in beta! (Yes, it's a joke. I'm better at being oblivious than at being ass-smoochy. ;-) )

Grant said...

I remember Valve doing this, specifically with Half Life 2 Episode 1. Their data showed that a huge number of players were quitting around the elevator sequence. It was where you had no lights, had to fight off zombines, exploding barrels, grenades, AND do this until the elevator arrived.

Valve went back and patched the area to be a tad more forgiving and hopefully bring players back.

Mike Darga said...

@Ysh: Thanks, the notes for this post have been gathering dust for two months, and I've really been wanting to finish it.

Yeah, I want to keep this blog completely separate from work stuff, so you probably won't see me talk about it much here.

Speaking of, work and some other projects have been crazy lately. Sorry for the long lag between posts this time.

@Grant: Valve has been really great about publicizing their datamining efforts, and so has CCP. Google is also another example of a company that does extensive datamining.

Can you detect a pattern here? I'd wager that Blizzard also does quite a bit of work on this sort of thing.

Mike Darga said...

Regarding the privacy issue, I don't think it should be a problem, though I'd love to hear your thoughts on it once you've seen the rest of the posts.

All of the information the system will be collecting is stuff that GMs etc would already have access to, it's just a matter of compiling it all in an intelligent way.

The last step at the end, contacting players with relevant information to their concerns, could possibly pose a problem, but even that should be possible to handle in a way that doesn't infringe on private information.

Ysharros said...

Considering that games companies already have information about me that they can abuse, mining my behaviour in a game doesn't really bother me per se, though I suspect it would still get a reactive paranoia-twitch from many (myself included). Remember that kerfuffle about the EQ2 logs a few months back?

It's a very irrational thing for most (or at least for me): I don't mind being watched when I'm not conscious of being watched (stalkers excepted), but once I know I'm being watched I get all freaked out -- and that goes for more than just MMOs. Try typing accurately when someone is watching over your shoulder...

So what I was really aiming at was, aren't companies in a damned if you do/damned if you don't situation of they black box and tell players about it -- especially if, as you say, they then contact players? Would they stand to actually LOSE players from the initial wave of "zomg they're watching meee!"?

Figured I'd throw those out now while they're actually in my brain and not buzzing around 3 states over where I'll never remember them.

Grant said...

I DO know Blizzard sends a sad orc to ask my why I quit every time...

I think the other similarity (other than post-ship data mining) is pre-ship player testing. I know Blizz does insane internal playtesting and Valve supposedly grabs people off the street to try their game daily.

Basically, the more chances you give yourself to judge your players' feedback (and not just your own opinion on the game) the better.

Mike Darga said...

Yes, the Observer Effect is very strong. People will do all kinds of weird things as soon as they get the idea that someone's watching them. Combine this with the fact that most players have no real way to describe how or why a game makes them feel a certain way, and you start to realize that datamining is really the only way to see what's actually happening.

Testing is great too. I wish everyone would adopt kleenex testing as a basic practice. I'll probably post about both of these things in more depth later.

Tesh said...

Kleenex testing? Quien es?

I've been looking for way to data mine playtesting for a project I'm working on. I don't know enough coding to put it in, but I know it's important, especially while I still have time to do some tuning.

As for using data mining to diagnose MMO quitting... I suspect that by the time someone threatens to quit, some of the really juicy data is gone... unless you're recording everything and can go back and look for patterns. (Here I'm assuming there's some window where recorded data gets purged, if only for storage considerations, let alone privacy. If all data is present in perpetuity for sifting, never mind.)

Mike Darga said...

Yes, I'm sure there's some sort of upper limit on how much data you could reasonably keep around, but it should hopefully be very large. In that recent example, EQ2 turned over 60 TERAbytes of data to researchers.

Kleenex testing was the term we used at Maxis to refer to "disposable" testers. This is more like focus testing than like QA testers.

You bring in your family, or your spouse, or a group of high school kids, or a person from elsewhere in the company, and sit them down in front of your game. You see how they do, and possibly make changes, but you never use that person again. They're now contaminated by knowing too much about the game and you have to throw them out, like Kleenex. I'll write a more in-depth post on this later.

Tesh said...

Ah, gotcha. We do that with our games here in my little company. Having a game designer who works at a local college (as well as two other colleges nearby) means there are usually plenty of Kleenex testers on hand for any given project. That's even before making the data public. :D

Their experience with our games has been extremely valuable.