December 21, 2008

Think more like a scientist

Game design is not a science, but like almost anything, science teaches skills which can be extremely useful in game design. If you've followed good design practices, but your design isn't working how you expected it to, it's a good time to put on your scientist hat and start investigating.

Before starting to track down a problem, take 5 minutes to define the problem, and make sure you and your coworkers actually agree. This will save tons of time. If the cause of the problem is exactly what you need to deduce, at least agree on what the symptoms of the problem are (for example, "nobody seems to want to play a tank in our game").

The scientific method

This is actually a very broad topic, but the 4 steps that we all learned in 5th grade science class are described by the Hypothetico-Deductive Model:

  1. Gather data ( observations about something that is unknown, unexplained, or new )
  2. Hypothesize an explanation for those observations.
  3. Deduce a consequence of that explanation (a prediction). Formulate an experiment to see if the predicted consequence is observed.
  4. Wait for corroboration. If there is corroboration, go to step 3. If not, the hypothesis is falsified. Go to step 2.
Let's walk through this process for our class shortage problem, which is a pretty common type of problem for MMO designers.

1. Gather Data

Is there actually a tank shortage? Everyone at your company seems to think there is, and there's a lot of wailing and gnashing on the message board from players, but it never hurts to actually check the data.

Every development team should be able to get access to statistics related to how players are playing and what they're doing. Engineering and Operations should be able to help you out with this.

Say you have 5 classes. Are 1/5 of your players playing tanks? You'll need to define some rules for how this can be evaluated. You can't just check how many tank-class characters have been created. Maybe they're not used. In fact, the ratio between actively used tanks and abandoned tanks may be useful info to you.

If more than one class can tank, or if there is a class that can choose through spec to either be a tank or not be a tank, you'll need to find a way to evaluate that as well.

Eventually, you'll have a clear definition of what it means to be playing a tank, and how many poeple are doing it, in relation to how many people you'd like to be doing it. If the ratio doesn't match your intentions, then you do in fact have a problem.

If the numbers match up to your intentions, but people still complain, it may be possible that you need to redefine your problem. This is why spending time defining problems and gathering data is so important.

Your intentions could be wrong. If there are 10 classes in the game, and only 1 of them can tank, and each team of 5 requires 1 tank, you won't have enough. Or perhaps it turns out that players find it necessary to bring 2 tanks to every fight. These are other flavors of the problem that you should also consider: maybe you need to add more tanking classes to the game. Maybe you need to change your instances so that 2 tanks aren't considered necessary. Keep digging through data until you identify a problem that sticks and your team agrees is worth solving.

2. Hypothesize

Assuming we decide there actually is a tank shortage, it's time for us to brainstorm reasons this shortage may be the case.

  • Tanks don't have enough interesting things to do in a fight, and are boring.
  • Tanks have too many difficult things to do in a fight, and are too high pressure.
  • Tanks aren't easy to level up with, because they deal too little damage.
  • Tanks aren't effective in pvp, because abilities like taunt don't affect enemy players.
Rank these hypotheses in order of likelihood. If your game emphasizes pvp a lot, the pvp cause may be a likely solution. If your game has a lot of solo content, it may be that leveling isn't fun for tanks.

Decide which problem to try fixing first. At this point, you'll need to back through the first few steps again, and agree exactly what the problem is, as well as determine some way to gather data on these sub problems, leading back to this step, developing hypotheses.

For example, tanks may be boring. If we can agree that may be true, we can gather data on this problem. How do we do this? Let's monitor some expert players and some unskilled players fighting through an instance, and gather data.

  • How many times does the tank switch targets compared to other classes?
  • How many times a minute does the tank use powers compared to other classes?
  • How many different powers does the tank end up using per minute?
  • How much chatting do tanks do while tanking, as opposed to other classes?
  • Etc.
Keep discussing more and more specific problems, until you feel you've arrived at something you can actually experiment with solving, e.g. "Tanks are required to use too few powers in most fights, and are boring."

3. Predict/Experiment

Make a prediction as to how to solve the problem: "Adding some more tanking powers to the tank class will make them less boring, and thus more desirable to play."

Develop an experiment to see if this is true. For our purposes this can be as simple as adding some new powers to the tank on one server, and seeing if usage rates improve on that test server.

It is important to follow some rules of good experimental design though: Make sure there is a control group of tanks with the old abilities, and that there is only one variable changing at a time. You can't test this change the same week you make overhauls to a bunch of other classes, and then draw useful conclusions about the fact that more people seem to be playing tanks. Solve one problem at a time.

4. Corroborate

If people seem to think playing a tank is much more fun with more abilities, the experiment has succeeded, but we still need to figure out if we solved the real problem, or if we solved the problem under all conditions.
We can add those powers to tanks, and wait to see some new data come in for tank usage. We may have solved the problem, but don't forget that we initially had a list of 4 problems:
  • Tanks don't have enough interesting things to do in a fight, and are boring.
  • Tanks have too many difficult things to do in a fight, and are too high pressure.
  • Tanks aren't easy to level up with, because they deal too little damage.
  • Tanks aren't effective in pvp, because abilities like taunt don't affect enemy players.
We may now see that people start playing tanks once they've leveled up, but that they are still very unpopular for pvp and leveling. Maybe there is a 5th problem that we haven't identified yet. Or maybe new players have started playing tanks more, but existing players may still need to be educated on the new changes before their behavior changes.

Of course, you may always end up solving a problem too well, and have to tune it back down a little bit. Nerfing overplayed classes or buffing unpopular classes is continuous process in games. No matter what you do, some players will always deny the problem was solved, say the problem was solved too well, or claim that the problem never existed in the first place (and often all 3 opinions will be present in a single forum thread).

Just don't even consider any problem solved to the point that you have to stop paying attention to it. This this process can help you refine the game further and further over time to match your goals, or even change to match new goals.


Tesh said...

Huh. I missed this one earlier. As an avowed fan of polymath thinking, I applaud the notion of being more intellectually rigorous in design. Nice article!

Mike Darga said...

Thanks, I like this one too. There is a fair amount of old stuff that may be just practice, but from time to time maybe i'll dig up or rewrite something older that I know nobody ever saw.

I intend this blog to be the kind that is friendly to digging around in, but I think most people don't ever have time to do that sort of thing.