Gamelab associate producer Scott Price delves into his take on the many forms of game testing he’s come to know during his career at Gamelab and beyond.

If you're here on our blog, you know that games need to be tested (a lot) before they ever get posted to a website for download or appear on a CD in a store. You've certainly tested games yourself, at the very least by "playtesting" as you play. But do you know how many kinds of testing a game often goes through?

When I came to Gamelab, I'd been a tester on a few projects, including a couple of games and a bunch of databases and websites. I knew about "quality assurance testing," and "beta testing," but in order to pioneer a full testing program, I needed to improve my 'game'. I dug around online, talked to other testers, went to all the QA-related sessions at GDC '07, and made a list of every kind of testing I could find. How many can you come up with? I found 16 different kinds of testing that might happen all in the course of a single game's development. If you want to look shiny in an interview for a position as a game tester, have these terms handy.

Quality Assurance (QA) Testing is a catch-all umbrella term. Anything that's making sure that the game that's released is fun, properly written, and functional is technically assuring its quality. Usually "qa testing" means testing for bugs, though, and if you say that you're a qa tester people will either ask what that means or they'll assume that you're hunting bugs. Often, that's the case: if you're a QA Tester for a game company you probably spend most of your day hunting bugs and writing bug reports, whether the bugs are typos on in-game text, glitches in the functionality of the game, or problems with the graphics of the game. Because it's such a broad term, I find it more useful to break QA Testing down into some sub-categories:

• Content testing includes anything that appears in the game, but isn't the way the game works. The text that appears in the game, art assets, and the storyline are all content. It's useful sometimes to separate this from Functional testing because you can start functional testing as soon as there's 5 minutes of playable game in front of you, but until alpha or beta a lot of the content is probably placeholder and not worth testing. Content testing comes in during late alpha and beta right up through release.
• Functional testing addresses the way the game works, and is the classic bug testing. It's where you'll find the coolest bugs, too, where your character skips levels, racks up infinite points, crashes your computer, or falls off the world. Functional testing can include ad-hoc testing and many other types; it's any time you're testing how the game works rather than what is in it.
• Ad-hoc testing goes by many names, but it's pretty straightforward: It's testing off the plan. A test plan for a game is based on the Design document and the functional specification for the game - it's how the game should work. Actual players, though, neither know the proper way to play the game, nor do they often care. They'll play however they want or can, and so they'll discover bugs that are outside the plan. A major part of QA testing any product should be ad-hoc testing by an experienced and creative tester, who will think of all the ways to exploit the system, combine power-ups, run the character where there's seeming no reason to go, quit when they shouldn't, and so on. Personally, I find this the most fun kind of testing, too, because you can get to see some of the wildest bugs where no one expects you to go.
• Documentation testing is a critical and often-overlooked part of QA testing. While 95% of your players will probably never look at the manual or the installation instructions, it's important that the instructions be there for the other 5% if for no other reason than having a call center do customer support for a user is often more expensive than the profit you made off selling it to them. If the in-game instructions, the FAQ, and the installer instructions are all correct, you can save some calls. Even better, if your team has testers check the internal documentation, so that the programmers know what the designers want and vice versa ... well, that team is a keeper.

Many kinds of testing come and go throughout the development cycle. When a game is still being designed, there's nothing digital to test but might be a paper prototype worth testing; later, when a project is about to be released, playtesting the core mechanic of a game isn't very helpful because you can't change the game so fundamentally. Hence, it's useful to break other kinds of testing down by when the QA department performs them.

Pre-production - Before programmers even start on a game, there are things to test about what the game will be.

• Playtesting is probably the most fun form of testing, but done well it's no less rigorous than the others. Playtesting evaluates how the game plays by playing it--- whether all of the elements of the game are balanced, whether a novice player understands how to play, and whether the players are having enough of the right sort of fun. While it's easy to do playtesting simply (just watch someone play and take notes), good playtesting involves a lot of careful work. Planning what the test is going to look for, what to tell the player, what to ask the player and how to ask it, what to look at in the player's unspoken reaction to the game, what kind of data to collect from the game itself about the player's performance ... you can break a sweat running a thorough playtest. In digital games, playtesting can and should start as soon as there are 5 minutes of playable game, though you have to be careful about what you're really testing. Players might really enjoy the way the game plays but be unable to get past the placeholder art in an alpha build. You can (and even should) playtest a paper version of your game- , if at all possible. Many of the best digital games were prototyped on paper in some way- even action games might have a trading or crafting element that is worth modeling on paper first.
• Focus group testing often doesn't count as QA at all, and instead comes under marketing or design. You get a group of people together, all focused around some feature: the intended audience for your game, or people who like some other game similar to yours, or people who think cilantro tastes like soap. Then you test something with them, often the core content or mechanic of a game, to see whether it appeals. Focus group testing is often done during the design phase of a game to make sure that all the time and money that you're about to spend on a game is going to come back to you. Mikey was a focus group tester.

Proto, Alpha, and Beta - Obviously, most testing occurs during production; that is, during the prototype, alpha, and beta phases of the project.

• Beta testing is a term that almost everyone knows. Technically, it means any testing that's done on a game that's in the "beta" stage of development. Often, though, it means the first version of a game that anyone outside the company sees... a 'public beta'. Public betas are great because thousands of eager fans will often find bugs that your small group of testers couldn't. It's also useful to distinguish beta testers from QA testers: beta testers might test from home, or casually on a single project, while QA testing is generally a full-time, trained job.
• Regression testing is a stage of testing. Once a bug has been fixed by the coders, it's returned to the QA department for regression testing. QA checks to see whether the bug is still there (regression) and then runs similar tests to see whether the fix broke something else. That second stage is often called halo testing; it involves no space marines, but rather tests all around a bug, looking for other bugs.
• Compatibility testing is one of the geekier kinds of testing, in my opinion: it's for the hardware mechanics. In order to get the list of system requirements for a game, you need to test it on a bunch of different machines, each with specific video cards, amounts of RAM, operating systems, processors, and so on. You can often start with a good guess (if a game was built for Flash, it has the requirements that Flash does) but you've still got to test from there. Compatibility testing pretty much has to come toward the end of development, when the game is coming together, because so much of the compatibility depends on exactly how the programmers work their magic. This means that if you can afford it, it's often good to run at least two rounds of compatibility tests: one early in beta to highlight some problems with plenty of time to fix them, and one late in beta or during GMC (gold master candidate, when the game is ready to ship) to really figure out exactly what the system requirements are.
• Load testing tests the limits of a system. The system could be the number of players on an MMO server, or the number of sprites active on the screen, the amount of traffic running over an internet connection, or the number of threads running in a particular program. It's most commonly used in the first sense, to mean the number of players or processes running on a server. Load testing generally requires a lot of people (like a beta test group) or software that fakes heavy activity.
• Unit testing is another pretty technical term, and it may not be the responsibility of the QA department. A unit test is run on a piece of code, often automated, to test whether it works at a basic level. For instance, if a piece of code is supposed to take in a username for an MMO and return an error if the name is already taken, then the unit test would automatically input a name that is known not to be used and one that is used and make sure that the proper response is given. Often programmers write unit tests for their own code, and QA only gets to see that part of the game once it has passed all of its unit tests. A good QA department, though, may help design unit tests to make them rigorous. Unit testing happens throughout a project... anytime the programmers are working!
• Soak testing is a very technical term that will make you look good in interviews but which might land you a tedious job. Soak testing involves running a game for a long time to see whether performance degrades. Very subtle bugs like memory leaks might not come out in playing the game for five minutes but could ruin a saved game or crash the computer after an hour and a half of playing. Soak testing is often automated, with software faking mouseclicks or otherwise making sure the computer doesn't go to sleep. Many soak test bugs end up being filed unfixed as PUB ("Psychotic User Behavior", a useful term, really) because a normal player wouldn't play the same Pac-man level for 175 minutes in a row. But if they do.....

Beyond Beta - Testing for a game doesn't always end when the game is released. These are tests that you might run in the weeks preceding launch or even after a game goes public.

• Compliance testing ensures that a game meets the standards for a game console manufacturer. Before Nintendo will make and publish a game, for instance, it needs to go through compliance testing. Because it's so strict, done by a third party, and often pass/fail without much feedback, compliance testing often comes very late in production. You want your game to put it's best foot forward, and the publisher needs to know that what they're approving is what is going to be released. Every public scandal like Hot Coffee or Pokemon's seizures makes compliance testing a bit more rigorous.
• Localization testing is all about translation... both in terms of language and in terms of culture. When a game made in Japan gets brought to America, testers ensure not only that the text in the game makes sense in English, but that idioms are translated, that visual gags make sense, and so on. Budgets are often tight for localization, so it can be a very difficult process. It often happens after a game has been initially released in the home country, because where it will be released may depend on its initial sales. This is very common to outsource because it doesn't make sense for every game developer to have, on hand, people fluent in 15 languages.

I should write an "End of the World As We Know It" parody with all these terms.

If you're interested in kinds of testing and game testing as a job, these links fed this article:

Wikipedia on "Game Testing."
Tom Sloper wrote the best article that I've ever read on game testers and testing .
Testing is a job, and at many companies, it's a really tough job.