One month from today, the video-gaming world will converge on Los Angeles for the annual Electronic Entertainment Expo, better known (and more quickly referred to) as E3. This is the time when developers and publishers pull out all the stops, unveiling and demonstrating their premier games for the year ahead.
As we prepare for E3 during the next four weeks, DailyGame will publish a four-part series of conversations we recently had with Korey Krauskopf (shown at right), the Test Manager for the Roleplaying, Adventure and Technology Studio at Microsoft Game Studios. E3 2003 is looking to be a breakout show for the Xbox and its role-playing lineup (Sudeki, Fable and others), and that?s not counting the games Microsoft and its outside developers have yet to announce.
For this first interview, we broke the ice by asking Korey more about how games are tested at Microsoft, how the Studios? teams work together and how it feels living the sleep-deprived life of a game tester.
DailyGame: How?d you land a job as a professional game tester?! Did you get a specific degree, did you work up from janitorial in the campus cafeteria, and what?s your mother think?
Korey Krauskopf: It all began in a rural, peaceful village? wait, that?s too far. Short answer: I stumbled into computers, found I was good at it, wound up doing NT support for Microsoft, then stumbled into testing and found it to be a pretty cool discipline to focus on. After learning the ropes of software testing for a few years, a friend in the games group said they had a test lead spot opening as they were ramping up for this console push. If working in the computer field was cool, and then focusing on testing in the computer field was extra cool, then testing GAMES was just about the pinnacle of cool. So I became a test lead, and about six months ago finally rose to my proper level of incompetence to become Test Manager.
Lots of gamers would say you have the dream job. Are there any aspects of the gig that might surprisingly turn people away? Don?t worry; we won?t tell your boss.
No worries, I tell my boss constantly. There are three scopes I?m now intimately familiar with for games testing: Test Engineer, Test Lead, and Test Manager. These are somewhat specific to Microsoft though, as other games companies don?t necessarily apply the rigor to building a solid test effort as Microsoft does.
Loosely defined, a Test Engineer owns areas of a game thoroughly (such as AI, performance, or specific levels), a Test Lead owns a project, and a Test Manager owns the test efforts of a studio (games of a certain genre, or other such logical grouping). I tell you this as a way of answering the question (in case you were wondering where the hell this was all going).
To that point I?ll say that most people think of games testing as playing a game 10 hours a day and reporting things that are wrong with it, essentially making it better in the long run. The person who has this as a dream job is going to be a passionate gamer, and as such will pour their passion into the game they are working on. This is can be great for the game, but hard on the person. The down side for each of the roles then:
- Test Engineer: what you feel most passionate about may not make it into the final version of your game for various reasons. This is really tough because everyone puts their heart and soul into a game, but in the end someone has to have the grand vision and make the call on what it will be and what it won?t be.
- Test Lead: Your job becomes more about tracking work, making sure things are done on time, and that your people have all the info they need to be as effective as possible. Because of this, some days you?ll go home and realize you haven?t actually been able to play the game for a few days, just reviewed bug reports, talked to developers and wrangled schedules.
- Test Manager: No down side. I eat ice cream all day and play the games with my toes (usability test case, ya know).
You focus on the RAT (Roleplaying, Adventure and Technology) Studio segment of Microsoft Game Studios (MGS), but how much overlap do you have with games in other genres? Say, for example, you wanted to ?test? Halo 2?.
This is where it becomes really great to be in the test function at MGS. We?ve got all these killer titles in the works in many different studios, but the test teams all lean on each other for help from time to time. So if it happens you are needed to ?test? a little Midtown here, a little Halo2 there, maybe some Rise of Nations on the side, I just have to be sure to enlist the other groups to help out a bit with Sudeki so it?s all equitable in the end.
To be clear though, some games (like Halo2, for instance) keep a very tight lid on their exposure. They don?t want any details being leaked prematurely thus ruining the surprises they have in store. So the only people who have a lot of access are the ones directly on the team.
How many people are currently on your testing team?
I?ve got 8 test leads, and around 30 people on the team total, though this varies a lot depending on what projects are at what point of their life cycle.
How many testing teams are there for Microsoft Game Studios? What genres does each focus on?
Many. Unfortunately I couldn?t give you an accurate count, as I?m mostly focused on the teams in my studio, not the ones in the other studios. As for genres, we seem to cover all the major ones, but the focus is now more about making the best titles in the world than about covering a genre. This may sound a little PR-ish, but that?s the honest direction we are all pushing in.
Is there a separate team that tests third-party games, or is there more than one team so each group can focus on a specific genre?
Each game that Microsoft publishes has its own test team dedicated to it, whether that is first-party (developed in house) or third-party (external developer). Each team will start with one (the test lead) and grow to around 8-14 dedicated testers depending on the scope. In addition to that, there are shared resources that each test team can call on for specific efforts. Not sure if that answers your question enough, but without writing a dissertation on Microsoft test procedures, that?s probably as clear as I can be ;-).
What?s your team?s schedule like? Do you find yourself testing three, four or more games at a time, or do you focus on one game for a week, then the next game, etc.?
Since each of my test teams ?owns? a game, each studio has many games in progress at any given time. So while I split my focus throughout the day, the teams stay focused on one at all times. Besides the occasional ?help? given to other games as alluded to earlier.
How long would you say you generally spend testing a game? Does it vary by genre and/or platform?
This is actually a great question that gets frequent analysis as Microsoft tries to figure out the best way to go about this; how long to schedule for each phase of a game, how to estimate content and scope against projected bug find rates, fix rates, game play length, and how much time a game needs in the ?polish? phase. If you looked at the time a test lead will stay focused on a project, a good guideline could be around 2 years. However, the bulk of the testing happens later in the cycle, so the fully staffed test team will be on the project closer to a year in length. Surprisingly it doesn’t vary by platform as much as one might expect, but part of this has to do with the fact that MGS has built a great config lab that can cover the majority of configuration issues, so the core team doesn’t have to.
As for varying by genre, again, not so much as you might think. Every genre can argue that their game is unique with unique challenges. And they are all right, too. But the actual staffing doesn’t vary wildly. Again, some of this is mitigated by the fact that we can cover a lot of tasks with our shared resources.
The major exception I?ve found to this is MMOG (Massively Multiplayer Online Games). These are unique in that you have a live service to run and regular content to create. Time, team size, and process vary a lot for this.
How many stages/builds of a game do you go through on a typical testing schedule?
Depends on the game. Most games are done on a milestone schedule with a build devoted to each planned milestone, and this can be anywhere from 5-20 or more, depending on how the planning goes. As for stages, most games go through an early stage working toward a prototype, then work toward getting into full production mode, then move into being feature complete, then asset (or content) complete, then into the end-game where things get really interesting ;-). Again, I could write a dissertation here, so I?ll just leave it at that.
How early do you typically begin testing a game before it?s even announced, or before preview builds and demos are released?
It would be rare for Microsoft to have not had some test attention on a product a year before it went public with preview builds. The test team is all about analyzing, evaluating, and reducing risk. So before a game goes public, there?s a significant amount of analysis that the test lead will do to help everyone on the project better understand the risks, probable timelines and overall work involved to achieve the designer?s core vision. So when Microsoft announces a game, it?s usually had quite a lot of work done on it already from many different disciplines within the studio, including the test group.
Does testing a game in progress spoil the final experience of playing the finished product?
Another great question. Honestly this was one of my worries when I got the job in the games group. People told me that I?d go home at night and the last thing I?d want to do was boot up a game. At the time I was having regular LAN-nights at my house to host StarCraft tournaments, so I couldn?t even begin to picture such a fate.
What I?ve found is that testing a game gives you an entirely different appreciation for what it took to get it to it?s final stages, and of all the things going on behind the scenes, so you can enjoy it at a different level. It?s also held true that there are so many types of games out there, that if you are burnt out on, say, Half Life, you can still be feeling the itch to destroy someone in Age of Myth. Heck, even when you burn out on the complexities of today?s games, you can go back to the simple joys of something like Lemmings. So one can never really be burnt out for long.
But in all honesty, sometimes I find that indeed, I get home and the only thing I want to do is draw with my son, or chill out with a book, and stay far away from any glowing screens. That usually lasts a couple hours ;-).
Do you ever test a game for so long during development that you avoid it like the plague once it ships?
Right after I?ve shipped a game (or even just helped on it), I?m jazzed to get the shrink-wrapped copy in my hands, and I grin like an idiot parading it in front of friends, neighbors, strangers, inanimate objects and small rodents; I can?t wait to get it home. Then as soon as I?ve seen it running, I probably won?t touch it for a couple months. After I?ve been away from it for a bit though, it?s great to go back to it and play it like the completed game it is without having to pick it apart from a testing mindset anymore.
Stay tuned next week for DailyGame?s second conversation, in which Korey talks about the differences in testing for the PC and Xbox and Microsoft?s early stages of Xbox testing and planning.