Last Monday we brought you the first of four interviews with Korey Krauskopf, the Test Manager for the Roleplaying, Adventure and Technology Studio at Microsoft Game Studios (MGS). The interview was an icebreaker of sorts, with Korey telling DailyGame readers about all of the hardship, heartbreak and hellacious tasks (yeah right) of being a paid game tester.
In this week?s interview, we delve a bit deeper into the mind of the Microsoft game-testing guru, hearing his thoughts about testing for the Xbox versus testing for a PC. He does, after all, test games for both. And he?s been busy testing several un-announced games on Microsoft?s E3 slate that, in his words, are ?pretty freaking killer.?
What the heck are they? You?ll have to tune in during the next two weeks to see whether Korey springs any clues. For now, sit back, relax and revel in his unedited Xbox vs. PC game-testing comments.
DailyGame: How does testing a game for one platform affect your approach to testing if/when that game is ported to another platform? For example, Halo for Xbox vs. Halo for PC.
Korey Krauskopf: I’ve been involved more from the sidelines on porting efforts, but I’ll answer as best I can from what I gleaned from watching this process. For example; porting RalliSport Challenge to PC was an educating experience, even though I didn’t get to lead it. A co-worker of mine at the time, Dustan Gourlie, had several advantages while taking this on; he could analyze the Xbox version’s bug database to find where bulk of the issues had been, he could talk to the team and find out what processes worked well and what didn’t, and of course he had well understood gameplay and dynamics to work with. In addition he could utilize the tools that the original test team had built and leverage them against the port effort.
On the other hand, he had to treat it like a full product effort (ports are not easy, nor quick) and had to deal with the uncertainties of new features (huge things like Multiplayer), platform issues that hadn’t existed before, performance tuning for the minimum config, and a really tight schedule because you want the game to come out as soon after the first platform as possible. Also, he had to ensure that the port, with all its changes, held true to an unchanging mirror (the released Xbox version) for things like difficulty, balance, control, etc.
I guess a good way to summarize it would be that you have some advantages going into a port effort, but those advantages will probably have hidden costs that you won’t discover until well into the effort, and you’ll have unexpected challenges, so you have to plan accordingly. Lastly, you have to make sure everyone involved treats the port effort with the seriousness and respect it needs to ensure you get the time, resources and support to do it right.
Generally speaking, what are the primary differences between testing a PC game and testing an Xbox game?
The most obvious and major difference would be the amount of configuration testing. Depending on your target minimum spec for a PC game, you’ve got a huge matrix of OS, CPU, memory, audio and video concerns to cover. As mentioned before, MGS has some great folks that give us statistics on what the most popular hardware is out there and provide a lab of configurations for us to ensure the most global coverage.
After config, the next major difference is the tools available. With a PC you have pretty much everything you can think of at your disposal. Want to watch the file reads and writes going on during the game? No worries, there’s a tool for that. However, all the tools that are available are PC-generic, not game specific.
For the Xbox this need had to be filled from the ground up. This means there were fewer tools in general, but the tools that were created were very specific, focused and useful.
How does the number of hours required to test a PC game differ from the amount of time you and your team spend testing Xbox games?
Not a big difference, really. Mostly due to the MGS shared resources and tools available. The difference is more in the game, how it’s designed, and the game’s features and breadth, not on the platform.
Do you prefer testing PC games over Xbox games or vice versa, or do you even have a preference?
That’s kind of like asking whether I like playing console or PC games; they are both pretty cool experiences, so ideally I won’t be limited to one. At first I was excited to work on console games, because dealing with config issues is a huge problem and working with a locked configuration greatly increases the likelihood that a bug found on one system will reproduce elsewhere (no more bugs resolved as “works fine on my machine!”). However, working on a PC game, you sometimes have greater control and exposure to what is going on beneath the covers, which is great too. So they both have their joys and pains.
What are the differences in testing an online vs. offline game?
Online games take more resources to cover, no two ways about that. Along with the technical implementation, the connection types involved (and thus another layer of config testing), online games have yet another path of user input (specifically, multiple users at one time), and user input is where things break. For example: if you run a game through a predictable pattern, you’ll get much fewer bugs because you know what will happen and when. Once the user plays with it, you get things happening that nobody ever expected, thus you have bugs. Then you put two people playing it together, and things go downhill fast. Add to that further problems of latency, synchronization, etc, and you get the picture.
Do you find yourself in longer testing cycles for a Live-enabled game than you do for non-Live games, or is the testing just different in nature?
Same answer as above; Live is an online feature with unexpected interactions happening. However, longer testing cycles are never easy to come by (games gotta ship, right?), so you end up trying to be smart about it by adding more people to the test team earlier, getting the live features added as early as possible, etc.
It boils down to being another example of the simple equation of ?Features = Time multiplied by Resources.? If you add features, then the other side has to move, too.
How long did you test MGS? PC games before beginning to test Xbox games, and what lessons did you learn doing that that you could apply to testing console games?
When I first started, I was on the racing group and got to help out with Midtown Madness2 and Motocross Madness2. After those, I worked on an internal title that ended up getting canceled before taking on RalliSport Challenge as Test Lead. Much of the processes learned from testing PC games helped form the processes and tools we used to approach Xbox testing.
For instance, we knew we’d need some way to test performance of a game streaming off of DVD media, and that the layout of the files from outer track to inner could make a huge difference, so we started pushing for tools to simulate this early. Another example was controller input: the racing team had a lot of expertise in testing the various controllers available to PCs, so there were a few people in the group that were very proactive at specifying how Xbox games would ideally be set up, and the team had already built a lot of practices and technical knowledge about what subtle nuances and tweaking need to be done to make control input simple yet deep for a game.
Overall there were a vast number of processes and knowledge that were immediately applicable to the Xbox games, but there were just as many things that were new and had to be learned quickly and refined as the projects went on.
How long before the Xbox and first Xbox games were shown publicly did you begin testing games? Did pre-Xbox testing begin sooner than ?normal? because Microsoft was new to the console world?
Anyone familiar with the two years prior to the Xbox launching is probably pretty well informed of how this process went, because details seemed to constantly show up on news sites. However, in general we got “preview hardware” a little more than a year before launch to develop (and thus, test) on. During that time there were regular updates to this hardware as chips were produced and the OS was refined.
As for whether this began earlier than normal, I’d have to say no. Most teams tried to build buffer into their schedules to deal with the changing hardware and OS platform, but when it came down to it, the earlier games were all trying to make the launch date, and that wasn’t moving ;-).
Do you find it easier to test console games because the Xbox hardware is static?
Hardware specs certainly affect the test effort. In that way, testing PC games is harder. However, there are other aspects of testing Xbox games (such as certification testing) that can be just as much work.
Given the current release estimates for the next Xbox, when do you anticipate first testing Xbox 2 games?
Details are sketchy; seems game sites have more information about this than I do sometimes ;-). This is on purpose, though, as Microsoft is very conscious about not treating its own publishing department any better than other publishers, so we don’t get privy to information that other folks don’t have. As for how soon we’ll begin testing, I’ll have to stick to the pure conjecture answer of “as soon as humanly possible,” which usually means as soon as a title gets signed up.
You?ve tested RPGs for PC. How do you think RPGs differ for consoles, and what are your thoughts about the future of RPGs on Xbox, a console that is arguably ?lacking? in that category?
So to be clear here: I joined the RAT group as the test manager about six months ago, and since I have a high-level view of all the projects, I mostly have to soak up the expertise from my leads and their teams. That being said, I have some seriously experienced people who work for me, so I should be able to give at least a meagerly informative answer here:
RPGs on consoles differ in a few respects. Some are due to technical limitations, and some are due to the designer’s best information of their target audience. Past sales data drives a lot of the monetary decisions of what games will get the go-ahead to be made. A few more specifics on these two points:
Technical: When designing for a console, you have to always think of the interface and how all the goodies you design will be found, used and managed. Just like with an RTS or an FPS, when you have an RPG you are limited by the buttons and triggers available, as well as the resolution of the display. Designing for a PC gives you a much broader toolset for input, so you’ll find PC games naturally gravitate toward more complete control, simply because it’s possible.
Target Audience: There is a huge amount of research that is done on what sells well and why. This directly affects what games get ultimately brought to market. To generalize here, action-based and story-based RPGs excel on consoles, while character-based and multiplayer RPGs excel on the PC. These lines are starting to blur though, so I don’t think it’s that simplistic anymore.
Fun question: what?s your PC like in the office, how?s it compare to your PC at home, and what do you play on your free time?
Office machine is a P4 2.4G 512Mb with GF4 Quadra. At home I’ve got a horrid compilation of parts that I upgrade every few months, it seems. I won’t bore you with Mobo stats or those gory details, but I’ve got an ATI 9700 and a gig of RAM, so I can handle whatever work I bring home with me. As for what I play, I find that I have to do a lot of competitive research at home, so that includes anything with some graphics, an interface and a rough theme to it. Yep, it’s a hard life being me.
Yeah, we?d sure say so. Be sure to come back next week for the third installment in DailyGame?s four-part Microsoft Files interview series, when Korey gets down and dirty with massive details about Microsoft?s upcoming RPG Sudeki.
