Simply put, it's testing by creating games. We prepare for it by creating a number of scenarios – a short pitch for a game project. Examples could be “3D Racing Game”, “2D Platformer” and so on. Then we group participants into teams, assign each team a scenario and then the teams have one week to create a game based on that simple premise.
For testing our most recent release (at this time of writing), Unity 5.5, we had 14 scenarios and 53 participants, including testers, student workers and developers.
But scenario testing is more than a game jam. It's a testing phase with a very specific goal: Find bugs by using the Editor in a credible way. We have made a breakdown of Editor features and platforms and mapped it to the different scenarios to get as wide coverage as possible. We make sure that features are covered by more than one scenario. By having scenarios be different game genres, we ensure that the features are used in different ways.
Areas mapped to different scenarios/teams for 5.5 Scenario Test Week
Different test approaches find different problems. We have more than 60000 automated tests with millions of executions each month. Our manual regression suite – Full Test Pass – consists of thousands of manual test-cases that provide a comprehensive sweep of the functionality in Unity. This is all great and invaluable! But broadly speaking, these kinds of tests tend to exercise functionality in isolation. Some errors appear when you combine features. Or they appear over time. Something might work the first five times you do it, only to fail the sixth. Furthermore – again, broadly speaking – test automation and manual test-cases repeat the same test over and over again. And while the scope of what we achieve with automation is impressive, it's still only a tiny fraction of the theoretical possibility space for how Unity can be interacted with.
With Scenario Testing we address some of these shortcomings. When you create a game, you don't simply methodically go over each feature of a product. You get repetition of actions, you get randomness and the Editor will be interacted with in ways that are impossible to predict when designing a feature or even a set of tests. And most importantly, the Editor is used in a way that is credible. Even if it’s an approximation, we get to feel some of the joy and frustration that our end-users feel. This is powerful. Chomping on your own dog-food is always a good thing…
Bugs reported during 5.5 Scenario Test Week
Here is an example of the kinds of feedback yielded by scenario testing. One of the scenarios was to create a Real Time Strategy game (screenshot below). It was intended to obtain coverage for AI, Particles and Terrain. However, once the game was taking shape, naturally the idea came up to add multiplayer. And very quickly the team ran into problems adapting their game to the current networking system. But the upside was that they were able to take this feedback to the networking team who could use it to inform future decisions regarding networking in Unity. We got this feedback because the conventions of a particular game genre inspired us to use our game engine in a certain way. And we got it even though networking was not part of the coverage scoped for the scenario. Automation or manual test-cases will not give you this kind of information.
We are continuously looking for ways to improve how we do Scenario Testing. One thing we are looking at is continuing work on some of the scenarios across multiple Scenario Test Weeks (i.e. multiple versions of Unity). Until now, we started over with new game productions each test-phase. We hope that this can give us a better approximation of both upgrading Unity during a production and also working on a project for longer than hacking the first prototype together.
We don't presume that our little scenarios can accurately replicate the experience of working with Unity on a real project. Still, we feel that we can make a pretty good approximation Scenario Testing. And – not to forget – it's actually a fun way to test Unity.
Here are a few small samples of projects that were created during Scenario Test Week for Unity 5.5. Note that in addition to the coverage mentioned for each project, all projects also used Unity Collaborate and Unity Cloud Build. We dog-food our services offerings as well. Asset Store is also tremendously helpful as it allows teams to use high-quality assets in their projects. Not only does this boost the production value of the games, it also contributes to testing by allowing more complex and realistic data/assets to be used.
FPS game Team size: 4 people. Target platform: Desktop. Focus areas: Audio, General Graphics, Substances, Post-processing
3D RTS Team size: 4 people. Target platform: Desktop. Focus areas: AI, Particles, Terrain/SpeedTree integration
2D JRPG Team size: 3 people. Target platform: Android. Focus areas: 2D, UI, Particles
2D Platformer Team size: 3 people. Target platforms: iOS, TVOS, PS Vita. Focus areas: 2D, Animation 2D, Physics 2D
VR Museum Team size: 2 people. Target platform: VR. Focus areas: Graphics - general, Lighting, Post Processing
Slender Man clone Team size: 3 people. Target platforms: PS 4, VR. Focus areas: Graphics - general, Terrain/SpeedTree integration, Substance integration