Runtime Tests - Unity’s Runtime API Test Framework

Hi, my name is Kasper Amstrup Andersen, and I am leading the Toolsmiths Team here at Unity. The Toolsmiths Team consists of 6 developers, and we work on tools, frameworks, and infrastructure - and then we also spend a great deal of time actually testing Unity. The team is located in Copenhagen (Denmark), Odessa (Ukraine), and Helsinki (Finland). In this blog post I will tell you about one of the frameworks we are currently working on: The Unity Runtime Test Framework.

At Unity QA our passion is "Quality". We like testing and we love to be smart about the testing we do. We now have 4 teams of very dedicated and professional testers and developers that spend every day making sure that Unity "just works" for everyone. The teams are: Test Engineers, Test Developers, Toolsmiths, and Student Workers (we have previously written a bit about the teams). A huge challenge when testing Unity is actually breaking down the product in logical components where it can make sense to talk about test coverage. The main headache is Unity’s many platforms which are in many ways (but not all) orthogonal to the core feature set. To give you an example: We have areas and we have platforms, but then we also have platform-specific areas (and features). A testing matrix basically explodes.

In our Runtime API Test Framework we treat platforms as being completely orthogonal to features. This assumption allows us to make a specialized framework that gives a lot of coverage with very little effort. You write a test once; the framework will make it run everywhere.

Before kicking off, I’ll give you some numbers and quick facts:

• Today, we have >750 unique Runtime API test cases.
• More than 9000 Runtime API test cases (across more than 10 platforms/players) are run in a single automated build verification test suite. The number is rapidly increasing.
• We easily execute more than 150.000 Runtime API test cases per day.
• The framework runs on all of Unity’s supported platforms/players.
• The framework (and our build system) runs tests on multiple platforms in parallel. Average execution time for the entire suite is less than 30 minutes, and half that time is spent setting up the environment and building players (actual test execution time is about 15 minutes on average).
• The framework is also used by external companies to integration test their products with Unity players.

We have named our Runtime API Test Framework "Runtime Tests". So, what is a Runtime Test? It's basically a test case that verifies some functionality in the Unity Runtime Classes. The Runtime Test Framework allows us to write a test case in C# and then it will run on all platforms without any boiler plating code. The framework is integrated into our build system and developers and testers run all tests in the branches where they are working + in Trunk, our shared code repository, whenever there are changes. Some of the framework’s core features are:

• Write tests once. The framework will make them run everywhere.
• A library of player-side assertion methods that are supported on all players.
• Tests are fast (execution time ranges from milliseconds to a few seconds).
• Tests run in scene isolation. Each test is executed in a separate scene.
• Tests can easily be disabled, also on specific platforms only.
• The framework is connected to a powerful back-end where we have close to real-time monitoring of key metrics.
• Automation for new players can be added basically without modifying the core framework.

The Runtime Test Framework consists of the following modules:

• Unity.Automation: Functionality for starting and interacting with the Unity Editor.
  
• Unity.Automation.Players: Functionality for starting and interacting with players and for using the build pipeline.
• Unity.RuntimeTests.Framework: The core framework logic.
• Unity.RuntimeTests.Runner: The test runner. Will load the module that contains test cases and start test execution.
• Unity.RuntimeTests: Runtime API test cases.

The different modules, and how they depend on each other, is shown in the figure below.

The framework consists of runner-side part and a player-side part. The player-side uses a very barebones implementation (no use of interfaces, generics etc.) since it must run on all players (also players we might consider to add in the future).

An example of actual test execution is shown in the figure below.

The runner will load the test assembly (Unity.RuntimeTests) and then pass control to the framework (1, 2). The framework will start a Unity Editor process (3). Then, it will create (or copy in) all assets required to run tests, and start building a scene for each test (4). This is done by invoking saved delegates inside the Editor process. Pre-made assets are stored together with the test assembly. For tests using the WWW class the framework will start a webserver where [request, response] pairs can be saved during setup (5). At runtime a test can then make a www request which will be redirected to the webserver by the framework. Finally, a player is built and started, test results are consumed, and a test report will be generated (6, 7a, 7b, 8).

The reporting backend scans the test assembly (Unity.RuntimeTests) and monitors all tests. This allows us to track various metrics. An example is shown below.

This chart shows all Runtime Tests grouped by area. The horizontal axis shows "area" and the vertical axis shows "number of tests".

So, there you have it. That’s our Runtime Tests. We like them and they make writing tests for all Unity players easy and fun. And then they simply give a lot of "test-bang-for-the-buck". But this is just one of the many things we’re currently working on in the Toolsmiths Team. I look forward to share more in future blog posts.