Hi, my name is Kasper Amstrup Andersen, and I am leading the Toolsmiths Team here at Unity. The Toolsmiths Team consists of 6 developers, and we work on tools, frameworks, and infrastructure - and then we also spend a great deal of time actually testing Unity. The team is located in Copenhagen (Denmark), Odessa (Ukraine), and Helsinki (Finland). In this blog post I will tell you about one of the frameworks we are currently working on: The Unity Runtime Test Framework.
At Unity QA our passion is "Quality". We like testing and we love to be smart about the testing we do. We now have 4 teams of very dedicated and professional testers and developers that spend every day making sure that Unity "just works" for everyone. The teams are: Test Engineers, Test Developers, Toolsmiths, and Student Workers (we have previously written a bit about the teams). A huge challenge when testing Unity is actually breaking down the product in logical components where it can make sense to talk about test coverage. The main headache is Unity’s many platforms which are in many ways (but not all) orthogonal to the core feature set. To give you an example: We have areas and we have platforms, but then we also have platform-specific areas (and features). A testing matrix basically explodes.
In our Runtime API Test Framework we treat platforms as being completely orthogonal to features. This assumption allows us to make a specialized framework that gives a lot of coverage with very little effort. You write a test once; the framework will make it run everywhere.
Before kicking off, I’ll give you some numbers and quick facts:
We have named our Runtime API Test Framework "Runtime Tests". So, what is a Runtime Test? It's basically a test case that verifies some functionality in the Unity Runtime Classes. The Runtime Test Framework allows us to write a test case in C# and then it will run on all platforms without any boiler plating code. The framework is integrated into our build system and developers and testers run all tests in the branches where they are working + in Trunk, our shared code repository, whenever there are changes. Some of the framework’s core features are:
The Runtime Test Framework consists of the following modules:
The different modules, and how they depend on each other, is shown in the figure below.
The framework consists of runner-side part and a player-side part. The player-side uses a very barebones implementation (no use of interfaces, generics etc.) since it must run on all players (also players we might consider to add in the future).
An example of actual test execution is shown in the figure below.
The runner will load the test assembly (Unity.RuntimeTests) and then pass control to the framework (1, 2). The framework will start a Unity Editor process (3). Then, it will create (or copy in) all assets required to run tests, and start building a scene for each test (4). This is done by invoking saved delegates inside the Editor process. Pre-made assets are stored together with the test assembly. For tests using the WWW class the framework will start a webserver where [request, response] pairs can be saved during setup (5). At runtime a test can then make a www request which will be redirected to the webserver by the framework. Finally, a player is built and started, test results are consumed, and a test report will be generated (6, 7a, 7b, 8).
The reporting backend scans the test assembly (Unity.RuntimeTests) and monitors all tests. This allows us to track various metrics. An example is shown below.
This chart shows all Runtime Tests grouped by area. The horizontal axis shows "area" and the vertical axis shows "number of tests".
So, there you have it. That’s our Runtime Tests. We like them and they make writing tests for all Unity players easy and fun. And then they simply give a lot of "test-bang-for-the-buck". But this is just one of the many things we’re currently working on in the Toolsmiths Team. I look forward to share more in future blog posts.