Leveraging Unity Cloud Build for testing large projects

This is a story about how we are using Unity Cloud Build internally and how it can make life easier for you, our users, as well. Read on to learn how we used to deal with large project testing and which awesome new possibilities are available now!

Once upon a time

During development of the massive Unity 5 release, and the extensive changes it entailed, we frequently ran into issues with importing and building projects. For Unity 5 we wanted a cleaner API, which in some cases meant we had to break backwards compatibility. Often, when importing an older project in Unity 5, we had to fix scripts manually. We were also hitting major bugs and regressions in graphics, physics and performance related areas.

Our testers do a very good job at making sure that projects import, build and run properly in every new build on all our supported platforms, but since we are constrained by time we usually used small projects (like AngryBots or Nightmares) to run these tests. These projects don’t cover many of Unity’s features – any of which might break in a new version – and they are nowhere close to the size and complexity of some of the projects developed by our users.

We were fortunate enough to have a few major studios share the full project folders for some of their completed games with us (for example, Republique) and we started manually importing and building these games on every new build during the beta and release candidate phases of development. We found and fixed many issues before any of our users would have been affected by them, but it was a tedious and time-consuming task.

This is how the testing process worked back then:

1. Install the new Unity build on Windows and OSX.
2. Open a large project and reimport it. Wait a long time and check back from time to time to see if it has finished (often the Script Updater dialog would require a confirmation prompt which meant even more waiting).
3. Fix any scripts or other issues and build the game for Mac and Windows Standalone.
4. Switch platform to Android. Wait a long time again and check to see when it is finished.
5. Run the game on Android.
6. Switch platform to iOS. Wait again.
7. Run the game on iOS.
8. Repeat from steps 2-7 for all the other large projects we had (7 in total).

This was usually done by one person and it could take a few days.

The glorious present

We quickly started discussing how we could automate some of this. While we were busy figuring that out, in another part of the company work was underway for the official release of what is now Unity Cloud Build. Cloud Build seemed like a perfect fit for automating testing of these large projects.

Fast forward to the present day,  and after we released Unity 5.1, and Cloud Build has been out there for a while, testing large projects goes through an entirely different process (described below with pictures for your viewing pleasure):

This saves us a tremendous amounts of time, since all it requires to make the build is a few clicks and all builds for all projects are done in parallel. As soon as the builds are ready we receive the notification e-mail, and testing can begin on all platforms. If anything fails during importing or building we also get a notification and we can act on it immediately.

Unity Cloud Build can also be configured to automatically poll a repository for changes and rebuild the projects automatically. It can rebuild projects on any supported Unity version.

A brighter future

Since we want to be able to scale up to testing more projects, the limiting factor now is running the builds on devices. The more projects we have in Unity Cloud Build, the more builds we have to install and run on devices.

The biggest problem with testing on mobile is device fragmentation (especially on Android) and we can only test on a few of the most popular models. We would like to know how these builds run on most devices (including some of the more esoteric ones). To that end, we are currently investigating services like TestDroid and AppThwack.

These services give us access to hundreds of devices and we can run our projects on any number of them. They offer a REST API that we can use to feed builds from Unity Cloud Build directly into them. What we get in return is performance data (CPU, Memory, Threading), screenshots of the game while running on the device, the ability to run custom testing scripts, get device logs and more.

By feeding all this data back into our own data warehouse, we can keep track of metrics across Unity versions, projects and devices and quickly pinpoint performance, rendering, and input issues.

Unity Cloud Build and you

Unity Cloud Build is the solution of choice for us when it comes to removing all the time-consuming tasks and bottlenecks involved in importing projects to newer Unity builds, switching platforms and building projects. But we built Unity Cloud Build first and foremost with you, our users, in mind. If you are just getting started on a new Unity project, sign up for the free Cloud Build option and see how easy it is to have us do the heavy lifting for you and share the final build results with your entire team. If you are a veteran Unity user working on one or multiple projects, you will find something suitable for you in one of our other licensing options.

So, what are you waiting for? Give Unity Cloud Build a try! It might just be the best thing that happened to you since we introduced the Cancel Build button!