Search Unity

Accelerating player builds with an incremental build pipeline

December 20, 2022 in Technology | 13 min. read
Accelerating player builds with an incremental build pipeline | Hero image
Accelerating player builds with an incremental build pipeline | Hero image

For the past two years, our team at Unity has been working to speed up both the script compilation pipeline and the player build pipeline – the parts of the Editor that run when you hit Build in the Build Player window – by making them incremental. My name is Jonas Echterhoff and I’ve worked on the engineering team at Unity since 2005. In this blog post, I’ll cover how we made incremental builds happen.

What are we trying to solve?

To compile scripts and build players in the Editor, Unity previously used custom C# code to run a sequence of steps, often invoking external tools. Over the years, builds (and the code used to implement them) have become more complex, and more time-consuming for our users. To solve this, we sought to optimize the time spent on builds for a better user experience, ensuring that:

  • When doing subsequent builds, the Editor only reruns the steps that actually need to be rerun because their inputs have changed. In other words, hitting Build twice, without making any changes in between, should be as close to instant as possible for the second build.
  • The Editor runs build steps in parallel, whenever possible, to fully utilize all CPU cores.

What did we do?

General idea

We needed to stop treating builds as a list of sequential steps executed linearly, and start using a proper build system instead; one that understands the inputs and outputs of each build step, and the dependencies between steps. Such a dependency structure is called a “build graph.”

Fortunately, we already have a graph-based build system at Unity, mainly used for building the Editor and Player runtimes. This system is called the Bee build system and it uses a C# API to describe the build graph. Seeing as it supports all the platforms we care about, and that it’s already familiar to many internal Unity developers, we decided to reimplement the Editor script compilation and player builds using Bee.

Implementation details

At the heart of the Bee build system, there are two programs:

The Build Program

The Build Program is a .NET program that takes, as input, all of the source and configuration files on disk and uses them to generate a build graph. The build graph is a description of:

  • All the things that need to be built (or, “build nodes”) 
  • The commands to run in order to build them
  • The dependencies between build nodes

The information is then stored in a .dag.json file. There are different build programs for building different things – one for script compilation and one for each platform player build.

The backend

The backend program, called bee_backend, reads the build graph in the generated .dag.json file and checks which input files on disk have changed since the last build, and which corresponding build nodes need to be rebuilt.

To do that, the program launches processes to run all the commands required to rebuild the necessary nodes, and distributes them in parallel across the available cores (wherever the dependency requirements permit). bee_backend is based on the Tundra build system, written by Andreas Fredriksson.

Rollout

We began working on this by rewriting the script compilation pipeline to use Bee. This enabled us to replace a lot of custom Editor code used to spawn Roslyn C# compiler processes with a simpler and more efficient build system.

This change first shipped in the Unity 2021.1 Tech Stream – but note that there are long-term plans to replace the entire script compilation system in Unity with an MSBuild-based system for better .NET ecosystem compatibility.

Next, we converted the player build code to run in Bee. The bulk of this work involved rewriting all the platform-specific build code. This took some time, but as a result, the code is much faster, cleaner, and more easily maintainable, as it is now shared between platforms.

We rolled out this change over a few Unity releases:

  • Desktop standalone players, WebGL, Android (2021.2)
  • iOS/tvOS, all Xbox platforms (2022.1)
  • All remaining platforms (2022.2)

We also integrated Burst into the incremental player build pipeline so that the Burst compiler could run in parallel to other build tasks and only run if assemblies have actually been changed. This shipped with the Unity 2022.2 Tech Stream and Burst 1.8.0 release.

Data builds

The build graph described here includes all the steps for building players that can be run as external tools with clearly defined dependencies, such as:

  • Script compilation
  • Il2CPP conversion and C++ compilation
  • Code stripping via UnityLinker
  • Burst compilation
  • Code signing
  • Packaging and compression

It also includes many platform-specific tools. It does not, however, encompass the serialization of scenes and assets into player data files (which is currently done by native code in-Editor).

Unfortunately, there’s more to building a fully incremental data build pipeline than adding nodes to a build graph. Due to the nature of Unity’s data file layout, each scene depends on all previously built scenes, so we cannot simply extract small changes to build independently of everything else. This is a shortcoming that is being addressed by other teams at Unity, and we hope to see significant improvements in the coming years.

But even with these constraints, our team didn’t want to force any unnecessary work when users hit the Build button. That’s why we decided to automate the feature formerly known as Script only build. Once a checkbox in the Build Player window, this feature served to tell Unity to reuse the player data from the last build (ignoring any changes), and to only recompile scripts instead – in order to allow for faster iteration times on code. But we felt that this put the burden of tracking changes to player data on the users, a task that computers can handle with ease.

In the new incremental build pipeline, we track whether any of the scenes or assets being used in the player have been changed since the last build. If there are no changes, the Editor now automatically skips the data build, and reuses the data from the last build. This way, users get quick builds by default, without the mental overhead of having to track changes manually.

However, it is still possible to skip the data build – even if there are data changes – by clicking the Force skip data build option in the pop-up menu of the Build button. This can be useful if you know that there are data changes, but want to quickly test a code change, without applying the pending data changes for this purpose.

Build > Force skip data build
Build > Force skip data build

That said, if you want a more incremental approach to player data builds, we recommend switching your project to use the Addressables package. This replaces the built-in data build pipeline with an AssetBundle-based data build and distribution mechanism. The Addressables package lets you define Addressable Groups containing assets and scenes, which will be built into AssetBundles independently of each other (and only if the data for a group changes).

The Clean Build option

As mentioned earlier, the incremental build pipeline will skip build actions that don’t have any changes for their inputs, and reuse build results from previous builds. However, there are situations where this is undesirable. Typically, this might be the case when you have custom callbacks modifying assets or scenes during the build and you want to change the behavior of those callbacks, requiring a new build. But another reason could be that you suspect that previous build results have become corrupted, for example, due to a bug. This, of course, requires a new build.

For these scenarios, you can select the Clean Build option, which deletes all cached build artifacts from disk and rebuilds everything. When building a player from the Build Player window, you can find this option in the pop-up menu on the Build button. When building a player from the BuildPipeline API, you can use BuildOptions.CleanBuildCache to do the same.

Profiling builds

To optimize the time spent on player builds, it’s important for us to truly understand where the time is being spent. Our Bee build system already supports writing out profile files in the Chrome Trace Events format, which can be visualized in Google Chrome. So we just added the build steps from BuildReport to that file, to put those build pipeline events into the context of a full build session. This lets us inspect a full, multithreaded timeline of everything that occurs during a build.

If you’re curious to learn more about how Unity builds your own project behind the scenes, you can do the same! To do so, make a player build, and then open Chrome, navigate to chrome://tracing, click Load, and select the Library/Bee/buildreport.json file from your project folder. Note, however, that this is unsupported internal territory – meaning it could change in any future version of Unity, so don’t depend on this for your workflows.

Profiling builds via Google Chrome
Profiling builds via Google Chrome

The results

Finally, what benefits does this bring? Let’s take a look at some numbers from the BoatAttack demo, built as a MacStandalone IL2CPP Player on a 2021 16” M1 MacBook Pro.

Numbers are “seconds to build a subsequent build” with script changes, or with no changes at all. You can see that Unity 2021 has significant overhead, even when doing “nothing,” because many build steps are being rerun. This includes Burst compilation, assembly stripping with UnityLinker, code signing, and more. With the incremental build pipeline, all these steps can be skipped (if there are no changes that would require rerunning them).

The results
The results

For platforms supporting incremental deployment, like Android, the incremental build pipeline only impacts files on disk if they need to be changed. This allows Unity to only transfer files to the device, resulting in faster iteration for the Patch and Run workflow.

For more tips on player builds, visit us in the forums or feel free to connect with me directly on Mastodon at @jonas@mastodon.gamedev.place. Be sure to watch for new technical blogs from other Unity developers as part of the ongoing Tech from the Trenches series.

December 20, 2022 in Technology | 13 min. read
Related Posts