One of the biggest bottlenecks in film production is waterfall development. Read how Neill Blomkamp’s Oats Studios used Unity’s Timeline feature and Multi-Scene editing so their teams could work in parallel on characters, sets, lighting, and cameras, then real-time render at 30 FPS.
I’m SerEn (Sean) Low, an Art Tools Engineer specialized in Tools and Pipeline on the Made with Unity team. Prior to joining Unity Technologies, I was at Disney Interactive, developing animation tools for their console games, which collaborated with Pixar, Marvel and Lucasfilm. For the ADAM films, I helped integrate tools such as Alembic and Timeline, making sure the animation pipeline is streamlined and easy to maintain.
Timeline was first introduced in Unity 2017.1 as our answer to artists and designers who wanted Unity to become a more intuitive tool for game, VR/AR and film development. They use Timeline for a variety of tasks, including creating and managing cinematic content, game-play sequences, audio tracks, and complex particle effects.
Timeline was used to assemble all types of assets in the Adam films
During production planning, Oats identified Timeline as the backbone of their real-time film project. “I don’t think we could do what we’re doing without Timeline,” said VFX supervisor, Chris Harvey. “We hit Play and we watch – this is real time.”
Unity’s Multi-Scene editing capability is commonly used in game production to allow various teams to work on different scenes individually and then combine them into a final large world.
Oats used Multi-Scene editing to organize their scene setups by team discipline – Main assets, Set (environmental/background), Lighting, Special Effects, etc. As well, they further organized assets in a scene into several Timeline instances. This is useful when dealing with a larger subset of an asset. A good example is for main characters because they typically require many tracks to assemble a full performance. These nested Timeline instances can be achieved using a ControlTrack.
Oats created several ControlTracks in a main Timeline, each for character animation, crowd animation, Alembic animation and sometimes even for lens flare. Besides permitting better organization, this nested structure also helps different teams to work on individual Timeline instances associated with their tasks only. It also makes version control simpler because changes are limited to individual Timeline assets and instances – meaning that there are fewer conflicts to be resolved each time an update is made.
A ControlTrack used in the main Timeline instance
Inside a “Character” Timeline instance (nested)
Oats’ technical director, Mike Ferraro, explains: “Simultaneously, the animation team is doing final animation but everybody else can still work, as opposed to the old linear pipeline where it’s a waterfall, and each department has to finish before you can move on to the next stage. This is a fluid development experience for us.”
This parallel workflow is one of the main reasons that the two Adam episodes were completed in just five months, which is relatively short compared to a traditional filmmaking pipeline.
Given how complex and long Oats’ Adam project was, there were a large number of assets in every scene that required careful planning and architecture in order to ensure the production was manageable at every point.
Besides using Multi-Scene editing and Timeline assets as a container, Oats also utilized Timeline’s Track Group to group similar asset types in the Timeline editor. This makes an assets group visible to only the team that works on it. Other team members work with this track group collapsed in the Timeline editor so they won’t risk introducing unnecessary changes to the assets group. For example, a lighting artist will expand the lighting track group and work on it, while the camera artist will collapse the lighting track group and focus on working in the camera group only.
An expanded Track Group in Adam: The Mirror
A collapsed Track Group (note that you can still see the clip thumbnails in it)
By default, Timeline includes an Animation Track, an Activation Track, and an Audio Track. But it can be further extended. One of its more potent features is the ability to write custom playable tracks. This enables you to control any property of any asset in a sequential manner.
Oats wrote many custom tracks for their production needs. For example, there is a Lighting track to set the main light, Skybox material, ambient, and fog. There are also custom tracks for controlling shadow cascade settings, for adjusting fill light properties, and for swapping Post-Processing Stack profiles. These controls enable precise setting of assets for each shot throughout Timeline. For details about the Playables track, check out the documentation and video.
A custom track for lighting and camera
Another nice add-on to Custom Track is the ability to control how two Timeline clips calculate the differences in property values when they overlap. This blending behavior is really useful for enabling a smooth transition between two clips over time.
Overlapping two clips to create a blend
Sample code for manipulating property value with weight
To achieve this blending, Oats used a Track Mixer to control their camera properties via PlayableBehaviour. For example, they set up a custom track to control the main camera’s focus point. By setting up two clips with different focus points and then overlapping them in Timeline, Oats was able to manipulate the weight of each clip during the transition. This helped them achieve the focus-puller effect as shown below.
Focus-puller effect shown in The Mirror
With Custom Track and the Playable API, Timeline becomes an ideal framework for integration with Unity plugins such as Cinemachine, Alembic Importer, and Frame Recorder. When developing these plugins, the Unity R&D teams implemented them seamlessly within Timeline using the very same API available to the public, so once you install one, its tracks and associated clips immediately become available in Timeline for your production work. With the familiar Timeline workflow and structure, users will find it easy to use the new plugins.
Unity plugins integrated with Timeline
Given Timeline’s extensibility, there is still much potential, either to use it to architect your scene organization or to drive your asset properties in a sequential manner. Timeline will continue to improve as we receive more feedback and feature requests. For example, we’ve recently upgraded Audio Track to support waveform visualization in Unity 2017.2. Be sure to check out the Unity Roadmap for more Timeline features to come!
Special thanks to the Unity Timeline Dev Team: Sean Thompson, Julien Blais, Vlad-Andrei Lazar and Chafik Achache