Lights, camera, update: The latest on Cinematic Studio and Live Capture

Last year, Unity released a suite of virtual production and cinematic tools, followed by the launch of a closed beta program to gather information on what creators need to realize or animate films and television shows. And we’ve been thrilled by the success our users are seeing.

Now the Unity 2022 series gives us the opportunity to update and refresh many of these capabilities. Read on for the latest updates on Recorder, Sequences, Python, Live Capture, and FBX Exporter.

When creating linear content like a film sequence, animation, or cinematic, the desired end state is often a video. That’s where Recorder comes in.

Recorder enables you to export what’s in your Timeline and Game view – ultimately, time-based data, which in most cases means video and still images. However, it can also export animation data, audio, and intermediate Arbitrary Output Variables (AOVs) for compositing or effects work.

Key feature improvements include:

• Export video: H.264, WebM, ProRes
• Export frames: PNG, JPEG, EXR
• Support for accumulation motion blur: High Definition Render Pipeline (HDRP) only
• Export AOVs: For effects and comp work
• Export animation data: Anim, FBX, Alembic for DCC roundtripping
• Export WAV audio: For improved audio

For this release, we focused on movie quality, codec support, and overall performance. We also updated our new accumulation motion-blur feature based on your recent feedback.

Here’s a summary of these updates:

• Encoder public API: You can now implement your favorite video codecs using the new Encoder public API for Movie Recorder. This includes an FFmpeg Encoder sample that illustrates how to use the API to integrate with a standalone instance of FFmpeg. The sample can be downloaded from the Package Manager.
• Improved codec options: Fine-tune the quality of your movies with advanced H.264 and VP8 codec options in Movie Recorder. This includes options for bitrate or GOP size.
• Animated GIFs: Back by popular demand, we restored support for animated GIFs, including enhanced support for Apple Silicon.
• ProRes: The improved performance  facilitates the creation of multiple high-resolution ProRes videos in-Editor.
• Accumulation improvement: We addressed feedback around the accumulation motion blur feature through: 
  • An updated UI to ease the onboarding process
  • Reduced accumulation resource requirements (accumulation is now exclusively applied on frames that are recorded, and ignored on frames that are played but not recorded)
  • Accommodations for the following specific requests:  
    • Audio recording while using accumulation 
    • Support for accumulation when recording from a Timeline clip
    • Shadow map filtering
      • When accumulation is active, the shadows generated by Spot Lights are filtered at no additional performance cost, resulting in smoother shadows and more temporal stability.
    • Subpixel Camera jitter by default (anti-aliasing)
      • When accumulation is active, all subsamples are recorded with a slightly perturbed Camera position, resulting in better anti-aliasing.

Timelines in movies are different from timelines in games. In games, time can branch off in many directions. Sometimes you need to trigger multiple timelines at once or interrupt a timeline because, well, things happen. This can lead to some complicated wiring. Perhaps unsurprisingly, the in-Editor Timeline is optimized for an entirely different set of use cases than what a filmmaker might want.

A linear story is different. Everything happens in a predictable order, so most of the process can be automated, and many of the complex steps eliminated. You can organize your project using industry-standard terms and folder structures, just as Sequences does. It accelerates the creation of your cinematic while simultaneously maintaining flexibility.

Key feature improvements include:

• Creation of shots and sequences: With just a few clicks, all the under-the-hood wiring is handled for you.
• Working in parallel: Each cinematic element is an independently editable Prefab so you can avoid conflicts like overwriting.
• Nondestructive changes: Thanks to that same Prefab workflow, you can try out changes nondestructively.
• Recorder integration: This simplifies rendering output.
• Asset organization: Keep your Timeline and project structure tidy.
• Timeline Scene Activation Track: Switch scenes on and off as the story progresses.

In this release, we started to address user feedback around scalability, especially when it comes to collaborating on larger projects with more assets.

See a summary of these updates:

• Unlimited nesting: Our original implementation permitted only three levels of sequence nesting. Now you can create as much nesting in your editorial as needed, for example: Episode 1 > Scene 1 > Sequence 1 > Shot 1.
• UX improvements:
  • Sequences TreeView: Now using UI Toolkit, which is inherently more scalable
  • Sequence Assets Hierarchy: Now in a separate window, which allows for a lighter layout when this level of setup is not required by the project
  • Help buttons: At the top of the windows for easy access to the documentation
  • Context menus: Polished to streamline the experience
  • Deletion in the Sequences window: Now supports multi-selection
• Scalable Editorial data model: 
  • In the previous version of Sequences, the sequences were distinct from the TimelineAsset; Using the TimelineAsset directly removes both cognitive and engineering overhead.

Python is the most widely used scripting language in the animation and VFX industries due to its flexibility and simple syntax. In the vast majority of studio pipelines, toolchains are connected using Python, and all major Digital Content Creation (DCC) apps ship with Python scripting APIs (e.g., Maya, Blender, Houdini, and Nuke).

With Python in the Unity Editor, you can:

• Automate scene and sequence assembly, among other other time-consuming tasks, like duplicating or modifying a large number of assets.
• Write custom Editor tools in C# that interact with Python libraries, such as PySide2, shotgun-api3, and NumPy.
• Take advantage of stronger integration with existing pipelines and better collaboration with important DCCs.

Key feature improvements include:

• Bidirectional bridge between C# and Python: Call any Python API from C#, and any C# API from Python.
• Python Script Editor window: This is principally aimed at technical artists who want to prototype, run short scripts, and create new menu items that run Python code.
• Project Settings: Customize your site-package paths and launch a command line in the same environment.
• Installation and load of Python packages using pip: The Python package ecosystem is fully available.
• Import studio scripts and favorite Python modules: This way, you can work directly in the Editor.
• Reuse your Python production codebase: Quickly complete your work in-Editor.
• Automate recurring tasks: Think of asset transformations and dailies submissions.
• Use third-party Python libraries: All available in-Editor.

This release of Python focuses on stabilization, updating dependencies, improving pip access, and ensuring cross-platform support.

Here’s a summary of the updates:

• Upgraded Python to 3.9: To follow the VFX Reference Platform CY2022 recommendations
• Upgraded Python for .NET to a custom fork of 3.0.0: Rc4 plus a backported fix of a critical bug
• Support for Ubuntu 18 and 20
• Native support for Apple Silicon
• Pip now included: Embedded in the Python installation

FBX is at the heart of model and animation import into the Unity ecosystem. In a film environment, however, it’s often more useful to roundtrip this data between Unity and other programs. The FBX Exporter enables a straightforward workflow for sending geometry, lights, cameras, and animation data to any application that supports the FBX format.

Key feature improvements include:

• Integration with 3D modeling software: Effortlessly import and export assets between Unity and popular DCCs, such as Autodesk® Maya® and Blender. The 3D modeling software remembers where the files go, and which objects to export back to Unity (this is currently only available on Windows and macOS).
• FBX Exporter: Record gameplay and export geometry, animation, lights, and cameras as FBX files to make cinematics, or previsualize/grey-box with ProBuilder, then export your GameObjects to FBX until you can replace them with final assets.
• FBX Recorder: Export animations through the Unity Recorder, including Cinemachine camera animations.
• FBX Prefab Variants: The FBX Importer allows you to import an FBX file as a Model Prefab and create Prefab Variants from them. Since Prefab Variants can override properties and children without affecting the original Prefab, you can now use them without breaking the link to a file when bringing in updates.

While listening to user feedback, we decided to focus this release on extending the Export API.

See a summary of these updates:

• A new function that enables export options to Keep Instances and Embed Textures
• A default file version that allows for exporting of large binary FBX files
  • Note for Blender users: Be sure to update to Blender 2.79 or later to read these files.
• Improved documentation and bug fixes
  • FBX Recorder now correctly records blend shape animation, and can export up to eight UVs.

Live Capture is a core tool that supports motion capture in live performances. The Live Capture package is built to be extensible and act as a platform for a wide range of motion-capture systems.

To make everything work well together, Live Capture incorporates timecode synchronization, with support for sources such as TentacleSync and NTP servers.

In 2023.1, there are two versions of Live Capture available:

• Live Capture 3.0.0 is fully released and will appear by default in the Editor. It is also part of 2022 LTS, so its feature set and API will remain stable and receive occasional bug fixes and minor improvements. Use this version if you are happy with the existing feature set, or if you are starting a long-term project that requires a stable foundation. 
• Live Capture 4.0.0-pre.2 is in prerelease and only available if you opt in for prerelease packages. Its features are new and not as thoroughly tested, and we might make adjustments as we receive feedback. The final released version of Live Capture 4.0.0 is expected to land in Unity 2023.2 and might differ from the current prerelease. Use this version if you want to be ready for what’s next, while also providing feedback on earlier versions of the tools.

Key feature improvements include:

• iOS apps for performance capture ready to go: Unity Virtual Camera and Unity Face Capture are both available for free on the App Store and offer an entry point for real-time 3D film creation.
• Real-time motion tracking: Animate a camera or a character’s face using real-time motion-tracking data from your phone.
• Take Recorder: Use it to shoot takes, play them back, and quickly iterate. 
• Third-party tool integration: Integrate third-party motion-capture systems with Unity.

It is now easier than ever to shoot and review takes. Thanks to improved integration with Timeline and a simpler way to animate cameras, you’ve got it covered.

With Live Capture 4.0.0, you can leverage:

• Camera Anchors: The Virtual Camera can now be anchored to a Transform in the scene, allowing you to drive a handheld camera relative to another object (for example, a car in motion), without having to change your scene’s Hierarchy. 
• Range-limited take navigation: Timeline tracks can contain multiple takes and be quite long. When this happens, scrubbing can become cumbersome. That’s why we now provide the ability to use Range markers and the Rewind button to limit your navigation to the shot you are working on. 
• Take Organizer: Use this dedicated window to navigate your takes, edit their metadata, and rate them.

• Synchronization without timecode: We provide a new set of APIs that makes it easier to synchronize audio, video, and mocap devices. You can now genlock the engine, which makes it possible to synchronize devices without having to rely on timecode.

Prior to 4.0.0, Live Capture 3.0.0 took existing features and made them more powerful and intuitive to use. It is now part of the 2022.2 Tech Stream (with minimum Editor version 2020.3.16f1).

With Live Capture 3.0.0, you can leverage:

• Gamepad support: The Virtual Camera iOS app allows you to control your Camera using Made for iOS (MFi) controllers. Camerawork often demands more precise control than what an onscreen joystick can offer.

• Generic Transform Capture device: On top of recording your camera and face actors, you can make any GameObject transformation part of your take. This enables you to use different inputs in the same take, such as a player-controlled character or a gravity-pulled ball.
• Synchronization windows: In version 2.0.0 of Live Capture, we began to support timecode synchronization through timecode sources such as TentacleSync devices. We’ve since improved that support with new windows to visualize synchronization, which facilitates the process of adjusting buffers.

• Take Recorder improvements: There are a number of changes made to improve the user experience. For example, you don’t need a Take Recorder in your scene to play takes anymore. You can also lock a clip so that you can scrub your Timeline without changing the shot you are currently working on.
• Keyframe reduction: You can reduce the size of Animation Clips by allowing a margin of error in the resulting curve.

Discover how Unity can help bring your next production to life by joining our beta program for early access to the latest artist tools and features. If you need more bespoke support, reach out to our solutions team.