Search Unity

Reimagining in-vehicle experiences with real-time 3D

January 14, 2020 in Manufacturing | 8 min. read
Topics covered

Is this article helpful for you?

Thank you for your feedback!

Unity is teaming up with NXP Semiconductors, the world’s largest provider of automotive semiconductors, to demonstrate a human-machine interface (HMI) toolchain that operates on NXP’s popular i.MX 8QuadMax Applications Processor. The collaboration creates opportunities to bring real-time 3D experiences, including games made with Unity, to mass-production vehicles, regardless of their trim level and cost.

One of the biggest changes taking place in the automotive industry is the digitalization of the dashboard. From super-sized stretch screens that blend instrument clusters and infotainment systems to augmented reality-powered head-up displays (AR HUDs), human-machine interface (HMI) systems are rapidly becoming a battleground for automotive companies. Gorden Wagener, Chief Design Officer for Daimler AG, the parent company of Mercedes-Benz, made the stakes clear when he said, “Screens are the new horsepower.” 

It’s not just the screen hardware that’s changing, but what’s shown on them. We’ve already seen games made with Unity, such as Cuphead, become playable in vehicles. Drivers and passengers today increasingly expect automakers to extend the games and other cutting-edge experiences that are available on their smartphones to their vehicles. But turning cars into smartphones on wheels is a difficult proposition, as current development processes are splintered across multiple production tools, which are often not up to the task of creating modern, responsive user experiences. 

Unity has already become a platform of choice in prototyping and developing new HMI experiences. In fact, the first production vehicles with HMI systems running on Unity will hit the road this year. 

The interactivity that real-time 3D provides is just one reason automakers – and other manufacturers creating products with an HMI component – are turning to Unity. With WYSIWYG (what you see is what you get) development, they can reduce development time throughout the HMI design and review stages and directly deploy to embedded targets (e.g., chipsets), therefore bridging the huge gap between design and engineering. 

Designers and engineers no longer have to struggle with recreating the HMI numerous times to meet different design requirements, as they now can speak the same language and use Unity as an end-to-end HMI graphics toolchain across design, prototyping, development, and mass deployment. Learn more about Unity’s HMI toolkit here.

Teaming up with NXP to implement advanced HMI experiences at a lower cost

Historically, advanced HMI experiences were limited to consumers who could afford a higher-priced trim level or a luxury vehicle. That’s changing with Unity’s collaboration with NXP, whose i.MX applications processors have been deployed in more than 200 million vehicles. 

By bringing a Unity HMI toolchain to NXP’s popular i.MX 8QuadMax applications processor, Unity is demonstrating the potential for value-minded automotive original equipment manufacturers (OEMs) to place immersive HMI systems in all their vehicles. Consumers can enjoy a high-fidelity and immersive interface that enables them to play games, connect with their smartphone and interact with their car – just like they would in a high-end vehicle.

“Unity brings an optimized content rendering engine, content creation and toolchains which can be utilized to work with stretch screens and ‘normal’ resolution screens alike. The output of this powerful engine is stunning display experiences that have to be seen. Coupled with i.MX 8QuadMax’s performance and fail-functional display safety capabilities provide the consumer with a seamless, beautiful and safe experience,” said Kyle Fox, Director for the i.MX 8 series applications processor at NXP.

To demonstrate the possibilities of this collaboration at CES 2020, Unity worked with icon incar, the leading design consultancy for automotive user experience (UX), to build an HMI experience that runs on NXP’s i.MX 8QuadMax applications processor. Check out the project below.

This content is hosted by a third party provider that does not allow video views without acceptance of Targeting Cookies. Please set your cookie preferences for Targeting Cookies to yes if you wish to view videos from these providers.

Making the Unity-NXP HMI application demo

The demo is designed to support immersive, high-impact experiences on an ultrawide 49-inch screen (1.24 meters) with a 32:9 aspect ratio. It consists of three key components: 1) the HMI system, comprising the digital instrument cluster, navigation with 3D mapping, car status, and media player; 2) a game which is purposefully situated on the passenger side; and 3) a visual effects (VFX) experience. 

While icon incar designed the user interface (UI) and UX of the HMI system and programmed the interactions, Unity ported the 3D engine and integrated the game and VFX experience into the HMI. All interactions are done remotely via a tablet used as a touchpad and a regular game controller, but can be easily configured for voice controls or gestures. Both icon incar and Unity optimized the project to ensure the various HMI experiences were performant on the NXP chipset.

HMI design

The demo is built using Unity’s Universal Render Pipeline, which is optimized towards delivering best in quality graphics, performance, and scalability to reach many devices and wider audiences.

The HMI UX is organized across multiple tiles, which dynamically update to offer adaptive, personalized experiences based on user preferences or the content being displayed. For instance, when the game or the VFX experience is onscreen, the HMI changes from three to two tiles to deliver a wider-screen experience.

Users can select from multiple drive modes (Comfort or Sport). In Sport mode, the speed and power is more prominent in the instrument cluster, and the car visualization shifts from an aerial view to a sportier, close-up perspective. The HMI also adjusts dynamically based on time of day, offering light and dark modes for day and night driving.

Day mode with an aerial navigation view in the center tile
Night mode with car status in the center tile

“Unity offers an incredible advantage to designers by allowing them to develop an interactive HMI for a target system, and then experience the near-final output and actual performance long before deployment,” said Florian Gulden, Founder of icon incar.

The navigation also surpasses what’s currently available to drivers on their smartphones. In real-time, drivers can change their point of view, altering their perspective just like in reality. They can use a bird’s eye view to get a sense of scale or zoom in to get an intimate view from the front, back, or side of the car. To make the experience more immersive, the shadows of the buildings also adjust automatically in the navigation mode.

Real-time 3D maps bring greater context and interactivity than what’s available to drivers on their smartphones.

HMI design for gaming: Boat Attack

Reimagine in-vehicle entertainment by extending games made with Unity to the digital dash.

Building HMIs with Unity paves the way to extend games beyond familiar platforms into the new environment of cars. Tesla CEO Elon Musk is already making this a reality in their electric vehicles (EVs). 

To ensure safety on the road today, games are designed for downtime, such as arriving early to pick up the kids from school or waiting for an EV to charge. As the era of autonomous vehicles approaches and drivers relinquish control of the wheel, experiences like these will come to the forefront. Cars will evolve into the next hub of media and entertainment, and the onus will be on automakers to provide engaging content from point A to B.

To port Boat Attack, a Unity boat racing game, to this chipset, we made a number of optimizations to make the game playable and performant on the NXP System-on-Chip (SoC). 

These included:

  • Using static light for all objects in the scene, with the exception of the racing boats, as well as a simplified version of the lighting for the shaders 
  • For assets in the island environment like the rocks and moss, baking lighting into textures instead of relying on shaders to do the job
  • Reducing vertices and polygon count in some complex objects in the island environment (e.g., trees, terrain, rocks), including using Terrain To Mesh from the Asset Store
  • Using Level of Detail (LOD) for some objects
  • Swapping out the game’s water shader with a lighter weight version (uWater - Fast Water Shader asset) and modifying it to support depth-based transparency

HMI design for VFX

Create compelling visual effects by generating and animating large quantities of images.

Preeminent user experiences must be powered by graphics capabilities that surpass user expectations. We used the Visual Effect Graph and the 3D Visualizer Spectrum Vu Meter asset to create a variety of entertaining experiences synced with music.

Another advantage of Unity for HMI is access to thousands of free and affordable assets through the Asset Store. For prototyping or production, quality art and tools contributed by our developer community save valuable time and resources on HMI development. The following assets were used in this project:


Stay tuned for more Unity HMI projects this year. Learn more about Unity’s HMI Toolkit and other HMI experiences made with Unity Learn more about the transformation of HMI systems in our whitepaper: Top 5 ways real-time 3D is revolutionizing the automotive product lifecycle.

January 14, 2020 in Manufacturing | 8 min. read

Is this article helpful for you?

Thank you for your feedback!

Topics covered