Introducing: Unity Robotics Visualizations Package

With this toolkit for visualizing and debugging the internal state of robotics simulations, Unity can be used as an all-in-one ROS simulation and visualization tool.

One of the challenges faced by roboticists is the need to understand what's happening in their system. In a complex interconnected network of components, when something isn't working, how do you figure out which part is going wrong? Is component A generating bad data, or is component B processing it wrong? It's critical to be able to visualize the data travelling around the system.

With this in mind, today we're excited to announce the next release from Unity Robotics: the Robotics Visualizations Package is a new package for displaying and customizing visualizations of ROS messages.

The Unity Robotics team has been hard at work, releasing several Robotics example projects, such as Pick-and-Place and Object Pose Estimation. Most recently, we released our Nav2-SLAM Example, demonstrating an autonomous robot navigating and mapping an unknown space, all simulated in Unity.

The Robotics Visualizations Package builds on and supports these packages by offering a library of customizable visualizations for all the common ROS message types: shapes, poses, point clouds, images, sensors of all kinds, and more. It natively supports the ROS transform tree, and allows you to enable, disable and customize visualizations for any ROS topic at runtime.

Here's how easy it is to use the Robotics Visualizations Package to add visualizations to an existing robotics project:

1. Import the Robotics Visualizations Package into Unity using Package Manager.
2. Drag the DefaultVisualizationSuite prefab into your Unity scene.
3. Hit Play, and you’ll see some new buttons in the Heads-Up Display (HUD). Click on the Topics button to see the list of all the topics ROS knows about.
4. Click on whichever topics you want to see visualizations for!

Transforms represent the relationships between coordinate frames in a robotic system. All your data are generated in different coordinate frames. Lidar data are in the lidar frame, camera data are in the camera frame, and map data are in the map frame. In order to make sense of all these disparate data sources, we need to have a common frame of reference. Transforms help us do this by keeping track of the relationships between these frames. Debugging robots without putting the sensors and algorithms in this 3D context is nearly impossible. With the Robotics Visualizations Package you can now view data in real time alongside Unity scenes and assets and seamlessly switch between simulated and real data.

Robot mapping is tricky. Maps can fracture, robots can drift. Is your odometry tuned correctly or did the map just break in half because an optimization-based SLAM algorithm broke down? The Unity Robotics Visualizations Package enables you to visualize the occupancy grid, transforms, localization, and lidar point cloud all on top of one another, enabling you to visually see where algorithms break down.

The Robotics Visualizations Package supports most common ROS message types, including Transforms, Occupancy Grids, 3D point clouds, markers, laser scans, images (jpeg, png and uncompressed), and more. And has opportunities for customization if you have a unique data type you need to view!

The Robotics Visualizations Package also supports user-created visualizations: it includes a powerful set of tools to draw anything you need, and/or build on and customize the built-in visualizations. Here are some highlights:

Drawing3d is an easy-to-use utility class for drawing arbitrary textured/colored lines, shapes, meshes and labels in 3D space. For example, you could use it to draw the ghost of an object at the position where you predict it will be, a line showing the trajectory it will follow to get there, and more lines around it to indicate error bars.

PointCloudDrawing is a GPU-optimized point cloud renderer, which can display up to 10 million billboarded points each with their own size and color, at interactive speeds. You can use it to display volumetric data such as 3D scans or depth images.

Display historical data trends, 3D movement trails, and more with the HistoryDrawingVisualizer template, which maintains a configurable-length history of the messages sent on a topic. You can analyse and display that history however you want.

And of course, all this is backed up by the power and ease of use of the Unity Engine, offering features such as AR and VR support; custom shaders with Unity Shader Graph; compute shaders, cloud simulation, and more.

We're excited to see what you all do with the Robotics Visualizations Package!

To get started with the Robotics Visualizations Package, check out this new extension to the Nav2-SLAM tutorial that demonstrates how to use the new package.

Our Robotics Visualizations Package is just a part of our growing ecosystem of robotics packages and features that enable robotics in Unity. For more robotics projects, visit the Unity Robotics Hub on GitHub.

Be sure to visit us on the Robotics Forum, or email us at unity-robotics@unity3d.com with your feedback and suggestions!