Search Unity

Introducing: Unity Robotics Visualizations Package

October 4, 2021 in Engine & platform | 16 min. read
robot arm
robot arm
Share

Is this article helpful for you?

Thank you for your feedback!

With this toolkit for visualizing and debugging the internal state of robotics simulations, Unity can be used as an all-in-one ROS simulation and visualization tool.

One of the challenges faced by roboticists is the need to understand what's happening in their system. In a complex interconnected network of components, when something isn't working, how do you figure out which part is going wrong? Is component A generating bad data, or is component B processing it wrong? It's critical to be able to visualize the data travelling around the system.

With this in mind, today we're excited to announce the next release from Unity Robotics: the Robotics Visualizations Package is a new package for displaying and customizing visualizations of ROS messages.

The Unity Robotics team has been hard at work, releasing several Robotics example projects, such as Pick-and-Place and Object Pose Estimation. Most recently, we released our Nav2-SLAM Example, demonstrating an autonomous robot navigating and mapping an unknown space, all simulated in Unity.

The Robotics Visualizations Package builds on and supports these packages by offering a library of customizable visualizations for all the common ROS message types: shapes, poses, point clouds, images, sensors of all kinds, and more. It natively supports the ROS transform tree, and allows you to enable, disable and customize visualizations for any ROS topic at runtime.

Previewing a Trajectory message by showing ghost copies of the robot.

Workflow

Here's how easy it is to use the Robotics Visualizations Package to add visualizations to an existing robotics project:

  1. Import the Robotics Visualizations Package into Unity using Package Manager.
  2. Drag the DefaultVisualizationSuite prefab into your Unity scene.
  3. Hit Play, and you’ll see some new buttons in the Heads-Up Display (HUD). Click on the Topics button to see the list of all the topics ROS knows about.
  4. Click on whichever topics you want to see visualizations for!
 The Topics menu operates similarly to rviz: clicking the radio button turns on visualization in the Unity Scene.
The Topics menu operates similarly to rviz: clicking the radio button turns on visualization in the Unity Scene. This configuration tool controls the overlay of data like transformations, maps, lidar point clouds, planned paths, and more. It can also visualize data like camera images; on the right we visualize the Turtlebot camera image captured in the Unity Scene.

Debug your robot applications with all your favorite visualization types

Transforms

Transforms represent the relationships between coordinate frames in a robotic system. All your data are generated in different coordinate frames. Lidar data are in the lidar frame, camera data are in the camera frame, and map data are in the map frame. In order to make sense of all these disparate data sources, we need to have a common frame of reference. Transforms help us do this by keeping track of the relationships between these frames. Debugging robots without putting the sensors and algorithms in this 3D context is nearly impossible. With the Robotics Visualizations Package you can now view data in real time alongside Unity scenes and assets and seamlessly switch between simulated and real data.

robot in factory
Here, robot transform visualizations have been activated, allowing you to overlay data and debug in real time.

Robot Maps

Robot mapping is tricky. Maps can fracture, robots can drift. Is your odometry tuned correctly or did the map just break in half because an optimization-based SLAM algorithm broke down? The Unity Robotics Visualizations Package enables you to visualize the occupancy grid, transforms, localization, and lidar point cloud all on top of one another, enabling you to visually see where algorithms break down.

This image shows a robot mapping a new space. You can see the occupancy grid fill in as the robot runs its SLAM Algorithm. These occupancy grids are like a greyscale image of the surroundings where each pixel (or cell) represents the probability of that area being occupied by an obstacle.

The Robotics Visualizations Package supports most common ROS message types, including Transforms, Occupancy Grids, 3D point clouds, markers, laser scans, images (jpeg, png and uncompressed), and more. And has opportunities for customization if you have a unique data type you need to view!

Here, robot transform visualizations have been activated, allowing you to overlay data and debug in real time.
Visualize lidar data alongside transformations and Unity scene data for debugging; seamlessly switch between simulation and real data.

Customization

The Robotics Visualizations Package also supports user-created visualizations: it includes a powerful set of tools to draw anything you need, and/or build on and customize the built-in visualizations. Here are some highlights:

Drawing3d is an easy-to-use utility class for drawing arbitrary textured/colored lines, shapes, meshes and labels in 3D space. For example, you could use it to draw the ghost of an object at the position where you predict it will be, a line showing the trajectory it will follow to get there, and more lines around it to indicate error bars.

PointCloudDrawing is a GPU-optimized point cloud renderer, which can display up to 10 million billboarded points each with their own size and color, at interactive speeds. You can use it to display volumetric data such as 3D scans or depth images.

Display historical data trends, 3D movement trails, and more with the HistoryDrawingVisualizer template, which maintains a configurable-length history of the messages sent on a topic. You can analyse and display that history however you want.

And of course, all this is backed up by the power and ease of use of the Unity Engine, offering features such as AR and VR support; custom shaders with Unity Shader Graph; compute shaders, cloud simulation, and more.

What’s next?

We're excited to see what you all do with the Robotics Visualizations Package!

To get started with the Robotics Visualizations Package, check out this new extension to the Nav2-SLAM tutorial that demonstrates how to use the new package.

Unity Robotics Visualizations Package enables you to visualize robot data alongside Unity environments combining real data and simulated data in one place.
Unity Robotics Visualizations Package enables you to visualize robot data alongside Unity environments combining real data and simulated data in one place.

Our Robotics Visualizations Package is just a part of our growing ecosystem of robotics packages and features that enable robotics in Unity. For more robotics projects, visit the Unity Robotics Hub on GitHub.

Be sure to visit us on the Robotics Forum, or email us at unity-robotics@unity3d.com with your feedback and suggestions!

October 4, 2021 in Engine & platform | 16 min. read

Is this article helpful for you?

Thank you for your feedback!