Our Made with Unity: AI series showcases Unity projects made by creators for a range of purposes that involve our artificial intelligence products. In this example, ML-Agents empowered AI developers by allowing them to quickly and easily set up machine learning environments and to train an agent how to play soccer before finally transferring that agent to a real robot.
Unity Machine Learning Agents Toolkit (ML-Agents) allows users to easily get started with reinforcement learning (RL) using Unity. ML-Agents gives users a variety of sample environments and model architectures that they can use to start working with RL. Users can then tune hyperparameters to experiment and improve the resulting models. All of this can happen without the user having to worry about creating a Unity environment or importing assets – and there’s no immediate need for coding. This project out of Japan by Ghelia Inc. used the ML-Agents soccer environment to train an agent to play soccer. The resulting RL model was then deployed on real Sony toio robots to play soccer. This is an exciting example of simulation-to-real-world with robotics using ML-Agents to train.
We interviewed Ghelia’s Ryo Shimizu, CEO and President; Hidekazu Furukawa, Lead Programmer for Innovation and Brand Strategy Office; and Masatoshi Uchida, Manager for Innovation Section of the Innovation and Brand Strategy Office to find out what inspired them to build this project. Read on to discover how they used ML-Agents Toolkit for training a real-world robot how to play soccer and how a golf ball fits into this scenario.
Ghelia is a company that focuses on reinforcement learning applications. The founder of Ghelia, Hiroaki Kitano, launched RobocupSoccer and developed the AIBO at Sony. Our team had previously built an air hockey demo, but since it consisted of many different components, it was not very portable. When we started to discuss creating another demo to explain to customers what reinforcement learning is, we wanted something that was going to be easier to carry around. Since ML-Agents already had a soccer environment, it made sense to use the small and portable Sony toio robots to create a soccer game, which could also lead to viral content.
To apply reinforcement learning to a real robot, the robot needs to exist in a simulation environment. Luckily, toio already has a simulator called toio SDK for Unity. By adding the ML-Agents package to it, we were able to use it for training immediately. While the toio SDK provided the robot models for Unity, we still needed to create the ball. We used Unity’s physics engine to recreate the ball in the simulator and needed to find a real-world ball that would match the simulation results. It turns out that a golf ball produced real-world results that reflected the training results. The ball’s position was detected in the simulation by using Unity transform value, and in the real world by image recognition using OpenCV and a camera.
We used a golf ball to represent the soccer ball, but to increase the recognition rate, we painted it red. Amazingly, we were able to use just one iPhone and its camera to detect the ball, control all eight robots (it was a four-on-four soccer game), and perform inference with the ML-Agents model.
At first, there were many own goals, so we tried to provide a negative reward for an own goal. However, this resulted in the goalkeepers not defending their goal. When we tried giving a positive reward for moving the ball, both teams would simply go back and forth, not putting the ball in the goal, basically stalling for time. Finally, we decided to reward one point for putting the ball in the opponent’s goal and took one point away for being scored on.
It was sometimes difficult to ascertain why the actual robots did not work as well as the simulation. For example, sometimes the inference didn’t work because we operated the robot on a slightly tilted floor. Other times, the ball rebounded differently from the simulation, so the robots didn’t respond as expected. The positioning of the camera was also quite sensitive, requiring millimeter-order precision, making it difficult to adjust at the event site every day. After each major set of improvements to the model, we trained for about three days. In the end, we had about six training sessions to achieve our results.
In the ML-Agents demo, after a goal, the agents line up in their original position, but it’s not so simple for real robots. Some problems, such as avoiding collisions between toios, were difficult to solve through reinforcement learning alone. While we initially tried to design a reward function for this scenario, we eventually solved it heuristically.
If there were demand, we would definitely consider making this project open source. You can find additional details about this project in our blog post (in Japanese).
AI, especially deep learning, is fascinating, but it is not well understood. You can’t fully appreciate its beauty and complexity until you work with it firsthand, and that’s a shame, so we encourage Unity developers worldwide to try it. I want to emphasize how much fun machine learning is and that Unity ML-Agents is a system that allows you to get started with machine learning easily or incorporate it into your project.
Get started with Unity ML-Agents or learn more about the Unity Robotics packages today. If your project requires you to kick off multiple training sessions in parallel, contact us to learn more about our ML-Agents Cloud offering.
Hidekazu Furukawa has also published a Japanese book called Unity ML-Agents Practical Game Programming that details how to get started with reinforcement learning using ML-Agents.