Search Unity

Made with Unity: AK GO, XR content for aviation maintenance training

February 10, 2022 in Manufacturing | 10 min. read
Image showing a man performing airplane maintenance training in VR
Image showing a man performing airplane maintenance training in VR
Topics covered
Share

Augmented Knowledge Corporation is the creator of AK GO, an industrial solution built with Unity that enables immersive aircraft maintenance training. Text-based analog manuals have been replaced by smart glasses and augmented reality (AR) leveraging artificial intelligence (AI). 

The story behind AK GO began in 2008. We asked CEO and founder of Augmented Knowledge Corp., Kevin GS Jo, to tell us about its development and the role Unity played. As a long-time proponent of virtual reality (VR)  and AI, he has been studying their application to the field of aviation maintenance.

How the project took flight

During his time as a university professor in 2008, CEO Jo received funding from Airbus to conduct empirical research with the University of Southern California and Korean Air on the use of AR and AI technologies for conducting onsite aviation maintenance.

His research confirmed that using AR and AI technologies rather than paper manuals improved the efficiency of maintenance by more than 30%. Jo also found that even less-experienced workers could perform nearly zero-error maintenance in terms of quality. He presented his thesis on related basic technology and its application with these results at the Association for the Advancement of Artificial Intelligence (AAAI) Conference on AI.

At the time, AR and mixed reality (MR) glasses were scarce, but with technological advances, various smart glasses, such as HoloLens 2 and Magic Leap, are today readily available on the market, establishing the foundation for their practical use. Jo patented the results of his studies in the U.S., China, and Korea, and began the journey to commercialize the results within the industry.

AK GO: XR content for aviation maintenance training

Aircraft maintenance is an essential process as poor maintenance can lead to fatal accidents. According to a study by the International Civil Aviation Organization (ICAO) in 2016, about 14% of aircraft accidents are caused by faulty maintenance. Although the maintenance of aircraft or complex machinery is strictly controlled to ensure maintenance is done according to documented procedures, finding and understanding the considerable content in the PDF file or paper manual before maintenance work and then confirming how the two-dimensional drawings correspond to the real-life aircraft are complex and time-consuming processes. 

AK GO, an AI solution equipped with the aircraft maintenance manual, displays the three-dimensional aircraft parts and manual simultaneously through the smart glasses worn by the worker.  Workers can be efficiently trained on the maintenance process through simulations using extended reality (XR) content, AI, and voice interaction.

With AK GO, workers can learn and repeat realistic maintenance processes as if they were training on a real aircraft, while training institutions can provide high-quality training without the burden of having to acquire expensive equipment.

Immersive content identical to onsite work processes

Aircraft are expensive, with a single vehicle typically costing at least 100 billion KRW. Aviation maintenance training institutions have limited resources to acquire them. Training institutions often resort to using old aircraft, such as retired or light aircraft that are no longer flown, as training equipment. However, even old aircraft are now reduced to being mere display models due to concerns of further damage to the equipment.

To address these challenges, Jo aimed to develop immersive training content that would accurately mimic real-life onsite maintenance processes, an exemplary use case for XR technology.

For effective aviation maintenance training content, as for all work on industrial sites, users must have easy access to information and must have both hands free to perform the work. HoloLens 2, wearable smart glasses that are now readily commercially available, enables this functionality, so Jo’s team focused on developing aviation maintenance content for that device. The team also applied voice recognition technology so that workers could easily access information.

This content is hosted by a third party provider that does not allow video views without acceptance of Targeting Cookies. Please set your cookie preferences for Targeting Cookies to yes if you wish to view videos from these providers.

Maintenance procedure guide according to the aircraft’s main landing gear maintenance manual

When our development team first started development, we had difficulty selecting an engine because we didn’t have any prior engine experience. The Unity Engine had great appeal because, compared to other engines, it was easily accessible for novice users. Also, as it supported high-level development, we determined the Unity engine would be less burdensome over the long-term development process as we made changes. Furthermore, it facilitated the development of MR content for HoloLens 2 and provided an environment that made it possible to acquire and use such things as plug-ins or assets we needed for development. That is why we chose Unity.

– Kevin GS Jo, CEO and Founder, Augmented Knowledge Corp.

Sharing the video feed from smart glasses to tablet

Developing VR content for smart glasses can be problematic because only the person wearing the device can see the content displayed through the glasses. So, the AK GO team improved user-friendliness. Rather than each trainee being equipped with both the expensive smart glasses or VR device and a PC, one trainee can use the equipment, while the video feed from the glasses is shared to other screens, such as tablet PCs.

With AK GO, trainees can see images of the actual aircraft displayed through the smart glasses and receive guidance for the work procedure displayed in context. Also, with deep-learning voice recognition technology, the trainee can easily access the menu and content at each step of the procedure. The video feed from the HoloLens 2 device can also be transmitted to tablets, making it possible to share the feed with others so they can also view, in real-time, the content used by the trainee.

This content is hosted by a third party provider that does not allow video views without acceptance of Targeting Cookies. Please set your cookie preferences for Targeting Cookies to yes if you wish to view videos from these providers.

Simulation of disassembly and assembly procedures of an aircraft’s landing gear that combines AI and XR

In the video on the disassembly and assembly of an aircraft’s landing gear, the AI displays the procedure steps from the manual within the trainee’s vision by communicating with the trainee. By combining AI and XR, the trainee communicates with the maintenance knowledgebase to see the necessary information during maintenance, superimposed on a three-dimensional model in AR.

How Unity features add value

According to Chief Architect and AR Developer Chidon Ahn, Unity’s editor scripting was invaluable for this project. Not just for a single project, though. If you are conducting multiple similar projects, once you define the task with editor scripting, the task can be easily repeated.

When developing content that handles vast amounts of data that changes with each repetition, the developer has to handle everything manually. However, with Unity, this process can be automated by using appropriately produced metafiles and predefined pre-fabs.

Unity has various assets that can be effectively used by small development teams like ours. Unity is also optimized for codevelopment by using cloud technology, so we’ve been using Unity since the beginning of this project.

– Chidon Ahn, Chief Architect and AR Developer, Augmented Knowledge Corp

Future outlook

AK GO’s focus has been on the aviation maintenance field in the design, testing, and realization of AI and XR convergent content. AK GO plans to expand into the field of digital twins, where real-time maintenance is required for industrial assets such as vehicle electronics, home appliances, and factories.

Augmented Knowledge Corporation intends to supply cloud-based content of various aircraft types to the world as a software-as-a-service (SaaS) model. Unity will be actively used to converge content from various sources.

AK GO was built using Unity 2019.1.14. Find out more on Augmented Knowledge Corp.’s website.

Learn about how you can use Unity for digital twins, including for immersive training applications.

February 10, 2022 in Manufacturing | 10 min. read
Topics covered