Making Your Next Flight Safer and Smoother with Wearable AR+VR

Written BY

Emily Friedman

June 27, 2018

From building the actual plane to the in-flight experience, wearable XR (AR, VR, MR) devices have a role to play in multiple professions within the commercial aviation industry. Employees whose jobs affect every aspect of one’s trip, including aircraft maintenance workers and flight crew can make use of wearable XR technologies to ensure the end goal: A safe and satisfied traveler. Find out how XR might be used on the ground and in the air when you go on your next business trip or vacation:

On the Ground: AR for Assembly

Both Airbus and Boeing employ augmented reality (AR) glasses in the aircraft assembly process. Airbus workers follow plans directly in their field of view, superimposed on the plane’s interior during cabin installation. They use the same solution to check the accuracy and quality of their work (image recognition technology and artificial intelligence at work); while Boeing employees use smart glasses to view a heads-up, hands-free roadmap for wire harness assembly over their real-world view. In each case, AR functions to form a stronger connection for the user between textual or diagrammatic instructions and the real working environment.

Using AR glasses with software by Upskill helped Boeing save tens of millions of dollars, but it’s not all about money: By helping employees work faster without error, aircraft manufacturers can deliver defect-free planes to customers quicker. Airlines and other buyers thus receive faster-built, higher quality aircraft and parts that breakdown less often. Aircraft and parts engineers can also use AR and VR devices to collaborate on new designs from anywhere in the world, sharing and testing ideas and even simulating the assembly or installation process to foresee issues. New XR platforms are only making this collaboration easier.

VR for Training

After assembly comes maintenance: It can take up to eight years to train and license an aviation maintenance professional. This includes aircraft OEM mechanics and airline technicians who perform safety checks, prepare aircraft components for flight, make repairs, and more. While accessing real aviation equipment for hands-on training is costly and difficult, in VR trainees can practice skills in a realistic, accident-proof immersive environment with virtual parts and tools. For instance, a mechanic wearing a VR headset could walk inside an engine and examine its parts as well as simulate different repair scenarios. With advanced audio and haptics (like a haptic suit), the trainee could even hear the noise and feel the motion of the engine, better preparing him for the real thing.

A recent study at the University of Maryland found that people actually learn and retain information better through immersive experiences compared to using a computer or tablet. Enterprises are also finding VR to be superior to reading a manual, watching videos, or taking a lecture-style class. While not an example of full immersion, Japan Airlines used Microsoft’s HoloLens to improve training for its engine mechanics—in place of physical hangouts, trainees learned all the engine components by working on a virtual engine in mixed reality.

Learning by doing with AR is effective and cost-saving for training, as well. Aviation maintenance workers can learn on the job without risk of error by using heads-up, hands-free smart glasses to view fool-proof text and visual aids over their work. The technology can even validate each step of an inspection or repair to prevent errors. Static instructions can become interactive, with virtual arrows and labels appearing on top of real-life aircraft equipment, showing the user where parts and tools should go. The result: Faster training without sacrificing accuracy or quality = quicker maintenance, fewer flight delays, and happier travelers.

Once the engine has been overhauled, the plane is ready for service. Expensive and logistically challenging, pilot training is another opportunity for VR. In recent years, the burden of paying for flight school has fallen onto pilots themselves. The $60,000-$80,000 price tag explains why flight school enrollment has fallen in the U.S., leading to a growing shortage of trained pilots not all that unlike the troubling shortage of skilled workers in other industries. CAE forecasts that over 255,000 pilots will be needed in the global commercial aviation industry by 2027, yet less than half that number has even begun training. Some carriers and manufacturers are making efforts by sponsoring aspiring aviators or expanding their flight training services, but the cost and time is still too great.

For industries with large, complex and expensive equipment like aviation, VR offers the closest thing to hands-on training. Virtual reality, capable of simulating almost every aspect of flying, feels more real than many current flight simulators (essentially stripped airplane cockpits with screens for windows) and is adaptable to all kinds of scenarios. Rookie pilots can walk around the cockpit, interact with the plane’s controls, and even practice an emergency landing, with tactile feedback to increase the sense of realness and help build muscle memory. VR is already finding its way into flight training programs: Airbus, for one, has been able to reduce training time and train more people in limited space using VR to supplement training in real aircraft; while Future Visual created a simulation for Oculus which takes pilot students through the entire pre-flight process. And VR isn’t just for ground crew and pilots; cabin crew and even airport staff training could incorporate immersive tech, as well.

In the Air: AR for Guidance

The length of runway required for a standard aircraft to get off the ground can be calculated, but what if there are unexpected failures? What if the engines aren’t working to full capacity or the takeoff field is wet? Will the aircraft still reach the required speed for takeoff? According to Boeing, 13% of fatal aircraft accidents occur during takeoff. In fact, pilot errors, not maintenance failures, are responsible for the vast majority of all aviation accidents. This isn’t surprising considering it’s largely left to the pilot’s subjective opinion to determine a response when something goes wrong.

The problem lies in how information is presented to the pilot inside the cockpit. It’s hard to focus on flying when you have to read and quickly analyze the text on a bunch of small instruments and screens all around you. AR technology can display this information in a more intuitive format. For instance, with smart glasses, information like pre-flight checklists, step-by-step instructions, current weather and air traffic information, even a 3D graphic of the takeoff path can appear overlaid in a pilot’s vision before takeoff. Aero Glass actually has a solution that displays flight path and instrument data to small airline pilots wearing smart glasses. The same cockpit information a pilot might get using physical controls and touch screens can be retrieved instead by voice command; and when a snap decision needs to be made during a flight, AI technology can pick out the most relevant information to display to the pilot.

XR in Flight Service?

The benefits of integrating AR glasses and VR headsets into aircraft assembly and technician training are tangible today, but at this point airlines have merely proposed ideas for using XR in the air without seriously investing. This is probably due to the consumer-facing nature of the in-flight experience. Providing flight attendants with smart glasses to interact with passengers or offering VR headsets as in-flight entertainment are not critical use cases like the need to quickly train thousands of new pilots. Moreover, the timeline for mainstream consumer use of AR and VR is still unclear.

XR hasn’t yet transformed the experience of flying, but some airlines are considering it. Air New Zealand, for example, had its crew members try out HoloLens to expedite and provide more tailored customer service during the flight. To cater to individual passengers, flight attendants might access their flight details (to help make connections), food allergies (to personalize meals), even their emotional state (facial recognition tech). Air France trialed VR headsets for in-flight, immersive entertainment; and though not in the air Lufthansa has used VR to sell upgrades to premium class right at the gate. Who knows? Maybe one day those safety instructions in your seat pocket will be replaced by a virtual reality video. In the meantime, rest assured that XR technologies are improving aviation operations behind the scenes, from the hangar to the cockpit.

The Enterprise Wearable Technology Summit (EWTS) is an annual conference dedicated to the use of wearable technology for business and industrial applications. As the leading event for enterprise wearables, EWTS is where enterprises go to innovate with the latest in wearable tech, including heads-up displays, AR/VR/MR, body- and wrist-worn devices, and even exoskeletons. The 5th annual EWTS will be held October 9-10, 2018 at The Fairmont in Austin, TX. For more details, please visit the conference website.

Further Reading