Students at Texas A&M University built an extended reality (XR) flight simulator that could potentially be used by the Texas Air National Guard as a cost-effective training tool for pilots. The simulator has been designed to work in conjunction with a mixed-reality deep-immersion headset developed by Passenger Inc., a company out of Austin, Texas.
After seeing the Retro Rocket simulator created by students in the Department of Aerospace Engineering and Department of Mechanical Engineering, Ron Maynard, CEO and founder of Passenger Inc., connected with Dr. Darren Hartl, associate professor in the Department of Aerospace Engineering, about building the F16 XR flight simulator.
Although Hartl works with virtual reality (VR), XR was new territory for him and his students. XR is a combination of VR and augmented reality (AR). VR is an immersive experience; think about the headsets gamers use today. It uses a computer-generated environment to replace what the user is viewing. AR takes a computer-generated view and overlays that onto our reality, such as Snapchat filters or Pokémon GO. XR lies between these two by blending virtual and physical worlds almost seamlessly, allowing the two to interact.
“When you’re sitting in the simulator, I don’t know that you can get any closer to feeling like you are flying an F16, visually and tactically, in terms of how you use and see your hands and other features of the experience that we have made,” said Hartl.
The physical elements of the design include toggle switches, functional buttons, HOTAS (hand on throttle and stick) controls, rudder pedals and a vibrating seat that immerses the user in the flight. There is also a multifunctional display onto which different VR elements can be cast, and it serves as a touchscreen. All these are fully tactile, and pilots can see their own hands as they manipulate each one. This is all within the aluminum frame designed by the students to be light, portable and modular.
The simulator also has a virtual landing gear lever controlled by hand tracking, so as the user places their hand over the virtual lever, it moves accordingly. The students also created a VR terrain equipped with pylons for racing agility simulations, missiles that the pilot can fire and a fully viewable aircraft around the pilot.
“The purpose was to demonstrate that we can use this for any airframe,” said Jesse Cate, aerospace engineering student. “The hand tracking was there to show that we might not even need a full physical cockpit, and the touchscreen was also important to show that we could replace the physical components as well if needed.”
This design shows that a system can be built quickly, economically and flexibly to give more pilots a chance to stay proficient regarding a range of mission tasks when not at the airfield.
Outside the project’s outcomes, the students gained a unique hands-on experience that elevated their field knowledge and technical skills.
“I had no idea there was anything like this before the Retro Rocket project started, but ever since then, I’ve been really interested in this field, so I wanted to join this project,” said Jake Schrass, aerospace engineering student. “What can we do with virtual reality? What can we do with mixed reality or augmented reality? And how does that fit into an aerospace context?”
It also created an opportunity for collaboration. When Hartl was first approached about the project, he reached out to Dr. Ann McNamara, associate dean for research and creative works and professor in the School of Performance, Visualization and Fine Arts, about advising the project together. This opened the door for aerospace engineering and visualization students to work together and gain real-world interdisciplinary experience.
“This project represents the excellence that can be achieved through interdisciplinary collaboration; students across aerospace and visualization worked alongside industry professionals to create a truly remarkable, highly functional prototype on a minimal budget,” said McNamara. “The results are compelling and impressive, demonstrating the high caliber of our students at Texas A&M. Hopefully, this is the beginning of future collaborations across disciplinary lines.”