The AI Test Force

  • Published
  • By Tyler Prince
  • Airman Magazine


Artificial Intelligence & the Test Enterprise



  Autonomy is a perfect problem for test, this is where we’re the happiest, in the middle of a problem that is uncertain and ill-defined, that’s our bread and butter.

- Col. James “Fangs” Valpiani, Commandant, USAF Test Pilot School

 

  The U.S. Air Force’s test enterprise has been at the vanguard of some of the most consequential technological advancements in aviation’s history. From Chuck Yeager breaking the sound barrier in 1947 and the first landings of the space shuttle in 1981, to today’s testing of Artificial Intelligence in manned and unmanned aircraft, America’s Air Force has pushed the limits of human endurance and innovation for decades.

  While AI and autonomy might appear as recent phenomena, bolstered by the accessibility of Large Language Models like ChatGPT, Google Gemini, and others, their roots in aviation stretch back over a century. “Autonomy in the rules-based form has been around since really the dawn of aviation,” said Col. James “Fangs” Valpiani, U.S. Air Force Test Pilot School commandant.

  “The first automated aircraft, the first autopilots were implemented in the early 1910s, so right after the Wright Brothers. You can accomplish an incredible amount with that kind of autonomy, in fact, the Air Force has all manner of automated drones as well as automated space probes that use this type of coding.” 

  Recent developments have markedly increased the scope of AI, enhancing tasks as routine as paperwork processing and as complex as the advancement of kinetic warfare technologies. The Air Force Test Center is at the forefront of the Air Force's exploration into AI and autonomy and leverages its vast institutional knowledge, resources and the skilled ingenuity and creativity of its testing experts to craft innovative solutions that address the challenges confronting the Air Force today and into the future.

  Col. James “FANGS” Valpiani, United States Air Force Test Pilot School commandant, poses for a photo at Edwards Air Force Base, Calif., March 6, 2024. The school is where the Air Force's top pilots, navigators and engineers learn how to conduct flight tests and generate the data needed to carry out test missions. In addition to pilots, they also train weapons systems officers, combat systems officers and Guardians. (U.S. Air Force photo by Tech. Sgt. Janiqua P. Robinson)

 

  Recognizing AI’s nearly limitless potential to shape the future of defense and technology, the Air Force is rigorously exploring the vast opportunities of machine-learning and AI-enabled autonomy to expedite critical decision-making and ensure readiness for tomorrow’s challenges. (U.S. Air Force Video by Tyler Prince)

 

The Air Force Test Center



  When you look at leading-edge capabilities like AI, the Air Force Test Center is at the vanguard of refining and evolving these technologies.

- Maj. Gen. Evan C. Dertien, AFTC commander


  Originally the Air Force Flight Test Center, AFTC was redesignated in 2012 and is headquartered at Edwards Air Force Base, California. It manages the Air Force's comprehensive flight test operations, including three primary test bases: Edwards AFB, Eglin AFB, Florida and Arnold AFB, Tennessee. Maj. Gen. Evan C. Dertien, AFTC commander, emphasized the center's critical role in propelling the development of autonomous aircraft technologies. 

  The test enterprise’s culture of innovation and history of technological advancements plays a crucial role in shaping the integration of AI within the Air Force. Dertien explained, "We possess the necessary safety protocols, testing infrastructure, data and risk management capabilities. So, it's only logical as the Air Force starts to adapt across both warfighting and personnel platforms that the test center would be at the forefront. We're excited to be the caretakers of evolving that technology.”



  Maj. Gen. Evan C. Dertien, commander of the Air Force Test Center, poses for a photo at Edwards Air Force Base, Calif., March 5, 2024. The AFTC conducts research, and developmental tests, and evaluates air, ground, space, and cyber systems, which provide timely, objective, and accurate information to decision-makers about the systems being tested. (U.S. Air Force photo by Tech. Sgt. Janiqua P. Robinson)

 

X-62 VISTA



  The X-62 VISTA provides the world’s first and currently only platform in which we can test very low-maturity software solutions that may perform poorly … and the boundaries of the system are such that both the aircraft and the crew that serve as safety monitor are never really at risk.

- Col. James “Fangs” Valpiani, Commandant, USAF Test Pilot School

 

  The Test Pilot School (TPS) and the X-62 Variable Stability In-flight Simulator Test Aircraft (VISTA) are leading the way in AI and autonomy testing. This platform enables flight test professionals at TPS to delve into the optimal application of AI in kinetic warfare platforms. Initially developed as a custom F-16 test bed for exploring thrust vectoring — a series of tests that laid the groundwork for the F-22's thrust-vectoring capabilities — the X-62 (previously known as the NF-16) has been adapted for various purposes over the years. “What makes the X-62 unique is that it can mimic the dynamics and handling characteristics of a diverse array of aircraft, from a C-17 to a 747, and even a Cessna," Valpiani noted. This versatility renders it an exceptional resource for training TPS students on numerous aircraft types. "We recognized that by virtue of being able to simulate a wide range of flight dynamics, the X-62 had the potential, if we could layer on autonomy, to become a game-changing platform once again. We've collaborated with the Air Force Research Laboratory (AFRL), the Defense Advanced Research Projects Agency (DARPA) and AFTC in recent years to develop what we call an 'autonomy sandbox.'"

  The X-62 is designed for experimenting with autonomy based on machine learning, diverging from the more traditional rules-based programming historically employed in autonomous systems. There are challenges faced with this type of coding, however, namely, its necessity for a predictable environment, where human experts can code clear, universally applicable rules.


  The X-62 VISTA receives post-flight maintenance after a flight at Edwards Air Force Base, Calif., March 7, 2024. VISTA stands for Variable Stability In-flight Simulator Test Aircraft and is operated by the Air Force Test Pilot School with the support of Calspan and Lockheed Martin. (U.S. Air Force photo by Tech. Sgt. Janiqua P. Robinson)



  The X-62 VISTA is prepped for takeoff at Edwards Air Force Base, Calif., March 7, 2024. VISTA stands for Variable Stability In-flight Simulator Test Aircraft and is operated by the Air Force Test Pilot School with the support of Calspan and Lockheed Martin. (U.S. Air Force photo by Tech. Sgt. Janiqua P. Robinson)


  “If you think about the kinds of challenges that the Air Force and aviation in general faces going forward, we’re talking about very dynamic environments,” Valpiani explained. “To tie this type of autonomy into everyday life, consider the cruise control function on your car, it's a very basic system, it’s expert coded, you tell it what to do … if you set it to 30 mph, it’ll hold that setting until you tell it to stop,” Valpiani said. “But if you were to try to generate a set of code and rules for how the vehicle should drive through say, downtown New York City traffic, you can imagine how a dense urban environment — with pedestrians jay-walking, bicycles out of their designated lanes, etc. —  is an extraordinarily dynamic situation.” Machine-learning algorithms help to solve the complex problems presented by dynamic situations by using statistical weightings and mathematical formulas to derive patterns that humans wouldn’t necessarily be able to see on their own.

​  “Machine learning has great promise for Air Force aviation, but the nature of machine-learning algorithms is that it’s hard for humans to scrutinize and understand these algorithms, we can ask it to do the same thing twice, and potentially get two completely different outcomes,” Valpiani said. Solving this unpredictability element is particularly important for military applications, especially combat autonomy, where understanding and predicting behavior is crucial. “As a result, we need a more robust safety environment with which to test and develop machine learning before we can be certain that they’re usable in less controlled environments,” Valpiani explained.


  The X-62 Variable In-Flight Simulator Test Aircraft (VISTA) flies in the skies over Edwards Air Force Base, California, March 23, 2023. (U.S. Air Force photo by Ethan Wagner)



  The X-62 Variable In-Flight Simulator Test Aircraft (VISTA) flies in the skies over Edwards Air Force Base, California, Aug. 26, 2022. (U.S. Air Force photo by Kyle Brasier)


  The researchers at TPS don’t just test these platforms for efficacy. One of the key aspects of testing is ensuring that the machine learning algorithms used in the VISTA (and other platforms) conform to the same rules of responsible use that the Air Force expects of all human pilots, “so in our partnerships with DARPA and other agencies, our main focus has been ensuring that their algorithms developed in simulated environments comply with those rules and norms of responsible use,” Vapliani stressed. “Then when we bring them over into real-world application, we’re assessing the same compliance and responsibility characteristics to ensure that they behave according to all the same rules that we hold ourselves to as human pilots.”

  “You really can’t talk about AI without considering the ethical and responsible use of the technology. Everything we’re doing right now from the grassroots forward is to make sure we comply with the DOD’s five principles for the responsible development of AI.”
Maj. Gen. Evan C. Dertien, Commander, Air Force Test Center.

  The principles guiding these tests are applied Air Force-wide, irrespective of the platform being tested. “You really can’t talk about AI without considering the ethical and responsible use of the technology,” Dertien emphasized. “Ultimately everything we’re doing right now is trying to develop and mature AI responsibly and ethically. The DoD has its five principles for the ethical development of AI: AI must be responsible, repeatable, traceable, governable and equitable, and everything we’re doing right now from the grassroots forward is to make sure we comply with those principles.”

 

  The X-62 Variable Stability In-flight Simulator Test Aircraft is prepped for takeoff at Edwards Air Force Base, Calif., March 7, 2024. VISTA is operated by the Air Force Test Pilot School with the support of Calspan and Lockheed Martin. (U.S. Air Force photo by Tech. Sgt. Janiqua P. Robinson)

 

The Airpower Foundations Combined Test Force



  Being AI-ready is about a lot more than just hardware, it’s about infrastructure, processes, and policy.

- Lt. Col. Michael “Cap” Pacini, Commander, 416th Flight Test Squadron

 

  The primary focus of the Air Force's test enterprise is the development of hardware and software, yet the approach to advancing AI and autonomy for kinetic platforms encompasses much more. Lt. Col. Michael “Cap” Pacini, 416th Flight Test Squadron commander (FLTS) and the Airpower Foundations Combined Test Force director, lays out the myriad of necessities for conducting this research effectively. “Being AI-ready is about a lot more than just hardware. It’s infrastructure, it’s processes and it’s policy. It’s establishing how we’re going to test autonomy. Our Emerging Technologies Integrated Test Force is very much focused on that problem, making sure that we have the infrastructure we need to remain at the cutting edge of flight test.“ The Airpower Foundations Combined Test Force is comprised of three integrated test forces (ITF); the T-38 and F-16 ITF, the T-7A Redhawk ITF, and the Emerging Technologies ITF (ET-ITF), covering the entire spectrum of airpower development and ensuring that test professionals have the resources they need to succeed.
  This endeavor relies heavily on collaborations with the private sector and academic institutions, which are vital to the testing activities across the Air Force. “It’s critical. When I look back at the late ‘80s or early ‘90s. It used to be growing up as a kid that all the greatest technologies in the civilian world came from the military. GPS is a prime example of a technology that was developed by the military that we use in our everyday lives,” Pacini explained, “Now, that trend has reversed, with civilian advancements often leading the way. Our role in the military is to identify how we can adapt these commercial innovations to meet the needs of the DoD.”

  To bring these technologies to bear, the test enterprise balances rigorous test and evaluation methods with speed to ensure the return on investment meets the Air Force’s expectations. “Everything about flight test is about risk reduction,” Pacini explained. “We could conduct no test and turn over new platforms directly to the combat air forces … shifting all the risk to our operational warfighters. Alternatively, we could test platforms endlessly, reducing risk, but never delivering the needed capabilities. Maintaining a balance of diligence and agility is vital to ensure that we deliver novel warfighting capabilities safely and expediently.”



  Lt. Col. Michael “CAP” Pacini, commander of the 416th Flight Test Squadron, and director of the Air Power Foundations Combined Test Force, poses for a photo at Edwards Air Force Base, Calif., March 6, 2024. The Airpower Foundation's Combined Task Force comprises three different integrated test forces; F-16  & T-38, T-7A,  and emerging technologies, and covers the entire spectrum of air power development. (U.S. Air Force photo by Tech. Sgt. Janiqua P. Robinson)

 

The Future is Now 



   It’s the people and their focus on getting it right that will make the difference. Not just making sure these platforms are efficient and safe, but also responsible. That’s the most important aspect of autonomy, that it’s something we can trust and employ responsibly in the service of our national defense and ultimately, for the betterment of humanity.

- Col. James “Fangs” Valpiani, Commandant, USAF Test Pilot School

 

  The age of AI isn’t coming, it’s arrived, and its potential to augment the prowess of Airmen and Guardians that serve our nation is nearly limitless. The Air Force’s test community has long been at the forefront of researching, developing, and implementing these technologies, daring into the unknown to advance warfighting potential. “The test profession at the end of the day is about exploring the unknown in a disciplined fashion.” Valpiani described, “Here, we have a battle space where the dominant feature of the platforms we expect to see is the unknown. How do we ensure that these untested technologies are responsible and behave in a way that we expect? That is why autonomy is a perfect problem for test; this is where we’re the happiest, in the middle of a problem that is uncertain, ill-defined, with some risk, maybe a lot of risk … add in a little time pressure and some competition, and now you’ve got the perfect recipe for a test challenge.”

  AI and autonomous systems are set to fundamentally transform Air Force operations, from sophisticated kinetic platforms like the X-62 VISTA to the basic tasks of daily airmanship, ensuring they reflect the Air Force's ethos will have a significant and lasting influence. “I hope that in 10 or 20 years from now when people look back at this moment in history, they will look back as well at the Air Force’s test professionals and think of this community as a place where the focus on responsible development of autonomy for aviation really took off,” Valpiani expressed. “Ultimately, it’s the people and their focus on getting it right that will make the difference. Not just making sure it’s efficient and safe, but also responsible. To me, that’s the most important aspect of autonomy, that it’s something that we can trust and employ responsibly in the service of our national defense, and ultimately, for the betterment of humanity.”

 

Read more from the Airman Magazine issue exploring AI in the Air Force

 
AIRMAN MAGAZINE