1 00:00:02,119 --> 00:00:03,897 Machine learning algorithms are 2 00:00:03,897 --> 00:00:06,360 distinct from other kinds of autonomy 3 00:00:06,369 --> 00:00:08,449 in that they use these powerful 4 00:00:08,460 --> 00:00:10,329 statistical methods to identify 5 00:00:10,340 --> 00:00:12,210 patterns in large sets of data . 6 00:00:12,239 --> 00:00:15,130 They're not easy for humans to read and 7 00:00:15,140 --> 00:00:17,084 understand and predict how they'll 8 00:00:17,084 --> 00:00:19,489 perform once they're implemented in a 9 00:00:19,500 --> 00:00:20,889 real world environment . 10 00:00:28,500 --> 00:00:32,380 So I'm Colonel James Valiani call sign 11 00:00:32,389 --> 00:00:34,840 fangs . I'm the commandant of the US 12 00:00:34,849 --> 00:00:37,069 Air Force Test pilot school . The test 13 00:00:37,080 --> 00:00:39,869 pilot school has three related missions . 14 00:00:39,880 --> 00:00:41,713 One is to create highly adaptive 15 00:00:41,713 --> 00:00:43,769 critical thinking test leaders . The 16 00:00:43,769 --> 00:00:45,824 second is to develop our staff , our 17 00:00:45,824 --> 00:00:47,991 world class faculty here and the third 18 00:00:47,991 --> 00:00:50,069 is to advance test research for the 19 00:00:50,080 --> 00:00:51,524 overall test enterprise . 20 00:00:54,360 --> 00:00:57,150 So autonomy in the rules based form has 21 00:00:57,159 --> 00:00:59,437 been around since the dawn of aviation . 22 00:00:59,437 --> 00:01:01,381 The first automated aircraft , the 23 00:01:01,381 --> 00:01:03,381 first autopilots or in the early 19 24 00:01:03,381 --> 00:01:05,215 tens . So right after the Wright 25 00:01:05,215 --> 00:01:07,930 Brothers , this kind of autonomy takes 26 00:01:07,940 --> 00:01:10,550 the form of rules that experts derive 27 00:01:10,559 --> 00:01:12,769 and code . So if this , then this if 28 00:01:12,779 --> 00:01:15,470 that do that sort of coding and you can 29 00:01:15,480 --> 00:01:17,536 accomplish an incredible amount with 30 00:01:17,536 --> 00:01:20,110 that kind of autonomy . The problem 31 00:01:20,120 --> 00:01:22,176 with this approach though is that it 32 00:01:22,176 --> 00:01:24,231 requires a kind of environment and a 33 00:01:24,231 --> 00:01:26,342 system that people can understand and 34 00:01:26,342 --> 00:01:29,269 drive rules for that are clear and 35 00:01:29,279 --> 00:01:31,335 applicable no matter how the context 36 00:01:31,335 --> 00:01:35,040 changes go back 37 00:01:35,050 --> 00:01:37,050 to a traditional airplane where you 38 00:01:37,050 --> 00:01:38,939 literally have cables and pulleys 39 00:01:38,939 --> 00:01:41,106 connected . It is very clear like when 40 00:01:41,106 --> 00:01:43,106 I move the stick in a certain place 41 00:01:43,106 --> 00:01:45,161 that the flight controls should move 42 00:01:45,161 --> 00:01:47,272 exactly this much in relation to that 43 00:01:47,272 --> 00:01:49,494 input . And then in the F-16 , that's a 44 00:01:49,494 --> 00:01:51,717 digital fly by wire , I still generally 45 00:01:51,717 --> 00:01:53,939 so that if I put the stick in a certain 46 00:01:53,939 --> 00:01:53,775 place in a very specific flight 47 00:01:53,785 --> 00:01:55,754 condition that the aircraft flight 48 00:01:55,764 --> 00:01:57,820 controls should respond in a certain 49 00:01:57,820 --> 00:01:59,875 way as we continue to advance . It's 50 00:01:59,875 --> 00:02:02,042 not that way where I can get into an a 51 00:02:02,042 --> 00:02:04,320 very specific flight control condition . 52 00:02:04,320 --> 00:02:06,542 I can put in the exact same control and 53 00:02:06,542 --> 00:02:08,542 put two times and get two different 54 00:02:08,542 --> 00:02:10,264 answers that becomes much more 55 00:02:10,264 --> 00:02:13,070 difficult to test in some ways . I'm 56 00:02:13,080 --> 00:02:15,247 Lieutenant Colonel Michael Puccini , I 57 00:02:15,247 --> 00:02:17,302 go by cap , I'm the commander of the 58 00:02:17,302 --> 00:02:19,413 4/16 flight test squadron and I'm the 59 00:02:19,413 --> 00:02:21,580 director of the Air Power Foundation's 60 00:02:21,580 --> 00:02:23,691 combined test force autonomy ready to 61 00:02:23,691 --> 00:02:25,969 me is is a lot more than just hardware . 62 00:02:25,969 --> 00:02:28,191 It's infrastructure , its processes and 63 00:02:28,191 --> 00:02:30,339 policy . It's establishing how we're 64 00:02:30,350 --> 00:02:32,440 going to test autonomy . Emerging 65 00:02:32,449 --> 00:02:34,671 technologies has very much been focused 66 00:02:34,671 --> 00:02:36,505 on that problem making sure that 67 00:02:36,505 --> 00:02:38,727 Edwards Air Force Base specifically has 68 00:02:38,727 --> 00:02:40,727 the infrastructure to really be the 69 00:02:40,727 --> 00:02:42,949 center of the testing universe . And in 70 00:02:42,949 --> 00:02:42,460 order to be that we have to be able to 71 00:02:42,470 --> 00:02:44,581 test the latest greatest technology , 72 00:02:44,830 --> 00:02:47,139 the combination of that skill set and 73 00:02:47,149 --> 00:02:49,316 that team that's pursuing that problem 74 00:02:49,316 --> 00:02:51,427 along with traditional flight tests . 75 00:02:51,427 --> 00:02:53,427 So if we need those capabilities in 76 00:02:53,427 --> 00:02:55,649 combat , the work that we do in advance 77 00:02:55,649 --> 00:02:54,830 is what's going to determine how 78 00:02:54,839 --> 00:02:55,950 successful we are . 79 00:02:58,800 --> 00:03:00,578 If you think about the kinds of 80 00:03:00,578 --> 00:03:02,744 challenges and opportunities that face 81 00:03:02,744 --> 00:03:04,578 the air force and in aviation in 82 00:03:04,578 --> 00:03:06,800 general , going forward , we're talking 83 00:03:06,800 --> 00:03:08,911 about the kinds of environments where 84 00:03:08,911 --> 00:03:10,967 the context is very dynamic . That's 85 00:03:10,967 --> 00:03:12,800 where machine learning , a whole 86 00:03:12,800 --> 00:03:14,911 different kind of autonomy has proven 87 00:03:14,911 --> 00:03:16,967 very promising . So machine learning 88 00:03:16,967 --> 00:03:18,689 essentially uses very advanced 89 00:03:18,689 --> 00:03:20,744 statistical tools to derive patterns 90 00:03:20,744 --> 00:03:22,744 from large quantities of data . The 91 00:03:22,744 --> 00:03:24,967 beauty of the approach or the advantage 92 00:03:24,967 --> 00:03:26,967 and the promise of this approach is 93 00:03:26,967 --> 00:03:28,744 that we can derive very complex 94 00:03:28,744 --> 00:03:28,149 patterns that humans couldn't 95 00:03:28,160 --> 00:03:30,389 necessarily determine on their own 96 00:03:30,490 --> 00:03:33,380 problem is that those solutions are not 97 00:03:33,389 --> 00:03:35,830 easily evaluated . They take the form 98 00:03:35,839 --> 00:03:38,929 of statistical weightings and complex 99 00:03:38,940 --> 00:03:40,940 mathematical expressions that don't 100 00:03:40,940 --> 00:03:43,070 translate to an intuition about what 101 00:03:43,080 --> 00:03:46,389 the system is going to do and why that 102 00:03:46,399 --> 00:03:48,759 matters from an X 62 perspective is 103 00:03:48,770 --> 00:03:50,770 these machine learning applications 104 00:03:50,770 --> 00:03:52,830 have tremendous promise for aviation 105 00:03:52,839 --> 00:03:55,117 and Air force , aviation in particular . 106 00:03:55,240 --> 00:03:57,270 But because they're not easily 107 00:03:57,279 --> 00:03:59,501 scrutinized , it's difficult to predict 108 00:03:59,501 --> 00:04:01,612 exactly how they're going to behave . 109 00:04:01,612 --> 00:04:03,223 We need a more robust safety 110 00:04:03,223 --> 00:04:05,279 environment within which to test and 111 00:04:05,279 --> 00:04:07,446 develop them before we're certain that 112 00:04:07,446 --> 00:04:09,529 they're usable in less controlled 113 00:04:09,539 --> 00:04:12,550 environments . And the X 62 provides 114 00:04:12,559 --> 00:04:14,726 really the world's first and currently 115 00:04:14,726 --> 00:04:16,948 only platform on which we have a safety 116 00:04:16,948 --> 00:04:19,115 sandbox , an autonomy sandbox in which 117 00:04:19,115 --> 00:04:21,170 we can place software solutions that 118 00:04:21,170 --> 00:04:23,281 are not high in maturity and they can 119 00:04:23,281 --> 00:04:25,503 behave very poorly . But the boundaries 120 00:04:25,503 --> 00:04:27,615 of the system are such that they will 121 00:04:27,615 --> 00:04:29,726 protect the aircraft so that the crew 122 00:04:29,726 --> 00:04:31,892 that's sitting inside that serves as a 123 00:04:31,892 --> 00:04:34,170 safety monitor is never really at risk . 124 00:04:35,630 --> 00:04:37,797 Everything about flight tests is about 125 00:04:37,797 --> 00:04:39,852 risk reduction . We can do no flight 126 00:04:39,852 --> 00:04:41,908 tests and turn it over to the combat 127 00:04:41,908 --> 00:04:44,130 air forces . It's probably not going to 128 00:04:44,130 --> 00:04:46,352 go well , that would be the extreme and 129 00:04:46,352 --> 00:04:48,519 we're putting all that risk now on our 130 00:04:48,519 --> 00:04:50,741 war fighter . But if I can do testing , 131 00:04:50,741 --> 00:04:52,963 that doesn't involve a direct risk to a 132 00:04:52,963 --> 00:04:55,130 human , I can potentially advance it a 133 00:04:55,130 --> 00:04:57,352 lot quicker . And the fact that we have 134 00:04:57,352 --> 00:04:59,463 that luxury really helps posture risk 135 00:04:59,463 --> 00:05:01,519 to where we can potentially expedite 136 00:05:01,519 --> 00:05:03,630 delivery of capabilities . Due to the 137 00:05:03,630 --> 00:05:05,852 complexity of these modifications , the 138 00:05:05,852 --> 00:05:08,019 aircraft was officially designated the 139 00:05:08,019 --> 00:05:10,239 N F-16 D . So the X 62 began life as a 140 00:05:10,250 --> 00:05:12,799 bespoke F-16 and it was really designed 141 00:05:12,809 --> 00:05:14,587 as a research platform from the 142 00:05:14,587 --> 00:05:16,365 beginning . So it , it's , it's 143 00:05:16,365 --> 00:05:18,365 actually designed physically in its 144 00:05:18,365 --> 00:05:20,142 construction differently than a 145 00:05:20,142 --> 00:05:22,198 production of 16 . Its first mission 146 00:05:22,198 --> 00:05:24,869 was for thrust vectoring research and 147 00:05:24,880 --> 00:05:26,824 that really was the genesis or the 148 00:05:26,824 --> 00:05:28,769 precursor to the F 20 two's thrust 149 00:05:28,769 --> 00:05:30,880 vectoring capability which makes that 150 00:05:30,880 --> 00:05:32,880 aircraft the most lethal air to air 151 00:05:32,880 --> 00:05:35,102 combat fighter on the planet . So the N 152 00:05:35,102 --> 00:05:37,047 F-16 as it was known at the time , 153 00:05:37,047 --> 00:05:39,269 really began its life as an enabler for 154 00:05:39,269 --> 00:05:41,380 cutting edge technology . And that of 155 00:05:41,380 --> 00:05:43,436 course , is its story throughout its 156 00:05:43,436 --> 00:05:47,209 entire life , really 157 00:05:47,220 --> 00:05:49,459 allowed the air force to test a whole 158 00:05:49,470 --> 00:05:51,414 series of different flight control 159 00:05:51,414 --> 00:05:53,637 systems as it went on to develop future 160 00:05:53,637 --> 00:05:55,940 platforms . What makes that aircraft 161 00:05:55,950 --> 00:05:59,059 unique is that it can represent the 162 00:05:59,070 --> 00:06:01,579 dynamics and the feel of a wide range 163 00:06:01,589 --> 00:06:03,700 of aircraft . We recognize that the 164 00:06:03,709 --> 00:06:05,876 aircraft had the potential if we could 165 00:06:05,876 --> 00:06:09,429 layer on autonomy to it to be once 166 00:06:09,440 --> 00:06:11,440 again a game changing platform . So 167 00:06:11,440 --> 00:06:13,162 over the next several years in 168 00:06:13,162 --> 00:06:15,162 partnership with a FRL with DARPA , 169 00:06:15,162 --> 00:06:17,051 with the Air Force Test Center to 170 00:06:17,051 --> 00:06:19,384 create what we call an autonomy sandbox , 171 00:06:19,384 --> 00:06:21,384 which allows us to place autonomous 172 00:06:21,384 --> 00:06:23,607 agents that control the aircraft into a 173 00:06:23,607 --> 00:06:25,218 bounded safety environment . 174 00:06:28,200 --> 00:06:31,350 The aircraft can still represent ac 17 175 00:06:31,359 --> 00:06:33,850 or 747 . And on top of that , we can 176 00:06:33,859 --> 00:06:36,279 layer on autonomous control of those 177 00:06:36,290 --> 00:06:39,140 aircraft to see how they perform . And 178 00:06:39,149 --> 00:06:42,049 so once again , the N F-16 , which as a 179 00:06:42,059 --> 00:06:44,570 result of this autonomy became the X 62 180 00:06:44,730 --> 00:06:46,769 is on the cutting edge of aviation 181 00:06:46,779 --> 00:06:49,359 research down 182 00:06:54,970 --> 00:06:57,369 test pilot school and the X 62 in 183 00:06:57,380 --> 00:07:00,029 particular , give us the ability to 184 00:07:00,230 --> 00:07:04,089 develop an intuition for how and when 185 00:07:04,100 --> 00:07:06,044 to trust and employ these kinds of 186 00:07:06,044 --> 00:07:07,933 systems . And a key aspect of our 187 00:07:07,933 --> 00:07:10,100 autonomy testing here is ensuring that 188 00:07:10,100 --> 00:07:12,044 the algorithms that we are testing 189 00:07:12,119 --> 00:07:14,589 conform to the expectations of 190 00:07:14,600 --> 00:07:18,309 responsible use . There are training 191 00:07:18,320 --> 00:07:22,230 rules and expectations of conduct that 192 00:07:22,239 --> 00:07:24,589 all human fighter pilots are expected 193 00:07:24,600 --> 00:07:26,829 to adhere to in our partnership with 194 00:07:26,839 --> 00:07:28,728 DARPA and other agencies that are 195 00:07:28,728 --> 00:07:30,672 focused on developing A I agents . 196 00:07:30,679 --> 00:07:32,720 We've studied closely as they've 197 00:07:32,730 --> 00:07:34,529 developed their algorithms in a 198 00:07:34,540 --> 00:07:36,651 simulation environment to comply with 199 00:07:36,651 --> 00:07:38,760 those norms of responsible use . And 200 00:07:38,769 --> 00:07:40,936 then when we bring them over into real 201 00:07:40,936 --> 00:07:44,005 world application in the X 62 from the 202 00:07:44,015 --> 00:07:45,855 simulation environment , we are 203 00:07:45,864 --> 00:07:48,024 assessing the same responsibility 204 00:07:48,035 --> 00:07:50,202 characteristics to ensure that they do 205 00:07:50,202 --> 00:07:52,368 behave according to all the same rules 206 00:07:52,368 --> 00:07:54,313 that we hold ourselves to as human 207 00:07:54,313 --> 00:07:56,445 pilots . The X 62 allows us to build 208 00:07:56,454 --> 00:07:58,454 confidence in the system . Not only 209 00:07:58,454 --> 00:08:00,287 from the perspective of are they 210 00:08:00,287 --> 00:08:02,343 efficacious ? Do they do the mission 211 00:08:02,343 --> 00:08:04,454 that we set them out to do ? What are 212 00:08:04,454 --> 00:08:06,510 they responsible ? Do they behave in 213 00:08:06,510 --> 00:08:09,000 such a way that they adhere to the 214 00:08:09,010 --> 00:08:11,279 norms and expectations of responsible 215 00:08:11,290 --> 00:08:13,279 use that we have for all humans in 216 00:08:13,290 --> 00:08:17,100 these environments ? You know , 217 00:08:17,109 --> 00:08:19,165 test really at the end of the day is 218 00:08:19,165 --> 00:08:20,998 about exploring the unknown in a 219 00:08:20,998 --> 00:08:23,220 disciplined fashion . So here we have a 220 00:08:23,220 --> 00:08:25,570 battle space where the dominant feature , 221 00:08:25,730 --> 00:08:28,519 the data driven characteristic of the 222 00:08:28,529 --> 00:08:30,418 platforms we expect to see is the 223 00:08:30,418 --> 00:08:32,529 unknown . How do we trust ? How do we 224 00:08:32,529 --> 00:08:34,696 know how do we , how do we ensure that 225 00:08:34,696 --> 00:08:36,640 they're responsible as the perfect 226 00:08:36,640 --> 00:08:38,751 problem for test ? That is our , it's 227 00:08:38,751 --> 00:08:38,659 our bread and butter . Right . This is 228 00:08:38,669 --> 00:08:40,640 where we , our happiest is in the 229 00:08:40,650 --> 00:08:42,950 middle of a problem that's uncertain , 230 00:08:43,169 --> 00:08:45,391 ill defined , there's some risk , maybe 231 00:08:45,391 --> 00:08:47,558 a lot of risk . And now you've got the 232 00:08:47,558 --> 00:08:49,669 recipe for a perfect test challenge . 233 00:08:49,669 --> 00:08:51,780 That is what the test pilot school is 234 00:08:51,780 --> 00:08:52,780 for . 235 00:08:56,030 --> 00:08:59,130 I hope that 10 years from now , 20 236 00:08:59,140 --> 00:09:01,362 years from now when people look back at 237 00:09:01,362 --> 00:09:03,418 this moment in history , I hope that 238 00:09:03,418 --> 00:09:06,200 they will look back as well at the test 239 00:09:06,210 --> 00:09:09,020 pilot school in the X 62 . And they 240 00:09:09,030 --> 00:09:12,590 will think of this community as a 241 00:09:12,599 --> 00:09:16,020 place where the focus on 242 00:09:16,030 --> 00:09:18,780 responsible development of autonomy 243 00:09:18,789 --> 00:09:22,700 really took off in aviation . The X 62 244 00:09:22,770 --> 00:09:26,400 is really just one piece of technology . 245 00:09:26,750 --> 00:09:28,972 It's the people here and their focus on 246 00:09:28,972 --> 00:09:31,194 getting it right with respect to combat 247 00:09:31,194 --> 00:09:33,306 autonomy and autonomy and aviation in 248 00:09:33,306 --> 00:09:35,417 general and making sure that it's not 249 00:09:35,417 --> 00:09:37,361 just efficient and safe , but also 250 00:09:37,361 --> 00:09:39,528 responsible that , that I hope uh is a 251 00:09:39,528 --> 00:09:41,417 hallmark of our contribution as a 252 00:09:41,417 --> 00:09:43,583 community . That is the most important 253 00:09:43,583 --> 00:09:45,361 aspect of autonomy is that it's 254 00:09:45,361 --> 00:09:47,528 something that we can trust and employ 255 00:09:47,528 --> 00:09:49,528 responsibly in the service of , you 256 00:09:49,528 --> 00:09:49,380 know , national defense and really , 257 00:09:49,390 --> 00:09:51,223 ultimately for the betterment of 258 00:09:51,223 --> 00:09:51,099 humanity .