Pages

Wednesday, September 9, 2020

AI Beats Fighter Pilot - Not Really

You’ve all heard about the recent test which pitted an artificial intelligence (AI) against a human fighter pilot in a simulator and the AI won 5 of 5 dogfights.  I can’t tell you how many people I’ve heard use that event as ‘proof’ that AI is more capable than humans and that we should be moving immediately to pure unmanned air combat aircraft.  Well, the results are not what they appeared to be.  The link below is to a review of the event by a former pilot and includes the actual simulator footage of the fights, analysis of the fights, explanations of the conditions of the fights, and discussion by the AI programmer representative.

I was stunned by the overwhelming advantage that the test conditions gave the AI.  For example, the AI was given perfect 360 degree situational awareness and perfect knowledge of not only its own performance but that of the human pilot’s aircraft which is utterly unrealistic.  Another example is that the AI merely had to have the target in a conical field of view to count as a gun hit – not actually place the piper on the target.  And so on. 

The results proved absolutely nothing.  I’m not going to go any further with this because the linked video does a much better and more complete job of analysis than I can.  All I’ll say is that you owe it to yourself to watch the video.  It will completely change whatever impression you had about the results.

I went into it thinking that I’d see some pretty impressive AI and I came away from it thinking the event was almost a waste of time for all parties and was tantamount to the usual staged war games and exercises that the military is so famous for.

For those who can see the imbedded video, below, check it out for yourself.  If you can't see the video, follow the link and see for yourself.


Link:  https://www.youtube.com/watch?v=ziCQqmEllZo



17 comments:

  1. Same here, was very disappointed, was expecting lot more. I almost feel DoD is getting shafted again by industry, IMO, what was the difference between this AI and just a regular flight sim game that I can get at Best Buy for $50 bucks?!? Some are saying this was spectacular, sorry, not seeing it, I played video games in the 80s!, yes, I'm old, that the computer put up a better and more interesting fight....

    ReplyDelete
  2. Yeah, the media overly hyped the competition. The machine-learning AIs all lost to the one that was programmed by a human.

    This is basically the equivalent of an air-to-air missile in terms of technology where you spend time programming in how the computer handles different situations to give it optimum performance. Similar techniques are used to program computer opponents in flight sims.

    "Artificial Intelligence" still has a long way to go since it clearly lost to the computer programmed by a human.

    ReplyDelete
  3. "the event was almost a waste of time for all parties"

    Except for those who profited off it, yes.

    ReplyDelete
  4. Probably the best test would be to take some drones, add some air to air weapons and watch them try to kill each other over the desert. Even without human pilots in the loop this should allow a good evaluation of the capabilities of the software. If you can use cheap enough drones, it need not be too pricey.

    ReplyDelete
    Replies
    1. We have boneyards full of actual aircraft that could be fitted out for such battles.

      Delete
  5. I kept noticing this trend with the military about how they are treating DARPA projects. I believe that historically (i might be wrong), DARPA programs are either basic research, foundational technologies and proof-of-concepts. Their research isn't aimed at making actual systems and weapons, they cooperate with larger defense companies and made it a reality. However the military decided to use whatever DARPA researched and hail it as the latest weapons that will be widely adopted without regards of how they are designed for (a laboratory condition). They use these demonstrations (historically they are as top secret as it gets and are only supposedly only shown to top generals and senators) as a publicity stunt and a promotional tool. Unfortunately for them, real military observers and analysts can see right through their PowerPoints and praising words. What a shameful display in my opinion.

    ReplyDelete
  6. I am somewhat reminded of video game AI like the "superhuman" who sometimes blow themselves up trying to shoot through walls, like Soldier of Fortune 2 review below:

    "Which brings us to the game's single biggest flaw - the AI. Put simply, every scruffy looking rebel soldier you come up against is in fact Superman wearing a really cunning disguise. He can see through walls (and will sometimes attempt to fire through them, with often hilarious results), poke arms and legs through solid objects, and consistently land a grenade at your feet from fifty feet away, over a hill, without being able to see you. "

    https://www.eurogamer.net/articles/r_sof2

    And of course below:

    https://tvtropes.org/pmwiki/pmwiki.php/Main/TheComputerIsACheatingBastard

    I'm sure had it actually been "physical" the AI may have/would crashed a long time ago (you can't noclip cheat in real-life)

    https://en.wikipedia.org/wiki/Noclip_mode

    Of course to experienced video game vets this is a somewhat well known common problem (at least from what I known, if you don't, I apologize and now you know) when the AI behaves in ways the player can't, as show by links as below:

    https://www.pcmag.com/opinions/unfair-advantage-dont-expect-ai-to-play-like-a-human

    ReplyDelete
  7. There is a miss concept among many that AI means computers think as human.

    NO! Today's tech is far far away from that.

    Basically, AI are written by human being. An AI runs on computers won't out smart than the one writes it. What a computer can outrun human are its consistency, fast response, precision memory, etc.

    Basically, the so called AI fight human was done on an F-16 simulator. Practically, you can say an AI software wins a video game over people.

    In real combats, many so called self-initiate weapons are piratically AI. For instance, if sensors detect a missile approach, it automatically fires a missile to intercept it. Like many technologies, in some cases, AI are great tools but not all cases. How good an AI depends on the one who write it (mainly algorithm behind it).

    ReplyDelete
  8. After watching and re-watching ComNavOps's linked youtube video, I think that the AI (not to mentioned the human pilot, Banger, was relatively inexperienced. Heck use a real experienced DCS player as suggested, into the sim and they may make the AI crash like 128th TIE Interceptor Squadron crashing into the surface of Gerrard V (in the video game)
    https://starwars.fandom.com/wiki/128th_TIE_Interceptor_Squadron )

    Also the AI would have shot itself down had it actually been physical like below doing that "really bad BFM" face-shots:

    https://www.popularmechanics.com/military/aviation/a27078461/dutch-f-16-flew-into-its-own-gunfire/

    As already typed there is no "noclip" cheat in real-life no matter how "in the cone" it is ,like the poor dutch F-16.

    https://arstechnica.com/tech-policy/2019/04/dutch-f-16-takes-cannon-fire-from-itself/

    Of course as in the video "we trust what works" and what what will likely work is the the AI will run into its own fired ammo in real life and likely shoot itself down (or damage itself)as plenty of real-life examples like below shows:

    " Sept. 21, 1956, test pilot Tom Attridge began a shallow dive in his F11F. As he did, he fired two short bursts from the aircraft's four 20mm cannons, and thought nothing of it – until he got to the end of his dive, and the bursts began to shoot up his aircraft. He started at 20,000 feet and then went into a Mach 1 dive as he fired. He accelerated with afterburner and at 13,000 feet, fired to empty. "

    (like the stuff the AI f-16 did in that video for example as explained in the "training rule thing" with all those face shots, unlike a skilled human pilot)

    https://www.wearethemighty.com/history/f11f-shot-itself-down

    And what's to prevent below if it really had been "physical"(using two pieces of tape no less) since there was no collision logic...

    https://www.technologyreview.com/2020/02/19/868188/hackers-can-trick-a-tesla-into-accelerating-by-50-miles-per-hour/

    ReplyDelete
  9. Thought somewhat the same thing, if a "face shot" is so difficult a shot and not really a guarantee for a human or AI, wouldn't you just let the AI go for face shots, let the AI F16 run out of ammo then shoot him down? Its tricky scary but if the AI stupidly wastes its ammo, why not???

    ReplyDelete
    Replies
    1. I noted that the human never made any adjustments. He just rode the exact same flight/fight profile to five straight defeats. Didn't exactly make humanity proud, did he?

      The human pilot seemed completely lost in the simulator - like he had no idea how to operate it or what it could do or not do. Those giant, wide circles that he was flying, for example - he seemed to have no idea of his speed, g's, or enemy position; I can't believe he'd do that in real combat. If he would then he's a horrible pilot. It just seemed like it was the first time he'd seen a simulator (and it was the first time he'd seen this one!) and he was just flying around trying to figure out how to do it.

      If an enemy, real or AI, had 'shot' you in the face three or four times in a row, wouldn't you change tactics or, at the very least, fire back, even if it thought it was a low probability shot … cause, clearly, in this simulator, it WASN'T A LOW PROBABILITY SHOT - so take advantage of it.

      I really hate to say it but the pilot could have been my pet cat for all the intelligence and adaptability he exhibited.

      So, what this spectacle demonstrated was that an AI, custom programmed for this simulator, and with all the advantages of perfect awareness, is capable of beating a pilot who was lost in his first time in a low quality simulator and made no adjustments as time went on.

      Delete
    2. I really wonder if the pilot had "orders" not to do anything fancy or change tactics. As you said, he pretty much flew in a circle the whole time, never tried anything different or try to go much into the vertical plane....

      Delete
    3. "The human pilot seemed completely lost in the simulator - like he had no idea how to operate it or what it could do or not do."

      Did he even have experience with the simulator before?

      Alternatively, NICO's explanation makes a lot of sense as well.

      Delete
    4. In the video, the YouTuber did mention about his doubts of the pilot competent in the simulator due to the same reasons listed before. It's highly possible that he was brought to be used as a test case scenario without any prior training with the simulator.

      Delete
  10. Food for thought:

    What if how the human pilot flew in the simulation is how pilots are taught how to fly in combat nowadays?

    ReplyDelete
    Replies
    1. Yikes! I sure would like to believe that's not true but …

      For the last several years our non-deployed pilots have barely been getting enough flight hours to stay flight certified let alone combat proficient.

      Delete

Comments will be moderated for posts older than 7 days in order to reduce spam.