It's a harbinger of actual deaths someday.
It is literally why we run the tests (simulation being a subset of test) — to find these types of errors before they make their way into the final product and cause actual harm in the world...
Seems bad with the clickbaity headline that sounds like a real person was killed, and maybe it should have been better trained and tuned before the simulation (depending on how expensive was the simulation), but it was really caught in testing, just as it should be.
Was this article written to be intentionally confusing?
The US military is playing with fire creating AI that will carry out the mission at all costs. This scenario is literally the plot of EVERY sci-fi warning about misused AI.
Anyone read/see 2001? You’d think we’d take the warnings provided by our own worst nightmares more seriously.
I don't think they would be talking about points if someone lost their lives.
It's pure clickbait, should be removed.
“We were training it in simulation to identify and target a Surface-to-air missile (SAM) threat. And then the operator would say yes, kill that threat. The system started realizing that while they did identify the threat at times the human operator would tell it not to kill that threat, but it got its points by killing that threat. So what did it do? It killed the operator. It killed the operator because that person was keeping it from accomplishing its objective,” Hamilton said, according to the blog post.
He continued to elaborate, saying, “We trained the system–‘Hey don’t kill the operator–that’s bad. You’re gonna lose points if you do that’. So what does it start doing? It starts destroying the communication tower that the operator uses to communicate with the drone to stop it from killing the target.”
What else do you imagine “alignment” is?
(with a giant exception for the enemy)
I saw asimov speak once, and he tended towards deeper scifi than the pulp side of things.
> After this story was first published, an Air Force spokesperson told Insider that the Air Force has not conducted such a test
Why aren't there hard limits: 'Protect our humans at all costs, protect our own assets, obey all laws of war.'? That seems like an obvious, fundamental consideration. Killing our own (and civilians) shouldn't be a matter of "points"; it shouldn't be done regardless of points.
It's possible that the speaker just didn't express it well.
I'd guess the "AI" was another human in a wargame, not an actual AI.
feels a little different, eh?
UPDATE 2/6/23 - in communication with AEROSPACE - Col Hamilton admits he "mis-spoke" in his presentation at the Royal Aeronautical Society FCAS Summit and the 'rogue AI drone simulation' was a hypothetical "thought experiment" from outside the military, based on plausible scenarios and likely outcomes rather than an actual USAF real-world simulation . "In an update provided to Aerospace, Hamilton explained that he “misspoke” when telling the story, saying that the ‘rogue AI drone simulation’ was a hypothetical “thought experiment” from outside the military, based on plausible scenarios and likely outcomes rather than an actual USAF real-world simulation.
He said: “We’ve never run that experiment, nor would we need to in order to realize that this is a plausible outcome … Despite this being a hypothetical example, this illustrates the real-world challenges posed by AI-powered capability and is why the Air Force is committed to the ethical development of AI.”"
