Why did the account recovery system need AI. Surely just an email would do? What added value would AI add?

a human would have noticed something different about the requests it was getting, or the frequency of requests, and as soon as it noticed a shift, it would have carried that knowledge forward and intensified the scrutiny if something seemed off- eventually communicating it up the chain.

- instead of the ai context dying.

in the ai case, information only survives to the extent where the ai is empowered to store a note or notify a manager of an observation. Anything that does not result in sending a message/storage is wiped

One concerning feature of AI is the speed and volume it is capable of failing at if poorly controlled, whether or not it’s more accurate than humans.

Even if humans failed at the same rate, if you tried to exploit at scale you’d be throttled by the size of the support team. The failure would happen at human-scale time frames and throughput.

There's no social engineering here, since all they have to do is copy and paste. This is a complete process design fail.