I've been a part of a system like this (a financial system that happens to not be technically a bank). The system was constantly under attack by fraudsters: You could find subreddits with guides on how to try to outright rob us. So we had teams building detection systems that tied to detect said fraud as early as possible: Hopefully before we handed the fraudsters any money. Depending on how bad the score was, there might not even be a manual review step before we closed the account, because the numbers were that blatant.

As with any classification system though, 100% accuracy isn't going to happen. But there's always some customer service rep that can look at the details of the account, and see why in the world the system said what it did. But a detailed explanation of why we thought something was fraudulent could (and sometimes would!) just lead to another fun reddit post where someone describes how to hide the fraud a little better.

For any given system like this, how much harm is actually being done, vs how much is being prevented (as fraud just leads to raising prices to cover for it: financial companies are not charities)? I've read way too many CSR conversations where a blatant fraudster with world-class chutzpah would claim that we were destroying their family for no reason, when the data was damning. But this doesn't mean that everyone who isn't a fraudster really reaches out to the CSRs, and has the energy to prove there was no fraud. The actual levels of damage are just hard to measure.

We should have sensible, mandatory, available customer service access, which costs just enough to access to not be hammered by bots, but that is completely refunded in case of error. But what is really causing this is that many companies have lowered the barrier of interaction so much that we are letting a lot of fraud through the door. Remember how getting a merchant account in a real bank is a multi-day affair? How getting hired to become a delivery driver needed an interview, with a real person, and a manager checking between deliveries? The price of not having to interact with a human to sign in is fraud detection that isn't a boss you interact with every day, makes sure you are working, and is paid from the work you do. Companies with billions of customers and probably hundreds of millions of suppliers aren't exactly workable without automating a lot of those intermediate jobs away.

Maybe we made the wrong call across the board, and lower-productivity, but far higher trust commerce is the way to go... but a lot of that commerce is losing in the market, right now. So if we like it, we have to be willing to pay extra for it.