If the money is to mitigate risk, then why can't the fee be (mostly) refundable? The bank takes on the risk of processing some enormous transaction and takes a percentages of the transaction as insurance in case of fraud. Then after a period of time with no issues, the money flows back, less some nominal amount for the actual work involved processing the transaction.

Not from the bank, they just roll back the transaction. Some merchant might end up on the hook, though.