It's easy to describe a business process with written down rules, and those are easy to find in legal discovery. It's much easier to obfuscate with an AI model, because "nobody knows what it's actually doing - it's AI!".
I am personally 7 for 8 in lifetime wins in my city's parking ticket appeals process. That doesn't mean that I think that 7 out of 8 tickets my city issues are incorrect.
Do you have actual knowledge of this? If not, the most obvious counterpoint is that the AI will need to give the reason or reasons for denial, and recording them for audit. Just like a human or a rules-based system.
Although I would still agree that there would need to be a mechanism for escalation to a human.
I suppose whole life where there is a cash value and investments being managed might have a more ongoing service need, but I'm not familiar with that.