The real problem is how humans react. Imagine a doc comes in, and is very wrong about a diagnosis.
Say... you have a broken arm, doc says it is sprained. You say No! and point out why.
AI says this, you say no, and the minumum wage person who works with the AI, shrugs and says the computer says so.