The practical one is that errors in a machine system scale, as do most things with machines. If I have a single bad X-ray tech who is applying the wrong medical process because I have a different race, for some reason, the damage that tech is doing is limited to whatever specific set of patients they are seeing. If a similar error occurs in a popular machine classification tool used widely by a hospital network, the damage is widespread. It is a plus that the machine can be corrected and the correction also scales, but with the (relatively speaking) stone tools we use to understand why a CNN makes its decisions these days, every fix risks breaking something else we're not testing for.

The first psychological reason is that machine learning systems break in "alien" ways. They don't make the kind of mistakes humans make... They make mistakes as a product of their machinery, which means it's much much harder to predict what those mistakes will look like for an average operator. As a frequent example, it's pretty rare for humans to misclassify human beings in photographs as apes, or to fail to recognize a face in an image because the skin is too dark. That's a failure mode that happens over and over again with image recognition systems.

And the second psychological reason is that humans don't trust machines to make human decisions yet. And that mistrust doesn't extend to other humans, even though we're incapable of cracking open another human's mind and understanding their thought process at the mechanistic level. It doesn't matter... we are the same organism and have a shared experience and empathy with them that we lack with machine recognition systems. It's semi-irrational, but it can't be wished away. A system for understanding why a machine makes decisions would be a step in the direction of addressing those concerns.