It’s a black box in the sense that we know the process used by LLMs to “reason” is very different than how humans reason and make decisions.
The whole point of “jury of your peers” is that your guilt or innocence is being decided by people with common life experience to you and thus the possibility of empathy and judging you fairly.
We only know it’s different. But we don’t know how it reasons or how humans reason. Just because it is different doesn’t make his black box argument valid. It remains invalid despite your new irrelevant point.
Ignore the fluff. Focus on the logic and stay rational. The black box argument is not at all valid.
We know other humans reason something like us because we know how we ourselves reason through the most direct experience possible and know that our fellow humans are very similar to us.
We know the LLM can be similar because we train it on data from us as well. We also know the LLM can be different, but we also know humans are different too.
Examples like psychopaths tells us how different humans can be. It’s all black boxes.
