It’s a black box in the sense that we know the process used by LLMs to “reason” is very different than how humans reason and make decisions.

The whole point of “jury of your peers” is that your guilt or innocence is being decided by people with common life experience to you and thus the possibility of empathy and judging you fairly.

No, your reasoning is flawed: a judge or a jury is voted, appointed or selected through a process which is the result of centuries civilization, norms and legislation. As corrupt it might be in some cases, it is expression of mankind.

They are not a black box, they passed rigourous examinations and they stick to principles enshrined in constitutions and laws.

Technology, or better, the productization of it, is instead the result of the interest of very few powerful individuals or corporations and aggressive product market fit iterations.

The bullish discourse around the PMF loop says that this is good because it is better at solving people's problems. But after a few decades it is obvious there's a gigantic risk the process actually exploits people's problems instead of actually solving them.