The validity of the algorithm can be - and apparently has been - reliably tested and been found to be useful and mostly unbiased. This analysis has been performed by both the algorithm's creators and highly adversarial third parties, such as the author of this article. Both found that whatever bias there is is small, and cannot be distinguished from random chance.

For example, the author of this very article has done such an analysis. Here's her R notebook:

https://github.com/propublica/compas-analysis/blob/master/Co...

Her analysis shows (within the limitations of the frequentist paradigm) that:

a) the predictor is useful - score_factorHigh and score_factorMedium both have p-values that are essentially zero.

b) The predictor is not racially biased that much - race_factorAfrican-American:score_factorHigh and the other bias terms have p-values that are > 0.05 .

Look, I'd love it if we required such algorithms to be open source. I'm a huge proponent of both open science and open government. Nevertheless, there is an entire discipline devoted to evaluating predictive algorithms without needing to care about their details - it's called "machine learning".

The wonderful thing about statistics is that even a highly biased person (such as the author of this article) can still reach a correct conclusion that goes against their biases.