Machine Bias (opens in new tab)

Where there was absolutely no effect, one out of every twenty combinations of all other variables would show significance in combination with the current value of that particular variable in the likelihood of future crime.

Furthermore, the algorithm would simply extend existing biases in arrest and sentencing, because it simply can't account for crimes that are uncaught and unpunished. Groups that are stopped, searched, arrested, and convicted at greater rates would without fail be sentenced to more time. Just another benefit of being white in America.

You end up using the fact that some groups are punished more often to justify punishing them more harshly.

Even worse, I bet that the fact that it thinks that women are at a higher risk for recidivism means that somewhere within the algorithm it's using the fact that women in general are less criminal than men to decide that women who do commit crime are more exceptional (within women), and therefore more deviant. It's disgusting. If you can't legally discriminate against a person on particular grounds, you certainly can't feed those grounds into an algorithm to let it discriminate for you while you shrug and feign innocence.

The algorithm is the innocent one - it's just attempting to reflect the system as it is. It's like an algorithm you would write to predict the winners of horse races, or the sports book. And just like one of those algorithms, if you stuff it with garbage (the kind of garbage that makes it wrong 77% of the time), it will result in garbage. If you use the results for something not external to the system, bad variables will feed back into themselves and make the results progressively worse - what's the effect of a longer sentence on recidivism? How does profitable is the arbitrage on your sports book algorithm if people use the results to bet, and the distribution of bets shift the odds?

In particular, the cases are heard in a particular order. For each prison, the prisoners with counsel go before those who are representing themselves. As in the US, those representing themselves typically fair worse. The judges try to finish an entire prison's worth of hearings before a meal, so the least-likely-to-succeed cases are typically assigned to spots right before a break.

There are some other bits of weirdness in the original data too. They found a statistically significant association between the ordinal position (e.g., 1st, 2nd, ..., last) and the parole board's decision, but failed to find any effect of actual time elapsed (e.g., in minutes), even though the latter is much more compatible with a physiological hypothesis like running out of glucose.

I also worry greatly about diagnostic predictive models that maximise overall prediction success but don't balance the relative consequences of false positives and false negatives.

> Black defendants are 45% more likely than white defendants to receive a higher score correcting for the seriousness of their crime, previous arrests, and future criminal behavior.

> Women are 19.4% more likely than men to get a higher score.

> Most surprisingly, people under 25 are 2.5 times as likely to get a higher score as middle aged defendants.

> The violent score overpredicts recidivism for black defendants by 77.3% compared to white defendants.

> Defendands under 25 are 7.4 times as likely to get a higher score as middle aged defendants.

> [U]nder COMPAS black defendants are 91% more likely to get a higher score and not go on to commit more crimes than white defendants after two year.

> COMPAS scores misclassify white reoffenders as low risk at 70.4% more often than black reoffenders.

> Black defendants are twice as likely to be false positives for a Higher violent score than white defendants.

> White defendants are 63% more likely to get a lower score and commit another crime than Black defendants.

Calling out one specific section that doesn't show bias doesn't magically exonerate the rest.

Larson is still the second author so it is certainly a big question how he can present data showing no statistical correlation between race and score then have his name on an article saying the exact opposite that is clearly pushing an agenda. And as noted, one where the owners of the publication are also involved in a competing risk assessment product.

https://github.com/propublica/compas-analysis/blob/master/Co...

Their own analysis shows that (p ~= 0) that high and medium risk factors are predictive. They also showed that the racial bias terms (race_factorAfrican-American:score_factorHigh, etc) are probably not predictive (p > 0.05).

Your quotes are not evidence of bias, though I see how they might confuse an innumerate reader. It's interesting how good a job this article is doing confusing the innumerate - it's almost as if it was written to mislead without technically lying.

For example, black defendants being pegged as being more likely to commit crimes can be caused by one of two things: bias or perhaps black defends actually are more likely to commit crimes. According to ProPublica's own analysis (see race_factorAfrican-American), the latter is actually the case. This is true with p = 4.52e-06 - see line [36].

I'm confused though; the mood affiliation of your post somehow suggests that her less than perfect choice of a statistical methodology somehow supports her claims. Could you explain that? Or am I simply misunderstanding what you are trying to say?

Also, lets suppose we just take her own analysis at face value, and don't view it through the p-value lens. The maximum likelihood estimate suggests that even if this effect is not random chance, it's not very big. I.e., the "score factor high" estimate is >8x larger than the "score factor high, race = black" estimate. Isn't this really good? Do you really think the human biases that this algorithm mitigates are lower than this?

Lastly, what specific analysis would convince you that this algorithm is predictive and non-biased (or more realistically, not very biased)?