undefined | Better HN

0 pointscivilized3y ago0 comments

Too fancy for anyone to be confident in what it's doing or whether it reflects real patterns.

0.90 AUC doesn't mean everything you can get this model to tell you is real, especially if you are trying to tell a causal story.

Now, show me a plot of your data with a trend line and I'll often be able to tell you if there's a real pattern in your data.

0 comments

w-j-w3y ago

Are you trying to claim that the model would better understand the complicated, real world of law enforcement if it were MORE reductionist?

civilizedOP3y ago

No, I'm not talking about how best to set up the model. I'm saying that complicated neural network models have no track record of yielding reliable insights about social phenomena. They are untested. Any insights they supposedly provide must be verified by a human analyst checking the data to see if it really looks like the model says.

This isn't a new or surprising principle, and it also applies to much simpler models. Any scientist knows that, if you fit a line to data, you better plot the data with the line before you make any big claims about what that line tells you about the world, because the data underlying that fit could look lots of completely different ways with different implications [1]. These authors did the equivalent of fitting the line and telling us all about its formula and the big implications of the formula without ever plotting it with the data.

[1] https://en.m.wikipedia.org/wiki/Anscombe's_quartet

j / k navigate · click thread line to collapse