undefined | Better HN

0 pointslacker14d ago0 comments

Yes there are studies, for example last year Pangram's false positives were measured to be under 0.5%.

https://www.pangram.com/blog/third-party-pangram-evals

Personally, at first I thought these sorts of tools were dumb and wouldn't really work, but I think it works because it just isn't designed to be "adversarial". If you want your AI to trick Pangram, you can make an AI to trick Pangram. It just catches people who are cutting and pasting from the AIs without putting any more effort into hiding it.

0 comments

3 comments · 1 top-level

sdenton414d ago· 2 in thread

Any binary classifier can have a FPR under 0.5% if you don't have any restriction on FNR...

D-Machine14d ago

While I am quite skeptical of the claims linked above, that link does indeed cover the FNR at the FPR of 0.005, and finds it broadly to be on the same order of magnitude, i.e. also below 0.005.

BobaFloutist14d ago

If the FPR is very low, the FNR rate doesn't really matter if you get a positive result (unless the pre-test probability is very low, which is not the case here)

j / k navigate · click thread line to collapse