Good question. We measure these using "precision" (of the users that users Sift flags, which percentage are actually fraudsters?) and "recall" (what percentage of fraudsters on the site does Sift flag?). We can get 90% precision or 90% recall, although not currently both at the same time, and it's the customers choice as to which to optimize for. We can just adjust a threshold to tune our system to their needs.

Companies that have high transaction amounts often use the machine learning system to detect likely fraudsters, but then have a human review each one and make the final decision to approve/deny. We have a visualization "widget" that shows the reviewers which signals made a particular user look suspicious. The advantage of using machine learning is then that you: a) catch fraudsters you wouldn't have noticed otherwise, b) don't have to review every single transaction, just the subset that are most suspicious, c) make it faster for your staff to review transactions since the visualization tools will help point them at what to look at.

Does that make sense?