Almost all data science workflows treat the annotators or subject matter experts as secondary. The tooling isn't set up to put them at the centre of the process and make it easy for them to collaborate with the more technical folks.
Perhaps it should be obvious but its definitely over looked in much of academic ML and in MLops.
right - but it seems like a, if not the, first lesson you learn when you leave the classroom for the "real" world.
It's not surprising to me that any people think this way, but it seems to be a characteristic of inexperience (or narrow experience).
> Every time I fire a linguist, the performance of our speech recognition system goes up. - Fred Jelinek
I also wish it was 'one lesson from four years of building tools for ML'.
On a serious note, there is a book on Human-In-The-Loop ML by Robert Monarch, published just a few weeks ago [2], where concepts like "active learning" are elucidated. Also, Andrew Ng recently started 'Data-Centric AI' competition, focusing on improving the data but keeping the model fixed[3].
There seems to be a growing emphasis on data quality while models become commoditized and outsourced to 'ML as a service' (MLAAS) platforms. If I understood correctly humanloop project aspires to be 'all-in-one' MLAAS serving both the models/predictions but also taking care of data annotations, targeting the market currently served by e.g. Scale.AI and Salesforce Einstein.
[1] Bayesian Reasoning and Machine Learning http://web4.cs.ucl.ac.uk/staff/D.Barber/pmwiki/pmwiki.php?n=...
[2] Human-in-the-Loop Machine Learning https://www.manning.com/books/human-in-the-loop-machine-lear...
[3] https://https-deeplearning-ai.github.io/data-centric-comp/
You seem to be pretty clued up on the area, what do you see as the pros and cons of an end-to-end approach?
Normally we do this kind of benchmark internally by sending the same dataset to each service and running some stats on the results, but if a vendor comes in with a ready to use comparison report that would be easier sale.
As for end-to-end you would be competing with large internal ML teams and revenue bringing internal ML engines, i'm probably not the right audience for that type of product. Salesforce seems to be doing alright on that front, but from my discussions with them there is a lot of hand-holding and customizations for each client use case, it's a high-touch thing.
If this is meant to imply it predicted them all correctly, that rings alarm bells to me. 100% accuracy is much more likely to mean something is wrong (label leakage?) than it is to mean you have an amazing model.
