I recently demonstrated a really simple bagged-decision tree model that "predicts" if the scanned part will go on to fail at downstream testing with ~95% certainty. I honestly don't have a whole lot of background in the realm of ML so it's entirely possible that I'm one of those dreaded types that are applying principles without full understanding of them (and yes I do actually feel quite guilty about it).
The results speak for themselves though - $1M/year scrap cost avoided (if the model is approved for production use) in just being able to tell earlier in the line when something has gone wrong. That's on one product, in one factory, in one company that has over 100 factories world-wide.
The experience has prompted me to go back to school to learn this stuff more formally. There is immense value to be found (or rather, waste to be avoided) using ML in complex manufacturing/supply-chain environments.
First, bagged decision trees are a little hard to interpret; what is the advantage of a bagged model vs the plain trees? Are you using a simple majority vote for combination? What are the variances between the different bootstraps?
Second - what do you mean by 95% ? Do you mean that out of 99999 good parts 4999 are thrown away? and one bad one is picked out as bad ?
Third - what is this telling you about your process? Do you have a theory that has evolved from the stats that tells you why parts are failing? This is the real test for me.. If the ML is telling you where it is going wrong (even if it's unavoidable/too expensive to solve) then you've got something real.
Unfortunately my concern would be that as it stands.. you might find that in production your classifier doesn't perform as well as it did in test... My worry has been generated by the fact that this same thing has happened to me !
Several times...
I'd hazard to guess that the 95% is the reduction in how many parts made it through the first test only to be caught later at the more expensive stage. So instead of binning 100 parts a month at that second stage, they now bin 5 parts a month and catch way more early on.
It sounds like the OP is using ML to identify flaws that simply just occur due to imperfections in the manufacturing process. That's life, it happens. You can know that they will occur without necessarily being able to prevent them because maybe there's some dust or other particulates in the air that deposit into the resin occasionally, or maybe the resin begins to cure and leaves hard spots that form bond flaws. There's heaps of possible reasons. It sounds more like the ML is doing classification of 'this too much of a flaw in a local zone' vs 'this has some flaws but it's still good enough to pass', which is how we operate with casting defects. For example, castings have these things called SCRATA comparitor plates, where you literally look at an 'example' tactile plate, look at your cast item, then mentally decide on a purely qualtative aspect which grade of plate it matches. Here [1] are some bad black and white photos of the plates.
[1] http://www.iron-foundry.com/ASTM-A802-steel-castings-surface...
By the way, control engineering for industry used to be very difficult (but is paid very well), and requires knowledge of systems theory, differential equations, and physics. But with the advent of ML, I suspect that might change; things may get a lot easier.
http://blog.mldb.ai/blog/posts/2016/01/ml-meets-economics/ http://blog.mldb.ai/blog/posts/2016/04/ml-meets-economics2/
In my case the % occurrence of the defect was very high and the False-Positive cost is also very high so my tool could provide value without being too stellar of a model.
How easy will it be to update your model if/when the downstream process changes?
At a previous job we had a process that relied heavily on visual inspection from employees. I often considered applying ML to certain inspection steps, but always figured it would be most useful for e.g. final inspection to avoid having to update the models frequently as the processes were updated.
I think I'm somewhat lucky in that with my product downstream processes are unlikely to change in a significant enough way to warrant "retraining" the model, but I guess that's probably the only way to handle that - retrain in the event of a significant process change. Our product stays fairly stable once it releases to production and the nature of the downstream processes is that they would have very little effect on the perceived severity of the defect at the final electrical test.
My only two misgivings about the program thus far: It is 1) pretty expensive and 2) geared towards working professionals rather than academics, but my employer is helping pay for a good chunk of the degree and I'm more interested in acquiring the skills and tools to go solve problems in industry as opposed to doing research.
Otherwise it has been great thus far. The program was attractive to me because it is somewhat marketed towards those that may not have a software background, but have problems in their industry that could benefit from a "proper" data science treatment. I've been referring to my application of the principles as "Six Sigma for the 21st Century" with managers/directors. I think the vast majority of HN would groan at that term, but it helps communicate the potential value to someone who has no technical background with software whatsoever (think old school manufacturing/operations types): Process improvement for environments with many variables that have practically unknowable inter-dependencies (as is the case with the project described in my original comment).
I don't think there's anything wrong the GP's achievement or post (it's all interesting stuff) but if something has not yet been implemented, it's worth nothing since there is "many a slip 'tween cup and the lip"
We use ML/Deep Learning for customer to product recommendations and product to product recommendations. For years we used only algorithms based on basic statistics but we've found places where the machine learned models out perform the simpler models.
Here is our blog post and related GitHub repo: https://aws.amazon.com/blogs/big-data/generating-recommendat... https://github.com/amznlabs/amazon-dsstne
If you are interested in this space, we're always hiring. Shoot me an email ($my_hn_username@amazon.com) or visit https://www.amazon.jobs/en/teams/personalization-and-recomme...
I haven't looked at coffee tables specifically, but I know when I've looked at home products in the past I've been surprised at how frequently people will buy two large items, e.g. TVs or furniture, within a short period. That said, I agree there is room for improvement here. We're constantly running experiments to improve the customer experience, I have faith that in the limit things will improve. Again, we have no shortage of experimental power so if you'd like to join in the experimentation let me know :)
its amazing that the vaunted Amazon technology is unable to figure out an algorithm that would satisfy a user's deep desire "please stop plastering Jeffrey Tambor's lipstick and mascara covered face on my startup screen, I've gotten tired of looking at it for the past year"
Maybe in-market* furniture shoppers tend to spend a lot of money. Maybe furniture is a very profitable category. Even if the system is smart enough to assume there's only a 20% chance that you're in the process of significant furniture purchases, furniture ads may still be a better use of the ad slot than a lower value item where you have an 80% chance of being in-market.
Then why show the same damn coffee table over and over? Maybe that's more likely to return your attention to your furniture purchasing? I have no idea. Most likely, they don't know exactly either. Most likely, that's just what the highest-scoring current algorithm decided.
*The duration of "in-market" varies by category. Some product categories have a long consideration phase. For example car shoppers tend to spend 2-3 months considering alternative brands and models before they spend a few weeks narrowing down on a specific car configuration and exact pricing.
Seriously though, I don't understand why it's so hard to take this effect into account, as there should be a very strong negative correlation between a purchase in a given category and the probability of buying an article from that category in the near future, so even a simple ML algorithm should be able to pick this up easily. Anyone here who can explain why this is difficult?
Not saying it works, but that'd be the goal.
I get suggestions for travel guide books for the same country I visited a year ago for which I purchased a guide book.
I guess this would make more sense.
For years we used only algorithms based on basic statistics but we've found places where the machine learned models out perform the simpler models.
One of the products the company I work for sells more or less attempts to find duplicate entries in a large, unclean data set with "machine learning."
The value added isn't in the use of ML techniques itself, it's in the hype train that fills the Valley these days: our customers see "Data Science product" and don't get that it's really basic predictive analytics under the hood. I'm not sure the product would actually sell as well as it does without that labeling.
To clarify: the company I work for actually uses ML. I actually work on the data science team at my company. My opinion is that we don't actually need to do these things, as our products are possible to create without the sophistication of even the basic techniques, but that battle was lost before I joined.
There's an interesting group of marginal approaches that have existed as-is for years, but have increasingly focused their branding on machine learning as its profile has risen.
You can reduce 90% of ML to this. Even neural networks are based on statistics.
If I have to draw a line between statistics and ML is that ML learns, it means it can predict things, however statistics only gives you information about the data you have. But for sure statistics and ML overlap a lot.
Statistics is about modelling the underlying probability distribution that generates your data. A convergence/generalization/etc result will usually be dependent on this underlying distribution.
ML is when you don't care much about the underlying distribution (modulo regularity assumptions), and your model doesn't even come from the same family at all.
I.e. linear regression is usually statistics, because you often believe the underlying data looks like f(x) ~ f(x0) + f'(x0)(x-x0)). Random forests are machine learning because you don't actually think the real world secretly has a random forest flotaing around.
Ha, sounds like a classification problem! Let's use ML to find the boundary.
Yes, this means ML is "just" statistics - the distinction being that it is automated so you can run it on larger amounts of data quickly.
I thought this was pretty much an accepted definition.
In brief, you're going to run up against two types of data - categorical and continuous. (There are facets to this, eg ordinal, but these are really the elemental types of data). The relationship of datatype to independent/dependent variable is what determines what kind of analysis you may conduct.
Categorical Independent vs. Categorical Dependent, for example, is fairly restrictive, as makes logical sense. You may cross-tabulate, you may score likelihood based on previous observation, but obviously, because all of the data involved are non-numeric, there's no chance for regression, ANOVA, etc. Linear Regression is used when both independent and dependent variables are continuous, and cross-category differencing techniques like ANOVA may be used when the independent is categorical and the dependent is continuous.
The part you don't typically learn until grad school is when the independent is continuous and the dependent is categorical, ie, in ML, a classification problem. The standard statistical methods used as foundation for these problems are logistic regression, logit/probit. It's expansion of these methods that lead to ML in the first place.
So you are misleading your customers through omission? This is the kind of thing that makes people question anyone stating they are using ML. Those of us actually implementing ML techniques (aka training neural nets and automating processes with data) are met with unnecessary skepticism as a result.
edit: OP clarified his position since this post so take that into account when reading.
My point was that using ML, even though we don't need to, "adds value" by virtue of the hype train. We need ML to sell products, not to create them.
I do agree that this sort of arrangement lends itself to supporting skepticism around AI and ML. On the other hand I don't think that's a bad thing.
All identification and extraction in our APIs is based on our ML models, which have been fed hundreds of thousands of data-point examples from annotated web pages. Basically: our back end has reviewed millions of web pages to learn what various components of a page are -- and even what "type" of page a page is -- and uses that to make judgments on ones submitted via API.
I would classify something like this blog post as ML, would you? http://stackoverflow.blog/2016/11/How-Do-Developers-in-New-Y...
A basic linear regression probably isn't ML, a backprop neural net clearly is, but somewhere between the two is a very fuzzy line between "statistics and data cleaning" and "actually machine learning". I think a lot of people have just pushed the ML angle of an already-reasonable approach to tie into that popularity.
Off topic: somehow the people not doing data cleaning because it is so boring and ML techniques do not need clean data end up with the worst overfitting I have ever seen.
In our space, the recent AI / ML advances have made things possible that were simply not realistic before.
That being said, the hype around Deep Learning is getting pretty bad. Several of our competitors have gone out of business (even though they were using the magic of Deep Learning). For example, JustVisual went under a couple of months ago ($20M+ raised) and Slyce ($50M+ raised) is apparently being sold for pennies on the dollar later this month.
Yes, Deep Learning has made some very fundamental advances, but that doesn't mean it's going to make money just as magically!
There's a lot of "DL allows us to do X so we should make a product / service using DL to do X", rather than "We think there's value in something doing Y, what allows us to do Y? <research> DL allows us to do Y better than anything else, lets use DL"
You gave the example of Slyce. Their products are cool, but I can't help but think "is DL the best way to get that end result?" for lots of the things they do.
The exact things we do depends entirely on which department(s) are licensing it. Basically, anywhere there's a product image (from their own inventory to mobile to social) and we can provide some kind of help, we do. Every department needs totally different things, so it varies quite a bit...but it's all leveraging our core automated detection + recognition + search APIs.
The primary advantage for customer is easier to use and troubleshoot faster.
https://www.sumologic.com/resource/featured-videos/demo-sumo...
overall the pattern matching stuff is pretty cool. Also, would like a see raw logs around this for when I am trying to debug event grouping errors based on the starting regex.
E.g. With LogReduce you can click on group and see log lines that belongs to it. IS that something that solves your problem, or are you looking for something else.
Feel free to send me an email (it is on my profile).
The same coworker also used decision trees to analyze query performance. He trained a decision tree on the words contained in the raw SQL query and the query plan. Anyone could then read the decision tree to understand what properties of a query made that query slow. There's been times we're we've noticed some queries having odd behavior going on, such as some queries having unusually high planning time. When something like this happens, we are able to train a decision tree based on the odd behavior we've noticed. We can then read the decision tree to see what queries have the weird behavior.
Marketers create their messages and define their goals (e.g., purchasing a product, using an app) and it learns what and when to message customers to drive them towards those goals. Basically, it turns marketing drip campaigns into a game and learns how to win it :)
We're seeing some pretty get results so far in our private beta (e.g., more goals reached, fewer emails sent), and excited to launch into public beta later this month.
For more info, check out https://www.optimail.io or read our Strong blog post at http://www.strong.io/blog/optimail-email-marketing-artificia....
1. Course Recommendations. We use low rank matrix factorization approaches to do recommendations, and are also looking into integrating other information sources (such as your career goals).
2. Search. Results are relevance ranked based on a variety of signals from popularity to learner preferences.
3. Learning. There's a lot of untapped potential here. We have done some research into peer grading de-biasing [1] and worked with folks at Stanford on studying how people learn to code [2].
We recently co-organized a NIPS workshop on ML for Education: http://ml4ed.cc . There's untapped potential in using ML to improve education.
[1] https://arxiv.org/pdf/1307.2579.pdf
[2] http://jonathan-huang.org/research/pubs/moocshop13/codeweb.h...
One way we're applying this is automatic creation of panoramic tours. Real estate is a big market for us, and a key differentiator of our product is the ability to create a tour of a home that will play automatically as either a slideshow or a 3D fly-through. The problem is, creating these tours manually takes time, as it requires navigating a 3D model to find the best views of each room. We know these tours add significant value when selling a home, but many of our customers don't have the time to create them. In our research lab we're using deep learning to create tours automatically by identifying different rooms of the house and what views of them tend to be appealing. We are drawing from a training set of roughly a million user-generated views from manually created guided tours, a decent portion of which are labelled with room type.
It's less far along, but we're also looking at semantic segmentation for 3D geometry estimation, deep learning for improved depth data quality, and other applications of deep learning to 3D data. Our customers have scanned about 370,000 buildings, which works out to around 300 million RGBD images of real places.
We use "real" ML for sentiment classification, as well as some of our natural language processing and opinion mining tools. However, most of the value comes from simple statistical analysis/probabilities/ratios, as other commenters mentioned. The ML is really important for determining that a certain customer was angry in a feedback comment, but less important in highlighting trending topics over time, for example.
A few machine learning-based classifiers (we've used Bayesian and SVM approaches). Word embeddings and topic modeling (similar to word2vec) which are based on shallow neural networks.
Those are a few of what I would consider the "real" machine learning tools we use. Most of the application, though, is statistics/pattern recognition/visualizations on top of the data calculated by the ML approaches.
The interesting thing is (in my opinion/experience) that a 10% improvement in some of the ML performance (a 10% increase in accuracy, for example) will translate to a 1-3% improvement in end user experience (they see slightly better insights and patterns, but it is a marginal improvement). On the other hand, layering a new visualization or statistical heuristic on top of the data can lead to a significant boost in user experience.
Again, this is just for our specific application/domain, but we focus on making the ML results more accessible to users instead of focusing on the marginal accuracy of the ML results themselves.
We work with B2B and B2C SAAS, mobile apps and games, and e-commerce. For each of them, it is a generalized solution customized to allow them to know which end users are most at risk of churning. The amount of time range varies depending on their customer lifecycles, but for longest lifecycles we can, with high precision, predict churn more than 6 months ahead of actual attrition.
Even more important than "who is at risk?" is "why are they at risk?". To answer this we highlight patterns and sets of behavior that are positively and negatively associated with churn, so that our customers have a reason to reach out, and are armed with specific behaviors they want to encourage, discourage, or modify.
This enables our customers to try to save their accounts / users. This can work through a variety of means, campaigns being the most common. For our B2B customers, the account managers have high confidence about whom they need to contact and why.
All of this includes regular model retraining, to take into account new user events and behaviors, new product updates, etc. We are confident in our solution and offer our customers a free trial to allow us to prove ourselves.
I can't share details, but we just signed our biggest contract yet, as of this morning. :)
For more http://appuri.com/
A recent whitepaper "Predicting User Churn with Machine Learning" http://resources.appuri.com/predicting_user_churn_ml/
Generally speaking, I think if you know your data relationships you don't need ML. If you don't, it can be especially useful.
Is this one of those things like the bin packing problem [1] where on first glances you'd expect it to have a definitive solution but it's actually deceptively very hard?
Not really a new application though...
On the other hand I found an internal fraud costing us 2-3 M €/year applying only the weak law of big numbers. Big corp, big numbers.
Now I build a similar system for a smaller company. I think we will stick mainly to logistic regression. I actually use "neural networks" with hand-crafted hidden layers to identify buying patterns in our grocery store shopping cart data. It works pretty well from a statistical point of view but it is still a gimmick used to acquire new b2b partners.
We use hierarchical clustering for topic detection. We also work on topic models (Blei and his legacy). We use word embeddings for information retrieval and various ML algorithms for different applications of mood and emotional learning: Bayes, SVM, Winnow (linear models) and sometimes decision trees and lists. We also learn from past events and crises in order to create models, mostly statistical, and try to estimate how an event might evolve in the future. We have also tried graph-based community detection algorithms on Twitter (min-cut). Finally we have experimented with non-linear statistical analysis on micro-blogging data, by applying methods such as correlation functions, escape times, and multi-step Markov chains (but with limited success).
I 'd like to add here that I feel ML is well defined (supervised, semi-supervised, unsupervised and using unlabeled data), statistical learning is more fuzzy (a good starting point is Vapnik's work) and regarding AI, I am not sure I know any more what it means! I am always open to discussion and ideas. Let me know.
We also support linear regression in the product itself - it was actually an on-boarding project for one of the engineers who joined this year, and he wrote a blog post to show them off: https://www.periscopedata.com/blog/movie-trendlines.html About 1/3rd of our customers are using trendlines, which is pretty good, but we haven't gotten enough requests for more complex ML algorithms to warrant focusing feature development there yet.
[1] https://people.eecs.berkeley.edu/~jonlong/long_shelhamer_fcn...
We also use ML to classify bittorrent filenames into media categories, but it's pretty trivial and frankly the initial heuristics applied to clean the data do more of the work than the ML achieves.
We use neural nets to generate descriptors of videos where motion is observed, and classify events as normal/abnormal.
There are a number of other statistical techniques you can use for this but scikit-learn makes this very very easy to do.
The bulk of what we do is anomaly detection.
[1] https://skymind.io/case-studies [2] insights.ubuntu.com/2016/04/25/making-deep-learning-accessible-on-openstack/
Trad learning for many applicatons : fault detection, risk management for installations, job allocation, incident detection (early warning of big things), content recommendation, media purchase advice, others....
Probabilistic learning for inventory repair - but this is not yet to impact, the results are great but the advice has not yet been ratified and productionised.
The first pass is usually a regex to find names, then for what's left run a natural language tool to find candidate names, and then manual entry.
Also, some GPU goodness for 10-100X visual scale, and now we're working on investigation automation on top :)
https://skillsmatter.com/skillscasts/9105-detecting-antisoci...
(We're still beginners as will be apparent from the video but it's proving useful so far. I should note, we do have 'proper' data scientists too, but they are mostly working on audience analysis/personalisation).
Wrote a system for automatically grading kids' essays (think the lame "summarize this passage"-type passages on standardized tests). In that case it was actually a platform for machine learning - ie, plumb together feature modules into modeling modules and compare output model results.
(www.queckt.com is anyone's interested)
Without AI/ML, we wouldn't have a product.
LEGAL INDUSTRY
Aka e-discovery [2]: produce digital documents in legal proceedings.
What was special: stringent requirements on statistical robustness! (the opposing party can challenge your process in court -- everything about way you build your datasets or measure the production recall the has to be absolutely bullet proof)
IT & SECURITY
Anomaly detection in system usage patterns (with features like process load, frequency, volume) using NNs.
What was special: extra features from document content (type of document being accessed, topic modeling, classification).
MEDIA
Built tiered IAB classification [3] for magazine and newspaper articles.
Built a topic modeling system to automatically discover themes in large document collections (articles, tweets), to replace manual taxonomies and tagging, for consistent KPI tracking.
What was special: massive data volumes, real-time processing.
REAL ESTATE
Built a recommendation engine that automatically assembles newsletters, and learns user preferences from their feedback (newsletter clicks), using multi-arm bandits.
What was special: exploration / exploitation tradeoff from implicit and explicit feedback. Topic modeling to get relevant features.
LIBRARY DISCOVERY
Built a search engine (which is called "discovery" in this industry), based on Elasticsearch.
What was special: we added a special plugin for "related article" recommendations, based on semantic analysis on article content (LDA, LSI).
HUMAN RESOURCES (HR)
Advised on an engine to automatically match CVs to job descriptions.
Built an ML engine to automatically route incoming job positions to hierarchy of some 1,000 pre-defined job categories.
Built a system to automatically extract structured information from (barely structured) CV PDFs.
Built a ML system to build "user profiles" from enterprise data (logs, wikis), then automatically match incoming help requests in plain text to domain experts.
What was special: Used bayesian inference to handle knowledge uncertainty and combine information from multiple sources.
TRANSPORTATION
Built a system to extract structured fixtures and cargoes from unstructured provider data (emails, attachments).
What was special: deep learning architecture on character level, to handle the massive amount of noise and variance.
BANKING
Built a system to automatically navigate banking sites for US banks, and scrape them on behalf of the user, using their provided username/password/MFA.
What was special: PITA of headless browsing. The ML part of identifying forms, pages and transactions was comparatively straightforward.
--------------
... and a bunch of others :)
Overall, in all cases, lots of tinkering and careful analysis to build something that actually works, as each industry is different and needs lots of SME. The dream of a "turn-key general-purpose ML" is still ways off, recent AI hype notwithstanding.
[1] http://rare-technologies.com/
[2] https://en.wikipedia.org/wiki/Electronic_discovery
[3] https://www.iab.com/guidelines/iab-quality-assurance-guideli...