Having said that there must be a beancounter in your organisation whose job is making forecasts. That person wouldn’t hesitate to lay you off if the company hits a rough patch. I think you get my meaning.
I don't think it is true.
ML is legit. The dark secret is that a lot of it isn't new.
It is only now, that the data, compute and hype have caught up to how useful good-old applied statistics is.
Now a days, a lot of older areas have been repacked into a fancier term of ML and AI. This goes hand-in-hand with some legitimately awesome cutting edge research with wide spread uses (Smart phone cameras, Self driving, Language translators, Massive social media knowledge graphs).
Places such as MIT and CMU would not have started undergrad programs in AI, unless they though it was here to stay.
But often the bulk of the work is getting or cleaning data to make ML work in the first place. And even after that, the results need to be better or cheaper than having an employee personally look at the same data sometimes (i.e. a low tech expert system).
If you have a thousand emails and you want an emotion score assigned to the contents, it's often cheaper to use temp workers or something like Mechanical Turk.
It's also worth noting that many interesting automation tasks don't have room for 99% confidence intervals. It's fine if Google flips it's search result rankings rarely. It's not OK to only miscalculate 1% of account balances. And predicting which balances are bad might be useful, but the bulk of the work is the boring enterprise work of getting the concerns in front of a human or something.
That's why ML seems like a solution in search of a problem. Unless your problem is getting people to click on ads.
That doesn't mean that there aren't people applying ML techniques to problems that can easily be solved in other ways, but that happens with every hyped technology. And for developers learning these techniques, there's going to be a certain amount of applying them more widely than is justified until they learn them well enough to improve their judgment. That's okay and all part of the learning (no pun intended) process.
As far as the original poster is concerned, since he works at a shipping company, a problem as simple as "find the address on a webpage" could be a neat proof of concept to show his bosses. Imagine if they could offer customers the ability to "ship" to a URL. That kind of thing might be doable with regexes and such if you constrain the problem to only addresses in the US/Canada, but worldwide address formats are so diverse that ML really helps.
In reality no one cared. We went on placing a disk order whenever the first dumb nagios alert paged someone. We bought more CPUs when the moaning from the analysts reached the CEO and he in turn moaned to our manager. It was fun to do and I learnt alot but realistically the impact on the business was minimal...
2) You could assess different delivery routes/ regions to determine if they are more/less on-time than other routes/ regions. Are the number of delivery vehicles adequate? When should you adjust the number of vehicles or change the routes themselves (like moving some peripheral regions to another route, or adjusting the cost charged when delivery is delayed).
3) When do external factors (like weather, esp rain or snow) introduce delays? Can you predict these delays, and ideally, compensate by changing routes or adding more delivery vehicles?
4) Should you more dynamically adjust your shipping fees to reflect faster/slower delivery time targets? This way you can tune your routes and manpower to save money for those who aren't as time sensitive, and improve the response time for those who are.
A lot of this is basic operations research. But you can call it AI, or use AI techniques just as well as traditional OR methods. Nobody will care what math/methods you use if you can add value.
Keep in mind that there may be flow on effects and ethical considerations. Once you assign a metric to individuals, someone is going to start attaching it to KPIs, ranking individuals by it, and ultimately firing individuals by it.
Ethics is an increasingly prominent aspect of ML.
The "AI" software basically just applied common sense and basic regression modeling to the situation. Originally, SAP (used for resource planning) took input data at face value - i.e. if a carrier said they have X capacity along Y route with a 7 day lead time, that was statically entered into SAP and all dependencies on that information took it as gospel[1]. The secret sauce of the new system was that it looked at actual data in the system to calculate those variables. It'd pull raw transaction data out of SAP for stuff like when a pickup was requested/scheduled, when a pickup actually occurred, when a delivery was anticipated, when it was actually marked as delivered, etc. Run some basic SQL and regression analysis over that, then override the static values in the system so they were more realistic.
Not super sophisticated, but had a pretty substantial impact at scale. Unfortunately in the US specifically, that impact was negative, due to certain cultural traits that impacted user adoption[2]. But in most other regions globally, it was integrated into the planning process much more successfully and drastically improved forecasting accuracy.
[1] This was how it was configured where I was at, at least. I'm not familiar enough with SAP as a whole to know if this was a peculiarity of that specific SAP instance or if it was a limitation of SAP itself.
[2] I came onto the team right after initial rollout, at which point every region globally was showing positive returns from the rollout. Except the US, which had actually shown negative returns (not just neutral). My role was to basically "redo" the US rollout, and almost all of the issues were based off of cultural differences in how work is approached.
do it anyway with Machine Learning and see how it goes. at least you will know the expected results.
I get the machine learning is interesting to some people, and it certainly has its place, but applying needlessly complex solutions to simple problems is how software developers get a bad reputation.
If you don’t have the need don’t use: ML / AI / Cloud computing / Hadoop / or any other random technology.
I personally have to build some things that turn out to be the wrong approach so I really understand how to build the right thing later.
Some small problem that could be replaced is a great opportunity to really understand the solution space ml offers.
It really boils down to how important people are to your organization. There is a spectrum between cog and irreplaceable, sometimes growing people is totally worth the cost.
https://updates.moonlightwork.com/updated-skill-search-now-w...
This book is kind of dated, but it's what I used for inspiration and reference: https://www.amazon.com/gp/product/B00F8QDZWG/ref=oh_aui_d_de...
First, coding toy problems (related to shipping or not) that implement linear regression, genetic algorithms, or neural networks, etc. will be a useful start
Analyze shipping and tracking EDI data to predict whether a shipment will be late (0.0 to 1.0 output, 1.0 being it will be late for certain)
Predict the likelihood a customer will churn (stop using your services) based on changes in volume, billing amounts, and other characteristics
Predicting this year's peak season shipping volume based on past years' data. See if you can beat the marketing/sales folks' predictions
Identify factors correlated with the most profitable shippers
Predict the likelihood a package is damaged
Use a genetic algorithm to improve driver routing
Reconfigure pickup times / drop off times to improve profitability
Use EDI shipping data to build a network graph of who is shipping to whom, segmented by type of some sort. Say you find that many A-type firms are shipping to B-type firms; any B-type firms that are not already customers could be interesting targets.
Score prospects to estimate their profitability by comparing their characteristics to existing customers' profitabilities
Use a neural network (or something else) to analyze EDI shipping data, damage data, and make packaging recommendations to customers
Analyze tracking EDI data, segmented by delivery area (zip+4?) and see if there are areas where drivers are more efficient at delivering faster. Maybe start an initiative to look at what separates the most efficient drivers from the least.
Reporting: not sexy, but really useful in this space
Bona fides: I used to work in the supply chain consulting space and consulted at firms like yours. Things are surprisingly basic in the shipping space - less meaty data science than one might think.
Edit: Formatting
> I can't come up with any problems
> that I couldn't solve with a
> relational database (postgres) and
> a data transformation step.
"I've just learned a neat new tool but I never apply it because I can solve all the problems in front of me with the tools I have."
In effect, you've found a local maximum where every direction seems like a step backwards, or an investment of time without any reasonable payoff.
Here are two general strategies to deal with this:
1. Take a well understood, and well documented existing need, and replicate the solution with the new toolkit. Acknowledge from the outset that this will be a step backwards, but go through the details anyway to better understand the technology. The goal isn't to make the system better, but to improve your understanding of ML and it's real world application. By choosing a well understood system, you are only learning applied ML rather than trying to simultaneously learn ML and the problem. Work toward parity with your existing methods. This part is rarely a big step forward, but I guarantee that this process will generate 100 good ideas about where to go next.
2. Find problems that were previously ignored, because they couldn't be solved. Something no one is even thinking to ask for, because none of the prior tools could do the job. This is the ideal situation because you are in a greenfield space where anything is an improvement. For ML specifically look at anywhere a lot of data is being generated but no one has the time to read it all unless something goes wrong.
When learning any new technology there is always a gap between learning it in the lab, and trying to execute with it IRL. The best way to maximize your own ability is to simply start applying it and building experience. Don't wait for a perfect halo project.
You mentioned using Pytorch. Instead, I recommend a classical machine learning using a library like scikit-learn (https://scikit-learn.org/). Use a random forest classifer and you'll get pretty good results out of the box.
If your data is in a postgres database across multiple tables, you will likely have to perform feature engineering in order to get it machine learning ready. For that, I recommend a library for automated feature engineering called Featuretools (http://github.com/featuretools/featuretools/). Here's a good article to get started with it (https://towardsdatascience.com/automated-feature-engineering...)
Finally, you will need to define a prediction problem and extract labeled training examples. I see people in this thread have suggested ideas of problems to work on. The key here is make sure that you pick a problem that you can both predict and take an action based off the prediction. For example, you could predict that there will be an influx of shipments to fulfill tomorrow, but that might not be enough to time to hire more people to help you fulfill them.
If you're curious what the process looks like end-to-end check out this blog series on a generalized framework for solving machine learning problems that was applied to customer churn prediction: https://blog.featurelabs.com/how-to-create-value-with-machin...
Full disclosure: I work for Feature Labs and develop Featuretools.
One big example is fraud: it's next-to-impossible to define a 100% accurate set of rules to filter fraud, but it's often easy to train an algorithm to catch the worst offenders, or flag suspicious cases to significantly narrow the amount a human needs to review.
Note: your ML system will likely be less explainable than the existing rules. This won't matter as much if the current rule collection is already more complex than a human can deal with. It will matter a LOT if your decisions are subject to regulation.
If your problems aren't audio / image based, then consider using traditional ML instead.
If you are just starting out. Check out SKLearn, Scipy and graphical models like CRFs. They are tried in tested methods that also require less specialized skills.
As someone else said, a lot of AI, ML tools are simply repackaged old school OR methods. The older methods get 95% there, with <50% of the effort.
Cutting edge ML isn't required for most problems. Especially non visual or time series problems.
It sounds like you have the right tool for the job now, so great. Keep using it. Dependencies should be added to projects as conservatively as possible. The best dependency is no dependency. You shouldn't go seeking out dependencies. Your app doesn't depend on machine learning, so why would you make it depend on something it doesn't actually depend on? Future maintainers (including yourself) would hate you for it.
There's also scope of using ML in analytics and monitoring side apart from the main application and is generally better tolerated by the product team.
https://www.ibm.com/watson/supply-chain/resources/csc/deskto...
One concrete example IBM likes to talk up is predicting shipping delays due to weather events and automatically recommending alternate suppliers.
There may be a way you can do some computer vision tasks for quality control in some parts of your business -- most businesses that deal in physical goods have quality control by visual inspection and in most of those you can end up with a CNN that provides a quick enough, good enough solution. However, sometimes for regulatory reasons it's not practical, or it's something that is not a critical part of the chain, and so on. But you could ask operations staff about whether they sometimes do that kind of task, and whether it takes up a lot of their day. It's not like you have to find the good idea alone.
My problem statement includes classifying hundreds of thousands of PDFs to different categories based on the content/first few pages. That is, if you have a pdf of a novel by Jeffrey Archer, it should be categorized as Entertainment or Novel etc. If you have a e-book of say Python for Dummies, it should be categorized as Engineering or Technology or Education or Programming and the like.
5 minute deck: https://github.com/codingitforward/cdfdemoday2018/blob/maste...
Feel free to shoot me a message.
If you have access to production logs and metrics, try to model things like page load times, server load, network latency, errors/timeout, number of page views/unique visitors.
You might hit on unexpected correlations and maybe unknown bugs (e.g. when page X is loaded with input Y, timeouts and server load increase because of a broken SQL request), or insights on the health of the production platform.
Also consider that ML simply isn't a useful tool for your company. "Let's not spend (more) time trying various ML techniques" is a perfectly valid and useful outcome of an experiment using ML.
In the end, is it really worth all that effort if you end up using Cognitive Services on Azure or ML on AWS? How to you deploy your model and actually USE it in a WPF CRUD app?
It shows a link to an article called: "No, you don't need ML/AI. You need SQL"