We held firm to the NO CUSTOMIZATION rule, we re-engineered our processes to fit SAP. Other than a few hiccups when integrating all aspects of a company that has 100 sub companies, and is under federal, state and local regulation the project basically went off without a hitch.
They are still taking upgrades 6 years later, without issue.
Most companies think they are unique, and special, and feel justified in needed to customize, all those companies are wrong.
That sounds just about like the most revolting and negative endorsement you can make for a business solution.
Every company is special and the flexibility to adapt to a changing environment is what keeps a company alive during those times of crisis, that always come. "We can't do this because the system does not allow it" is great and expected at the warehouse worker level, annoying at the branch manager level and catastrophic at senior management level.
I've seen the flip side where people custom code the heck out of the system. That becomes impossible to upgrade and impossible to take advantage of new features.
I've also seen lots of bad processes continued that could have been re-engineered to be good processes if the implementer and business had an honest conversation. SAP for better or worse forces these conversations by limiting the ways that you can implement the solution.
SAP failed implementations are not unique from other software failed implementations. It doesn't fail because of the product; it fails because of the people and politics of the organization. SAP tends to make the headlines because of the size of the implementations and the eye watering costs.
This is a fun analogy, but in the world of enterprise-level business, the business often is software. Entire businesses today are ran by software, and that software is written according to the same set of specifications: the law. The small particulars that don't have to do with law or common business practices are pretty much reducible to implementation details.
SAP is one such very comprehensive implementation, that is very possibly the only software suite that has a nearly full coverage of the entire "spec" for business purposes. If you can't translate your business operations to SAP, beg your pardon for my abrasiveness but you're doing things wrong.
I’m strongly agreeing with the GP: it’s both better and cheaper to at least meet half way with software like this. It’s simpler to change processes than to customize business systems.
What becomes painful is that the deployment of some new business system now became a massive reorganization and a complete overhaul of all business processes. But this is still less painful than customizing SAP to any large degree.
All the processes had become "unique" over time, not because the company was special, but because they were coding their own solutions and either the IT staff or the business wanted something done for the benefit of the specific department.
Once the processes were evaluated with a critical eye toward what actually needed to be accomplished, the uniqueness of the existing process couldn't stand up to scrutiny.
Luckily the CIO and the CEO had the courage to face down the IT staff and business leaders, the rank and file cheered for adopting standard processes, they knew the existing processed sucked.
https://www.amazon.com/Softwar-Intimate-Portrait-Ellison-Ora...
A lot of these larger software packages support different levels of customization, and they usually press people to use the simplest stuff.
It's not completely possible to guarantee reliability once the customer is writing their own code. You can guarantee the app doesn't fall over, and data isn't lost, but ultimately the customer is free to shoot themselves in the foot, and they will.
Yes, businesses often have unique needs and sometimes they're so unique and so critical that they're worth heavily customizing for.
On the other hand, I like telling a story from not that many years ago when a very small business (I'm talking like 10 people) wanted to get off running their own Exchange server. Exchange made sense when they set it up but Gmail and the like had come on the scene since.
The CEO eventually had to mandate it because the person who ran the business side of things was incredibly strongly opposed because they were used to storing emails associated with contracts in a hierarchical folder structure and, at the time, Gmail didn't support nested labels.
In that case, change how you do things to match the software was absolutely the right answer.
FTFY. There's huge benefits in using standardized solutions, like faster upgrades, and fixes to bugs you even didn't know you had but that were solved for other customers.
Look at it like any other utility infrastructure: if you want your factory to hook up to the same power grid as everyone else, it’s your factory that’s got to change, not the power grid. Or do you want to spend half of every new factory bring-up winding your own three-phase-220V-to-five-phase-39.2V transformers to meet your “unique” needs?
A company must consider whether its business processes are a competitive advantage. In well established businesses, chances are they aren't. In that scenario, it really doesn't make much sense to stick with them.
And for a customer of enterprise software I would so much agree. In my experience the people who manage the integration with third party software usually have no clue what the software is about. Instead of being arrogant it would be much smarter to learn from the software vendor, who's an expert in his field and has seen dozens of customer environments, how to solve certain problems.
exactly. People from "unique" camp fail to recognize that SAP isn't just software, it is "best [business] practices" coded into that software, the practices distilled from huge experience during many years with many customers. Granted, for many techies any business practices/processes and related software look disgusting - this is why we're techies, not accountants. At some point in the bright future we'll develop super AI which will take over the world and abolish business processes....
It is ridiculously expensive and complex. I know that the problems it aims to solve are also highly complex and very specific, but reading stories like this, plus (if you live in Germany and work in the IT industry) sometimes hearing unbelievable stories from your peers, it always confuses me that companies still want so much SAP.
In the late 90s/early 00s a lot of people had seemingly infinite trust in IBM. Maybe the same logic applies here as well.
Let me try to explain the mystery around SAP.
1. Most of the situations where SAP projects go haywire is largely due to (a) incorrect implementation by the service integrator and (b) challenges between business processes and systems support for those business processes. Sally in accounting wants to see sales per unit for the Germany promotional rollout, while Karen in marketing wants to see overall sales for 100 store locations in both Germany and Netherlands. The NL's retail stores don't split out VAT because their POS systems don't allow you to do that...etc etc. You can see where this is going...
2. Implementing ERP at the global level is incredibly complex. There's this idea (especially in the HN circles) to just "throw SaaS" at the problem, but it's not that simple. If you haven't been involved with an implementation at this size, it's not worth discussing. It's a wonderful hodgepodge of culture challenges, project management methodology clashes, and misalignment of strategy.
3. Yes, no one ever got fired for hiring SAP, just like they did with IBM. This is obviously changing, but still has some truth to it (see next point).
4. Honestly, being a CIO of a global F500 company absolutely sucks. If you try to stick your head out and do anything innovative and it fails, your axed. Many CIOs simply go into turtle mode - protect the budget you're given by the CFO and just don't f anything up. A CIO who choses to build a custom ERP and fails will be fired 100x faster than one who uses a packaged solution and it fails.
5. What's the alternative? Oracle? At this level, they're all the same (ok, not totally true, but generally accurate).
Happy to answer any questions...this area is unfortunately/fortunately close to my heart.
How do you see the chance for classical “low-end disruption” like Christensen described? i.e. small players buy small solutions because they’re cheaper and have less scope. Then larger companies start using the low-cost software etc.
I don't think it's possible for software to suck worse than the Oracle E-Business Suite expenses reporting software.
Scale: at this size, you pick SAP or Oracle. There's very little else out there in the market.
Proven implementation: speaking for most clients, you cannot get a major ERP implementation done without outside help. You will turn to either the vendors or major integrators. You will then ask them to give you advice and prove their expertise. They will give you a list of prior successes, which is in my sector always going to favour SAP as Oracle have a bad track record. You will then pick the system and configuration and maybe even architecture they are recommending.
Industry Expertise: SAP, in many industry sectors, know their shit. You can try making your own tools, you can try pulling 20 different apps together to get the functionality you want. But that increases cost, complexity and often violates architectural principles. So you just buy SAP and all the modules, because they have every possible business process mapped out, data structures configured, suppliers and product catalogues ready to go, etc. Oracle has this too, but fewer products support or integrate well in my limited experience.
Of course, SAP isn't without its issues or drawbacks, but i have consulted with clients across many projects including the initiak upstream business cases, and these are often the key deciding factors.
Of course, user's experiences don't matter when selling or supporting a software like this. I don't think it's part of the decision maker's conscious that usability (in the sense of being able to accomplish tasks efficiently) for a software that will be used by a larger part of the corporate workforce might be important not only for the employees using it, but also for the company. These systems can eat up so much work time of employees and create so many issues within a company and in customer or supplier relations...
Then, once the implementation is ongoing and it turns out that there are some corner cases which were overlooked and that don't fit into the SAP model at all, SAP has another option available: hire their expensive consultants. Depending on how locked-in they are and how big their projects, this can completely destroy some companies.
ERPs are tedious, uninspiring, production-level software that have to deal with all manners of ill-defined business rules and variations. It's almost impossible to come up with an elegant product because the problem space itself is inelegant and in many cases irreducibly so. I imagine it's hard to attract talented developers (except those solely attracted to money) to this space because the feedback loop and intellectual payoff is so unrewarding. Most of the time the implementation problems have to do with human problems, not technical ones. Most ERP customization is outsourced to pools of fungible labor.
But when you run a large enterprise, ERPs are absolutely necessary, so someone's gotta do it.
1. The first reason for this is compliance. Most large companies are worried about compliance and trust only comes with many years of successful deployments. For a new entrant to be recognized can take upto 10 years and for clients to accept the solution as a solid one.
2. The second reason is complexity. When you consider the scale and complexity of an implementation, most new ERP companies do not understand the effort goes into pre-sales and going through hundreds of pages of RFPs. There are system integrators, consultancies and entire ecosystems locked in to this duo-poly.
But things are changing. We run an open source product (https://erpnext.org) and we have seen people are looking to move away from the SAP-Oracle duopoly. Innovative companies where the CTO has as much as a say as a CIO in such decisions are willing to look outside at Open Source. It is still very early days, but this space looks exciting right now.
It's the same situation with Oracle, Microsoft Windows and countless other "corporate" software solutions. If you look at it from a hacker point of view it makes no sense, it's ridiculously expensive, it's often overkill, it's not elegant, you could do the same thing for 1% the cost and it'd work better etc... Now if you look at it from the point of view of a company that needs something that will work 100% by a given deadline and you want somebody you can yell at if it doesn't (and you want to know they'll still be here 6 months from now) it makes total sense.
I used to think that - then I did some work with large scale ERP systems and realised that I was wildly wrong. Fortunately I moved into other areas while I still had some sanity.
Except an expensive fashion item doesn't ruin your ability to do your job and is still probably cheaper in relative scales.
They can get projections of what the number are going to be for the quarter continuously so in the last few weeks of the quarter they can speed up or slow down sales and spending to hit the earnings numbers they want.
One guy who worked for me did a 4 day week - he sometimes took half a day of leave, and when he got to his last half day he could never book it, because the system was adamant he only had 0.49 days of leave remaining.
Not sure if that was bad configuration when it was implemented ... or the product being a bit ropey when something doesn't fit their model perfectly, but it stuck with me and made me forever wary.
Supply chain may be complex but for heavens sake, 500M euros! and not complete. I can do it for half :)
You probably conflate that period with the 60s-70s.
In the late 90s/00s it was when IBM regained some mojo by adopting Linux and OSS and building on top of them.
It was not its time of "infinite trust", but a time of a recovery into "some trust".
Of course, at the fraction of the the cost of the SAP solution.
One of the simplest things an SAP implementation team can do to promote their success is to strictly limit the number of ABAP developers on the team. Developers are going to develop and while some customization will be needed, with a team of 10 or 20 ABAP developers you are going to generate way more customizations than you will with only 3 or 5 developers. Resource and skill scarcity will drive the discussions of "do we need this?"
More rare is that you also you need an exec sponsor at a CxO level (other than the CIO) who has the intestinal fortitude and leadership skills to tell their employees "we are going to conform our process to what the software offers" rather than the opposite. Your unique methods of fixed asset management and accounting (to take one example) are not where your company needs to invest in being different from every other company, much as your accounting manager who rails "this is how we always have done it because of our special situation" will complain to the contrary unless the CFO is telling them otherwise.
Because introducing a new data system didn't work out, the Deutsche Post (German Mail) has already booked higher losses a few years ago. The same now happened to Lidl. The planned system still doesn't work smoothly after seven years and more than half a billion euros in costs. Now, the discount store pulled the kill chord.
For years, Lidl is expanding operations. The discount store from Neckarsulm has stores in almost all European countries and is now also expanding in the USA. A new inventory management system should support purchases and logistics to keep track of ever more complex business. That was the decision in 2011.
System is not suitable for high-turnover countries
The software of the development concern SAP from Walldorf should be adapted to the needs of Lidl. Until now, the system is only being used in small places in Austria, Northern Ireland and the USA. It was clearly shown that the by over a hundred IT specialists developed SAP version is not suitable for high-turnover countries. Now, Lidl has stopped the project. In a paper called "Heilbronner Stimme", an article writes to the employees that the goal is not achievable "with reasonable effort". Until now, the project cost over half a billion euros according to expert opinions -- for example for costly IT consultants and SAP licenses. Now, Lidl says they want to further develop their old inventory management system.
---
Edit: I've never had much to do with SAP, but just today I've been applying for jobs in Germany and two of the places (out of five or so) use some SAP system for their online applications. The former threw a HTTP 400 and later HTTP 500 errors so I couldn't complete the application. The latter is currently stuck on registering: I've been waiting for the page to load for 4 minutes now. Pretty sure that one is broken, too.
pulled the plug
> For years, Lidl is expanding operations.
For years, Lidl has expanded operations.
> high-turnover countries
high-volume or high-revenue
> pulled the plug
Kill cords are a thing (on powerboats, particularly) and the original metaphor is fine.
> > For years, Lidl is expanding operations.
> For years, Lidl has expanded operations.
'For years, Lidl has been expanding operations'. Past continuous tense is better if the expansion's still going on.
> > high-turnover countries
> high-volume or high-revenue
High-turnover is completely correct, at least in British English.
I realize these words are not directly from the company and I am reading them through Google translate, but I do think they accurately reflect what makes these implementations work or not. If you are getting into SAP and "adapting it to your needs" then you will almost certainly fail.
Scenario: I am implementing SAP's invoice approval workflow in a company that already has an invoice approval workflow that a former CFO designed quickly after a problem a few years earlier. Its worked fine.
Companies who are successful implementing SAP: "I mean SAP's workflow makes sense, is already implemented, works for much larger companies, and has no bearing on our core business, so lets just switch to that.
Companies who fail at implementing SAP: "Lets hire some consultants to adapt SAP to our needs"
Source: I did process flows in preparation for SAP implementations in the late 2000s.
On the other hand I know some consultants making millions by creating customizations to SAP specific in their vertical and then sell that as a packaged product to all competitors. Usually these products are incredibly niche (like a specific workflow for Scandinavian utility companies that implements some stuff all Scandinavian utility companies require by law but SAP does not provide. Picking between then implementing it by hiring Accenture or buying off the shelve from someone who did this before is easy.
This is a tricky topic. If you're getting into SAP and you're not adapting it to your needs, you're almost certainly doing something wrong.
The trick is that it _has_ to be a "meet in the middle" kind of situation, where SAP wants to be used in a certain fashion, but also allows a certain amount of leeway — any non-trivial implementation project will involve both tweaking the knobs SAP provides to fit the customer, and developing bespoke components for those scenarios that SAP either doesn't cover at all, or doesn't handle the way you'd like.
90% of back-office stuff is exactly the same between very large companies no matter the industry or history, but that is very hard to get across to the career accounting manager who designed those processes and now is having SAP shoved down their throats.
Secretly, I always suspected that one reason for bringing in SAP was that the senior executives recognized how inefficient their backoffice was but weren't willing to force huge process changes and create ill-will for barely marginal benefits, so they hired SAP to come in and be the bad guy to argue about it for them. Then they can say "Yeah man, SAP sucks, soooo inflexible, I argued against it but what are you gonna do, changing it is really expensive, I guess we'll just have to do it their way" and message boards fill up with complaints about how inflexible and expensive SAP is. Funny how that works!
But, they won't make the sale if they honestly say "use it the way it is or else don't buy it", so they say something more along the lines of "you can customize it".
Sometimes the project cannot be stopped immediately during development, even if it is already known that the project is a failure. The reason is that some companies capitalize the cost of the development over many years, and if they were to cancel the project they would have to book the entire amount as an expense in a given year, instead of depreciating it over multiple years.
I know of one company that worked on an SAP implementation for about 5 years, when it was only supposed to take 6 months or so, and the reason was that if they stopped the project they would have to book the entire cost as an expense in the current financial year. The exec team, who had financially-based bonuses, weren't keen on that.
Instead the project was put together to the extent possible so that it could be declared usable, and then promptly shelved. However, it was "completed" so in theory the sunk cost could have been spread out over multiple years.
Sometimes, you just have to follow the money.
In defense of ... well actually I don’t know whom I defending here, but what is undereported by German media in this case is “Scheinselbständigkeit”. The remainder of the story then fits the common narrative that developers share about SAP (look, enterprise software in action) and consultants (all overpaid underperformers).
“Scheinselbständigkeit” is a legal term, translatable as “fake entrepreneurialism” and a common problem for German IT freelancers and contractors because it basically states, that you aren’t a freelancer if you serve one client exclusively for more than 6 consecutive months. This could result in huge fines for you but more likely your client as you become retroactively an employee of your client which in turn must pay social security for the years you served him.
It is still a vague term though and in result causes a giant legal grey area, which in turn results in each sector copying what its dominant company’s legal team declared.
For Lidl this means that in the middle of this project, they just happened to fire all of the freelancers.
You can imagine the impact: the most capable and experienced developers had to be replaced with inexperienced freshmen or stuff had to be outsourced to agencies staffed with inexperienced freshmen.
The way I see it is that it's tough, solutions are rarely chosen based on technical merits. It's a mix of "nobody gets fired for choosing SAP/Oracle", decisionmaker's ignorance, having someone who will be there to point the finger at in 5 years and... politics.
The bigger the company, the harder it gets to find anyone who would know or care what the hell is going on in the company and what their needs are. The few cases where you can win over SAP/Oracle is when a company is cornered because they have either failed to implement one of the big player's solution or they are out of money so they have to "compromise". IMHO having to "compromise" has turned out very well for some. The level of customization that we did would be very hard or impossible to achieve with SAP/Oracle.
I know all non-routine maintenance, mandatory inspections, contractors and warehouse supplies etc is booked in it. Stuff like fire extinguishers, elevator mechanic call outs for breakdowns, replacement hydraulic fluid stuff like that is done via SAP. I think even general 'consumables' stuff like toner for printers in offices / control rooms, fluorescent light tube replacement etc are managed by it.
When you think about it these systems are horrendously complicated.
Of course the project was killed after 3/4 years and millions spent.
A good thing for SAP that in the large firms they target, mediocrity floats upwards!
The main problem is not the actual software it is the people who implement it.
Sap as a company has no control over the implementors. The implementors don’t understand how the software is intended to work.
I can compare it to Samsung Android vs Google Android.
Actually, this leads to the same types of problems that everyone is discussing here over 'customizing'... at the root it is a collision of ever larger growing global companies with ever-increasing complex process that must rely on older and older code (or customizing) bases, because stopping everything and doing a total overhaul just isn't feasible at such scale.
Every custom program or customized process brings rise to a future risk of headaches/breakages when the system is upgraded, a process is once again changed, or something should be added to the process.
I guess the best analogy I can think of as an ABAP developer, is that it sometimes feels like running on a treadmill that just keeps getting faster...
Someone commented above how IBM used to be the company that was infinitely trusted... and indeed, SAP itself as a company is certainly not at fault, but until the environment changes with SAP customers and all their custom code and processes... I guess only time will tell...
Most non-enterprise people don't get it (as seen in this thread where most people are only taking into consideration the cost of development): In a software suite's lifecycle, the initial implementation cost is nothing compared to the total cost until sunset.
Application lifecycle management (or product lifecycle management in a broader context) is a hugely important thing for a corporation that plans on existing for more than a couple years.
This is a good illustration of the mindset difference between Enterprise and Tech.
The Dutch government is a very different story. Until we have a few parallel governments between which we can easily choose , without having to uproot our entire lives, it is a pathetic waste of tax payer money. I dream sweet dreams of accountability and proper incentive structures for government and public servants.
One day.
One of those companies also wen't bankrupt a couple years later. They weren't doing all too well before either, so wasting a huge load of money on a system that basically makes your internal processes grind to a halt probably didn't help.
Hearing that you waste half a billion on this really makes me wonder if it isn't much more practical to just develop custom made software in house.
Lidl is wasting 500 million euros
Because the introduction of a new data system did not work out, Deutsche Post already had to record a high loss several years ago. The same thing happened to Lidl. After seven years and costs of more than half a billion euros, the planned system is still not running smoothly. Now the discounter has pulled the ripcord.
Lidl has been on an expansion course for years. The discount store from Neckarsulm now has branches in almost every country in Europe and is now also growing in the USA. A new merchandise management system was needed to easily keep track of the increasingly complex business processes and to control branches, purchasing and logistics. Therefor the decision in 2011.
System is not good for high-turnover countries
Software from the Walldorf-based software company SAP was to be adapted to the needs of Lidl. So far, however, the new system has only been introduced in some small agencies in Austria, Northern Ireland and the USA. It has been shown that the SAP version developed by over one hundred IT specialists is not suitable for high-turnover countries. Now Lidl has stopped the project. In one of the newspaper "Heilbronner Stimme" present letter to the coworkers it is said that the actual "goals" are "not reachable with justifiable effort". So far, according to expert opinion, the project has consumed more than half a billion euros - for expensive IT consultants and SAP licenses, for example. Now Lidl wants to further develop its own inventory management system.
Screen are rarely referred to by name, but have weird codes like XQ44 and are ugly AF. It also has a whole bunch of utilities for fixing up the referential identity on records, importing data etc.
If they're doing anything beyond vanilla consulting work with SAP then I doubt they're using cheaper offshore talent. Or maybe they are and that's why this project has failed so spectacularly. (Note: I'm not saying all offshore talent is hot garbage, I'm saying the cheap talent is hot garbage, especially SAP consultants).
ERPs are hard. ERPs are not software for modelling business processes, they are software for creating software that models business processes. And at the next layer you have all of the shortcuts that the vendor and the customer took during getting something that works out the door. Which happens a lot, at both levels. This is of course a recipe for fun interesting times with lots of billable hours.
Just one example : we had a big implementation in the big region where Implementation manager for SAP introduction was originally from. He was approached by some factory director and asked if we could change the standard process as "You know we have these guys working for years and they are not very adaptive for changes..let's make it easier for them, will ya?" And Implementation manager agreed. "As ERPs come and go, but connections and people stay there". So it took 3 consultants + 2 developers three months to rework the process. It later was not working perfectly, so many bugs and enhancements(as these guys couldn't stop asking for more changes) caused more money spent in support.. And it's just one episode in several years long implementation cycles.
On other hand I had very smooth implementation in TOP 50 company. But for them it was not only to implement as software, but also to audit their internal processes and to eliminate waste. There dedicated Implementation Manager started introduction to SAP with words: "We are Columbus, SAP is as ship which will bring us to America. And wont be easy, but I hope you will all jump on this ship otherwise we will have to leave you..".
SAP ERP is made for standardizing and simplifying processes, so just don't invent light-bulb every time you need a light in a room.
I would estimate using the SAP systems costs about 10% of productivity (net after gains through digitalization of processes).
Problems with new data system: Lidl blows 500 million euros
The Lidl brand plate at a branch. Picture: BR/Herbert Ebner
Lidl has been expanding for years. The discounter from Neckarsulm now has branches in almost all countries in Europe and is now also growing in the USA. A new merchandise management system was needed to provide an easy overview of the increasingly complex business processes and to control the stores, purchasing and logistics. That was the decision made in 2011.
System is not suitable for high-turnover countries
Software from the Walldorf-based operating software group SAP was to be adapted to Lidl's needs. So far, however, the new system has only been introduced in the small agencies in Austria, Northern Ireland and the USA. This has shown that the SAP version developed by more than a hundred IT specialists is not suitable for high-revenue countries. Now Lidl has stopped the project. In a letter to the staff of the newspaper "Heilbronner Stimme" it was stated that the actual "goals" could not be achieved "with justifiable effort". Experts believe that the project has already cost more than half a billion euros - for costly IT consultants and SAP licenses, for example. Lidl now wants to further develop its old merchandise management system.
In a letter to the staff of the newspaper "Heilbronner Stimme" it was stated that the actual "goals" could not be achieved "with justifiable effort".
That's like management was: "What the hell, the memo gets leaked anyways. Let's send it to the newspaper directly."
The IT part of SAP is quite small and easy to grok if you have CS training, what you will learn is mostly SAP specific terminology, the details of the business process(es) you're working with, learning to work with outdated software packages and learning to work around people in an IT role who have no interest (or skill) in IT.
That is completely unrealistic, it arguably won't even cover the operations teams that you need to have in place for keeping it running and the teams for maintaining and updating the software are going to be a lot more than that.
A colleague and myself handle many such applications by automating the living daylights out of everything around them. I just finished SAS integration system engineering project, fully automated. Two weeks’ worth of work which took the previous engineer six months non-stop. 750 grand was a generous overkill.
I bet the problem was at scaling/updating system on individual locations because I heard about the similar problems in small remote groceries/shops.
https://www.computerworld.com/article/2493041/it-careers/air...
The article is fairly thin on content, though. It barely contains more information than the Hacker News headline.
