The Canadian public service is really complex. There are multiple unions with multiple overlapping collective bargaining agreements, where the public service is allocated to different classes. These classes are paid specific rates, with retroactive pay being common for changing classes. The majority of the problems with Phoenix have been employees moving from their classes and being paid the correct amount. It has also adversely affected non-unionized positions.
My understanding is that the Harper government (Prime Minister until 2015), who was responsible for the negotiation and for laying out the requirements, was trying to save a money and not responibly create a pay system. Two major factors that jump out at me:
1) Requirements did not call for training. The system was implemented and IBM was not required to train any operators on how the system functions, which is important because all new staff were hired to run the system and,
2) Due to the need for cost saving measures (the government was trying really hard to balance the budget, as the election was coming up), the previous payroll staff were terminated and a new payroll centre was opened in Miramichi, which is a small town in the middle of nowhere.
So, on top of new software, the government lost all of its institutional knowledge regarding payroll and how things are supposed to work. It's actually hard to say how much of this is IBM's fault and how much is the governments, because the government doesn't know how to fix it. No one knows how Phoenix works and no one knows how it is supposed to work. It's just a big mess with no end in sight.
Could IBM have done a better job? Probably, but garbage in, garbage out.
For further reading, the Auditor-General's report: http://www.oag-bvg.gc.ca/internet/English/parl_oag_201711_01...
I respectfully disagree. IBM management either knew what they were contracting was a recipe for disaster, or they were incompetent. In both cases, as they were supposed to be the billion-dollar expert on the matter, they bear great responsibility for this failure.
And at an industry level, companies that promise the impossible push out of the market more honest ones, and they deserve all the bad PR IBM is getting on this one.
Any vendor will do exactly what is asked, no matter what. IBM, Dell, Accenture, Deloitte, McKinsey, etc. There is no reason why a customer should allow a contractor like this to run roughshod over them -- thats incompetence and bad governance.
I've run big projects with vendors like this. You write good RFPs and hold feet to the fire and they will deliver.
IBM have made billions doing so. Bad publicity is just a good start.
The really interesting point is that if this business plan is shown to be strategic (if I were Canada, Australia or Pennsylvania I'd be looking at full discovery of all email records), IBM would then be vulnerable to civil suit for fraud.
Customer even gets angry and asking for removal of such risks. Vendors comply by sneeking it in the contract in clever words.
In the end project is either canceled or goes over budged or barely manage to deliver anything.
Those companies did continue to do business with us though and we've normally earned back our losses and more on future projects.
That's what honest vendors do. That's not what IBM did.
The software developed by IBM was so complex, that it required the expertise of the people who got laid off to operate it. The novices who took over didn't know how to make the software work, and chaos ensued.
The Canadian government is as much at fault as the developers on this one. (I speak as one who is paying for this fiasco :o).
The first $200 million of cost overruns I'm willing to pin on the Government of Canada. The subsequent billion dollars and the failure of the system to work at all is the vendor's fault.
If the Government of Canada is entirely impossible to work with, the vendor should have shut the project down at 200% of budget rather than run it to six times budget and still fail.
It's obvious that there was zero actual knowledge involved related to implementing large systems - just by the fact that someone thought it was a good idea to take a behemoth system excreted out of Oracle, and then pass it through the corporate body of IBM, before inserting it into a huge government organisation, while at the same time getting rid of any process expertise and business knowledge they might actually have on-board.
However - I don't think it's probable that any amount of training could save the situation to any measurable degree whatsoever.
Is IBM to blame? Well, I think they knew what would happen.
If the project is estimated to take over a year to deploy, by then you're in a new budget cycle and the vendor has come to an agreement with the procurement department to charge for additional services as required in subsequent phases.
my earlier post: https://news.ycombinator.com/item?id=16495431
basically, it's a political game... the same jobs they "lost" in miramichi because of the firearms registry legal change, they tried to get back. Same pork barrel politics that congressmen in the US play with trying to bring jobs to rural districts.
I'm not sure when that changed, but I feel like these days, IBM is closer to Oracle than AWS.
"IBM is a multifaceted company.
These kinds of projects fall under what used to be called IBM Global Services https://en.wikipedia.org/wiki/IBM_Global_Services. It could be compared somewhat to EDS (HP), Accenture, Perot Systems (Dell) etc. I've never heard of over-delivery from any of these kinds of outsourcing arrangements, they always seem so obviously destined for boondoggle.
IBM proper, the one that makes mainframes and POWER and DB2 and a ton of operating systems and storage etc is very much a technology company. Some of their best products have the worst sales and marketing efforts. I'm working directly with the senior leadership of the POWER group right now and there are no salesmen in sight.. the technology will either sell itself or not. When we met in person the first time the GM told me "we can build any kind of computer you want" - meaning microarchitecture changes, SERDES configuration, new board layout, sheet metal, OS, application tweaks. Not a lot of companies can do that. There is hubris, less technology, and lack of technical value at FANG or most startups or whatever your benchmark is in comparison.
IBM Research is one of the only remaining great industrial research organizations https://en.wikipedia.org/wiki/IBM_Research. Their results speak for themselves."
The problem is that having a boondoggle division reflects on the whole company. When I'm making decisions for a client, I never even think about what IBM solutions I could apply to a problem.
I've only had a few projects using IBM products, admittedly most very old legacy systems -- but those introductions failed at just basic dev hygiene things. Documentation behind paywalls, couldn't get things to run on my dev machine without very experimental projects made by individuals - last updated in 2003. Just frustrating. I don't like advocating complete rewrites, I've happily updated a company's internal tool that was written in as3. Could have squeezed more hours out of them if they wanted to go with html5, but doing so didn't open up any new possibilities they wanted so I didn't.
If your company is running on RPG, and it's just a CRUD app -- I'm going to save you money by recommending a rewrite under the two circumstances I've encountered in the wild.
Honestly, this isn't their fault. It's awesome that their products have lasted so long for companies. I don't expect a company to keep up to date documentation / interoperable interpreters for 30 year deprecated tech. IBM's just so old that they've accumulated many layers of fossilized products, and those things have caused an unfair bias in my mind (and probably the minds of others)
I'm sure that there are problems IBM research is uniquely qualified to solve, my perception is that they're the wrong choice for most problems. I'm pretty thankful for companies like them. (IBM, Siemens, SAS etc) Their expensive ERP systems have enabled me to raise my rate over the years off the information asynchrony that forms the foundation of their success.
"No one ever got fired for buying IBM" because IBM was the safe, established choice. If you bought IBM and the project ended up failing, nobody would blame the failure on your decision, because buying IBM was what you were expected to do. If you bought from some other vendor and the project ended up failing, however, people would rush to claim that the failure was your fault for not doing the safe thing and buying IBM. In other words, you bought IBM to protect yourself from appearing to have taken a risk, not because you expected them to do a particularly good job.
If TFA is accurate, it would appear that "no one ever got fired for buying IBM" is as true today as it ever was. If the Canadian government had hired the Freshbooks guy to build their new payroll system, and the project had failed, the person who had made that decision would have had a whole lot of explaining to do. That person bought IBM, though, so everyone will just shrug their shoulders and assume the fault for the failure lies elsewhere.
The talk inside was the new CEO (Gerstner[1]) was leveraging IBM's size rather than the previous plan to break it into pieces and was somewhat successful at it.
I left to go back to grad school. And I think the new management went back to the "this is too big and unmanageable, its worth more broken up" mindset.
https://www.amazon.com/IBM-Redux-Gerstner-Business-Turnaroun...
(I also grew up professionally in the 2000s, well after IBM was widely considered elite by the tech nerd community.)
I never though of it that way but rather as a CYA move.
Employed elsewhere (o hai msft) but originated, under that particular acronym, at IBM.
The cconcept itself can be traced back at least as far as Sun Tzu.
Late 1980's. I was there, working for a skunkworks fragment in a Big Blue shoppe. To express doubt about IBM was to get no bonus. Then they started to haemorrhage cash to the tune of 5B$/quarter.
Really it's that Oracle is closer, these days, to what IBM was 30 years ago.
I'm afraid Watson is very far from a "cool new tech", many companies have described their experience with Watson as a money gouging disaster.
What day was that? Back in the 70's with the Selectric typewriters?
No, seriously, those were fucking fantastic machines!
That's a compliment. IBM is by far more dysfunctional.
It changed in early 2000s, I believe.
This is not true. IBM produces a ton of software (spanning a range from terrible to excellent, quality-wise). This is where much of their profit comes from now. They still produce a bunch of hardware, mostly at the higher (and legacy) end.
This attitude is shot through the whole piece (essentializing not just Europeans but also Indians), and it's weird and creepy.
I originally thought maybe he was from Slovakia (I have occasionally made jokes or generalisations about Irish people, as I am one) -- but then the footnotes make it clear it's not self deprecation. It's just prejudice.
I think we should do more of it on HN (and everywhere else). People should try harder to decouple their biases from facts especially in their public writing.
The work culture in France has been understood for a long time. I don't understand how you think it's bias, when you clearly have never worked there.
Pick up your suitcase and get outside North America for a bit. The world's a lot more diverse and interesting than the education system might have you believe.
Referring to Ukraine as "the Ukraine" has been inappropriate since the end of 1991. Just this detail paints the author as being out of touch.
Or maybe just not perfect? Google trends show that "the Ukraine" is about 1/3 as popular as "Ukraine" in common usage. It's more like calling the FSB the KGB - wrong, for sure, but was correct for decades, so not at all surprising that people still say it. http://www.businessinsider.com/why-ukraine-isnt-the-ukraine-...
:(
It's not uncommon for large government projects to fail with all the bureaucracy and politics that is in play. I am not saying IBM is great, but this article is just not worth it.
Can anyone share a good IBM experience? I'm honestly interested.
The IBM pc we bought in the 90ies was fantastic.
It had working suspend resume many years ahead of other computers and came with 365/24 free phone support for a year (and the next years was about 50USD). This being was very useful since I didn't have Internet access.
The support team had unlimited time, patience and knowledge and would walk me (15-16 y.o) through any procedure including if necessary: backup, reconfigure disk with fdisk, restore os from backup etc etc. I learned a whole lot from them both technical stuff as well as how to deal with customers ;-)
(And yes, old IBM ThinkPads were very popular IIRC.)
Edit: details.
Its almost seems like when you can get away with milking the system, nobody seems to have any moral qualms in selling the government a $200 screwdriver.
I remember hearing about it then, but I cannot now find details of the problems.
Who was responsible for picking IBM? Are they still working for the CA gov? Have they passed the hot potato to someone else? How were their technical skills and soft skills evaluated? Did they receive any donations for the contract? Where are the safety clauses in the contract? Is IBM the only benefactor of this contract? Any political implications?
Remember that Gov. point person also bears a lot of the responsibility for shopping for IT services in a magazine or trough their business network.
If IBM has been so ineffective at delivering services there would be more cases like this and it would ultimately hurt their bottom line. If this was wide spread practice across their business units. Maybe their business as a whole is insulated by the other better performing parts of it's corporation.
It seems to me that these "Governments" should investigate anyone who touched these contracts.
To the widespread corruption present in Eastern Europe. It does exist. However this article goes to show that corruption is present at a larger scale in some of the "most" developed nations on earth.
No wonder IBM has to go get developers outside of the US/CA/AU if they were the lowest bid.
Now it all fits together. Whenever i go to my states web services and they look odd. I now know why. That is changing though.
http://www.cbc.ca/news/canada/ottawa/phoenix-ibm-contract-un...
Few if any 'integrators' will touch these projects.
They were forced to put some IT guys in our office. Those are 2 kids working as cheap freelancers for a local subcontractor hired by IBM.
Service is horrible. Tickets stacking up and being deleted regularly. Project relevant IT not working, people leaving because of it. It's a mess.
Its just that most of my colleagues hate the software made by IBM, its slow, a pain to use and has a tendency to make things very very complicated. You cannot build great stuff with mediocre tools.
The IT industry has moved on to leaner and simpler tools and platforms. IBM, atleast the GBS group, is basically a sales company.
Note : I worked in IBM GBS for a year before I left for more interesting and rewarding jobs. This was in 2008 though.
On the other hand, talking with friends in the government and government contracting, they argue that IBM and the major contractors have perfected the art of exploiting government into maximizing billable hours at the expense of results.
It's a bit like the metaphorical malicious genie interpreting your request. It isn't possible to write specifications that can't be interpreted malevolently.
There's probably blame on both sides, but it seems nowadays that "it needs to be a roaring dumpster fire until we do things the right way". Unfortunately, either no one _in_ government seems competent to know that right way or be empowered enough to fight for it (maybe they've all left).
"Here is the tl;dr version of their story: Marketplace Lite, or “MPL” as they came to be known, devoted months to rewriting Healthcare.gov functions in full, working as a startup within the government and replacing contractor-made apps with ones costing one-fiftieth of the price."
https://www.theatlantic.com/technology/archive/2015/07/the-s...
On the government side, many of them realize they're playing a losing game, but they don't have the resources to change the system, and it's not a priority for people to get elected, so congress doesn't step in. (And would they know enough to do so?)
Consultancies mis-sell and misrepresent their services, whilst public service/government scopes the project poorly and is more than happy to buy into the fantasy. Every project is different so this is still an oversimplification.
Nevertheless, when an enterprise IT service provider is involved in a project it's often a red flag on successful delivery. Whether that's correlation or causation is up for debate.
No really.
The issues on the other side are the opportunity ibm, accenture, oracle etc exploit. They're not just aware of those issues going in, they are 100% counting on them.
They're not even interested unless those issues exist - it's one or two orders of magnitude less profit. In fact you can say that those issues are why these consultancies get hired in the first place. No competent manager would go near these carpet baggers.
You can blame the marks for being marks and say they deserve it (tax payers) but this has nothing to do with IBM being a con. A con they are. If they don't like that reputation they shouldn't do it, again and again. But it's so profitable it pays for the PR to restore reputation up to a point. We've now got that point. No PR can cover the stench.
IBM - only an idiot or the corrupt on kickbacks would engage them to con.
Did this article really compare FreshBooks to the payroll needs of the Government of Canada? Whatt??
I'm suggesting that Mike McDerment could build and lead a team who would successfully build a payroll system for the Government of Canada for far less than a billion dollars.
IBM has not built a working payroll system (despite starting with Oracle's Peoplesoft for that money). I suspect my company (or some of our best competitors) could build a working payroll system for about $50 million based on our success on smaller projects. Our specialty however is in publishing systems and online video. I'm specifically recommending Freshbooks as McDerment has a background in creating accounting and billing systems and scaling them.
Additionally, as a Canadian, McDerment is already familiar with many of the variables to do with Canadian tax and employment legislation and geography. Any foreigner would need three to six months to assimilate Canada specific information.
I also don't suggest that IBM could not have built this payroll system satisfactorily. Based on failed projects in Canada, Australia, Pennsylvania and Slovenia, big promises, poor delivery, unlimited billing and zero liability on government projects appears to be strategic policy (links to each of those projects appear in a new preface to my article). Such behaviour has made Big Blue billions.
Right but those transactions have all been more or less the same type of vanilla invoicing. Unless something has really changed in Freshbooks, that’s an app that handles very specific relatively simple needs. The government of Canada is a giant entity that is most seriously orders of mangnitude different and complex than any usecase for FreshBooks, and that’s just the reality.
> I'm suggesting that Mike McDerment could build and lead a team who would successfully build a payroll system for the Government of Canada for far less than a billion dollars.
Maybe, but probably not. The real problem is probably that the business needs of the government of Canada are not well defined and the bureaucrats cannot define them. The comparison to FreshBooks is just so off because it’s totally upside down. FreshBooks was built and provides certain functionality. Building a payroll system for The government of Canada is an exercise in reverse engineering the Government of Canada. It’s an exercise in squaring hundreds of not thousands of different circles and making them fit into a box.
Presumably you just have a table with people with salary, and every month you wire the money and record information about the wire or otherwise have a way to record when the person has taken cash.
You might want to have more complex rules for determining schedule and amount of salaries, and support for contractors, unpaid leave, taxes, etc. but that doesn't seem particulary complex either.
How can this possibly take more than 100 person-years (i.e. < $30 million) to make even a truly extravagant version of?
Not to mention that presumably such software already exists and could be used instead of writing a new one.
[obviously my explanation would be corruption, but I wonder if I miss something]
There are thousands of byzantine rules in the contracts and agreements, some mutually contradictory.
The old code likely contained a lot of business logic that represented an institutional memory of how to handle all these corner cases.
Anyone wishing to reimplement a system like that will have to rediscover these corner cases one at a time in production, while thousands of people complain that their paycheque is short.
Then there are also pensions to keep track of etc...
IBM promised a big bang that would be finished before the election. What could possibly go wrong?
I think that there is a positive aspect. Other governments will learn from mistakes when ordering expensive projects like this.
It's like dealing with any other service business like contracts, lawyers, wedding photographers or others. If you don't give them clear guidance the result will most likely be an expensive disaster.
I think governments should have a special organisation that just maintains the expertise for handling these sort of big projects all across the government.
"How IBM Conned My Execs Out Of Millions" http://atdt.freeshell.org/k5/story_2005_9_27_95759_4240.html (cache)
I remember the guy was harassed by IBM lawyers and even lost his job.
Small projects are better focused, cost less, are easier to understand, and therefore succeed more frequently (or fail more silently). It's big projects, frequently proposed and conceived by governments or enormous industrial conglomerates that are poorly thought-out, improperly managed, and suffer the worst of project management incompetence or hubris/excess. But who bids on that kind of project? Big companies like IBM. Let's be fair, they don't just bid on them, they also coax them into being, but my point stands: if you want to fail big, you've got to dream big. This isn't a defense of IBM, who deserves to own the shame of talking big but being unable to actually deliver. But it is a reminder that the project designers get equal blame for these sweeping, grandiose, visionary catastrophes.
I remember once talking about how they (big blue) see Atlassian as a competitor and how they have to try harder to convince companies to use IBM products... the very moment someone replaced Jira with some IBM shit or Gmail Business or even Outlook with Lotus Notes I would quit.
When I worked for IBM Global Business Service div as a dev, most of our payroll, billing, travel approvals and what not were done on LN. IIRC email was just another app on the platform.
http://www.abc.net.au/news/2016-11-25/ibm-to-pay-over-$30m-i...
http://www.abc.net.au/news/2016-10-25/turning-router-off-and...
https://www.itnews.com.au/news/queenslands-ibm-ban-lives-on-...
The quote from Queensland's Premier at the time of the ban:
“I don’t want companies that have this sort of culture doing work for the people of this state.”
IBM produced a $1.2 billion payroll project for Queensland heath that just never worked.
NZ also had a disastrous police database project fail that was managed by IBM.
From day one the client company gets swamped by an army of vendor analysts, the prime reason for this is to establish that it is the clients fault the project will fail as they couldn't respond to requests for information fast enough. Any information you manage to supply will be scrutinized for 'discovery' of 'change requests' to pad the meters and CYA.
Meanwhile, a 'technical specialist' room chock full of developers is installed to (a) put on extra time pressure and (b) start the billing engine in top gear. When you went into that room there were literally people sitting at desks watching the vendor's equivalent of CS 101 video courses.
Also, a team of lawyers is already preparing the documents for the 'settlement' in case of the (very likely) project failure.
All status information to the project's steering committee gets 'green shifted' until the supplier is ready to shift to litigation mode and then overnight the project's near unrecoverable disaster gets revealed alongside a proposed 'rescue' plan that is priced so ridiculously it makes the 'settlement' look cheap.
In the private sector these train-wrecks are often settled with non-publish clauses as making the press would make both parties look bad. In our case the supplier dropped a small percentage on the billing. The client was left with a room (literally) full of boxes of A4 'analysis' documents and 25M€ out of pocket. (this was in the financial industry, so it was basically pocket change)
All involved, both supplier side and customer side seem to have not suffered career wise from this disaster, moving swiftly to new clients and new projects.
P.S. Am I the only one that finds the author's sweeping generalizations of nationalities a bit in bad taste?
Generally you respond with a proposal that on the surface ticks ALL the boxes. Later you show the proposed product and state that all the missing parts will be implemented or altered during implementation.
You get a bunch of senior people or architects to estimate the work +/- 1000% (mostly for the time estimates). The difference between what is charged to the customer and the actual cost of the work is swallowed by the bidder with the idea that it will be gouged back during additional work that wasn't in the initial scope.
Then you present your proposal and if you win the bid you then try everything to make it profitable from that point on.
ps. None of this is compatible with agile.
So they outsourced the work to Oracle which is known to churn out garbage. Also just a lot of blame shifting between garbage-quality companies.
Okay, so Canada signed a bad contract and didn't make IBM promise to delivery the goods.
> In cases like this in the past, the Canadian Government would just be able to tell IBM to deliver the goods as promised or IBM would be banned from doing business in Canada - effectively frozen.
But you just got done telling us that IBM didn't promise to deliver the goods.
> Under NAFTA and similar trade pacts, governments have lost all leverage and these sweetheart deals continue to be pushed through.
NAFTA and similar trade agreements neither require governments to sign contracts that don't require delivery of working solutions, not require governments to not sue to enforce contract terms once signed. If IBM broke the contract terms, Canada can sue. But if Canada signed a really dumb contract, there's a number of remedies here, but violating the contract terms and just demanding other parties do things they have no obligation to do to make up for your mistake doesn't seem like a good choice. You know, "rule of law" and all that?
> Why should the Canadian taxpayer foot the bills for corrupt contracts with devious suppliers? The
They shouldn't. But, well, was it corrupt? Prove it in court, and the contract can be voided. NAFTA won't stop that either. If you can't prove it in court, they maybe it was just dumb (and not bribery), in which case yes, the taxpayers should foot the bill. If they don't like it, they can vote in a better government. That's democracy.
I have a really hard time wrapping my mind around a project that expensive
Many of us work in these kinds of industries.
