To be precise it is finally getting some of the attention it deserves, both from the public and from the government.
The miscarriages of justice date from 1999 until 2015, and the high court ruling (about the software being faulty) that finally stopped the flow of convictions dates from 2019, almost five years ago. Very little happened since then, and in particular no one has been held accountable.
If that delay sounds absolutely bonkers to you then yes, that is what everyone else thinks as well.
Even crazier. The Post Office just recently lowered(!) the amount of money they allocated for compensations: https://www.bbc.co.uk/news/business-67784706
They basically allocated a pot of money to pay the people they have harmed off. But these people had to actively go out and request their conviction to be overturned. Which of course many people were reluctant to do, since it means they would be upturning their life again, and going to court and finding lawyers and etc. They understandably have very little trust in the system. So Post Office just shrugged and decided they don't need to keep that much money around.
From https://www.theguardian.com/uk-news/2024/jan/09/how-the-post...:
> One [bug]...would see the screen freeze as the user was attempting to confirm receipt of cash. Each time the user pressed “enter” on the frozen screen, it would silently update the record. In Dalmellington, that bug created a £24,000 discrepancy, which the Post Office tried to hold the post office operator responsible for.
The fact that this is even possible, apart from the sheer incompetency of a software design that permits it, means that no person can reasonably be accused of stealing cash if that accusation is based on records from this system.
Consider the shittiest homegrown shopping portal probably doesn't do that and any commercial point of sale system absolutely won't do that either. The system has to be such a POS internally that no other vendor will take the liability for it.
Reminds me of the Therac-25 that killed people. This thing probably killed more people than that.
Jail. The people who knowingly lied and sent people to prison need to be jailed. Not fined, not given a handie, not allowed to simply retire. J.A.I.L.
They callously wrecked people's lives, including some that committed suicide.
It seems that some people there abused their power and need to cool down behind bars.
From the CNN article it seems like the police is going to get involved in this, so at least some good news for the people wrongfully prosecuted by the people charge of the Post Office.
>not sure if it was a bug or someone remotely changed the account records
It seems to me that determining whether this was a bug or a deliberate backdoor that allowed someone connected to Fujitsu to change account values should have been one of the first priorities of the investigation. The CNN article I read before the BBC article didn't even mention this, it seemed to just imply that this was all a mistake or at worst the government/Fujitsu trying to coverup the fact that the software was shoddy.
This meant those that fought the charges and lost were then forced to "pay back" the money they "stole" and also pay the Post Office legal fees and being sent to prison. The scandal is sickening on literally every level you can imagine!
https://www.theguardian.com/news/audio/2024/jan/08/revisited...
https://www.theguardian.com/uk-news/audio/2024/jan/08/revisi...
Also some Private Eye podcast episodes:
Also, strange seeing my hometown mentioned on here!
I've been following this story for years, I know BBC's Panorama covered it in 2015, and was really surprised there wasn't more of an outcry in the UK about such blatant unfairness, guess technical cases like this are really helped by a TV drama with quality actors making it easily accessible to the general population, as soon as ITV aired the miniseries, the petition to have Vennall's CBE stripped jumped from 7K signatures to 700K+, and she said she would return it. (not the golden handshakes and "performance" bonuses though lol)
I just can't believe that no-one has been held criminally accountable (yet) for the sheer (to quote the judges in the civil case) affront to justice that this represented - whether it was Post Office Limited (POL) misleading (well, lying to) the courts, or Fujitsu employees lying in prosecutions, or POL deliberately withholding evidence in their role as a prosecutor (wild that this power to prosecute as transferred from Royal Mail to the spun off entity), and some of the cynically legalistic games they played (like asking the judge to recuse himself for being biased against them, to further delay the outcome of the civil suit).
And the crap they pulled with the Second Sight independent analysis that they commissioned themselves...
I was just reading the Clarke Advice(s), and god-damn, I can't see how any former POL executives can claim "we honestly believed all that stuff we said in court, honest", after their own goddamn independent legal advice told them "stop these prosecutions, like now, your expert witness is full of shit" The 2nd Clarke Advice is a great read too, basically "STOP TRYING TO HIDE EVIDENCE, JESUS CHRIST".
Oh, and some of the code that turned up from Horizon in an internal Fujitsus report is egregiously bad (how do I turn x into -x or -x into x? What's -(-x)? nevermind, I got this), from what I've read, the team that wrote Fujitsus was considered low skilled and mocked by other groups within Fujitsu as basically a group of clowns merely lacking a circus.
Clarke Advice 1: https://www.postofficescandal.uk/wp-content/uploads/2022/10/...
Clarke Advice 2: https://www.postofficescandal.uk/wp-content/uploads/2022/10/...
Fujitsu report: https://www.postofficehorizoninquiry.org.uk/sites/default/fi...
Discussion of recusal application by Court of Appeal: https://taxpolicy.org.uk/assets/recusal_judgment.pdf
But this was at least a deliberate, criminal cover up at the expense of hundreds of innocent people, maybe worse, implemented with tools intended for fighting serious crime. It seems unbelievable that this could happen in a civilized country.
https://www.theguardian.com/technology/2007/apr/26/comment.s...
https://insidetime.org/newsround/massive-miscarriage-of-just...
A quick search shows me 371 different matching acronyms, ranging from the "Calgary Board of Education", to "current best estimate", to "Central Bank of Egypt".
This doesn't help.
The poverty levels are extreme in some parts of the U.K. that are far away from London and the high population density belt that goes through Birmingham to Manchester, lower than in some of the places in the E.U. that one thinks of as the poorer places in Europe. And the infrastructure gets almost zero attention and lip service from London-centric thinking. (Remember the cancellation of the northern parts of HS2?)
But some things are the same as London. I'd mention some, but they would be highly ironic given the subject headlined. (-:
Certainly Paula Vennells should absolutely be in prison for her oversight and deliberate destruction of people's lives.
For context, it led to (sometimes huge) discrepancies in what it thought should be in the till based on sales data from the day.
I don't build systems anything like Horizon, so this is probably a massive over simplification, but surely tracking stock and cash in tills should be really simple? Does anyone know if there are public details on what exactly the technical bug was?
Two bugs are detailed here: https://www.theguardian.com/uk-news/2024/jan/09/how-the-post...
> One, named the “Dalmellington Bug”, after the village in Scotland where a post office operator first fell prey to it, would see the screen freeze as the user was attempting to confirm receipt of cash. Each time the user pressed “enter” on the frozen screen, it would silently update the record. In Dalmellington, that bug created a £24,000 discrepancy, which the Post Office tried to hold the post office operator responsible for.
> Another bug, called the Callendar Square bug – again named after the first branch found to have been affected by it – created duplicate transactions due to an error in the database underpinning the system: despite being clear duplicates, the post office operator was again held responsible for the errors.
There weren't just bugs, either Fujitsu had the ability of modifying or adding transactional records remotely, but always denied they had the ability to do so, as did the Post Office.
I have only skimmed this so far but it seems there were many bugs.
Also this stood out to me.
> 4. Fujitsu inserted transactions. These are injected into branch accounts by Fujitsu. They may be performed in order to ‘balance’ a discrepancy. These do NOT require acceptance by SPMs in the same manner as TCs and TAs.
They inserted transactions for corrections of discrepancies where you could not see they were authored by them; it was just assumed the postmaster did.
They did not let postmasters know about this; probably an order by the post office.
These corrections where incorrect, and caused discrepancies to increase; they mixed plus and minus sign quite often.
They added lines of code to the system running on the counter in the local post office.
They used the message store system like wild west without defining a data schema.
We're trying to figure out the architecture over on this other thread - https://news.ycombinator.com/item?id=38954516
But basically it was based on a sortof replicated XML database that sent text files back and forth, so the basics of synchronisation and consensus were just not covered properly. e.g. if someone kept pressing a key when the screen was frozen, duplicate transactions got created. Or if they turned their machine off earlier than 5pm, end-of-day processes did not get run properly. Or if their node was trying to synchronise with the central node a bit later than normal and the central node was doing a 'reindexing' operation, their messages got lost. Lots of edge case synronisation bugs basically.
edit: also here's a good computerphile video about the bugs https://www.youtube.com/watch?v=hBJm9ZYqL10
It was not only the bugs but also the UX and the overall process that made things worse.
The system did things that the postmaster was either unaware of, or did not intend to do. When there was a customer session and a session time-out occurred it was changed into an actual transaction without the postmaster intending it; like you have a shopping cart on an e-commerce site and it transforms the shopping cart automatically into a sale after timeout.
There were screen freezes and when the postmaster pressed enter while the screen freeze happend, it multiplied the amount of money, without the screen being updated.
Postmasters are responsible for discrepancies, and when they called the telephone support they were occasionally told "not to worry, it will sort itself out". There were corrections from the outside and postmasters sometimes waited for weeks and even months for these corrections to happen to resolve the discrepancies. Sometimes they never happened.
You'd think so, except they wanted a custom system built at incredible expense.
Private Eye were one of the few people reporting on this regularly. I've been reading about it in there for close on ten years, and am still astonished that it's taken this long to really hit home what happened to these individuals. Bravo to the makers of the recent show that's brought it back into the spotlight. It's truly shocking what the Post Office and Fujitsu did, and one can only hope prosecutions arise from this.
For anyone working in IT, there are lessons to be learned here about what impact software can have on individuals' lives, and bravo to any whistleblower that came forward to speak out.
Fujitsu acted like thugs not just to save face, but because important shareholders would lose money if the truth came out.
Also, this kind of thing:
https://www.opendemocracy.net/en/fujitsu-post-office-scandal...
The PM's father in law is the head of Infosys. The PM's wife still has significant holdings in Infosys. Infosys and Fujitsu have a close partnership.
And so on. It's corrupt from top to bottom. The UK government is effectively the marketing wing of the huge public sector corporates, who invariably seem to have senior Tories and Tory donors on their boards.
Rishi Sunak -> Wife (Met 2004, Married 2009) -> Father -> InfoSys -> Fujitsu
Add up to any proof or even suggestion of corruption - just rich people know other rich people.
I’m sure that you already know about it, but the bit that gets me is the Russian oligarchs and their money, honours, property and influence.
Assignations, poisoning etc and still the situation is tolerated.
Agreed. For those more audio inclined, I linked to these in another comment but I originally discovered their reporting on this via their "Page 94" podcast:
(I have the same feelings as mhh__ in https://news.ycombinator.com/item?id=38967529. Just a remarkable and extremely slow miscarriage of justice.)
I mean how in the world can you accuse and convict someone of theft when there is zero evidence outside of the IT system. And how in the world was the IT system never scrutinized?
I personally think the prosecutions were a sign of the times when people were still far too trusting of computer systems. I feel like these days, everyone would realize that there would be at least reasonable doubt as to the accuracy of the system, yet when these prosecutions were mostly taking place it feels like everyone just assumed the system was perfect.
A friendly yet rueful amendment.
I think the idea that complex systems are assumed to always be correct is... dicey at the best of times and even more so when it's critical to a criminal case.
..absent evidence to the contrary. But in the Post Office case, the Post Office had all the evidence, and refused to disclose it. As far as I'm aware, failure to disclose evidence that might help the defence is perverting the course of justice. I don't know why no manager's been charged.
I shudder at how many scandals never come to light as the process to clear things up and get the truth out is an uphill battle. Oh and my local MP is Ed Davey, who is deeply linked to this PO scandal, though in fairness, he does seem to be singled out over all the others who did less on their watch.
I somehow wonder if society is on a path of race to the bottom at times and amazed how it works with all the flaws that just seem to grow.
Heck whilst typing this I get an email from police about some bail I never attended and yet again, it's the wrong person and has nothing to do with me beyond causing me more grief and stress. I don't even have a criminal record and due to do jury duty later this year, which is unlikely as being driven to wits end.
But hard to really deal with it due to PTSD of it all and decade of abusive neighbours who literally tried to get me killed and not only proof of that ignored but made out the other way around when it is clear as crystal with evidence submitted that is not only untrue but downright blatant lie. Police ignored that and more so, much more in past that it is a history of pure and utter failings and criminal.
Glad I have video and audio recordings proving it and that includes meetings in which police and housing lie and damming as hell. Yet, you would think somebody would care, but dam as I'm not alone in these situations. But look at Post Office scandal - how many people died over that and efforts to get to light. Then the pressure to pursue the truth and effort when others died trying.
It is scary how often things get covered up. There again, I used to work for the BBC and mindful how things can get swept under carpets and ignored like Jimmy.
Bureaucracies don't scale well.
And it failed multiple times.
Note that Scotland has its own legal system and it is generally saner. Not sure if any sub-postmasters were prosecuted in Scotland.
Scotland's legal system prosecuted and jailed Craig Murray. It also prosecuted Alex Salmond, who was unanimously acquitted by a jury; but Craig didn't benefit from a jury. The chief prosecutor is a member of the government, and controls police investigations.
In the case of a political prosecution, I wouldn't describe the Scottish system as "saner".
They were, though through the usual public mechanism by the Procurator Fiscal
I mean, they had expert witnesses and stuff. But I've been reading about this in Private Eye for years, and as an IT person you read it and think 'yes those people are totally being fucked over because the organisation is hiding the bugs that obviously lurk in their enterprisey system' whereas for non-technical people it was less obvious.
If there were duplicated entries, they would have stuck out like a sore thumb.
So even before the system came into question, basic accounting should have been used.
I'd equate that to torture.
I have some relatives who tried out ITVX for the first time especially to watch the drama, which they didn't catch as it was broadcast. They reported that ITVX was almost unusable, and also that the app crashed their telly and caused it to lose the ability to output sound until they rebooted the telly and let it go through an update cycle. "BBC iPlayer is much better." came the grumble down the telephone. (-:
I told them that it was ITV, which has 4 channels and is almost bound to repeat it.
>Post Office Scandal Explained https://www.youtube.com/watch?v=LdQQib3rmkE
The balancing. Perhaps because more was cash, but I would have expected some logging to have shown "at 4pm, the account had cash on hand of X pounds. at 4:08pm, the account had 2300 pounds less". Would that not have been useful? But watching (yes, it was a dramatization, but it didn't seem like much), it seems like PO did not want to admit any fault, ever, at all, and just kept sticking to that far beyond the time when they could have just cut their loss, held up their hands, and said "hey - problem identified, we'll fix it".
If I sold 5000 pounds of stuff, and should have received 5000 pounds, and... I have 5000 pounds on hand, but the computer says I have a shortfall 1000 pounds... how is that defensible?
Where did the money 'go'? If I had 4000 pounds on hand, but the computer then said I should have 6000, and I'm short 2000 pounds... why wasn't that discpreancy noted?
Also.. what was the motivation behind all the changes? The whistleblower seemed to indicate this was remote people at horizon "fixing" things by hand. Were these all just 'accidents' no one owned up to? The drama seemed to indicate there was some intentional retaliation against some people, but it couldn't have been intentional in all cases. But also... it doesn't sound like anyone at Horizon stole that money, they were just changing figures in a system.
These were _incredibly obvious_ duplicate entry errors when the Post Office investigated but they blindly took the system as perfect and automatically assumed criminality on the part of the postmaster.
NOTHING about this makes any sense if you assume people on the Post Office/Fujitsu side were acting in good faith.
They did not blindly take the system as perfect, but instead they were well aware of the errors and instead decided to bully the sub postmasters and gain bonuses for successful prosecution. They knew exactly what they were doing and took pains to hide any evidence of the shortcomings of Horizon.
This system works well enough that it can be done with pen and paper. Perhaps consolidating some stuff from time to time (but keeping the calculations for X years).
As far as I can tell Horizon wasn't playing by the rules, either because of bugs or because of badly designed secret admin access (maybe to fix those bugs, but you can guess how well that goes).
These changes via remote access were not logged.
There was also a point in the drama where the Fujitsu support staff talked a subpostmaster through fixing the errors herself; that is, accounts could be tampered with without using remote access.
I suppose all this unaudited frigging with balances was enabled to cover up the fact that Horizon had serious bugs.
What I'm waiting to find out is why the Post Office managers were so desperate to protect the reputation of Horizon, that they were willing to risk being charged with perverting the course of justice.
They received bonuses for each successful prosecution and it would be career suicide for any of them to blow the whistle.
https://www.postofficehorizoninquiry.org.uk/evidence/pol0002...
An interesting incidence/problem report is in [2] at 1h:03m:00s, where an replication error in their Riposte message store system is described. After a defect counter hardware was replaced, it will fill its empty store with messages from other counters.
[1] "Technical Appendix to Judgment (No.6) “Horizon Issues”", https://www.judiciary.uk/wp-content/uploads/2022/07/bates-v-...
[2] "Alvin Finch - Richard Coleman - Day 54 AM (17 May 2023) - Post Office Horizon IT Inquiry", https://www.youtube.com/watch?v=H0_-ebCVU5k&t=3780
[3] Steven Murdoch: "What went wrong with Horizon: learning from the Post Office Trial", https://www.benthamsgaze.org/2021/07/15/what-went-wrong-with...
>Is Horizon still being used? >Yes. There have been several versions of Horizon since its introduction in 1999 and the current version of the system, introduced from 2017, was found in the group litigation to be robust, relative to comparable systems. But we are not complacent about that and are continuing to work, together with our postmasters, to make improvements. We will be moving away from Horizon to a new IT cloud-based system that will be more user-friendly and easier to adapt for new products and services. This is currently being developed with the involvement of our Postmasters.
via: https://corporate.postoffice.co.uk/en/horizon-scandal-pages/...
How could that happen outside of the Post Office? ICL/Fujitsu had remote access to all the remote boxes and could add/amend records without any audit trail and without knowledge of the sub postmasters, but I can't see how they could get money to go into their own pockets unless they entered into a deal with the sub postmaster operating the branch. Hypothetically, they could delete some transactions and then the sub postmaster would have more money in their till than accounted for and could split the proceeds. (I don't think this ever happened).
News like that make me very angry.
From the little I find to read, looks like it would take a 'basic' IT Auditor could in a few days figure out (and fail) basic ITGCs.
And it amazes me when I hear the term "DevOps" (cringe). But hey, I get it.. we need to make money, A LOT of money, and we need to make it FAST because of reasons (competition, greed, lack of talent, timing, etc.). Who has time and money to spend on Dev/Staging/Prod? Only stupid highly regulated environments need to be stupid enough to do this..
Going back to the story, THIS happens, people go to prison, people die. I have (in my IT Audit years) discovered/uncovered SO MANY tragic things.
And what scares me.. when I see YC 'offsprings' hiring and 99.9% of jobs are "stuff-building/money-making". Zero audit roles. And I am not angry because "damn - please hire me".. it's that how many of these places have an experience, IT savvy auditor? Not someone's cousin that is an accountant.. (no disrespect - it will take a CPA 10+ years to not be tricked).
And with that, I leave you with a friendly note on IT Auditors find and hire one with strong IT knowledge/background. They are worth their weight in Latinum!!
Indeed, they were the exact opposite: huge multinationals with presumably gigantic legal, infosec and HR policy departments who could probably jump through the various compliance hurdles required by this procurement process in their sleep.
By comparison the startups you're complaining about usually take years to reach sufficient maturity to take part in large public sector procurement processes like this.
Clearly in this case maturity and scale were not a bulwark against incompetence, opacity and mendacity. Indeed the opposite - as I type this the Post Office's lawyers Herbert Smith Freehills are making mealy mouthed justifications for witholding reams of technical evidence from the public enquiry for months. It's disgusting, and as someone running a small business and endlessly having to justify our commitment to information security, data privacy and transparency I find the hypocrisy infuriating.
On the world of RACI, the client is always the A. I don't expect the guy who gets paid to be honest. I expect the payer to do their checks.
And stuff like that could have been picked up but an ITGC audit, Project Audit (reqs), SOX, any type of break/smoke test.. and so on..
Somebody dropped the ball - hard. This could have been prevented and/or detected and/or corrected.
Having served as Internal Audit for many many years, I get angry because I/someone in my line of work should have caught this.
Now.. WTF was the internal audit of Royal Mail/Post Office? Why isn't the CAE brought in for questioning and what was the scope of their audits?
Yes, definitely NOT YC company. But I don't see any YC companies hiring auditors, only engineers ;)
As long as it is logged in some system, you’re golden. This is somewhat enforced through regular publicly disclosed audits. I suppose that’s what Ernst and Young did when they noted in an audit that Fujitsu has unrestricted access to modify accounts without the postmasters knowledge, and this poses risk.
What strikes me as bizarre is that no regulatory body took any action on that report. Maybe the report was private, I’m not sure - I don’t know how things work in the UK. In any case, this really takes your trust away from public institutions. You’re left to wonder, if this was possible, what else might still be possible?
One of the features that the customer desired was essentially an overriding, free-form, editor for the recorded information, as part of the "get all of the information for a case to be submitted to the court" bit.
We didn't do it, because the entire development team thought it was ridiculous, and the person requesting it retired before issue boiled over.
I understand this whole thing is not just a software development failure. But it also feels as if the Fujitsu developers had done a better job we wouldn't be having the whole debacle.
So what are the best practices with a system like this? Are there good and practical books for example on the topic?
"Any software with nf certification must not allow any concealment of data essential for VAT assessment. It must be equipped with a system for identification and tracing of all processes and information relating to collections.
Concerning the registration, the security is done by electronic signature. All information recorded in the cash register system cannot be modified or deleted. Errors and returns are automatically considered as new transactions.
As regards the retention of data, the minimum period imposed by the standard is six years. Furthermore, during a tax audit, an archiving system must facilitate the interventions of the tax authorities, by quickly restoring the data necessary for the calculation of VAT.
Note that archiving is systematic before any purging procedure. This rule ensures that the data is available at each intervention of the administration’s agents."
It's perhaps a bit more like signed git commits than blockchain; the requirement is that every transaction be signed and immutable. If "corrections" are required, the correction has to be appended to the log leaving the original in place. I believe there's also a requirement to feed all this receipt data to the tax authorities.
Jeez, as I'm betting almost all readers of HN know, if it's connected to the internet, it's not always secure. I'd suggest this is a blatant lie, the truth of which could have easily been verified by asking their in-house tech. After being told by Fujitsu about remote access possibilities, they didn't want believe them?
https://www.bbc.co.uk/sounds/brand/m000jf7j
Check that out to see how the U.K. establishment works.
If I were to hazard a guess, you haven't chosen to read the article, and have no motivation to actually substantiate your distrust for them past a vague "it's bullshit".
https://en.m.wikipedia.org/wiki/British_Post_Office_scandal#....
If the computer says "You took £x thousand and only banked £Y thousand", the first thing to do is audit the receipts and (unless there were fraudulent entries being made, while noting this was allegedly not possible) as the first rule of forensic accounting is 'follow the money'.
What's so bad here is that the executives of a well respected UK institution acted purely from a profit/reputational motive, and dug themselves deeper and deeper into the hole as a result.
More scarey part of this is that it's Minority Report - "Computer says you're guilty, end of story". Positive take would be the courts are less likely to believe the 'computers are infallible' line in the future.
Inquiry website https://www.postofficehorizoninquiry.org.uk
A new four-part dramatisation about the Horizon scandal aired from January 1st - 4th in the UK.
As a result, the scandal became a popular topic for discussion publically and in the media, as well as introducing the issue to a whole bunch of people who were unaware
I was a bit confused myself as to why this was getting coverage out of the blue until I learned about the ITV series yesterday
