They emailed me back, they said that the floppy thing makes a good headline but is really just the tip of the iceberg. It's really the whole system that's like this at every layer, it needs replacing they say.
If they had good schematics for all of the parts it might be possible to keep the the system running for a long time with a couple of smart EEs who are comfortable with the scope and soldering iron, but eventually they're going to run out of some obscure part and be up a creek.
Or maybe they could replace entire boards with home designed versions condense all of the old logic down to one chip and a handful of support components and start in-place upgrading without a total system revamp. Still an expensive process, and one that requires some hard to find engineers on staff, but theoretically spreads out the upgrade process over many years. It also loses out on functional improvement opportunities while your system is made up of a hodgepodge of old and new hardware.
The computers in question are, I believe, just thin clients.
Since I experienced 5.25 inch floppy disk era...and even the occasional bernoulli disk...we could simply say: the system had its run, replacement is reasonable. A lot of stuff had changed since then, and not just in storage media.
Until you have directly used "legacy" machine that are mission critical, and understand how even a tiny error could cause the failure of a business and all the jobs that go with it, you can't avoid underestimating the true scale and scope of the problem.
It may feel emotionally like this at every layer, but the layers that are not floppy discs are completely different from floppy discs.
Other than electrolytics caps (easy replacement items), old electronics is reliable.
Moving parts need service; that's life.
> needs replacement
Not believable without detailed justification.
In the present article, someone is literally quoted as everything working just fine.
Good luck getting the city to see it that way. If you'd like help, let me know.
this is the agency that brought you the central fucking subway, a $300M sewer project masquerading as some red paint on van ness, that picked the already-mostly-grade-separated M Ocean over the N Judah to subway-ify and that is incapable of flicking the traffic preemption switch on the rest of the T to the on position without a decades long pedestrian detection LIDAR project for the unique in the world needs of san francisco.
(they probably are…)
It sounds like they let this problem fester until it has reached this existential end. They clamor for new technology, which they get, then they use the most risk adverse management strategy and never upgrade or change it, until it reaches this problem state.
They either need significant third party help deploying and managing this system, or they should go back a few generations of technology and use the simplest possible system that meets their needs. Pen and paper should be considered if it can be made more efficient.
I guess the people who work on public transit aren't interested in having the best public transit system available. They're only interested in keeping it running for as long as possible with zero changes or responsibility.
This is why I'm strongly doubtful on public transportation in the US. Our bureaucracy can't handle it.
Let me guess, you also think public servants are lazy, overpaid and unskilled? Comments like this belie how many people form opinions of public agencies without having any real experience with government.
The problem is the public and the politicians - transit employees largely do the best they can with the resources and constraints they are given. In order to succeed transit needs municipalities and states to: pay more taxes; accept that transit won't generate a profit, but does generate non-monetary public value; and gather their resolve to deal with NIMBYs.
It was more than a few years ago, but I remember the posts on Slashdot where tech people were wringing their hands over the fact that election offices were not adopting touchscreen ballots fast enough. Some posters were also promoting things like internet voting. How did that work out?
Sometimes it actually is better to move slowly instead of adopting something new just because it is new. With old equipment and older technology, we know the failure modes, we have the procedures in place for addressing them. Whether it's floppy disks, paper ballots, (or human beings,) older does not necessarily mean worse.
They haven't shown any interest in updating the system. It works, they can get service, and get "new" replacements for things that go bad.
What they might not know though is that there is basically just one engineer we have (and probably the only one on Earth) who knows how to work on these things. He's getting old, and obviously none of the younger engineers really have an interest in learning ancient forgotten systems.
"Jeffrey Tumlin: "It's a question of risk. The system is currently working just fine but we know that with each increasing year risk of data degradation on the floppy disks increases and that at some point there will be a catastrophic failure."
This seems to imply they have been using the exact same disk for the past 20 years (absurd), they have absolutely no idea what is written on the disk and how it can be safely backed up or restored. This would be a problem regardless of the medium used.
Although I hold the line at using paper tape there is nothing wrong with using floppies other than it seems antiquated. It certainly is reliable and cheap. Maybe the only thing that needs replacing is the people running the Muni.
There’s also the chance that they take a disk that’s on the verge of failure, plug it into the system, and some corrupted commands get loaded into the system. That could easily result in a “catastrophic failure”.
Floppy disks are not reliable or cheap. They physically degrade over time, and at this point are nowhere near cheap for “new” disks.
> hey haven't shown any interest in updating the system.
So expensive to update that there's a calculated end-of-life to the system. They'd love to know about your engineer situation. That'd trigger plans put in place a while ago.
Your company could do a last big batch for the city and send the old guy out with a nice bonus.
At my workplace, we still ship devices with AMD 8086 processors to the military. And AMD still makes us 8086 processors on special order.
We got a few guys under 40 that work on that project once in a blue moon when a change is required.
This is the problem, not that they're using a floppy. This isn't web dev where you get to rewrite everything every 6 mos. Systems have to have decades long life cycles BUT THEY EVENTUALLY NEED TO BE REPLACED and that's not happening quickly enough here.
Edit: It was last updated in 1998, so it's due now not a decade from now.
* They must be experts, otherwise how could they get away with charging so much, right?
I rode Muni for 10 years almost every working day. Do you want to know what reduced congestion the most. Buying new muni cars because the old muni cars door mechanism had been repaired so much they did not open or close %50 of the time and people had to use other doors to get on and off a train on a daily basis.
What's really happening is going to be: Emerging new use-cases and rising costs from specialized vendors for replacement parts that match the existing system are driving a desire for the agency to replace the control system. The floppy system bit can 100% be replaced with a solution that isn't a floppy, but that wouldn't help them with the rest of what they want.
"because life cycles" is a lazy description.
> "We were the first agency in the U.S. to adopt this particular technology but it was from an era that computers didn't have a hard drive so you have to load the software from floppy disks on to the computer,"
In 1998, most personal computers already had hard drives [0]. From Wikipedia "The IBM PC/XT in 1983 included an internal 10 MB HDD, and soon thereafter, internal HDDs proliferated on personal computers."
The 3.5" floppy is from the mid 80's, again from Wiki [1] "In the early 1980s, many manufacturers introduced smaller floppy drives and media in various formats. A consortium of 21 companies eventually settled on a 3½-inch design..."
[0] https://en.wikipedia.org/wiki/Hard_disk_drive
[1] https://en.wikipedia.org/wiki/Floppy_disk
Why do I have to do this research instead of the "journalist"?
The other good thing about the floppy is it can't hold very much code. So the system has a tight upper bound on how bloated and complex it can get. Simpler systems are more maintainable.
These things seem like great assets for maintaining critical infrastructure.
EDIT: Another great thing is such a system will be stateless. No disks, no filesystems, no databases. Sign me up.
(These days, you could also replace the floppy with a USB drive: they make adapters/emulators.)
Edit: Even the news article hints at 5 inch floppy disks...which makes 1998 make absolutely no sense to me at least.
https://abc7news.com/san-francisco-train-system-has-been-run....
My family's first PC in 1987 had a hard disk. The Wikipedia quote they provide lines up with that, and provides a more authoritative point of when it was introduced.
And yeah, the "5 inch floppy" quote paired with a photo of a diskette. God only knows what actual hardware the system uses. But point being … the journalist doesn't seem to have found out.
Why do I have to do this research instead of the "journalist"?
Meanwhile in Germany, we have had moving blocks from the late 80s, based on a technology developed from the mid-60's and production-ready by the 70s [1]. Incredibly, the LZB technology never had an actual accident happen in all the time, only three "bare misses" (one of which was pretty spectacular in that it caused a train to pass over a switch rated for 80 km/h with around 185 km/h without derailing).
[1] https://de.wikipedia.org/wiki/Linienf%C3%B6rmige_Zugbeeinflu...
He pointed out glaring factual errors in the story, which should not have made it through any kind of editorial review. For example: 25 years ago pretty much every computer had a hard drive. And the disk depicted in the article is obviously a 3.5", not a "five-inch floppy."
Probably this. Probably they got the plans worked up in 1990, but didn't finish everything until 1998.
Probably cost like hell and was decade or two behind in technology, but I didn't buy it and definitely wouldn't have. Salespeople get people to do stupid stuff.
I'd guess these are used a lot in industrial settings where the code and task is actually very simple. Would run no problem with a $1 microcontroller and a hundred lines of C and with a lot less hassle. Likely vendors have gotten a reputation and keep on selling them with FUD.
What is it about public transit in the US that it is so... bad? Inadequate funding seems to be the easy one, but the MBTA (Boston) doesn't even handle the funds it has well. Yeah it needs more funding but there is also just a core issue to how it's run.
It is sad to see the state of public transit in this country, particularly in dense urban areas where we should be discouraging Car use as much as possible.
I am very curious what other countries are doing that we are not.
Now Governor Pritzer has authorized a new high speed rail commission to try and get this project going again.
In a nutshell .. Rail project between Iowa City and Chicago was fully funded as of 2016… all that actually happened is some rail road crossing were improved.. lines were inspected and no actual new lines constructed. Instead 1 out of the 2 billion was spend on “engineering” costs and compliance paperwork … which now has expired and need to be redone if the project is to be completed.
The amount of money spent on compliance paperwork and “engineering” is staggering. Many six figure salaries depend on slightly altering existing engineered projects to meet compliance requirements for projects that are never built. From waste treatment upgrades, water treatment, road improvement, traffic studies, on and on. The amount of money spent on services that are not finished or lead to an actual project is absolutely staggering … entire industries depend on this inefficiency and lobby effectively to keep things “obtuse”.
It just feels like the money is going to the wrong places like you said.
I would probably argue that we still need more funding even if we fixed how we used the money, but we need to fix how we use it first.
A lot of transit could be fixed by just taking a regular routine user, empowering them to become a dictator for a week and point out all the friction points they hit actually using the system. But then that would make the entire bureaucratic system that is the transit agency look like idiots who don't understand their own jobs, so it will unfortunately never be done.
And in some ways I don't blame people for doing this, because you need really supportive stakeholders to work in an iterative fashion. Otherwise you get a ton of nitpicking that amounts to a lot of "why didn't you guess 100 perfectly everything up front" and "how can we start if you can't give me a full plan and a firm price?" In a blame-hungry environment, waterfall is the safest choice for the people doing the project.
When the Democratic National Convention came to Boston (2004), the subway got a lot of nice "which way do I go? what are these stairs? if not these, then where?" temporary orientation signage. It lingered afterward, with some becoming permanent signage, and others puzzlingly not. Such an event involves massive interdepartmental communication, tight timelines, altered incentives and constraints, and additional resources. Some of the altered communication channels are said to have persisted. I might be interesting to look at events where things are "shaken up", to better understand the steady-state tangle.
It is now a fairly regular occurrence to have a train show up that the screen had no reference too existing a minute or two before.
But yeah it feels like they were not designed or setup by people that actually use the trains and were setup by a comity thinking they know best.
Maybe that is the big difference, in other countries the people using it are also the ones managing it?
It works because the companies own land and facilities around every station. Grocery stores, office buildings, shopping centers (stores at many stations), apartments, etc. This creates a virtuous cycle where the more riders the more people use their other services and visa-versa.
As for semi-competitive, at least in Tokyo, Osaka, and Kyoto there are enough lines that you often have a choice. For example Tokyo (area) to Yokohama there's JR, Keikyu, Tokyu, (3 different companies). They have slightly different routes so if you're closer to one you might take one or the other but they do advertise trying to get you ride their's over the other's. To Hanada there's Keikyu and the Tokyo Monorail (it's own company). To Narita there's the Narita Express (JR) and Keisei (a different company) as well as local lines from both. Same in Kyoto. You can go to Kyoto to Nara via JR or via Keihan. You can go Kyoto to Osaka via JR or Hanshin.
So, if one company offers easy pay methods and another doesn't it quickly gets the reputation and "a crappy old line" (who wants to live there, open an office there, etc...)
- chronic underfunding - we all know how problems build up when maintenance is deferred
- waterfall planning - Mary Poppendieck has a nice talk on how much trouble this causes: https://www.infoq.com/presentations/tyranny-of-plan/
- political point-scoring - blame-oriented cultures discourage experimentation and incremental improvement
- political polarization - one-party areas can more easily slide into cronyism, and fighting between parties makes it hard to compromise even on things like fixing infrastructure
- classism - in a lot of places, transit is for the poors
- racism - many don't want transit bringing Those People around
- manager culture, not engineer culture - as we see with Boeing, standard MBA thinking doesn't work well for long-term safety and reliability; the focus on short term metrics, mostly financial ones, leads to underinvestment and decay of infrastructure
One option is ranked choice voting.
other cities (in the east, with the best public transit) don't let unions grab the entire city by the balls
government employees in other cities aren't as hellbent on extracting their pound of flesh from the taxpayer
I live in New Zealand public transit in NZ used to be incredibly bad during the heights of private operation in the 1990's but has improved heaps over the last 10-20 years as local/central govt have progressively taken back control over more aspects of public transit. However public agencies are still not allowed to directly operate bus or train services which still sees private companies failing to provide good service with no consquences and public agencies unable to take over to provide the service improvements that are needed. It is interesting to compare New Zealand in the 1950's had one of the highest levels of public transit usage per capita under public ownership of buses and railways. This dropped down to embrassingly low levels after out-sourcing to private entities to a low in the 1990's.
It has only been under government leadership that we have seen a revival to the current levels (e.g. doubling to 100+ million trips per annum in our biggest city from well below 50 million in the mid-1990's) due to local government coordinating public transit systems (and contracting private entities to operate to the timetables set by public authorities) that meets the needs of the users and not just private entities making a profit. Unions have been vocal too for improving systems not only for transit employees but also recognising that what's best for transit users also benefit transit employees.
YMMV and possibly depends on the political environment in each country.
Massive money and entrenched behind preserving a car-based life.
Car companies, auto workers unions, railroads can siphon off funds from promised improvements to cover deferred maintenance and give themselves bonuses for being so clever… Hell, AAA which one might expect to spend its income on serving its members instead diverts money to proactively lobbying against improvements in public transit in an ongoing display of cynical self-preservation
It’s basically hopeless.
If you want to help, vote. Or even run for something on a pro-transit, YIMBY platform.
I was expecting it to be some kind of utopia, with futuristic technology on every corner.
In reality, it is roughly the same as anywhere else in America. A bit of a let down. The innovation does not take place in the infrastructure.
Beautiful place, though!
Upgrading those systems is expensive and difficult, so it's not surprising old prices of equipment hung around until the end of their useful life. Especially since BART is going through a massive budget crisis at the moment (and this isn't the only one in its history).
*: Many people don't. Self-driving cars are better than human-driven cars, but affordable and reliable trains are even better.
[1] https://www.sfchronicle.com/sf/article/S-F-voters-narrowly-r...
Sounds like they are using the floppy as an excuse to push for an upgrade that has nothing to do with the floppy drives.
I’ve ridden Muni in SF for years, drivers control most of the decision making.
There’s a handful of single track tunnels/sensors/necessary software-based coordination, but as another commenter pointed out, the doors cause more issues than signal problems.
yes, I know safety-critical systems are different. I also expect that the floppy-disk issue is just the easiest problem to explain of a long chain of terrible legacy lock-ins. However, if they're literally holding their breath every morning when it's time to IPL the system off a floppy...that part sounds solvable.
That's not the 3.5" floppy disk in the video. This is the old floppy disks[1]
1. https://www.digitaltreasures.ca/img/level2_floppy_525.jpg
The OG floppy disks were 8 inches, though, 5 inch floppies were the small ones (hence why 3.5" disks were "microfloppies"):
https://images.techhive.com/images/article/2015/12/floppy_di...
> "We were the first agency in the U.S. to adopt this particular technology but it was from an era that computers didn't have a hard drive so you have to load the software from floppy disks on to the computer," said Mariana Maguire, SFMTA Train Control Project.
> SFMTA's train control system relies every morning on 5 inch floppy disks.
This doesn’t make any sense. 5.25-inch floppy disks and no hard disks was not “cutting edge technology” in 1998. It arguably wasn’t even “cutting edge technology” in 1988
Perhaps ATC was layered on top of the original signalling system, which is the part that uses the floppy disks?
The technology works, there is a replacement outlined, there is no shortage of floppy disks - even 5 1/4 ones.
- "It's a question of risk. The system is currently working just fine but we know that with each increasing year risk of data degradation on the floppy disks increases and that at some point there will be a catastrophic failure."
"Data degradation on the floppies" should not, by itself, cause any sort of "catastrophic failure" in a sound system. If one software disk fails, you should have five identical copies in a drawer, five more in a different room, ten more off-site, plus a disk image on cloud storage that you can write fresh floppy disks from.
I mean, it's probably a good idea to change the storage medium too—but that's not the root problem.
"The San Francisco Municipal Transportation Agency (SFMTA), which runs the city's Muni Metro light rail, claims to be the first US agency to adopt floppy disks. But today, the SFMTA is eager to abandon its reliance on 5¼-inch floppy disks—just give it about six more years and a few hundred more million dollars."
yea for Unicode Character “¼”
It's unlikely a system deployed in 1998 would be using three separate 5¼" floppies. I'd believe a single 5¼" floppy, but not three of them.
It's much more likely the system boots off uses a single 3.5" floppy, and the reporter forgot to read out the "point"
This naivety is not Katie's fault. We who work in tech are to blame for constantly pushing our half-baked experimental garbage as if it was "engineering" on par with civil or aeronautical systems. We can't blame people for occasionally believing the lies.
The tech-washed version of this quote might go something like "wow I thought everything was moving to the multi-cloud serverless kooberneetus now, why is it still running on a computer?"
> It is easy to run a secure computer system. You merely have to disconnect all dial-up connections and permit only direct-wired terminals, put the machine and its terminals in a shielded room, and post a guard at the door[1].
This is the kind of thing I want running the trains. Give it ECC RAM too, please.
[1] https://en.m.wikipedia.org/wiki/Robert_Morris_(cryptographer...
Yeah, that's called a cold boot. Moving to not-floppies doesn't mean you can avoid this. Clearly it's off of floppies instead of ROM so you can more easily update the software, but I am wondering how often that ended up happening. Maybe EEPROMs would have been better.
> Luz Pena: "How dire is it to change the system to upgrade it from a floppy disk to a wireless system?"
I agree that floppies aren't the peak of reliability, but "a wireless system" also sounds like a disaster. I don't want critical urban infrastructure running on extremely hackable OTA updates. For the love of god, SF, you can avoid pretty much all potential cybersecurity problems by just not putting your trains online.
I feel like neither the interviewer nor the interviewee really had the technical expertise to speak to this. This entire piece is just, "oooooo, floppies are old. Old bad! Why not new yet? New good."
The retro community has proven reliably that a simple Raspberry PI can easily bit-bang floppy controllers. We have myriad floppy-to-SD card adapters.
Surely a plug-and-play solution that removes the area of most concern (reliance on the media itself) should be easily achievable in a few months?
I am sure they have copies of the floppy disk, and likely have images on other machines that can be used to create a new floppy even if all the copies are lost.
I feel like the SFMTA is mainly using the existence of the floppy disks as a marketing point for their desire to get funding to update the system. It is something that the average person will be able to look at and know it is out of date in a visceral way.
The reporter seems to read this as if we are one bad floppy away from failure, but that is not the actual case.
The floppy disk angle is there to make a good headline. The article makes it clear this is a much bigger project than just replacing the floppy disks.
"The detail[ed] project schedule will be finalized once we have a contractor onboard. This is effectively a multi-phase decade long project that starts with pieces of market street subway and pieces in the surface. Ultimately our goal is to have a single train control system for the entire rail system," said Tumlin.So SF always does the big thing because otherwise the overhead dominates.
Lawyers will probably spend a year wrangling liability concerns.
I could probably replace a floppy drive on any system within a couple of weeks. I would not accept legal or financial liability for any such solution without an extremely thorough and slow review of all aspects of the project. edit: and an astronomically high paycheck.
> Much more critical than the dated use of floppy disks is the system's loop cable, which transmits data between the central servers and the trains and, according to Roccaforte, "has less bandwidth than an old AOL dial-up modem."
Also, they're already late as hell, what's 5 more years? 5 1/4 isn't going to become more obsolete than it is at this point.
>The transportation body says the train control system was built to last for just 20 to 25 years, meaning it surpassed its expected lifetime in 2023. In 2020, the Muni Reliability Working Group, said to be composed of local and national transit experts, recommended replacing the transit control system within five to seven years. [...] "We have to maintain programmers who are experts in the programming languages of the '90s in order to keep running our current system, so we have a technical debt that stretches back many decades," Tumlin told San Francisco's KQED in February 2023.
Clearly no one wants to do that.
I would guess higher ups don't want to spend money on a band-aid without really fixing the problem.
Do they know that floppy can be backed up?
This kind of knowledge and experience doesn't come cheap, and we all know how much US city governments pay. I was at one point, very briefly, motivated to apply to a city job before I learned the pay is approximately 1/3 to 1/4 of what the private sector pays. The USDS routinely posts on "who wants to get hired" and the comments on that and other thread also mention a 66% to 75% salary reduction from baseline.
This is even before we get to legal liability and political risk shifting that can happen when there is a fully responsible contracted company involved.
https://www.theverge.com/2019/10/25/20931800/usa-nuclear-8-i...
And in the video, she says “on 3x 5 inch floppy disks like this one <shows a 3.5 inch floppy>”
Woulda been a nice time to clean up some of this technical debt!
Or how about the SF Emergency Sirens, taken offline in late 2019 for a "2 year" upgrade plan that officeholders implied was already in place?
In August 2023, with no progress whatsover, with the Maui fire disaster fresh on their minds, Mayor Breed & Supervisors President Peskin touted they'd finally funded a plan to return them to service soon: https://www.sf.gov/news/mayor-breed-and-board-president-pesk...
In that same August 2023 timeframe, Peskin said the plan would bring this "need to have" system "up and running" & to "state of the art" by end of 2024, for $5.5M: https://www.cbsnews.com/sanfrancisco/news/san-francisco-city...
Of course, this was just more blatant self-exonerating bullshit from our local political machines immune from any real accountability for incompetence in basic public functions.
A mere 6 months later in February 2024, nothing's been started, Peskin admitted "we don't even have a plan", the department is still waiting until "funding is identified", and the cost estimate has ballooned to $20.5m: https://abc7news.com/san-francisco-sirens-emergency-911-aler...
That works out to $170K+ for each of 119 units – units that each could probably just be a weatherized consumer-grade handheld device with multiple mobile/packet/sat radios, & a simple authenticated-playback app, mounted on existing poles that presumably already have power and even loudspeakers.
More history, through January 2022, via JWZ's blog:
https://www.jwz.org/blog/2022/01/the-reason-the-tuesday-noon...
The key words in the sentence are: "it's working just fine".
Data degradation of floppy discs is easy: just copy them to fresh ones, and verify that you have a good copy. The images should be safely backed up so they can be regenerated. (Plus there are emulators; a topic covered elsewhere under this submission.)
I mean, are they really using the same 30 year old floppy discs over and over again until they degrade?
"it was from an era that computers didn't have a hard drive"
Absolute BS. Pretty much every computer had a hard drive in 1998, and most had CD-ROM.
Then they referred to the 3.5" disk as a "5-inch floppy."
<sigh>