undefined | Better HN

0 pointspixel_popping1d ago0 comments

I feel the pain OP.

Over the last decade, I've ran hundreds of servers if not thousands, and I entirely stopped using hard drives, now it's solely SSD/NVMe where the failure rate in practice is incredibly lower, I've had my fair share of middle-night runs because websites are offline or whatever to end-up in a hard drive diagnosis circus.

Imo, the peace of mind you get worth the cost, it also allows you to rethink development entirely, typical example would be that suddenly, copying all node_modules or rust deps is a great idea with 10Gbit/s bandwidth and fast drives (yes, I expect people to shit on me for saying this, please give me the counterarguments if you downvote me), many things change if you have a higher base performance assumption, storage is relatively cheap as well. I would never advise anyone that wants to run continuously in prod with low friction to get servers with HDD.

I get that for some use cases it's not possible, but for large majority of use cases, it's clearly not HDD that is the cost burden. $50 servers gets you TBs of SSD, of course don't go with VPS or "Cloud" if you intend to change your development based on new performance assumptions, it blows my mind the numbers of people paying thousand of dollars just to handle what, 100K visitors a day? That fits on a $100 server and a bunch of Kimsufi hosted across the world as a CDN.

People are overcomplicating infrastructure, big time (which leads to more problems, higher maintenance, security issues and so on).

0 comments

toast01d ago

> Over the last decade, I've ran hundreds of servers if not thousands, and I entirely stopped using hard drives, now it's solely SSD/NVMe where the failure rate in practice is incredibly lower, I've had my fair share of middle-night runs because websites are offline or whatever to end-up in a hard drive diagnosis circus.

My experience is that (most) spinners give off reliable pre-failure indicators (if you take the time to look/script looking), but SSDs fail by disappearing from the bus. The SSDs do fail much less often, but they still fail from time to time and recovery is harder.

Either way, if your data is important to you/your customers, you really need a backup/recovery plan.

I dunno about recent pricing, but not so long ago, it felt like spinners had a pretty high price floor and SSDs didn't... If you don't need a lot of space, you could find a small SSD that was still around the same $/GB as a medium sized SSD, but for spinners, there's a floor in dollars and space. So if you don't need a lot of space, you save money with an SSD and get better perf for free... If you need a lot of space and not a lot of perf, big spinners are more attainable than big SSDs.

ryandrake1d ago

> My experience is that (most) spinners give off reliable pre-failure indicators (if you take the time to look/script looking), but SSDs fail by disappearing from the bus. The SSDs do fail much less often, but they still fail from time to time and recovery is harder.

I'm not a pro, just a smalltime dork with a homelab. I use cheap WD HDDs on my NAS system connected to an LSI hardware RAID controller. I'll boast that I have a 100% record so far of preventing downtime and data loss by simply listening for the controller's audible alarm and swapping drives right away (I keep brand new spares). I also have offline backups, but have so far never needed them. Not sure how this would change if I moved to SSDs.

adastra2223h ago

I have a homelab with 24 disks (2x 12-drive raidz3 pools). In the past 15 years of operation I have replaced many drives. Usually my experience matches yours, but there is one time I got a simultaneous 3-disk failure. One disk failed, then when I replaced it and did a scrub, two more from the same pool failed in quick succession. I had to scramble to get more spare drives, and didn't sleep much while I waited for the pools to finish the scrub.

Coordinated failure is a thing, unfortunately. Drives bought together tend to fail together.

Felger1d ago

Well, SAS disk tend to go in failed state immediately or very quickly, most of the time without going first through the warning state.

SATA disk are indeed generally more predictable failure-wise. Most issues are related to a failing head stack assembly. Rarely platter demagnetization for some disks (Toshiba laptop).

Other failure issues are usually related to a friggin' manufactured firmware issue from Dell, HP or Lenovo corp.

pixel_poppingOP1d ago

Agree with the diagnostic part.

> Either way, if your data is important to you/your customers, you really need a backup/recovery plan.

You'd be surprised at how many devs/companies walk on eggshells all the time (praying that the fatal moment never arrive) because they aren't "brave" enough to do a proper backup system, which is often few minutes/hours of setup only.

Retr0id1d ago

It is quite remarkable how quickly a modern SSD can scan over TBs of data, I'm less afraid of O(n) queries than I used to be.

j / k navigate · click thread line to collapse