undefined | Better HN

0 pointsChuckMcM10y ago0 comments

AFR buddy, AFR. AFR is the annual failure rate, its typically 2 - 5% of the population per year. 4 drives you don't have a large enough set to see this in action, just every day you're in danger of losing a drive by a small statistical amount.

In the Blekko cluster we have just under 10,000 drives. We have a two 20 drive 'boxes' (40 drives) from Western Digital, as drives fail we pull replacements from the 'new/refurbished' box, and we put the dead one in the outgoing box. When we get up to 20 we RMA them in bulk, 20 go out, 20 more come in. That becomes the new 'new/refurbished' box.

It really isn't SAN vs non-SAN it is all statistics.

That said, if you're running your ReadyNAS with raid 10 (mirrored drives in a RAID 0 config) you may find some unpleasantness when a drive does fail. Statistically you have a 1/10 chance of not being able to re-silver the mirror for a 5900 RPM desktop SATA drive. That gets a bit painful.

0 comments

7 comments · 2 top-level

codezero10y ago· 4 in thread

Just out of curiosity, has anyone done any research into determining whether a drive which fails after X days/years has some properties in the first Y days that could be a signal for future failure?

Smerity10y ago

I'm certain there are more recent statistics but Google's "Failure Trends in a Large Disk Drive Population"[1] (2007) is a good start:

"In addition to presenting failure statistics, we analyze the correlation between failures and several parameters generally believed to impact longevity."

There is also a more recent open source dataset from Backblaze[2] that includes:

"Every day, the software that runs the Backblaze data center takes a snapshot of the state of every drive in the data center, including the drive’s serial number, model number, and all of its SMART data"

which forms the basis of an article correlating SMART data with drive failures at Backblaze[3].

[1]: http://static.googleusercontent.com/media/research.google.co...

[2]: https://www.backblaze.com/blog/hard-drive-data-feb2015/

[3]: https://www.backblaze.com/blog/hard-drive-smart-stats/

maccam9410y ago

The TL;DR answer is yes, there are some hard drive SMART values that can indicate failure is likely, but they vary by model and don't necessarily show before failure.

codezero10y ago

yeah, I was wondering if there were measurables that could be correlated with failure before SMART kicked in, even if they were something like date of year, or location of manufacture, or shipping route they took. :P

1 more reply

jks10y ago

Yes: https://www.backblaze.com/blog/hard-drive-smart-stats/

nadams10y ago· 1 in thread

> just every day you're in danger of losing a drive by a small statistical amount.

Which is why for important data I always use some sort of RAID (or cloud syncing). If I lose a drive I won't lose all my data (presumably though if I bought both drives at the same time there is a chance that both could fail at the same or close to the same time).

> That said, if you're running your ReadyNAS with raid 10 (mirrored drives in a RAID 0 config)

ReadyNAS and other products use a special type of RAID that is actually kind of clever (that will use all space regardless of disk size). I know many would criticize the special RAID but I've had more issues with Linux software RAID than I have had with ReadyNAS (nothing related to data loss). But suffice it to say if 1 drive fails then in theory the data should still be ok (at least that is what their claim to fame is).

My personal opinion - I feel like the ReadyNAS will die before the drives. I'm not saying they couldn't or won't die - but I feel like that would be highly unlikely and even more unlikely for more than 1 to fail at once.

Maybe I'm just paranoid - but when I worked on my last SAN it felt like the company used the cheapest possible drives with a short MTBF for the simple reason that my employer would have to keep using them and keep paying for their warranty service (this wasn't a big name like Dell).

Veratyr10y ago

> Which is why for important data I always use some sort of RAID (or cloud syncing). If I lose a drive I won't lose all my data

Just a friendly reminder that RAID != backup. There are numerous data loss cases that RAID does not deal with.

Personally I use striped ZFS with important volumes periodically snapshotted, replicated to external (and encrypted) disks and then stored offsite (cycle through a couple of sets of disks). Most important data is also periodically synced to cloud storage (as well as offsite disk).

This accounts for:

- Single disk failure (striping)

- Bitrot (ZFS scrubbing can reveal bitrot on disk and correct it from parity)

- Human error (snapshots)

- Catastrophic damage to home NAS (offsite backups)

RAID alone (depending on the particular implementation) will generally not deal with the 3 latter failure cases.

j / k navigate · click thread line to collapse