Also, i've never heard the expression 4 nines before, anyone explain that for me?
When you start to define reliability as a specific layer of your infrastructure, you are setting yourself up for problems. You should be redundant at all layers, thereby reliable across your entire stack.
These things are usually specified in SLA (service level agreements) where if the provider exceeds the amount of downtime in the SLA means that you don't have to pay. If Amazon has a 4-nines SLA that means that you don't get any discount if they have 1 hour down per year.
Wouldn't it look the same from my end?
Then people realized, that was just not happening in the real world:
- Dual-channel interfaces would still fail (bad line card on the switch).
- Redundant Storage frames could suffer catastrophic outages.
- Datacenters with dozens of layers of redundancy could still fail due to fiber cuts, or faulty transformers.
- Unexpected crap happens, and you can never account for all of it.
So gradually, a movement began: virtualized computing and automated provisioning tools really gained traction and maturity. A shift towards more defensive programming emerged: we now don't trust the bottom layers to work. We plan ahead for them, we expect failure. Google, publishing their server specs and approach, really pushed this forward: people realized you could run serious infrastructure on mickey mouse hardware.
No longer is reliability needed at the component layer. We can move the responsibility much farther up the stack, and just added resources to cope.
I think many people (especially in the enterprise space, where change is slow, and tech is slow to adopt) have not adjusted for this change yet. They continue to buy uber-expensive server gear with redundant power supplies, mirrored disk (for redundancy, not performance reasons), ECC checking memory, and so on. They don't realize the $10,000 server they spent on a single server, could have bought three less redundant options that would have tripled their computing power. For the most part, they are right: you can't do these things until you fundamentally change your approach to software.
There is a balance, and I don't feel that throwing out ECC memory is necessarily the right choice for the majority of server applications. Low hardware cost needs to be achieved with an holistic approach - simply buying the cheapest possible components is unlikely to lead to the lowest cost unless your software and datacenter designs are really specialized for it.
DRAM errors are rather common in real systems[1]. There are two big hidden costs to this. The first one is the risk of silent data corruption. Unless you are willing to write your software in a way that is very careful to check all calculations, you run the risk of getting the wrong answer. The other hidden cost is operational: memory errors are often difficult to diagnose and you have to pay a highly skilled human to do it, as well as lose the use of the server while it is being done.
It may be that buying ECC RAM decreases the cost-per-page reliably served of your entire operation. If you are Google scale then that may not be the case, but for nearly all smaller operations it is.
'Enterprise' type hard drives are another potential long-term saving by spending more up front. Having a human replace a disk, and having the server down for the time it takes for a disk to be replaced, is expensive. If you have a large number of disks, especially if you are sensitive to small numbers of IO errors, it may be worth paying more up front.
Using an external view of Google's architecture to say 'cheap hardware is always good' is too simplistic. Yes, there is good evidence that single-host reliability mechanisms like RAID might give a poor ROI. Yes, redundancy is a powerful way to get reliability. But, before you take this to the extreme, you need to have carefully designed applications, carefully designed datacenters, and extremely low per-host operations costs (probably through aggressive automation). Unless you have these things, the optimal cost-per-request server design for your company may be very different from the ones Google, Facebook and Amazon chose.
[1] http://www.cs.toronto.edu/%7Ebianca/papers/sigmetrics09.pdf
http://techblog.netflix.com/2010/12/5-lessons-weve-learned-u...
The point seems to be to expect failure and make sure you're prepared but increasing the likely hood of failures for a slight decrease in cost is not a good way to go about it.
The question then becomes, when is it economically more viable to use multiple low(er) availability data centers as opposed to one hyper available one? (And since you should be designing for data center failover anyway, that is a fixed cost).
It's not, and that's why we don't have it.
And if you're large enough to require 5-nines (or even 4) of uptime, you should be looking at data-center level redundancy anyway.
That is why redundancy is so important. Instead of fighting an a battle which is exponentially increasing in difficulty, you chose to optimize the reliability of a single component. You give up optimizing each of the really complex subsystems (datacenters) at a certain level (3 or 4 nines) and focus on optimizing the reliability of a really simple component for detecting failures and directing traffic to the online datacenter.
Reliability engineers have known this for a really long time. If you can fit redundancy into your design, it is almost always a cheaper way to approach high reliability than optimizing the reliability of each subsystem.
