Github is down (opens in new tab)

If only there was some way of using git locally without needing github!

Oh wait, this isn't svn.

This feels like a pretty standard pattern for a lot of services — fail, come back up on backup DB, fail again when backup proves to not be capable of handling the surge of load, then eventually come back up on the primary DB once people have gotten bored and stopped hitting 'refresh'.

Is that a function of not prewarming failover DBs, or is there something pathological about the primary-secondary pattern?

Frontpaging on HN will certainly help.

Question for the Github people: Why not keep serving non-stale cached data while your databases are down?

You can do this with proxies or by modifying your code to always serve out of cache, and the db updates the cache, so if the db is down, the cache is your temporary failover while you fail over to the secondary db. ('cache' is anything memcached-like that's separate from your db)

I've noticed that these are not immediately marked [dead] by the Hacker News admins like some other spammy topics, and I guess this is a sort of gray-area. On the one hand, it is in some sense a "new thing" and it affects the community; on the other hand it does not particularly gratify anyone's intellectual curiosity to be linked to that status page. The later explanation and diagnosis of what went wrong, if it's made public, might be interesting: just knowing "you can't use this now" isn't of that caliber though.

It's also puzzlingly dynamic: In other circumstances the "current status" pages linked during outages have not been blog-type, but have instead just literally reported the current status, leading to links which say "X is down" -- only for you to click it and see "X is functioning normally." This has already happened for the folks who linked the GitHub main page, which loads normally. (And won't it also prevent the same URLs from being submitted at a future date?)

Cool, an auto-refreshing status page that will mostly be looked at when the servers (possibly the network) are already stressed.

Now that is a good service status page. It made me go from "Github sucks" to "Go github." It is well pieced, informative, and upfront about the whole deal. Their auto refresher seals the whole deal. It shows they are confident on their skills to get the problem fixed. Bravo github. Well done. Now hurry up and finish so you can answer the support email I sent this morning. :)

I can see a github post-downtime analysis blog post coming up (I hope). :)

If only there was a better git hosting provider. Oh wait, I use bitbucket.org!

Yay!! We made it, frontpage on HN!