All HTTP-based services unresponsive (opens in new tab)

I think hosting the repository is one thing.

But integrating those two repositories with all the automated workflows is quite a headache. Jenkins won't automatically switch over to another git backend if one fails. Other automated tools like code review are mostly relying on a centralized repostiory. I don't see any simple solution to solve these issues. Bitbucket, GitHub, GitLab don't have any easy fallback solutions when you relying core workflows on their services.

Self hosting is maybe one solution (we do this with bitbucket), but this requires major administrative effort to keep it running reliably (always available, no data losses on hardware/software failures).

That's what I wanted to do. Thank you for the guide !

There are ways to engineer on AWS to basically never go down barring massive, systemic, cross-region failure.

Granted, there's no economical way you could self-host a Gittea instance to avoid this.

> Github can probably do better at SRE than I can with a self-hosted server

GitHub maybe but for BitBucket, I'm not sure at all that it's true.

You can self host JIRA.

That's something that is bound to happen more often. I cannot let that affect me. I cannot let their problems become mine.

My self hosted repos won't get DDoS. BitBucket is a large target and that's a problem for me.

StatusGator is neat, thanks for linking that. Do you have graphs anywhere to track the number of outages/problems over time? It would be nice to see if there's been an uptick in problems generally, across certain services, etc.

You would think Amazon would have sufficient backup power to make that a non-issue, as would any significant third party data centers involved in routing data. However, a downed data line might make sense as a root cause.

Amazon's description of S3: "Data is automatically distributed across a minimum of three physical facilities that are geographically separated within an AWS Region".

What are people doing wrong with how they use S3? Until AWS provides a cross-region S3 that is master-master or self-healing, the suggestion that people are using AWS improperly seems incorrect.

Because it was the first AWS region, it is known to have the oldest hardware, the most dirty hacks, and the most outages.

us-east-1 is the oldest region with the oldest hardware which is probably why it has more issues than others

I think that’s the problem with us-east-1: it’s the guinea pig region, so of course it’s going to have the most problems. If you don’t need cutting edge features, you shouldn’t be there.

us-east-1 feels like the region where Amazon rolls things out first. This has two primary effects:

* sometimes new things break in unexpected ways

* sometimes things get changed for the Rev. B and those revisions don’t get done in Virginia because they’ve already completed the Rev. A rollout.

Also, there’s a secondary effect: because it’s the “default” region, it has a LOT more tenants, which means it probably has scaling and HA problems that none of the other regions do.

Yea, this makes no sense to me. I have a pretty heavy workload in AWS (us-east-1), don't use DC AT ALL and nothing is down for me today (except Atlassian Jira/Confluence Cloud), we self host BB. Why their 'cloud' based application relies on DC is very odd.

Hello, I'm Irena from the Networking Engineering team at Atlassian. I have been directly involved with this incident and wanted to provide some answers to the questions. We’ve built our architectures based on the AWS Direct Connect service because it’s the most reliable and scalable solution based on our customer and network needs. The AWS Direct Connect service we use in the US East Region has multiple redundant links (4x 10Gbps) optimized for data throughput requirements and availability, and to our knowledge the AWS Direct Connect transit facilities have power backups that would help contribute to its reliability. But, as we saw from today’s event, something still failed.

I should note that we have both publicly and privately reachable resources in AWS. The publicly reachable resources have fail-overs built in for situations like these (it happens automatically), but the private reachable resources with our architecture depend solely on AWS Direct Connect. For example, our Bitbucket failure today was due to the fact that we rely on AWS Direct Connect to link between the Bitbucket Cloud components that we host in our data centers and others that we host on AWS. Bitbucket could continue connecting to services in our own data centers and the public Internet/AWS, but could not talk to the privately reachable resources in the Atlassian infrastructure hosted on AWS.

We understand the importance and the impact for our customers, and dedicated several teams to this issue as soon as it was reported. AWS has resolved the issue, but we will look into ways to help prevent and better mitigate these types of issues in the future as part of our incident review and improvement processes.

Keep trying, it eventually goes through.

Hmm, I just pushed some commits over SSH. Seems to work fine on my end so far.