Render Is Down | Better HN

Render Is Down | Better HN

26 comments

Their status page (https://status.render.com/) says:

Update - Many services recovered automatically, engineering is continuing to identify still affected services and mitigate issues as necessary.

Mar 26, 2024 - 16:28 UTC

Update - We are continuing to work on a fix for this issue.

Mar 26, 2024 - 16:20 UTC

Identified - We are encountering a broad range of outages across the Render Platform affecting connections and services. Engineering is working on mitigating the cause of these issues and narrowing down any non-affected components.

Mar 26, 2024 - 16:19 UTC

Whenever there's an incident like this, it seems like the status pages take a while to reflect the problem. Why is that? The part of the status page that can be manually updated is now correct, but most of the automated checks still show 100% uptime and only "degraded performance" in some cases despite being fully offline.

If I were implementing a status page, it might look something like a ping to some url from various regions. Assuming that's what these status pages are doing, why do they often say "All systems operational" until well into the downtime? Frustrating that I have to confirm on HN before I can know for sure something isn't just down for me.

anurag2y ago

(Render CEO) As an engineer myself, I understand your frustration. Updating the status page automatically is a hard problem when the systems in question are distributed and complex. Perhaps we could post a generic 'something is wrong' message using the automated checks we have in place, and go from there.

Sytten2y ago

For me it is the speed of update, I posted an outage on our own status a good 15m before they did on their status. I had to email them before I got an ACK.

anurag2y ago

(Render CEO) Things are recovering. We're still investigating, but the issue seems unrelated to Cloudflare. We'll post more updates on https://status.render.com.

Sytten2y ago

I expect to have a decent post-mortem for this major outage...

anurag2y ago

We will share a full incident report soon.

neb_bOP2y ago

Their status page says operational, but even their main website https://render.com is down

edit: website is back up

anurag2y ago

We're posting the latest updates at https://status.render.com. Databases are taking longer to recover but we're on it.

Sytten2y ago

Please give us an update if you have one, are we to expect a multi-hour recovery?

anurag2y ago

Recovery is well underway, but I don't have an ETA yet. Some regions like Ohio are recovering faster than others.

christophilus2y ago

It looks to me like it's related to Cloudflare: https://www.cloudflarestatus.com/

A bunch of my services (Postgres, webserver, workers) are still down. Some are up however in the same zone which is interesting.

anurag2y ago

We continue updating https://status.render.com, and your services should have come back online within the last hour. If not, please email me, and we'll figure it out.

Sytten2y ago

Our databases are up, but I am getting routing/dns issues leading to service failures

anurag2y ago

Things should be back up for you (Sytten). Email me if they aren't. We're still working on full recovery for other users.

abi2y ago

Our databases are all down :(

Our production Postgres instance is still down as of 16:46 London time.

We are 100% back up now

Sytten2y ago

All our services and databases are still down

anurag2y ago

We're on it. Some databases are taking longer to recover.

zegl2y ago

Does "some databases" mean all? I have three databases on Render, all are currently down.

j / k navigate · click thread line to collapse