That said, I don't think this would serve the intended purpose here. My specific use case was upgrading the postgres version I was using. To do that we introduced a follower, prevented writes to the primary (with the approach summarized in the post), upgraded the follower, and finally promoted the follower to be the new primary. Automatic failover to the follower during this process would likely confuse things.
[High Availability Postgres]: https://devcenter.heroku.com/articles/heroku-postgres-ha
1. Have app configured to connect to both main and replica.
2. Connect to the rails console and tell the app to stay in read only mode until told otherwise.
3. Disable replication
4. Upgrade main to new PostgreSQL version
5. Tell the app to move back to read-write mode
6. Re-create the replica
This flow helped us do hundreds of PostgreSQL major version upgrades in AWS RDS this quarter when we moved from PG 10 to 12.
And this is just a plus, using the gem during normal operations means that if a Redis or PostgreSQL main explodes for any reason the app keeps serving traffic, albeit in read-only.
> Automatic failover to the follower during this process would likely confuse things.
I believe here the problem is mostly naming. The gem "failover" to read-only mode to a replica, it doesn't promote replicas to main ever. Naming is hard.
I would want to test the end-user experience before using it in another app, especially one with very different usage patterns, but for this case it ended up being a great optimization between simplicity and robustness.
We also discussed the eventual solution I ended up with (which is summarized in this post) in this week's Bike Shed if you want to hear a bit more about it: https://www.bikeshed.fm/264.
i've done something analogous before and it seemed to me the exception framework was the right abstraction to use (though other opinions are welcome!). in this particular case, it seems like the main problem is that the trigger exceptions are not sufficiently granular/appropriate to describe the exact exception condition on which to trigger read-only mode:
* ActiveRecord::StatementInvalid
* PG::InsufficientPrivilege
rather than "DatabaseDownForMaintenance".
We retrofitted the above behavior to an ~400k line monolithic Python app with minimal changes a bit over 10 years ago (the web framework was thankfully agreeable), shed over half the load on the master day 1, and could happily bounce the master db or do schema updates at will without needing to touch the appservers. Plug it into your CI system and you can do both automatic deployments and database updates, often with unnoticeable downtime if you are clever about your database patches.
