undefined | Better HN

0 pointssemiquaver2y ago0 comments

In practice the only option that I’ve seen work for very large teams and very large relational databases is online schema change tools like https://github.com/shayonj/pg-osc and https://github.com/github/gh-ost (the latter developed for GitHub’s monolith). It’s just too difficult to model what migrations will cause problems under load. Using a binlog/shadowtable approach for all migrations mostly obviates the problem.

0 comments

scient2y ago

Rails migrations really fail to be viable at scale. And its not really because of the migrations in Rails, its because changes in PostgreSQL get very very expensive. Things have gotten better in PG 15, but its still not quite there yet.

I've ran into an integer PK being almost exhausted in values twice now. In one of these cases I used pg-osc to rewrite a 8TB table on disk without downtime over a period of a week to fix the issue. In the other case it was about 6TB within 2 days. Its doable, but the amount of planning and time it takes is non-trivial.

I so far have no found any other viable solutions either. I keep hearing about logical replication but I haven't seen a great real-world solution at scale so far.

semiquaverOP2y ago

What has worked in my experience:

- decouple migration execution from code deployment. Require that all migrations are in totally separate PRs can be run well-in-advance of the code that depends on the db change being run

- maintain a stateful migration execution system using an online schema change system supervised by DBAs

- PRs adding migrations still need to be merged and deployed, but can only do so after the actual migration has run.

tianzhou2y ago

+1. We have been working on a schema migration tool called Bytebase for 3+ years. And we follow the similar principles.

CubsFan10602y ago

You can also migrate it using logical replication.

j / k navigate · click thread line to collapse