undefined | Better HN

0 pointsi_like_waiting4y ago0 comments

But can you stream data from lets say MS SQL directly into Postgres? Easiest way I found is Kafka, I would love some simple python script instead

0 comments

3 comments · 1 top-level

BiteCode_dev4y ago· 2 in thread

Stream? Unless you have really hight traffic, put that in a python script while loop that regularly check for new rows, and it will be fine.

If you want to get fancy, db now have pub/sub.

There are use case for stream replication, but you need way more data than 99% of biz have.

Spivak4y ago

You don’t want stream replication for scale, you want it for availability and durability. The number of times replication has saved my ass is too damn high.

And you want Kafka (or something like it) the moment you need two processes handling updates which again you probably want for availability reasons so a crash doesn’t stop the stream.

You also don’t catch deletes or updates with this setup. But there’s a million db to Kafka connectors that read the binlog or pretend to be a replica to handle this.

BiteCode_dev4y ago

Sure, if you need high availability + replication cross db, go for Kafka, but that's the thread point: it's not a common use case.

Most people don't need high availability, and most replication use the same db which comes with tooling for that.

j / k navigate · click thread line to collapse