Persisting message offsets in DB has its own challenges. The apps become tightly coupled with a specific Kafka cluster and that makes it difficult to swap clusters in case of a failover event.
If you expect apps to persist offsets then it’s important to have a mechanism/process to safely reset the app state in DB when the stored offset doesn’t make sense.
Interesting, do you have any resources about how to best handle side effects with message queues (e.g. GCP PubSub)? Trying to find out if its worth the effort, or good practice to allow replay-ability (like from backup) of message and get back to the same state
Not sure about PubSub, but Kafka doesn’t store messages on disk forever by default. So hydrating the state from a message bus may not be a good idea depending on what you’re doing. I’d say DB is the way to go here.
Don’t have any resources though. I’m only speaking from experience.
