Ask HN: How do you version control your event stream schema?

I'm in the same boat - looking to consolidate a few smaller data pipelines that have organically grown over the years into a single pipeline and taking the opportunity to try and get the org to rethink how we emit events in various parts of the stack. As teams operate fairly independently, we need something that enables a flexible data structure whilst still enabling the ability to join data on common well-defined fields.

The current path we are going down is looking at using a 'nested' JSON structure that enables sub-processes in the various systems to inherit values from the parent. Something similar to:

{ "type": "schemaX", "version": 1, "payload": { "k1": "v1", "type": "schemaY", "version": 3, "payload": { "kk1": "v1" } } }

The structure itself will be documented using JSONSchema and hopefully we will be able to verify the validity of events as they are processed, though this might be too expensive to do in high volume scenarios.

The goal is to then build a low latency router that takes in routing policies to forward subsets of events to further data pipelines. The policies themselves will be defined using some kind of DSL (possibly using JSONPath?).

As a whole, this seems to be a fairly common problem[0] that other companies are trying to solve, but a lot of the low level details are not spoken about. One thing that does seem to be a common component to a service like this is Apache Flink.[1]

[0] Netflix Keystone - https://www.youtube.com/watch?v=sPB8w-YXX1s [1] https://flink.apache.org/

We're using Avro + Kafka's Registry. We try to append to our messages to keep backwards compatibility.

That being said, there are things we intentionally do to ensure we can support this. e.g, Allow for nullable properties.

We pass in the message type, version, and additional headers. It's fairly common to create an envelope around the actual message. I would also suggest consider doing this for a few reasons. Skipping the obvious benefits, there's one powerful benefit in adding meta data to messages without having to alter the actual event. e.g, security tokens, time to live, and more.

I'm sure this might be also fairly commonly known, but I would highly recommend adding information like request correlation information.