Designing Data Gateway for Cassandra in Open Source (opens in new tab)

Thanks, glad you enjoyed it!

Up until now we've primarily used our [Discord server](https://discord.gg/33mKDHHFUE) and Github issues for feedback. Starting with v2 we plan to start using Github Discussions as another avenue. Although we like Discord for free form discussion and quick questions it's not great for retention or searching for past solutions. Issues felt a bit constrained and tend to be directly tied to pull requests. Our hope is that with Discussions we'll be able to have more long form discussions that can be retained and made us of by others in the community. Also with Discussions we would like to start making use of an RFC process where anyone can propose designs, discuss, and then proceed with implementation.

Nice!

Stargate v1 is a monolith where everything runs within the same JVM which means service to persistence communication is just simple method calls. One of our main goals of Stargate v2 was to break up this monolith which meant that we could no longer communicate with just direct method calls and would instead need to bridge the gap between service and persistence with some other communication mechanism. We explored a few different options such as REST or CQL via drivers but ultimately landed on gRPC for a few reasons:

1. REST could be a viable option and has been used many times when splitting up a monolith but we didn't believe it could give us the performance we would need due to opening and closing connections

2. With REST we would essentially have to define our own schema definition language to describe the transport of messages between service and persistence which would also add the overhead of serializing and deserializing json payloads

3. CQL is a proven transport mechanism but requires the use of a driver which means whatever language used to implement the service must also have a workable driver (flexibility is a driving design principle for v2)

4. The session based nature of drivers would cause issues in a multi-user or multi-tenant environment

5. Although CQL fits most of our use cases it doesn't quite cover everything

So, yes performance was a factor in choosing gRPC but there was also the flexibility it gave us while being able to reuse CQL thus avoiding reinventing the wheel.

Thanks! The team put a lot of work into writing this and exploring many different options before arriving at the proposed architecture.

Initially we'll be using unary calls but I can see use cases where streaming would be useful. One might be for keeping a schema representation in sync between the services and coordinator (for those services that need to be schema aware). Another potential could be future implementations of reactive APIs and subscription or push models. Although in the near term we can leverage streaming for performance improvements by allowing the service to begin performing transformations before the entire response has been received.