Ideally something like Pulsars API (which more resembles something like GCP Pub/Sub) would be the de facto as it is capable of handling both cases seamlessly.
There was Kafka-on-Pulsar until recently until the developers essentially made it non-OSS which is pretty unfortunate for Pulsar adoption which is already low.
That said, it's basically Kafka with the topic-partition persistence swapped out. So the libraries etc are just the standard Kafka libraries because the frontend listener is unmodified.
I am open to new things.. with at least a tad of supporting data attached :D Ymmv.
Most of the Kafka deployments I've seen in recent years have been using cloud versions e.g. AWS Managed Kafka.
So it's great that you save money on compute but it does mean you are now operating it yourself which then costs money.
However, AutoMQ has adopted a cloud-native architecture, offloading storage to EBS and S3(https://docs.automq.com/docs/automq-s3kafka/Q8fNwoCDGiBOV6k8...), eliminating the need for replication and rendering the Broker stateless, which simplifies operations significantly.
