undefined | Better HN

0 pointsmanigandham9y ago0 comments

It's not a message queue, it's a logging system. Queues are meant for ephemeral messages that expire once consumed. Kafka is immutable log storage that can be read as many times as necessary by consumers.

Biggest competitors would be AWS Kinesis, Azure EventHubs and Google PubSub.

0 comments

saryant9y ago

The biggest difference, IMO, is that Kafka is typically used when a message will be consumed by multiple consumers, whereas RabbitMQ or SQS generally send a message to a single consumer.

We use it to ingest ~40mb/s and fan it out to a number of consuming applications.

I'll also add that if you put some thought behind your topic replication and partitioning you can build some incredibly resilient applications. Also that "immutable" isn't necessarily true, it's common for Kafka topics to roll off messages based on time or size. (That's just to clarify for those not familiar with Kafka. I realize that you mean messages are not deleted or modified once written to a topic, other than by topic retention settings)

manigandhamOP9y ago

It's very easy to setup multiple consumers with RabbitMQ topics though and SQS is a very basic queueing system that just doesn't support much.

To me, Kafka is just meant for much larger magnitudes of scale and persistence (of the entire log of messages for however long you need) as a core feature.

Google's PubSub is still the best blend of traditional queue semantics with Kafka scale and persistence though.

asymmetric9y ago

It's incorrect to say that RMQ sends message to consumers. What it does is it routes messages to queues/exchanges. It's then entirely up to you to decide how many consumers will effectively consume them, I.e. You can have as many consumers as you wish.

dkersten9y ago

messages are not deleted or modified once written to a topic, other than by topic retention settings

Except if you have log compaction turned on, I guess.

A key "selling point" of Kafka for me is that each consumer can decide from when they wish to receive messages. That is, you can replay the messages.

j / k navigate · click thread line to collapse