undefined | Better HN

0 pointstveita7y ago0 comments

From what I can see this doesn't have built-in consumer balancing and offset storage, like Kafka does. It also lacks more exotic Kafka features like topic compaction and exactly-once processing.

In Kafka bulk reading is very cheap, the broker basically just calls sendfile() to send a file segment with compressed message chunks. On the other hand only the leader of a partition can serve requests, so you are often limited by bandwidth. It looks like LogDevice has to do a bit more work server side, but may be able to read from all servers with a replica.

Kafka stores more metadata in the record wrapper, like client and server timestamps and partition key.

There are client libraries for C++ and Python.

Operationally they look similar - both require a Zookeeper cluster, and both require assigning permanent ids to nodes.

It would be interesting to see some benchmarks comparing LogDevice with Kafka and Pulsar. That said, I suspect from the lack of buzz around Pulsar that Kafka isn't a performance bottleneck for most people using it.

0 comments

1 comments · 1 top-level

manigandham7y ago

Kafka is also very embedded everywhere now, with a big first-mover advantage. Pulsar already does everything Kafka does but also supports custom functions, per-message acknowledgements, and native cross-region replication.

Unfortunately it's hard to change something that already works. Most users don't hit the performance limits of their tools so they'll just continue Kafka if it's already running.

j / k navigate · click thread line to collapse