Running Apache Kafka at Scale (opens in new tab)

(engineering.linkedin.com)

49 pointsboredandroid11y ago9 comments

9 comments

6 comments · 2 top-level

robalfonso11y ago· 4 in thread

This is very timely. We are just starting to work on a new centralized logging system using logstache->kafka->elasticsearch->kibana, we may also use kibana for some other tasks as well. Anyone have any experience using that setup? Pros/Cons?

We used graylog for a while but ran into some issues with back pressure on elastic search (hence kafka).

toomuchtodo11y ago

Look into the new Graylog (v1.0). It was recently released, and they use Kafka internally now for buffering. We're sending thousands of msgs per second to Graylog without issue.

waitwaitwhay11y ago

According to their documentation, the data storage for log events is Elasticsearch and if the Elasticsearch data is lost then the logs are gone. Consideribg that ES is not a database and may lose data this sounds a bit scary to me.

3 more replies

weego11y ago

Having started off planning the exact same stack (but for business intelligence event tracking) we ended up dropping kafka for kinesis. It felt like until we got to the scale where it tips in the favour of self managed infrastructure (if ever) then there was no value and only probable pain in managing that part.

robalfonso11y ago

We already self manage our own infrastructure. So yeah, we are in exact opposite positions, thanks for the insight though!

felipesabino11y ago

I heard comparisons with other technologies before, like RabbitMQ [1] [2] but I would love to see an feature/performance comparison between kafka and other similar solutions, specially with cloud based services like the new kid on the 'Google Pub/Sub' [3]

[1] http://www.quora.com/RabbitMQ-vs-Kafka-which-one-for-durable...

[2] https://youtu.be/MA_3fPBFBtg?t=35m27s

[3] https://cloud.google.com/pubsub

j / k navigate · click thread line to collapse