Interactive Queries in Apache Kafka Streams (opens in new tab)

Here are my notes on getting Kafka running on a system with limited resources (it's pretty easy):

https://kafka.blog/posts/scaling-down-apache-kafka/

It actually runs well in such scenarios, it's just the default settings assume you're going to be running at large scale because that's what it's designed to do.

As other people say, Kafka is overkill for smaller use cases, but using it sets you up to scale if you need it and it gets you thinking in a good way architecturally.

Kafka is pretty unique in what it is and how it does things and the way it is designed means you can build all kinds of solutions on it. If what you want is a more traditional messaging system, Kafka is probably overkill for your scale.

If on the other hand, you want an event store, it might be worth using. You can build event stores on top of lots of different data stores but you'll have to manage the edge cases your self usually. Kafka has them built in.

That said, Kafka is pretty complex to operate. I'd only start using it when it became obvious it was the solution to your problem or there was some external factor pushing you to use it.

Kafka's pretty neat and 'everyone' uses it. But for small scale stuff, this is easy enough to do with a queue, a timer (maybe even cron) and a small SQL DB (SQLite, H2, Etc.). Msgs flow into queue. Repeating timer wakes up, runs script/program to: create temp table in DB, dump queue to temp table, query temp table, write query results wherever, drop temp table. You can add further features/complexity by adding sliding/tumbling window, coalescing queue or other parts into the mix, per your specific requirements.

If they're evenly distributed, 100K requests per day are barely 1/sec. It's peak throughput for which you should be planning. At 1/sec you can use just about anything.

That aside, if you're not sure if you should be using it I'm going to guess the answer is, "No you shouldn't". It's very cool tech but most likely overkill for whatever you're doing. Also, until recently securing it was a particular pain due to lack of TLS and authentication (I think this may have been resolved now).

It really depends on what you need to do. That's around 1 request per second. I think Kafka might be overkill, but if you plan on needing to increase capacity then it might be worth it. It might just be easier to use RabbitMQ or even Redis at that scale though.

How small where the box? normally kafka dosnt need much (4GB/8GB) and just uses the rest for page cache which can be sacrificed for performance.

Confluent relies on it for their non-Java clients (Python, Go, .Net at least).