At the time, I was working on a team pushing terabytes of logs through Kafka. A stateless broker backed by S3 would have saved us many headaches from hand-holding the Kafka cluster. It really struck me as a simple and elegant solution for our use case. Ever since then, I've been slowly tinkering and rewriting various implementations to better understand how to build a similar S3-backed broker.
yaks aims to be simple and only require a S3 bucket and Postgres database. Metadata is written to the db using an event sourcing model and reads to S3 are cached using the popular groupcache library.
Thanks for checking this out.
No comments yet.
