undefined | Better HN

0 pointslichtenberger6y ago0 comments

I've already written with one of the core committers to BookKeeper and yes, I think I'll use it.

That said, I'm always looking for users and contributors to https://sirix.io/ or https://github.com/sirixdb/sirix. That would be super awesome, but I'm sure you have your own ideas already for using BookKeeper.

I'm currently not sure if I first want to build a frontend (I'm a backend engineer, but would like to learn some TypeScript along with using Vue.js and D3js), to interact with SirixDB and to build interactive visualizations to compare revisions of JSON- or XML-resources in SirixDB (stored in a binary format of course highly optimized for space-efficient snapshots).

I think as I'm lacking users it might be more useful, but for sure I'm at least as eager to put forth the idea of scalable SirixDB databases :-)

So, I'd like to use BookKeeper most probably (single writer, read your own write consistency, using synchronous -- for a quorum -- and asynchronous replication for the rest, exactly once semantics...). The thing I don't like is that we also need ZooKeeper, but yeah.

BTW: Why do you think BookKeeper is better than Kafka for this purpose? :-)

0 comments

4 comments · 2 top-level

eternalban6y ago· 2 in thread

This is a great review of Pulsar by Jack Vanlightly. If you haven't read it, I highly recommend it:

https://jack-vanlightly.com/blog/2018/10/2/understanding-how...

> The thing I don't like is that we also need BookKeeper, but yeah.

Word on the street is now you can swap the consensus module in ZooKeeper. [update: just re-checked Pulsar's dev channel. Apparently one can use etcd instead of zookeeper.]

> sirixdb

Sirix sounds interesting but is a bit close to home to a (currently dormant) temporal DB project of my own. But the real issue is I usually do significant OSS contribution if I'm on the bench and not working.

Wish you success in your effort!

atombender6y ago

Being able to use Etcd instead of ZooKeeper is huge. Etcd is considerably easier to set up and manage, in my experience; not needing to manage another JVM component is also a big win. Thanks for sharing!

eternalban6y ago

To be clear, it is BookKeeper that can alternatively use etcd not Pulsar.

https://bookkeeper.apache.org/bps/BP-28-etcd-as-metadata-sto...

https://github.com/apache/bookkeeper/commit/c7f1e70f437423ed...

1 more reply

rad_gruchalski6y ago

> BTW: Why do you think BookKeeper is better than Kafka for this purpose? :-)

One of the reasons is that your storage is not tied to ingest. With Kafka, your broker sits where the data sits. That has advantages and disadvantages. The disadvantages are that it's really difficult to do unlimited history topics / partitions in Kafka edit for significant volume topics. You can have only so much storage and only so many disks attached to your machine. It's not possible to add volumes at runtime. I could be wrong but I think Kafka still requires a broker restart after adding volumes to the server properties.

With BookKeeper, you just add nodes with more storage. BookKeeper tracks who was the owner of that partition at the time of write. It's possible to spread partitions across multiple machines.

j / k navigate · click thread line to collapse