https://bookkeeper.apache.org/bps/BP-28-etcd-as-metadata-sto...
https://github.com/apache/bookkeeper/commit/c7f1e70f437423ed...
I have to admit that I'm not entirely clear on what Pulsar adds, given that BookKeeper implements the distributed log. Pulsar looks like it's a slightly higher-level abstraction that gives you a more traditional topic-based pub/sub mssaging API on top of BK, plus things like schemas and "functions"?
From what I can tell, BK supports log streaming just like Kafka, so you can use it to build a pub/sub system if you don't need all of Pulsar's features — right?
BK provides ledgers, DistributedLog provides log semantics on top of those ledgers. Pulsar builds a full blown messaging/streaming system on top of that.
Per this thread's OP, it seems to make more sense to use BK/DL directly if you want a WAL component to build a DB.
I'm looking at using BK for a WAL-like component for a document store. It does involve multiple (many!) consumers tailing the log, but unlike your typical Kafka setup, there's need for no partitioning.
I was unsure of whether BK provides an API that's nice to use for non-Java languages like Go and Rust, which is what we use. Apache projects written in Java (or Scala), particularly those in the Hadoop ecosystem, tend to be very JVM-centric, and often they're impossible to use from other languages because there's no non-Java API. Kafka is an outlier here, but it certainly started out with a Java-only client and it took years for non-Java clients to mature enough to be usable.
I thought I saw that BK had a REST API, but now I see that it's just an admin API. From what I can tell, it's exactly like every other Apache Java project — there's a big monolithic Java client that's tightly coupled with the server, and to write a client in any other language you'd have to reverse-engineer it and essentially port the whole thing, because it's a "fat client" — it looks like it talks directly to ZooKeeper and "bookies", and has a lot of logic built in. The Bookie protocol looks like it's binary and hand-coded (not gRPC or anything nice). So disappointing! /rant
Edit: Looks like the files I was looking at are generated from Protobuf. So, yay. Unfortunately, that's just the Bookie protocol. The main client is still fat and in Java.