Democratisation of complex machinery like the raft and consensus algorithms, schedulers, append logs, query optimisers and so on is a superb thing. It is something that has only been possible in the last 3/4 years and would have been devilishly difficult before then without significant upfront work.
Fetishising complexity might be a bit strong here. Seeing more posts like this is a good thing.
At the same time, OP is right: a business doing this is making poor engineering decisions.
I also agree that the open source tech used in this solution is very powerful and it is great we are at a point where a solution like this could exist. I love avro schema registries and kafka just as much as the next guy.
Projects like this are how we get solutions that blow existing stuff out of the water. Although a lot just end up being fun projects that amount to nothing but an entertaining blog article.
What's extra cool about this for me is how it illustrates to me how far open source components that build on the Apache Kafka ecosystem have come, to the degree that each fairly complex component of relational databases that expose a full RDBMS system from the basic building block of a log have been fully developed (to some production capacity) in the Kafka ecosystem. That's amazing! What does this mean?
It means that you could theoretically (please don't hurt me) implement a relational DB backed by a twitter DB log, or a blockchain, or any other event log that you can't necessarily use traditionally as a WAL. It also introduces a lot of really interesting possibilities for data integration. Definitely has my mind reeling a bit.
Do you run away from microservices too?
Because I can - https://www.youtube.com/watch?v=NPWi5yJK3zo
1- RDBMS implement an imperfect view of the relational model. Is like to say Java is the only OO inplemented in the world.
2- SQL is the standard... and is not practical to say to eliminate it... BUT RDBMS are constrained for the subpar language SQL is. Modern additions are nice... but that is to say that postcript is a nice way to do apps.
Also, a much better language could expand the role of RDBMS.
This is not weird. I live this way when working in FoxPro:
- UI on fox lang. - DB on fox lang. - Reports on fox lang. - Scripting on fox lang. - Web on fox lang - Triggers on fox lang - OO on fox lang.
And all that still with clean separation of logic and components.
3- RDBMS were made by a certain mindset and use case of the 80s. Still so good that can be valid today, but it could get extended. JSON support is just a tiny example of that.
4- Why you can't do "SELECT .. FROM index"? Why you need to create a table to get an index or FTS? Is weird limitation.
The relational model is fine too for KV stores.
----
"Relational database" is a database made on top of the relational MODEL. You could get MANY IMPLEMENTATIONS of that with different use cases. Similar how the functional MODEL and the OO MODEL is not frozen with a single language... that is not that good for app development.
With a better lang, you could eliminate massive ORMs in a nice way!
P.D: Remember, this IS NOT teory. Is proven. This is how was with the dbase family of langs!
As I also want to use a distributed log in the future: Do you know Apache Pulsar or the underlying BookKeeper, which I think was especially made for a distributed WAL?
I might either want to use Apache BookKeeper or Apache Pulsar for a distributed log for scaling my Open Source temporal database, too. Furthermore I'd like to expose the API for streaming changes into the Browser or wherever you want to :-)
That said, I'm always looking for users and contributors to https://sirix.io/ or https://github.com/sirixdb/sirix. That would be super awesome, but I'm sure you have your own ideas already for using BookKeeper.
I'm currently not sure if I first want to build a frontend (I'm a backend engineer, but would like to learn some TypeScript along with using Vue.js and D3js), to interact with SirixDB and to build interactive visualizations to compare revisions of JSON- or XML-resources in SirixDB (stored in a binary format of course highly optimized for space-efficient snapshots).
I think as I'm lacking users it might be more useful, but for sure I'm at least as eager to put forth the idea of scalable SirixDB databases :-)
So, I'd like to use BookKeeper most probably (single writer, read your own write consistency, using synchronous -- for a quorum -- and asynchronous replication for the rest, exactly once semantics...). The thing I don't like is that we also need ZooKeeper, but yeah.
BTW: Why do you think BookKeeper is better than Kafka for this purpose? :-)
It isn't enough to show something can be done. One must also think if it should be done.
Not sure if thats an accurate description of ksql.
One idea I’ve had in the past is to implement a Kafka and Zookeeper layer for FoundationDB.
I don't see the article claiming people should drop PostgreSQL and use this, it's just asking people to give it a try if they are interested.
Much more interesting would be to discuss what capabilities would be different if this were a mature product.
For instance, this sounds interesting:
> One advantage of using Kafka is that multiple servers can all “tail” the same set of topics. This allows multiple KarelDB servers to run as a cluster, with no single-point of failure. In this case, one of the servers will be elected as the leader while the others will be followers (or replicas). When a follower receives a JDBC request, it will use the Avatica JDBC driver to forward the JDBC request to the leader. If the leader fails, one of the followers will be elected as a new leader.
