What am I missing? What would compel someone to abandon a battle tested system like PSQL for a critical storage engine for this?
You can do an append-only table like this in any SQL database. You can use composite keys to achieve the query+natural sorting abilities over a single field. The effort to replicate this feature in any other environment is so minimal that it the 'vendor lock-in' makes this a non-starter in my eyes.
Focusing on your abandonment part etc etc:
- Its not really trying to achieve that, this is simply a more efficient model, with a good storage coupling (for chat).
- What is being achieved here is a cohesive solution for chat specific workloads and optimising for it.
One of the benefits is speed (throughput, latency) - and very cheap calls.
More so - under sql, and for developers who have different demands and mostly changing demands - the effort is quite complicated situation (I speak from my own experience)
The focus here is a simple downmost layer - that allow all the varieties of top layers & requirements to be achieved easily.
In my blog post - i share a variety of these.
Also product is not ready - but thought i share this idea - as i think it can be of benefit to most devs who are starting out with a product.
You technically dont even need progressdb to achieve it - You can do it with a backend based pebble or rocksdb database.
There are no changes to the underlying core, i simply make it easy to use operationally, along with other chat centric demands in the AI age.
e.g Backups, Retention, Scaling it up etc.
So there is no "Vendor lock-in" - and i plan to opensource everything.
I dont even feel i own the idea - its pretty out there, am just bringing eyes towards it.
Which messages belongs together depends on the context and when you discover that the fancy thread store you built your app around can't return a list of MY messages without a full sequential scan of every single thread you'll wish you could solve it with an index.
This statement is in no way saying you dont need an index, its simply saying, this is the variety of ways people might end up choosing.
Which is just 1 example of the type of work - a thread model & compatible store can fix and fix more efficiently.
In no way fancy too - this is just redis with a disc and some steps imo.
