> The database is stored in memory. So it must be small enough to fit in RAM, and the full journal has to be replayed from scratch when opening a file.
For larger datasets, you really want disk support. Using something like SQLite or DuckDB as an append-only store is another way to achieve this effect.
Also lack of a proper query language will be a problem for long term serious use. A simple hand-rolled program API can only get you so far, until you need more advanced querying.
> Unlike XML or JSON, joedb is a binary file format that does not require any parsing. So, joedb files are much smaller, and processing data is much faster.
Some time ago I created a JSON-compatible serialization format that is zero-copy (no parsing required): https://github.com/fastserial/lite3
It doesn't do transactions or history versioning, but it is also very fast in memory. Something like jq or JSONPath on a disk-file version of this format could be interesting.
I have been developing this library for more than 10 years. I could not find a simple light-weight tool to serialize data to files with proper ACID transactions, and did not want to use a SQL database. Even SQLite is not that light, and using SQL strings as API from C is very unpleasant. I thought about the simplest possible way to implement ACID transactions, and came up with the design of joedb. It is orders of magnitude less complex than a SQL database, and provides the simple type-safe low-level access to data I want in C++ code.
My three cents: compact the journal when its size exceeds the actual data size. With thresholds or other knobs; with the point being the initial load time should be directly proportional to the amount of actual data. Everything else/older is a backup.
