https://github.com/starqueue/starqueue
The code is in there for Postgres, MS SQL and MySQL (which all support SKIP LOCKED) though at some point I abandoned all but Postgres.
If I was to write another message queue then I wouldn’t use a database, I’d use the file system based around Linux file moves, which are atomic. What I really want is a message queue that is fast and requires zero config, file based message queues are both…. better than a database.
I've experimented with using the file system for storing configuration data where each config value is a single file. Nested structures just use directories. The name of the file is the field name. The file extension is a hint about the data type contained within (.txt is obvious, but I also liked .bool). Parsing data is trivial. I don't need any special viewing or editing tools, just a file manager and text editor. You can see when a specific config value was changed by checking the file update time. You don't have to load the whole configuration just to access one field. And you could conceivably TAR up the whole thing if you wanted to transmit it somewhere.
I use it to configure little sub-projects in my personal website. I really like it, but I shudder to think of the complaining I'd hear from other developers if I ever used it in a work project, just because it's not whatever they've ever seen before and would require a moment of two of thinking on their behalf to get over ingrained habits.
However, and this is a big drawback, once you have too many config files and you start reading and writing from different processes, you get into bottleneck situations quickly.
First of all, if you have multiple processes trying to read/write the same config, that's kind of suspect, and if file I/O is a bottleneck for your config system, that's a different suspicious situation. Why are your processes writing to the config so... often?
But regardless, I can't see how those problems get immediately better by storing that config in a single file. If anything, having it split across multiple files would improve the situation, as different processes that might only be concerned with different sections of the config won't need to wait on file locks from process unrelated to their concerns.
Anyway it’s an elegant idea. Silly to have dozens of config file formats when the fs already has everything it needs. We have xattr too.
The flaw on the OS level is that it is hard to get everyone to change. For new apps not a problem, and any performance concerns are no longer an issue for config.
Not a bad idea at all, but perhaps re-inventing a very old wheel?
[UI]
DarkMode=1
Theme=solarized
[Options]
Indent=4
InventMode=tabs
<user-home>/.config/<app-name>/UI/DarkMode.bool --> 1
<user-home>/.config/<app-name>/UI/Theme.txt --> solarized
And what reinvention is there? If you're just using a system that already exists, you're not reinventing anything.
[0] https://github.com/gchq/Bailo/tree/main/lib/p-mongo-queue
I found almost all message queues to be horribly complex to configure, debug and run. Even database queues require a lot of config, relative to using the file system.
I did actually write a file system based message queue in Rust and it instantly maxed out the disk at about 30,000 messages a second. It did about 7 million messages a second when run as a purely RAM message queue but that didn’t use file system at all.
It depends what you’re doing of course… running a bank on a file system queue wouldn’t make sense.
A fast message queue should be a tiny executable that you run and you’re in business in seconds, no faffing around with even a minute of config.
I just hate configuration in general.
Bank transfers are done by a text file sent over FTP (file transfer protocol; circa 1971)
https://orientalbank.com/assets/Pdfs/ach/ACH_ORIGINATION_AGR...
Did you try an in-memory filesystem through tmpfs?
Shouldn't a message send be worst case a CAS. It really seems like all the work around garbage collection would have some use for in-memory high speed queues.
Are you familiar with the LMAX Disruptor? Is is a Java based cross thread messaging library used for day trading applications.