RX – a new random-access JSON alternative (opens in new tab)

I was instantly suspicious that a “new better format” for serialization didn’t open with the input/output. And this is why (fucking lol, gtfo):

    Q^mSat,3^b:d+s+E,4Fri,3^u:h+k+u,6Thu,3^P:j+

If you are effectively going binary, do it. CBOR or Protobuf or any dozen other binary serializations that would be far more efficient.

The author claims this is because of copy and pasting… cool, remind me what BASE64 is again?

Reminds me of https://en.wikipedia.org/wiki/PHP_serialization_format

That kind of feels a bit worst of both worlds. None of the space savings/efficiency of binary but also no human readability.

Being able to copy/paste a serialization format is not really a feature i think i would care about.

if one of the advantages is making it copy-pastable then I would suggest the REXC viewer should give you the option to copy the REXC output, currently I have no way of knowing this by looking at your github or demo viewer

another thing, I put in a 400KB json and the REXC is 250KB, cool, but ideally the viewer should also tell me the compressed sizes, because that same json is 65kb after zstd, no idea how well your REXC will compress

edit: I think I figured out you can right click "copy as REXC" on the top object in the viewer to get an output, and compressed it, same document as my json compressed to 110kb, so this is not great... 2x the size of json after compression.

Are there any examples? If it's ASCII I'd expect to see some of the actual data in the readme, not just API.

Unless, to read that correctly, it only has a text encoding as long as you can guarantee you don't have any unicode?

You don't want to copy-paste anything like that as text anyway. Just copy and paste files.

No human is reading much data regardless of the format.

What is the benefit over using for example BSON?

I have an idea, why don't we all go back using XML at this point, as any initial selling point / differentiator has been slowly eroded away?

Serialized just means encoded as a stream of bytes so that it can be transferred between systems. There are absolutely cases where you want to be able to query a value directly like a database instead of parsing the entire thing to memory before you can read it. Think of this as no-sql sqlite.

Not at all. What makes you say that?

many serialization format are just a memory structure dump.

That's not really random access, though. You're effectively just searching through the entire dataset for every targeted read you're after.

What might be interesting is to have a tool that processes full JSON data and creates a b-tree index on specified keys. Then you could run searches against the index that return byte offsets you can use for actual random access on the original JSON.

OTOH, this is basically just recreating a database, just using raw JSON as its storage format.

Or in this case, just do `rx file.rx` It has jq like queries built in and supports inputs with either rx or json. Also if you prefer jq, you can do `rx file.rx | jq`

Amazon Ion has some support for this - items are length-prefixed so you can skip over them easily.

It falls down if you have e.g. an array of 1 million small items, because you still need to skip over 999999 items to get to the last one. It looks like RX adds some support for indexes to improve that.

I was in this situation where we needed to sparsely read huge JSON files. In the end we just switched to SQLite which handles all that perfectly. I'd probably still use it over RX, even though there's a somewhat awkward impedance mismatch between SQL and structs.

Exactly. Low heap allocations when reading values is one of the main driving factors in this design!

What about CBOR that can retain JSON compatibility?

If you are working with an end you don’t control, this “newer better” format isn’t in your cards either.

The cost of using a textual format is that floats become so slow to parse, that it’s a factor of over 14 times slower than parsing a normal integer. Even with the fastest simd algos we have right now.

Yeah the raw parse speed comparison is almost a red herring at this point. The real cost with JSON is when you have a 200MB manifest or build artifact and you need exactly two fields out of it. You're still loading the whole thing into memory, building the full object graph, and GC gets to clean all of it up after. That's the part where something like RX with selective access actually matters. Parse speed benchmarks don't capture that at all.

Initial format docs are now here:

https://github.com/creationix/rx/blob/main/docs/rx-format.md

Railroad diagrams will come later when I have more time.

yeah, LuaJIT is one of the use cases I had in mind working on this. JSON is pretty fast in modern JS engines, but in Lua land, JSON kinda sucks and doesn't really match the language without using virtual tables.

JSON has `null` values with string keyds, but lua doesn't have `null`. It has `nil`, but you can't have a key with a nil value. Setting nil deletes the key

Lua tables are unordered. But JS and JSON are often ordered and order often matters.

RX, however matches Lua/LuaJIT extremely well and should out-perform the JS Proxy based decoder using metatables. Since it's using metatables anyway do to the lazy parsing, it's trivial to do things like preserve order when calling `pairs` and `ipairs` and even including keys with associated null values.

You can round trip safely in Lua, which is not easy with most JSON implementations.

it's not really possible to stay human readable and get the compression levels and random access properties I was going for. But it is as human tooling friendly as possible given the constraints.