undefined | Better HN

0 pointsdiarrhea2y ago0 comments

I wonder: can fast, special-case JSON parsers be dynamically autogenerated from JSON Schemas?

Perhaps some macro-ridden Rust monstrosity that spits out specialised parsers at compile time, dynamically…

0 comments

For json schema specifically there are some tools like go-jsonschema[1] but I've never used them personally. But you can use something like ffjson[2] in go to generate a static serialize/deserialize function based on a struct definition.

[1] https://github.com/omissis/go-jsonschema [2] https://github.com/pquerna/ffjson

atombender2y ago

Hey, go-jsonschema is my project. (Someone else just took over maintaining it, though.) It still relies on the standard Go parser; all it does it generate structs with the right types and tags.

marginalia_nu2y ago

A fundamental problem with JSON parsing is that it has variable length fields that don't encode their length, in a streaming scenario you basically need to keep resizing your buffer until the data fits. If the data is on disk and not streaming you may get away with reading ahead to find the end of the field first, but that's also not particularly fast.

Schemas can't fix that.

Groxx2y ago

Why couldn't they? Schemas can allow you to have that as part of your schema. E.g. JSON Schema lets you define max and min lengths on variable-sized things. You can avoid all dynamic resizing if you're careful enough.

I'll definitely agree that most things won't fully take advantage of that even if you provide that information, but it is definitely possible to do so.

marginalia_nu2y ago

Unless you have fixed field lengths, you're still doing twice the work either scanning or resizing the buffer (or over-allocating memory I guess).

That said, JSON is designed for human readability above performance, so it's a design concession that makes sense. What doesn't make sense is using JSON anywhere performance matters.

robocat2y ago

Only if you are using pointers/slices into the buffer as an optimisation.

Otherwise there is no need to keep a buffer of anything after it has been parsed.

marginalia_nu2y ago

I'm talking about during parsing.

Let's assume I send you a JSON object that is one very long string and nothing else. It's e.g. 1 GB in size. To know you need to allocate a 1GB buffer, you need to first scan it, and then copy it; or keep reallocating the same buffer until it fits.

It's an absurd case, but shorter strings face similar overhead.

galangalalgol2y ago

Doesn't the serde crate's json support do precisely this? It generates structs that have optional in all the right places and with all the right types anyway. Seems like the llvm optimiser can probably do something useful with that even if the serde feature isn't using apriori knowledge out of the schema.

dleeftink2y ago

Somewhat tangentially related, Fabian Iwand posted this regex prefix tree visualiser/generator last week [0], which may offer some inspiration for prototyping auto generated schemas.

dleeftink2y ago

Forgot to include the link:

[0]: https://observablehq.com/@mootari/regex-from-strings

atombender2y ago

You forgot to include the link?

mhh__2y ago

It's relatively common in D application to use the compile time capabilities to generator a parser at compile time

PartiallyTyped2y ago

Pydantic does that to some extend I think.

j / k navigate · click thread line to collapse

0 comments

minhazm2y ago

[1] https://github.com/omissis/go-jsonschema [2] https://github.com/pquerna/ffjson

atombender2y ago

Hey, go-jsonschema is my project. (Someone else just took over maintaining it, though.) It still relies on the standard Go parser; all it does it generate structs with the right types and tags.

marginalia_nu2y ago

Schemas can't fix that.

Groxx2y ago

I'll definitely agree that most things won't fully take advantage of that even if you provide that information, but it is definitely possible to do so.

marginalia_nu2y ago

Unless you have fixed field lengths, you're still doing twice the work either scanning or resizing the buffer (or over-allocating memory I guess).

That said, JSON is designed for human readability above performance, so it's a design concession that makes sense. What doesn't make sense is using JSON anywhere performance matters.

robocat2y ago

Only if you are using pointers/slices into the buffer as an optimisation.

Otherwise there is no need to keep a buffer of anything after it has been parsed.

marginalia_nu2y ago

I'm talking about during parsing.

It's an absurd case, but shorter strings face similar overhead.

galangalalgol2y ago

dleeftink2y ago

Somewhat tangentially related, Fabian Iwand posted this regex prefix tree visualiser/generator last week [0], which may offer some inspiration for prototyping auto generated schemas.

dleeftink2y ago

Forgot to include the link:

[0]: https://observablehq.com/@mootari/regex-from-strings

atombender2y ago

You forgot to include the link?

mhh__2y ago

It's relatively common in D application to use the compile time capabilities to generator a parser at compile time

PartiallyTyped2y ago

Pydantic does that to some extend I think.

j / k navigate · click thread line to collapse