https://github.com/fiatjaf/awesome-jq
https://github.com/TomConlin/json2xpath
https://github.com/antonmedv/fx
https://github.com/fiatjaf/jiq
https://github.com/jmespath/jp
https://github.com/cube2222/jql
https://github.com/borkdude/jet
https://github.com/jzelinskie/faq
https://github.com/dflemstr/rq
Personally I think that next time I might just fire up Hy and use its functional capabilities.
> Make JSON greppable!
> gron[1] transforms JSON into discrete assignments to make it easier to grep for what you want and see the absolute 'path' to it. It eases the exploration of APIs that return large blobs of JSON but have terrible documentation.
▶ gron "https://api.github.com/repos/tomnomnom/gron/commits?per_page=1" | fgrep "commit.author"
json[0].commit.author = {};
json[0].commit.author.date = "2016-07-02T10:51:21Z";
json[0].commit.author.email = "mail@tomnomnom.com";
json[0].commit.author.name = "Tom Hudson";
https://docs.microsoft.com/en-us/powershell/module/microsoft...
https://github.com/borkdude/babashka
https://news.ycombinator.com/item?id=24353476
Aside: another nice tool I recently discovered for working with JSON and YML, doing conversion and diffs (especially helpful for generated files):
It is not nearly as expressive as jq, but it is faster for my use cases (written in golang).
Personally I'd prefer Fennel, which is on Lua and thus a whole lot faster, especially in regard to the startup time—but as I noted in a thread on Fennel, Lua's omission of a proper ‘null’ makes it awkward to handle exchange and transformations of data from third parties. And, since I'm likely to fiddle with the queries for some time, startup delay is less important here.
2) doesn't accept values that jq accepts
% time jq -r '[expression]' < parcels | wc
365 1454 7978
jq -r < parcels 1.39s user 0.00s system 99% cpu 1.390 total
wc 0.00s user 0.00s system 0% cpu 1.390 total
% time ~/.yarn/bin/q '[expression]' parcels | wc
q: internal error, uncaught exception:
Yojson.Json_error("Line 56, bytes -1-32:\nJunk after end
of JSON value: '{\n \"OBJECTID\": 155303,\n \"BOOK\"'") echo '{"foo": "bar"}' | query-json ".foo" /dev/stdinTwo examples I can remember off the top of my head:
- Nix build scripts
- OpenMoko
query-json ".foo" <(echo '{"foo": "bar"}')I definitely agree that reading from stdin is critical if I'll be able to use it. Don't take the criticism too hard though (especially the "author doesn't appreciate unix" stuff. Sometimes we can be such assholes to each other).
Nice work!
The incompatibility is apparently due to the fact that jq is happy with a concatenation of JSON objects and q is not. For example {'foo':1}{'foo':2} as opposed to [{'foo':1},{'foo':2}]
Query CSV with SQL
Happy to rename it to qj instead, but the option of renaming the binary it's a good workaround.
He replied and thought about qj
I will try to bring it to the brenchmark, thanks for sharing
It is more verbose, like you get the size of something with array:size or map:size functions, so it is more readable
I am implementing it in Xidel 0.9.9+: http://www.videlibri.de/xidel.html
echo '' | fzf --print-query --preview-window wrap --preview 'cat test.json | jql {q}'
I often have to pluck out attributes from streams of json records (1 json object per line) - often millions/billions.
jq is almost always the bottleneck in the pipeline at 100% CPU - so much so that we often add an fgrep to the left side of the pipeline to minimize the input to jq as much as possible.
The main weakness seems to be streaming use cases (not having the whole file in memory at once). These are supported, but the syntax is quite awkward.
I should specify that on the performance section, Thanks!
The C code is clean enough as C code goes, but fairly monolithic. And it’s C, so it’s not noticeably slow until you start processing GB. But it would probably take a rewrite to improve its performance significantly.
https://mosermichael.github.io/jq-illustrated/dir/content.ht...
and there are a few techniques to "discover" the schema of the json file, I trend to read with '.' or 'keys' and later keep going.
I'm planning to implement a flag where each operation prints the internal state of the json, so you would see what are the "pipes".
I will pick a few of your cheatsheet to implement next in q, Thanks!
IMO, an individual dev making a fast useful tool should always be welcomed as a feat of worthy hacking.
But if I had done something like that, and then serendipitously discovered that I was exceeding the original's performance, I certainly wouldn't be shy about it.
Also, this comes across as armchair criticism purely for the sake of armchair criticism. My own experience has been that, when I'm doing ETL that involves wrangling JSON, the "wrangling JSON" bit of it is almost always the bottleneck. So any improvement is more than welcome and deserves to be cheered. Even if it's an improvement on something that's already the current fastest way to do it.
Reimplementing a piece of software that is 12 years old which mimics their UX and improves performance and error messages it's more than welcome in my opinion. My purpose was to learn the OCaml stack of writting compilers, so I personally found that I "needed" a language already created.
Thanks for raising those concerns
We'd all be better off if plebes grew their skills by reimplementing common tools.
I was somewhat surprised it didn't use an existing json parser library.
Replacing that with, say, traditional command line flags would make it a lot less useful for me, I'd probably have to build much longer pipe-chains to do things that are relatively simple and readable jq snippets (if one knows the syntax.)
Using an established scripting language in its place would make it pretty much just python -c/ruby -e or whatever with some pre-loaded functions, but what's the point? You can always just write a quick python/ruby/whatever script, jq to me is an alternative for cases where a script feels unnecessary. It would also mean everything gets more verbose, so less of my jq transformations can be inlined without loss of readability.
Aligning it to more established languages would probably cause confusion as well in those cases where it doesn't match the reference language 1:1. Looks like javascript, writes like javascript, but only for a tiny subset of the language, etc.
Doing this only for a few function names or syntax constructs still results in a pretty unique and unusual language that will require people to reference the docs a lot, just now lots of existing scripts break.
There're a lot of quirks from the usage of it and people struggling with learning such a great tool, so in the area of query-json it will try to make a better interface for users.
In this case the author is using Reason as an alternative syntax to OCaml. Reason resembles javascript a little more, and some people find that nicer to work with. So the idea is that you write Reason code, then translate it into OCaml code using the Reason tools, and then ultimately you compile it down to a native binary.
If instead you want to write a web-app which runs in a web browser or node.js, then you'd need to compile it to Javascript, which is what bucklescript helps you do.
Where does Rescript come in? As explained above, Reason can be used for writing either native apps or javascript apps. However, it's hard to evolve the syntax of Reason in a way which satisfies both aims. So they've now split the work -- going forward, Reason will specialize on native, and Rescript will specialize on javascript apps. Their syntax is expected to diverge from each other, in order to support those aims as best as they can.
I used Reason to compile to Native, so using OCaml's stdlib and OCaml's dependencies and compiling it with OCaml, but my source code is written in Reason syntax.
It wasn't.
Before doing any curl | bash, check what's on the install command, that's the entire point of it.
I'm sure other people have use cases where the browser wouldn't meet their needs, but for me, I find jq unnecessary.
Menhir/sedlex and others are pretty high accessibility barrier for new commers.
One of the nice things about all of it it's the discord, it's friendly and always helpful.
Hope it helps, just let me know if there's any specific!
> The report shows that q is between 2x and 5x faster than jq in all operations tested and same speed (~1.1x) with huge files (> 100M).
While faster for somethings....that's a pretty large set of caveats!
I have a issue to improve performance where I can push this forward: https://github.com/davesnx/query-json/issues/7
But sure, are caveats!
https://github.com/davesnx/query-json#performance https://github.com/davesnx/query-json/blob/master/benchmarks...
But all explanations aren't based by any evidence, just asumptions.
For the set of operations that I implement it it's faster, that's true.
The feature that I think penalizes a lot jq is "def functions", the capacity of define any function that can be available during run-time.
This creates a few layers, one of the difference is the interpreter and the linker, the responsible for getting all the builtin functions and compile them have them ready to use at runtime.
The other pain point is the architecture of the operations on top of jq, since it's a stack based. In query-json it's a piped recursive operations.
Aside from the code, the OCaml stack, menhir has been proved to be really fast when creating those kind of compilers.
I will dig more into performance and try to profile both tools in order to improve mine.
Thanks
It is a JSON parser in C without heap allocations. The query language is piddly, but the tool can be useful for grabbing a single value from a very large JSON file. I don't have time for it, but someone could fork and make it a real deal.