A Faster Alternative to Jq (opens in new tab)

I think the big problem is it's a tool you usually reach for so rarely you never quite get the opportunity to really learn it well, so it always remains in that valley of despair where you know you should use it, but it's never intuitive or easy to use.

It's not unique in that regard. 'sed' is Turing complete[1][2], but few people get farther than learning how to do a basic regex substitution.

[1] https://catonmat.net/proof-that-sed-is-turing-complete

[1] And arguably a Turing tarpit.

That’s interesting! Can you say a little more? I find jq’s syntax and semantics to be simple and intuitive. It’s mostly dots, pipes, and brackets. It’s a lot like writing shell pipelines imo. And I tend to use it in the same way. Lots of one-time use invocations, so I spend more time writing jq filters than I spend reading them.

I suspect my use cases are less complex than yours. Or maybe jq just fits the way I think for some reason.

I dream of a world in which all CLI tools produce and consume JSON and we use jq to glue them together. Sounds like that would be a nightmare for you.

Shameless plug, but you might like this: https://github.com/IvanIsCoding/celq

jq is the CLI I like the most, but sometimes even I struggled to understand the queries I wrote in the past. celq uses a more familiar language (CEL)

Funny that everyone is linking the tools they wrote for themselves to deal with this problem. I am no exception. I wrote one that just lets you write JavaScript. Imagine my surprise that this extremely naive implementation was faster than jq, even on large files.

    $ cat package.json | dq 'Object.keys(data).slice(0, 5)'
    [ "name", "type", "version", "scripts", "dependencies" ]

https://crespo.business/posts/dq-its-just-js/

It's because .json itself has so much useless cruft it's often annoying to deal with. I am forever indebted for younger self forcing me to learn Clojure. Most of the time I choose not even bother with JSON anymore - EDN semantically so much cleaner - it's almost twice compact (yet lossless), it's far more readable (quotes and commas are optional), and easier to work with structurally. These days I'd use borkdude/jet or babashka and then deal with data in Clojure REPL - there I can inspect it from all sorts of angles, it's far easier to group, sort, slice, dice, map and filter through it. One can even easily visualize the data using djblue/portal. Why most people strangulate themselves with confusing jq operators unnecessarily, I would never understand. Clojure is not that hard, maybe learn some basics, it comes handy a lot. Even when your team doesn't have any Clojure code.

Highly recommend gron. https://github.com/tomnomnom/gron

To fix this I recently made myself a tiny tool I called jtree that recursively walks json, spitting out one line per leaf. Each line is the jq selector and leaf value separated by "=".

No more fiddling around trying to figure out the damn selector by trying to track the indentation level across a huge file. Also easy to pipe into fzf, then split on "=", trim, then pass to jq

JMESPath is what I wish jq was. Consistent grammar. It only issue is it lacks the ability to convert JSON to other formats like CSV.

If we're plugging jq alternatives, I'll plug my own: https://git.sr.ht/~charles/rq

I was working at lot with Rego (the DSL for Open Policy Agent) and realized it was actually a pretty nice syntax for jq type use cases.

I just ask Opus to generate the queries for me these days.

LOL ... I can absolutely feel your pain. That's exactly why I created for myself a graphical approach. I shared the first version with friends and it turned into "ColumnLens" (ImGUI on Mac) app. Here is a use case from the healthcare industry: https://columnlens.com/industries/medical

Like I did with regex some years earlier, I worked on a project for a few weeks that required constant interactions with jq, and through that I managed to lock in the general shape of queries so that my google hints became much faster.

Of course, this doesn't matter now, I just ask an LLM to make the query for me if it's so complex that I can't do it by hand within seconds.

I agree, even trivial tasks require us to go back to jq's manual to learn how to write their language.

this and other reasons is why I built: https://github.com/dhuan/dop

When I need it i find that relearning the jq syntax is still faster than whatever other harebrained scheme I might come up with to solve my problem. It’s just so useful 2x a year when I really need it

I completely agree. I much prefer leveraging actual javascript to get what I need instead of spending time trying to fumble my way through jq syntax.

yeah I literally just use gemini / claude to one-shot JQ queries now

I also genuinely hate using jq. It is one of the only things that I rely heavily on AI.

Whenever you have this kind of impressions on some development, here are my 2 cents: just think "I'm not the target audience". And that's fine.

The difference between 2ms and 0.2ms might sound unneeded, or even silly to you. But somebody, somewhere, is doing stream processing of TB-sized JSON objects, and they will care. These news are for them.

I get the sentiment, but everybody thinks that, and in aggregate, you get death by a thousand paper cuts.

It’s the same sentiment as “Individuals don’t matter, look at how tiny my contribution is.”. Society is made up of individuals, so everybody has to do their part.

> 9/10 whatever tooling you are using now will be perfectly fine.

It is not though. Software is getting slower faster than hardware is getting quicker. We have computers that are easily 3–4+ orders of magnitudes faster than what we had 40 years ago, yet everything has somehow gotten slower.

True. I feel like the main way a tool could differentiate from jq is having more intuitive syntax and many real world examples to show off the syntax.

Maybe look at it from another perspective. Better performance == less CPU cycles wasted. Consider how many people use jq daily and think about how much energy could be saved by faster implementations. In times like this where energy is becoming more scarce we should think about things like this.

Was about to post exactly this... It is impressive engineering wise, but for data and syntax, ease of use or all the great features, I care about that more. Speed isn't that important to me for a lot of these tools.

If I/you was working with JSON of that size where this was important, id say you probably need to stop using JSON! and some other binary or structured format... so long as it has some kinda tooling support.

And further if you are doing important stuff in the CLI needing a big chain of commands, you probably should be programming something to do it anyways...

that's even before we get to the whole JSON isn't really a good data format whatsoever... and there are many better ways. The old ways or the new ways. One day I will get to use my XSLT skills again :D

I absolutely understand what you're saying. It makes complete sense. But I will never, ever shake the sense that software that isn't as fast as possible is offensive, immoral, delinquent -- the result of sloth, lassitude, lack of imagination, and a general hostility toward our noble Art.

"Fast enough" will always bug me. "Still ahead of network latency" will always sound like the dog ate your homework. I understand the perils of premature optimization, but not a refusal to optimize.

And I doubt I'm alone.

Then this is for the handful of cases for you. When it matters it matters.

I agree for some things, but not for tools or "micro-software" like jq that can get called a LOT in an automated process. Every order of magnitude saved for the latter category can be meaningful.

> I see the same thing repeated in various front end tooling too. They all claim to be _much_ faster than their counterpart.

>

> 9/10 whatever tooling you are using now will be perfectly fine

Are you working in frontend? On non-trivial webapps? Because this is entirely wrong in my experience. Performance issues are the #1 complaint of everyone on the frontend team. Be that in compiling, testing or (to a lesser extend) the actual app.

Yes

I don't think I remember one case where jq wasn't fast enough

Now what I'd really want is a jq that's more intuitive and easier to understand

Fair, but agentic tooling can benefit quite a lot from this

Opencode, ClaudeCode, etc, feel slow. Whatever make them faster is a win :)

I process TB-size ndjson files. I want to use jq to do some simple transformations between stages of the processing pipeline (e.g. rename a field), but it so slow that I write a single-use node or rust script instead.

Deal with really big log files, mostly.

If you work at a hyperscaler, service log volume borders on the insane, and while there is a whole pile of tooling around logs, often there's no real substitute for pulling a couple of terabytes locally and going to town on them.

I'll write a one-off shell pipeline to inspect something on 10^5 servers - it will be sent to each of those servers and run once or a handful of times, and the results will be transmitted back and that's that. Kind of a map-reduce shell thing, for ops type tasks.

Sometimes those will actually need to process through a bunch of data unexpectedly.

Sometimes those will be run on a loop - once per second, N per minute (etc), and the results will be used to monitor a situation until a bug is fixed or a spike in load is resolved or a proper monitoring program/metric can be deployed.

Sometimes those are to investigate a pegged CPU and the amortized lower runtime across all the tasks on the CPU is noticable.

We run our machines hot and part of the reason we can do that is being in the habit of choosing lower cost (in cycles) tooling whenever we can. If i can spend a little time and effort learning a tool that saves a bunch of cpu in aggregate, its a win. When the whole company does it, we can spend a lot less on hardware than it costs in engineer time to make these decisions.

Another way of putting it is: its a type of frugality (not cheapness, just spending wisely). If you save a dollar once, its nothing. If you have a habit of saving a dollar every time the opportunity arises, it adds up quickly. By having a habit of choosing more performant tools, you're less likely to hit a case where you wish you did use more performant tools, and are practiced at it when the need arises for pure parsimony and it's less painful.

We parse JSON responses for dashboards, alerting, etc. Thousands of nodes, depending on the resolution of your monitoring you could see improvements here.

It's a simple loop:

- Someone likes tool X

- Figures, that they can vibe code alternative

- They take Rust for performance or FAVORITE_LANG for credentials

- Claude implements small subset of features

- Benchmark subset

- Claim win, profit on showcase

Note: this particular project doesn't have many visible tells, but there's pattern of overdocumentation (17% comment-to-code ratio, >1000 words in README, Claude-like comment patterns), so it might be a guided process.

I still think that the project follows the "subset is faster than set" trend.

You don't know something is slow until you encounter a use case where the speed becomes noticeable. Then you see the slowness across the board. If you can notice that a command hasn't completed and you are able to fully process a thought about it, it's slow(er than your mind, ergo slow!).

Usually, a perceptive user/technical mind is able to tweak their usage of the tools around their limitations, but if you can find a tool that doesn't have those limitations, it feels far more superior.

The only place where ripgrep hasn't seeped into my workflow for example, is after the pipe and that's just out of (bad?) habit. So much so, sometimes I'll do this foolishly rg "<term>" | grep <second filter>; then proceed to do a metaphoric facepalm on my mind. Let's see if jg can make me go jg <term> | jq <transformation> :)

Not every use case of jq is a person using it interactively in their terminal, believe it or not.

Race between ripgrep and ugrep is entertaining.

Optimization = good

Prioritizing SEO-ing speed over supporting the same features/syntax (especially without an immediately prominent disclosure of these deficiencies) = marketing bullshit

A faster jq except it can't do what jq does... maybe I can use this as a pre-filter when necessary.

Speed is a quality in itself. We are so bugged down by slow stuff that we often ignore that and don’t actively search for another.

But every now and then a well-optimised tool/page comes along with instant feedback and is a real pleasure to use.

I think some people are more affected by that than others.

Obligatory https://m.xkcd.com/1205

Integrating with server software, the performance is nice to have, as you can have say 100 kRPS requests coming in that need some jq-like logic. For CLI tool, like you said, the performance of any of them is ok, for most of the cases.

> Having used `jq` and `yq`

If you don't mind me asking, which yq? There's a Go variant and a Python pass-through variant, the latter also including xq and tomlq.

I use jq to grind through gigabytes of GeoJSON files exported from ArcGIS, as an ETL stage. It takes a long time.

Yeah, turns out not everyone uses these tools the way you do. Weird!

+1. I switched to using Nushell as my daily driver around mid-2023 (0.84.0?) and use it in preference to other interactive tools. I do keep at hand jq, yq, and mlr because I need to exchange stuff with colleagues who don't use Nu.

Same! Nushell replaced almost all of them

Had to spend some efforts to set up completions, also there some small rough edges around commands discoverability, but anyway, much better than the previous oh-my-zsh setup

Ideally, wish it also had a flag to enforce users to write type annotations + compiling scripts as static binaries + a TUI library, and then I'd seriously consider it for writing small apps, but I like and appreciate it in the current state already

Same here, nushell is awesome! It helped me to automate so many more things than I did with any other shell. The syntax is so much more intuitive and coherent, which really helps a lot for someone who always forgot how to write ifs or loops in bash ^^

Something I find myself saying a lot, Nushell is a better `jq` than `jq`

I suspect the website is vibe-coded, like the tool itself.

I think this has just been fixed. A bit of dark mode was leaking into light in the css.

Fine in Firefox on Android. Note that the scales of the charts are all different, which makes them hard to compare.

Also, there are lots of charts without comparison so the numbers mean nothing...

Hey OP here! Sorry about this this is just laziness on my part because I never use light mode so I forget to test haha, will push a fix!

Modern programmers these days just give a shit about user experience. Better to just load up in reader mode.

White text with light background, yeah.

I had the same problem (brave browser)

Looks fine to me on Edge/Windows.

Should be fixed now! Let me now :)

Broken on iOS Safari too

I keep an eye on jaq, but there are some holes in the story. jaq 3.0 is faster than Linux distro builds of jq, but jq built correctly is faster than jaq. As far as I can tell the performance reputation of jq is caused by bad distro packaging.

Thank you for the recommendation.

It looks like jaq has already progressed much further in the right direction than jsongrep has just started in the not-quite-as-right direction.

Hey thank you! OP here, yes I was struggling to find large enough documents to run the benchmarks on, the range currently on the benchmark data is ~106 B - ~190MB, which I think covers the majority of quick task workloads, but would love to have large documents, if there's an public ones you can thinking of I'd like to know!

If rust is not in the HN title and fire emojis in the readme, it doesn't come from the Rust region of France.

It's just sparkling memory safe high performance software

The backtracking implementation in jq is really the secret sauce for how it handles those complex filters without getting bogged down

Looks like naive implementation of homemade bytecode interpreter. What's so mind blowing about that? Maybe I missed something.

OP here: sorry about that, the light mode inconsistencies should be fixed now. Will continue to work on making the site design better as well!

You don't learn it properly. It's not supposed to be intuitive, it's supposed to be concise at the cost of it being intuitive. Would be like somebody saying typing words in to Google is more intuitive than writing regex.

jq is supposed to fit in to other bash scripts as a one liner. That's it's super power. I know very few people who write regex on the fly either (unless you were using it everyday) they check the documentation and flesh it out when they need it.

Just use Claude to generate the jq expression you need and test it.

theory is one thing but the cpu cache is the real bottleneck here... here is a small visual breakdown of how these arrays look in memory and why pointer chasing is so expensive compared to the actual logic: https://vectree.io/c/json-array-memory-indexing

basically the double jump to find values in the heap is what slows down these tools most

jj was already taken by jujutsu, unfortunately.

LOL ... came here to grips about that!

Also "jg" reads very similar to "jq", and initially I thought he was talking about "jq" all along, and I was like: where can I see the "jasongrep" examples? Threw me off for a minute.

Just hit this too:

https://news.ycombinator.com/item?id=47542182

The reason I was interested, was adding the new tool to arkade (similar to Brew, but more developer/devops focused - downloads binaries)

The agent found no Arm binaries.. and it seemed like an odd miss for a core tool

https://x.com/alexellisuk/status/2037514629409112346?s=20

I’d install it via cargo anyway and that would build it for arm64.

If the arm64 version was on homebrew (didn’t check if it is but assume not because it’s not mentioned on the page), I’d install it from there rather than from cargo.

I don’t really manually install binaries from GitHub, but it’s nice that the author provides binaries for several platforms for people that do like to install it that way.

OP here: Releases have been updated! Also someone was kind enough to package it in Homebrew already :) https://github.com/micahkepe/jsongrep/pull/22

Really? That is your response? This is an high quality article from someone who spend a lot of time implementing a cool tool and also sharing the intricate inner workings of it. And your response is, "eh there are no official binaries for my platform". Give them some credit! Be a little more constructive!

How does it deal with nested objects? E.g. one of the fields/columns is an array of objects.

What were your syntax stumbling blocks? I must be honest I've used jq enough but can never remember the syntax. It's one of the worst things about jq IMO (not the speed, even though I'm a fan of speedups). There's something ungrokkable about that syntax for me.

*alternatives

no

OP here- I will add! Thank you for checking out the project!

They've been added!

The unlimited memory required for JSON is poor design

netstrings has no such issues

As someone who worked with Unix/Linux and command line arguments for 30 years and still "abuse" cat like the documentation, I regularly hear this complaint.

Yes, "cmd <file" is more efficient for the computer but not for the reader in many cases. I read from left to the right and the pipeline might be long or "cmd" might have plenty of arguments (or both). Having "cat file | cmd" immediately gives me the context for what I am working with and corresponds well with "take this file, do this, then that, etc" with it) and makes it easier for me to grok what is happening (the first operation will have some kind of input from stdin). Without that, the context starts with the (first) operation like in the sentence "do this operation, on this file (,then this, etc)". I might not be familiar with it or knowing the arguments it expects.

At least for me, the first variant comes more naturally and is quicker to follow (in most cases), so unless it is performance sensitive that is what I end up with (and cat is insanely fast for most cases).