undefined | Better HN

0 pointshnlmorg4y ago0 comments

The problem with auto-detection is that in POSIX-like shells it only works for the most simplest of problems (eg `ls` becoming a single column list when piped) because ultimately everything is treated by the shell as white space delimited list and treated by the OS as an untyped stream of bytes.

However more modern shells fix this problem with having typed pipelines and builtins written to understand more than just a flat file of bytes.

Take _murex_ for example (disclaimer, I'm the author of that shell):

  » jobs
  PID   State      Background  Process  Parameters
  2104  Executing  true        exec     sleep 9000000
  2240  Executing  true        exec     sleep 9000000

It's readable but what if I wanted to pass it as a table?

  » jobs | cat
  ["PID","State","Background","Process","Parameters"]
  [2104,"Executing",true,"exec","sleep 9000000"]
  [2240,"Executing",true,"exec","sleep 9000000"]

ok, so it auto-detects it is running as a pipe and outputs it as a jsonlines table. That would be annoying in Bash. But with a type aware shell, that shell knows it's a jsonlines table, eg

  » jobs | debug | [[ /Data-Type/Murex ]]
  jsonl

...but what can we do with a jsonlines table? Well you can select individual columns:

  » jobs | [ PID State ]
  [
      "PID",
      "State"
  ]
  [
      "2104",
      "Executing"
  ]
  [
      "2240",
      "Executing"
  ]

run SQL against it

  » jobs | select * where PID > 2200
  ["PID","State","Background","Process","Parameters"]
  ["2240","Executing","true","exec","sleep 9000000"]

iterate through each row

  » jobs | foreach proc { if { =$proc[0]>2200 } then { echo $proc } }
  [2240,"Executing",true,"exec","sleep 9000000"]

or even just convert it into another format, like CSV

  » jobs | format csv
  PID,State,Background,Process,Parameters
  2104,Executing,true,exec,sleep 9000000
  2240,Executing,true,exec,sleep 9000000

...or YAML...

  » jobs | format csv
  - - PID
    - State
    - Background
    - Process
    - Parameters
  - - "2104"
    - Executing
    - "true"
    - exec
    - sleep 9000000
  - - "2240"
    - Executing
    - "true"
    - exec
    - sleep 9000000

And it all just works without you having to think or even know what data format is traversing the pipeline.

However unfortunately none of this is possible with Bash. And thus the majority of tools are forced to be dumb to compensate.

» jobs | format csv - - PID - State - Background - Process - Parameters - - "2104" - Executing - "true" - exec - sleep 9000000 - - "2240" - Executing - "true" - exec - sleep 9000000

0 comments

4 comments · 1 top-level

preseinger4y ago· 3 in thread

I don't think typed pipelines in a shell is a virtue. I think dumb bytes are ideal. Text manipulation can be laborious but it's always straightforward.

laumars4y ago

I used to think that when Powershell was my only exposure typed pipelines. But after using murex I’ve totally changed my opinion. You still have POSIX pipes there for when you want dumb bytes. But you also have typed pipes layered on top to allow one to manage structural data without having to think about it.

It’s a bit like having jc integrated into the shell, except that jc supports TOML, YAML, tables of various formats and all sorts - and autodetects the content type too - so you don’t have to learn a dozen different tools for managing a dozen different content types.

So it’s the best of both worlds.

hnlmorgOP4y ago

It's only straightforward if you're using tools that understand what those dumb bytes are. Often you'll get away with a mixture of grep, sed and awk but the moment you turn to something like jq you've done away with the assumption that the pipeline is just dumb bytes. However rather than having to explicitly remember which laborious set of commands work well together, murex will provide you with a suite of builtins that have the same flags and behaviour regardless of the structure of the bytes being read in by them. And if you prefer your pipes to be untyped POSIX byte streams then murex still works here because it will suggest (via tab-completion suggestions) the right commands to use against each previous command.

preseinger4y ago

> It's only straightforward if you're using tools that understand what those dumb bytes are. Often you'll get away with a mixture of grep, sed and awk but the moment you turn to something like jq you've done away with the assumption that the pipeline is just dumb bytes

I guess my claim is that the minute you need to parse the dumb bytes as anything other than strings, your problem is complex enough that you shouldn't be using shell to solve it, and you should write a proper program instead. That's not hard and fast, for example I use fish and `math` is very powerful.

`jq` is also a bad example, I think, in many ways... it explicitly doesn't follow the Unix philosophy, and instead does it's work by parsing an opaque string according to it's own DSL. Much better is a tool like gron, which allows you to process JSON using familiar tools like grep.

j / k navigate · click thread line to collapse