I will donate $50 to you or your favorite charity to encourage a new feature: to-usv, which outputs Unicode separated values (USV) with unit separator U+241F and record separator U+241E.
Unicode separated values (USV) are much like comma separated values (CSV), tab separated values (TSV) a.k.a. tab delimited format (TDF), and ASCII separated values (ASV) a.k.a. DEL (Delimited ASCII).
The advantages of USV for me are that USV handles text that happens to contain commas and/or tabs and/or newlines, and also having a visual character representation.
For example USV is great for me within typical source code, such as Unix scripts, because the characters show up, and also easy to copy/paste, and also easy to use within various kinds of editor search boxes.
Bonus: if the programming implementation of to-usv calls a more-flexible function that takes a unit separator string and a record separator string, then you can easily create similar commands for to-asv, to-csv, etc.
A. use some "weird" separators and hope those don't appear in your input
B. bite the bullet and escape and parse properly
Option A is perfectly reasonable for one-offs, where you can handle exceptional cases or know they won't occur because you know what's in the data. However for reusable code, you need option B, which means not using `cut` to parse CSV files, for instance (since commas can occur inside double-quoted strings). In that case, what's the benefit of using USV over an existing, more common, format?
Orthogonal to escaping, the choice is what characters to use for unit separator and record separator.
If the data are for machines only, then for me the choice of characters doesn't matter. If the data are potentially for reading or editing, such as by a programmer, then my choice is to prefer typically-visible characters over typically-invisible characters and/or zero-width characters (e.g. ASV a.k.a. DEL a.k.a. ASCII 30 & 31).
My choice of USV is thus because U+241F and U+241E are visible, and also in Unicode they are semantically meaningful.
It is handy for pipelining UNIX commands so that they can handle data that includes commas and newlines inside fields. In this example, csvquote is used twice in the pipeline, first at the beginning to make the transformation to ASCII separators and then at the end to undo the transformation so that the separators are human-readable.
> csvquote foobar.csv | cut -d ',' -f 5 | sort | uniq -c | csvquote -u
It doesn't yet have any built-in awareness of UTF or multi-byte characters, but I'd be happy to receive a pull request if it's something you're able to offer.
Also, is your offer available for other tabular data tools? :)
For example I can write code samples such as:
echo 'a␟b␟c␞d␟e␟f␞g␟h␟i' | tr ␟␞ '\t\n'
a b c
d e f
g h i
- https://github.com/dkogan/vnlog
- https://csvkit.readthedocs.io/
- https://github.com/johnkerl/miller
- https://github.com/BurntSushi/xsv
- https://github.com/eBay/tsv-utils-dlang
- https://stedolan.github.io/jq/
- https://github.com/BatchLabs/charlatan
- https://github.com/dinedal/textql
- https://github.com/dbohdan/sqawk
(disclaimer: vnlog is my tool)
Here people emulate formatted and filtered ps(1) using GVariants and a bunch of CLI tools:
https://blogs.gnome.org/alexl/2012/08/10/rethinking-the-shel...
Here people use SQL to query and format data right from the shell:
https://github.com/jhspetersson/fselect
https://github.com/facebook/osquery
Also, libxo is a library that allows tools like ls(1) in FreeBSD to generate data in various formats (e.g. JSON):
https://wiki.freebsd.org/LibXo
(edit: formatting)
- https://github.com/adamwiggins/rush
- https://github.com/xonsh/xonsh
Its amazing that people still try this nowdays that pwsh solved it for all.
Those who fail to understand powershell are condemned to recreate it poorly.
It'd be great for GNU to create a standard for native structured output (as well as a converter tool like the one in this post), then have other tools be able to do it.
But realistically, pwsh is Open Source, runs just fine on Unix boxes and does this now.
How? `ps | kill node`. No pgrep hack because ps output a list of processes, not a line of text. As a Unix person Windows Terminal and pwsh is where I spend most of my day.
This is common misconception. Posh allows both verbose and shorten styles via various mechanisms - command aliases, parameter abbreviations and aliases, proxys, pipeline settings for objects etc.
If the above characterisation is right, it is a middle-ground between Powershell and traditional methods.
Also, this is not introducing a new shell language.
Isn't it better to forbid them? Presumably you are saving the space for further extensions, but this is allowing readers to interpret them as '?'
Similarly what is the rationale for interpreting control characters is '?'? Instead you can ban them, with the possible exception of treating tabs as spaces.
You could have the format definition forbid these characters; and then in the section about Postel's principle have "If you choose to accept forbidden characters, you MUST treat them as '?'".
I'm not attacking your overall idea, btw. I've just given this a bunch of thought myself, and the design space is very tricky. My current approach would be to use ASV (ascii codes 27-31) and abandon 'cat'-based readability in favor of a 'vcat' which gives you a better visual representation. Of course that has its issues too.. :)
This resource-wasting antipattern pops up over and over again.
Also, logs are message-oriented entries and serializing them as discrete, lengthy files is insane.
Structured data should stay structured, say a time-series / log-structured database. Destructuring should be a rare event.
Caveat, my Plan 9 experience is mostly theoretical.
> Aligns the data with the headers. This is done by resizing the columns so that even the longest value fits into the column.
> ...
> This command doesn't work with infinite streams.
Does this do nothing with infinite streams or does it do a "rolling" alignment?
Even with an infinite stream you can keep track of the max width seen thus far and align all future output to those levels. It'll still have some jank to the initial alignment but assuming a consistent distribution of the lengths over time it'd be good enough for eyeballing the results.
I was thinking about adding a 'trim' command that would trim long fields to fit into the default field size.
Have you considered having a way to render output in a graphical toolkit?
See for example:
https://github.com/dharmatech/PsReplWpf
which renders PowerShell output in WPF presentations.
uxy ls
uxy reformat "NAME SIZE"
uxy align
1) Developing a Linux command-line utility: an article I wrote for IBM developerWorks:
https://jugad2.blogspot.com/2014/09/my-ibm-developerworks-ar...
Follow links in the article to go to the source code of the tool described in the tutorial, and the PDF of the IBM dW article.
2) My comment, here:
https://news.ycombinator.com/item?id=19564706
on this HN thread:
Ask HN: Looking for a series on implementing classic Unix tools from scratch: