Awk: The Power and Promise of a 40-Year-Old Language (opens in new tab)

HN discussion threads for some of the links mentioned in the article:

* Using AWK and R to parse 25TB - https://news.ycombinator.com/item?id=20293579

* Command-line Tools can be 235x Faster than a Hadoop Cluster - https://news.ycombinator.com/item?id=17135841

* The State of the AWK - https://news.ycombinator.com/item?id=23240800

For awk alternative implementations, I'm keeping an eye on frawk [0]. Aims to be faster, supports csv, etc.

[0] https://github.com/ezrosent/frawk

"A good programmer uses the most powerful tool to do a job. A great programmer uses the least powerful tool that does the job." I believe this, and I always try to find the combination of simple and lightweight tools which does the job at hand correctly.

Awk sometimes proves surprisingly powerful. Just look at the concision of this awk one liner doing a fairly complex job:

    zcat large.log.gz | awk '{print $0 | "gzip -v9c > large.log-"$1"_"$2".gz"}' # Breakup compressed log by syslog date and recompress. #awksome

Taken from: https://mobile.twitter.com/climagic/status/61415389723039744...

I never use Awk until last year. I wanted to monitor an embedded device with little more than bustbox and python on it. There was quite a bit of information in the log files (I had already written a custom log file viewer with some highlighting) but I wanted to monitor in real-time. Somehow I decided to use Awk to monitor the tail of the log file and do realtime bar-graphs by generating appropriate cursor control sequences. In the end I had about 50 lines of Awk to upload to the board and run a command to pipe the log into it - very minimally invasive and very informative.

Would recommend learning Awk with some kind of real-world use of your own. BTW it reminded me of using XSLT which I think is another often overlooked "good thing".

> Very few people still code with the legacies of the 1970s: ML, Pascal, Scheme, Smalltalk.

Arguably, the software world would be better off if more people did code with those 1970s languages, than with the ones we are stuck with now.

And that applies to Awk, too. As the author quotes Neil Ormos stating, Awk is well suited for personal computing, something which we have gotten further and further from at the same time as computers have become more distributed. At what point in history have such a large fraction of the human race had the ability to calculate to such an amazing order of magnitude, and at what point in history have such a large fraction of the same human race not bothered with calculation?

Awk is a great tool precisely because it puts quite a lot of expressive power in the hands of an average user on a Unix system. Sure, on a Lisp machine or Smalltalk machine there really isn't the same need for Awk: the systems languages on such machines are safe enough and expressive enough to do what Awk does. But in the Unix context — which is basically what we're all living in, with even the VMS-derived Windows more-or-less adhering to the Unix model — Awk is a godsend.

edit: correct typo

Gawk's ability to extend it with C code is interesting as well, and pretty straightforward.

Here's the source for the fork() extension that ships with gawk...it's ~150 lines or so: https://git.savannah.gnu.org/cgit/gawk.git/tree/extension/fo...

I was able to make a (terrible/joke/but-it-kinda-works) web server with gawk using the extensions that ship with it: https://gist.github.com/willurd/5720255#gistcomment-3143007

My first and only real use of awk was around 1995. I was working at a new job doing embedded software work at GE and we had a lot of documentation in SGML, written/viewed using Interleaf. Interleaf was super slow on the HP-UX workstations we had and iirc search was even slower. I got the idea to convert all the SGML files into a single HTML file and I reached for awk as I had used it for some one-liners previously. I ended up writing an awk script that generated a frameset with one sidebar frame that was a treeish table of contents and the other frame the mondo html file with anchors for the table of contents. It loaded pretty fast in the HP-UX browser and search was really fast.

I've used Python almost my entire career, but started with out the UNIX tools. I never found awk interesting, then took a peek at it recently and understood: this was the pre-perl! it had scripting-language hash tables!

My company mandates Windows but Git Bash has been a backdoor into Unix tools and I've recently learned sed and awk to take full advantage of it. You need to think a bit about your one liners and they'll always feel very hacky, but sed/awk (with a bit of sort thrown in) are an amazingly powerful combination for dealing with all sorts of messy data dumps. In 10 minutes I can craft a one liner that replaces a 2 hours C# console app and runs just as fast. And, surprisingly, I often find it easier to go back months later and understand the messy looking one liner than the nicely formatted, well commented, unit tested console app.

My first job getting paid to program was in awk. Processing log files.

In the middle of that job, my supervsior, you know what, we're doing increasingly complicated things with awk and it's getting increasingly hacky... I've heard that Perl is like awk but better, do you want to learn Perl and switch to that?

And so we did. My thought then was there was little that was easier in awk than Perl, you could use Perl very much like awk if you wanted, you can even use the right command-line args to have Perl have an "implied loop" like awk... but then you can do a lot more with Perl too.

I don't use Perl anymore. Or awk.

i no longer use it but Perl was always the better solution when one thought AWK was the answer.

Perl will do those things where AWK really shines and if the problem got bigger, Perl was easier to deal with.

Learning awk is actually pretty simple. For years I just used the '{print $2}' version to extract fields, but after reading some short book I felt pretty confident of having understood the basics.

Sadly I don't remember which book it was, but this page looks like a good start: https://ferd.ca/awk-in-20-minutes.html

My own shameless plug: https://ketancmaheshwari.github.io/posts/2020/05/24/SMC18-Da...

There's a free awk course here for anyone interested https://www.udemy.com/course/awk-tutorial/

When you have a standardized problem setting like the implicit loop in awk, n alternative to a whole new programming language is a simple < 100 lines of code program generator [1].

This design lets you retain easy access to large sets of pre-existing libraries as well as have a "compiled/statically typed" situation, if you want. It also leverages familiarity with your existing programming languages. I adapted a similar small program like this to emit a C program, but anything else is obviously pretty easy. Easy is good. Familiar is good.

Interactivity-wise, with a TinyC/tcc fast running compiler backend my `rp` programs run sub-second from ENTER to completion on small data. Even with not optimizing tcc, they they still run faster than byte-compiled/VM interpreted mawk/gawk on a per input-byte basis. If you take the time to do an optimized build with gcc -O3/etc., they can run much faster.

And I leave the source code around if you want to just use the program generator as a way to save keystrokes/get a fast start on a row processing program.

Anyway, I'm not trying to start a language holy war, but just exhibit how if you rotate the problem (or your head looking at the problem) ever so slightly another answer exists in this space and is quite easy. :-)

[1] https://github.com/c-blake/cligen/blob/master/examples/rp.ni...

I use awk constantly in bioinformatics, for many of the file formats designed to store genomic data, awk is the easiest tool you can use for processing.

I use awk to auto-generate C header files from other header files. I work with $vendor's huge complicated kernel driver codebase. I need small pieces of $vendor's interconnected header files in order to make kernel calls to their drivers without pulling in all their code.

I only recently learned Awk enough to be useful. But I still don't reach for it when I probably should.

What are the most common cases where you reach for Awk instead of some other tools?

I recently used it to parse and recombine data from the OpenVPN status file. That file has a few differently formatted tables in the same file. Using Awk, I was able to change a variable as each table was encountered, this I could change the Awk program behavior by which table it was operating on.

And let's not forget about the amazing commercial offering of Awk, known as Tawk (by Thompson Automation). To this day some features from Tawk cannot be found in Gawk.

awk is great for data analysis - usually, I start with cut, then move to awk as complexity increases and finally to python.

I find it very unpleasant to read Awk code. It looks as bad as regex to me.

sed is pretty ancient too. I've used it a lot with Docker to alter parameters during builds.

awk is fast and really useful.

It's also generally unreadable.