I would consider myself an expert or at least near-expert in python, but I don't see opportunities to replace my shell scripts with python popping up left and right. Do you open files manually and set up pipe chains with subprocess.Popen? I've done this, and its generally many more LOC compared to the shell original, and harder to read.
On the other hand, I'd consider myself maybe 7/10 skill level with bash, but most developers are only ever a 2/10 or 3/10 with bash/shell. I can't help but think that the average developer's lack of shell understanding is where all these suggestions to convert to python come from. If it's that easy or beneficial to convert to python, then it probably should have been written in python originally.
When you use regexes, letting python use them directly seems to be much cleaner than quoting and passing them to awk/sed/grep or other text processing tools.
Additionally, python seems to have libraries to handle all kinds of input like json and csv.
One thing that python doesn't do as naturally as bash is invoke commands. For python I usually include some functions to run commands passed as a list, and check the return code or return the output. then 'cmd -f $arg1' becomes myrun('cmd','-f', arg1)
import sh
names = sh.ls("./src")
for name in names:
sh.mv(["./src/" + name, "./dst/" + name + ".out"])The main advantage of bash is that it exists on just about every unix machine.
Before Bun, Node+V8 was just too slow to start.
IMHO all scripts should be written in TypeScript...you get typechecking and all the rest of the editing experience. Plus things like Wallaby.js for live coding/testing.
My `.bashrc` now just runs a TypeScript script with Bun. Allows you to use proper structure instead of brittle spaghetti stuff. The number of times I'm debugging silly PATH stuff was too much...
This is true for a number of languages that can also be run without AOT compilation (Go, Rust, etc). This feels like some really weird, incoherent astroturf take for Bun promotion.
This pattern handles lots of styles of options: short and long options (-h, --help), `--` for separating options from positional args, with GNU-style long options (--output-file=$filename).
while getopts :o:h-: option
do case $option in
h ) print_help;;
o ) output_file=$OPTARG;;
- ) case $OPTARG in
help ) print_help;;
output-file=* ) output_file=${OPTARG##*=};;
* ) echo "bad option $OPTARG" >&2; exit 1;;
esac;;
'?' ) echo "unknown option: $OPTARG" >&2; exit 1;;
: ) echo "option missing argument: $OPTARG" >&2; exit 1;;
* ) echo "bad state in getopts" >&2; exit 1;;
esac
done
shift $((OPTIND-1))
(( $# > 0 )) && printf 'remaining arg: %s\n' "$@"- while...; do / if ...; then
- space before ;;
- no space before case match
- use util-linux getopt because getopts doesn't handle long args
- DRY with a die() function rather than exits littered all over
i don't know where you learned English composition, but I can't follow your critique. are you pointing out your preferences (which may well match gnu or bsd standards, i don't know tell me) or things that actually make a difference? Did he break emacs auto-indent? Are you pointing out all the flaws or the correct ways, i'm having to eyeball diff. Use more of your words.
while
do
doesn't seem substantially worse/confusing compared to while;do unless you have some reason?spacing out ;; does it make a difference? or do you like things spaced out?
etc.
Feel free to use your own formatting preferences.
I love working with bash, but it has some footguns that really require an expert hand.
I know bash as-is will always be around for backwards compatibility with the god-knows-how-many scripts out there. It'd just be nice if there were a widely embraced path forward that kept the shell scripting spirit while shedding some of the unintuitive behaviors
It's run thousands of lines of unmodified shell AND bash scripts for years, and provides an upgrade path to a new language.
The new language is described here, and you can try it: https://www.oilshell.org/release/latest/doc/oil-language-tou...
I've gotten some great feedback on it, but it's not stable yet. You can still influence the direction of the language (join us on Zulip)
Latest release gives the status of the project: https://www.oilshell.org/blog/2023/03/release-0.14.2.html
As an anecdote, one thing that was extremely difficult was fixing all the footguns around set -e / errexit in bash.
In shell and bash, "you're damned if you do set -e and damned if yuo don't"
However I believe we have done it: https://www.oilshell.org/release/0.14.2/doc/error-handling.h...
We simply provide options to make sure that every exit code is checked. That's really all. But shells do NOT do this. In fact the standard specifies that shells shouldn't, which has left people mystified for decades.
I've gotten good feedback about error handling and the rest of the changes. Again, you can try it right now if you want to verify that the footguns are indeed fixed.
Shameless plug: I've recently written a blog post about how set -e suddenly ceases to work when using function calls in flow control (what you refer to as "disabled errexit quirk"): https://snails.dev/posts/set-e.html
Your comment made me interested in Oil, thanks!
usage='
[-1]
[+NAME?cmd - XXX Synopsis here]
[s:sample?XXX Sample option with argument]:[MODE]
[v:verbose?XXX Show progress]
file ...'
while getopts "$usage" _
do
typeset "opt_$_"="${OPTARG:-1}";;
done
shift OPTIND-1 > Write bash code to parse the following arguments:
>
> $ ./export.sh --out-file dump.json --db foo.sqlite --include-headers --exclude-rev -vvv -p9000
Here's an example bash code to parse the arguments:
#!/bin/bash
# Default values for optional arguments
INCLUDE_HEADERS=false
EXCLUDE_REV=false
VERBOSE_LEVEL=0
PORT=0
# Parse the arguments
while [[ "$#" -gt 0 ]]; do
case $1 in
--out-file) OUT_FILE="$2"; shift ;;
--db) DB="$2"; shift ;;
--include-headers) INCLUDE_HEADERS=true ;;
--exclude-rev) EXCLUDE_REV=true ;;
-v) VERBOSE_LEVEL=$((VERBOSE_LEVEL+1)) ;;
-p) PORT="$2"; shift ;;
-h|--help)
echo "Usage: ./export.sh [OPTIONS]"
echo ""
echo "Options:"
echo "--out-file Output file path (required)"
echo "--db Database path (required)"
echo "--include-headers Include column headers in output (default: false)"
echo "--exclude-rev Exclude revisions in output (default: false)"
echo "-v Increase verbosity level (default: 0)"
echo "-p Port number (default: 0)"
echo "-h, --help Show help"
exit 0
;;
*) echo "Unknown parameter passed: $1"; exit 1 ;;
esac
shift
done
# Check required arguments
if [[ -z $OUT_FILE ]]; then
echo "Error: missing argument --out-file"
exit 1
fi
if [[ -z $DB ]]; then
echo "Error: missing argument --db"
exit 1
fi
# Print parsed arguments
echo "OUT_FILE=$OUT_FILE"
echo "DB=$DB"
echo "INCLUDE_HEADERS=$INCLUDE_HEADERS"
echo "EXCLUDE_REV=$EXCLUDE_REV"
echo "VERBOSE_LEVEL=$VERBOSE_LEVEL"
echo "PORT=$PORT"Being able to define your argument types and generate parsing code for them using an example CLI invocation feels very natural and expressive to me. I personally found it to be useful for my work.
For my own needs, I rely on a tiny function I wrote, called process_optargs. Example use:
source /path/to/process_optargs
source /path/to/error
source /path/to/is_in
function myfunction () { # example function whose options/arguments we'd want to process
# Define local variables which will be populated or checked against inside process_optargs
local -A OPTIONS=()
local -a ARGS=()
local -a VALID_FLAG_OPTIONS=( -h/--help -v --version ) # note -v and --version represent separate flags here! (e.g. '-v' could be for 'verbose')
local -a VALID_KEYVAL_OPTIONS=( -r/--repetitions )
local COMMAND_NAME="myfunction"
# Process options and arguments; exit if an error occurs
process_optargs "$@" || exit 1
# Validate and collect parsed options and arguments as desired
if is_in '-h' "${!OPTIONS[@]}" || is_in '--help' "${!OPTIONS[@]}"
then display_help
fi
if is_in '-r' "${!OPTIONS[@]}"; then REPS="${OPTIONS[-r]}"
elif is_in '--repetitions' "${!OPTIONS[@]}"; then REPS="${OPTIONS[--repetitions]}"
fi
if test "${#ARGS[@]}" -lt 2
then error "myfunction requires at least 2 non-option arguments"
exit 1
fi
# ...etc
}
It works as you'd expect, with appropriate checks for correctness of inputs, and is compatible with most unix conventions (including '--' and '-' as arguments).If anyone's interested let me know and I can share the code.
Enjoy :)
You must be joking.
It uses a similar style of deriving the arguments from the usage declaration, but it also includes some useful logging functions and is all in one script. There's some more info available on their style choices here: https://bash3boilerplate.sh/
# [ <-- needed because of Argbash
There's also this bit:The square brackets in your script have to match (i.e. every opening square bracket [ has to be followed at some point by a closing square bracket ]).
There is a workaround — if you need constructs s.a. red=$'\e[0;91m', you can put the matching square bracket behind a comment, i.e. red=$'\e[0;91m' # match square bracket: ].
That kind of kludginess is a turn-off.
Why ccouldn't I just go `source argbash _ARG_ --single option o --bool print --position positional -- $@` and get _ARG_OPTION, _ARG_PRINT, and _ARG_POSITIONAL environment variables set based on the commands passed in, without having to dump a hundred lines of code in my script?
Prompt:
> Write a bash script "foo.sh" that accepts a required filename, optional flags for "-r/--reverse" and "-s/--skip" and an optional "-o/--output=other-file" parameter. It should have "-h/--help" text too explaining this.
Then copy and paste out the result and write the rest of the script (or use further prompts to get ChatGPT to write it for you).
Could it be done better if I spent more time on it or was a Bash expert? Absolutely, but for most of the times when I need to do something like this I really don't care too much about the finished quality.
util-linux getopt exists.
I see docopts is ‘the same’ implementation but for shell, have never tried it though.
==== docopt helps you:
- define the interface for your command-line app, and - automatically generate a parser for it. ====
Going to Python/whatever is not quite the same — shell scripts are not as much written as extracted from shell history, so switching to a separate language is a large extra step.
I wish there were more projects on the other side of the spectrum: take the script's self-reported usage string, à la docopt [0], and derive argument-parsing code from that. After all, we have GPT-4 now.