No, it's not. It's 2020, we need to stop it with the stringly-typed, footgun-filled madness that is bash scripting. We deserve better than this.
Unless I'm missing something, this README example already contains a bug:
template $(dirname $0)/motd.template > /etc/motd
Personally I have to say that I started writing more bash scripts again recently, I even wrote one that interfaces with the GPIO ports on a Raspberry Pi. Works like a charm, is concise and (I find) also quite readable: https://gist.github.com/adewes/7a4c20a5a7379e19d78ba54521d3d...
But then again I'm also a person who enjoys Perl, so maybe it is time for me to be banished to a lonely island :D
Jinja2 can produce HTML pages, but it was never exclusively designed for that. In fact one of the earliest uses of Jinja2 was generating LaTeX.
Bash is far from perfect, but it's readily available at many people's fingertips, and it gets you to some solutions really fast. This is cool.
If you have two type systems and you don't want them to interact, gluing them together with stringly typed bash is a good way to keep their scope small. As a user interacting with multiple programs by different authors, that's a very common use case.
Giving your type systems boundaries is a good idea. The alternative tends towards one-type-system-to-rule-them-all, an idea that Russell thoroughly explored in the early 1900's, Godel proved to be unmaintainable in the 30's, and that everybody except the Java and C# crowd has since abandoned as a dead end.
I haven't yet taken the time to play with it but I'm hoping www.oilshell.org addresses some of the warts.
I do think there are a few issues that make bash non-robust for this kind of task. If you only look at this through the lens of the "type system", you have a good point about gluing views together, but I think "just strings" is not enough typing. I think I and the industry have had better luck with a base system more like JSON, instead of "just strings". It provides enough structure so that there's a straightforward way to turn wire protocols or other external data into your Java objects or whatever, let dynamic scripting languages handle it in their native base types (which often map pretty well to what JSON offers, while bash really doesn't--note that the author here isn') and are fairly reasonable to manage on their own without special tooling.
However, that aside, even if "just strings" were fine for everything, the fact that bash is designed as an interactive shell with barewords for strings really exposes a lot of gotchas for handling data from different sources, just from the sometimes-diabolical problems with quoting and I/O. In the absence of actual function return values, capturing output and storing a process exit code, which might sometimes be the last process's exit or maybe something inside the pipeline if you've set the right global options, is error-prone. It's not robust to hope that the only output coming (on two different output streams) constitutes a complete and correctly-anticipated return value from a subprogram. Errors associated with empty variable values are frequent for this reason.
Secondly, bash lacks a lot of features which you need external programs to provide, and the dependency management glue between them is not good. You'll need find, ls, sed, awk (and you may not remember the days with multiple significant Unix platforms and differences in behavior between gawk, nawk, awk-of-different-flavors), cut, sort, ps (/usr/bin/ps or /usr/ucb/ps) all stored in different platform- and environment-specific locations and exhibiting different behaviors without a clear way of constraining your dependencies or even asserting them correctly. In the modern world, you can add to that list curl, jq, parallel, and other shell tools. Fairly common tasks like YAML handling is even more of a clusterf. The fact that these tools were all designed to work for their own reasons and not as extensions to a common language isn't their* problem, but it becomes yours if you're trying to deliver a robust and serious piece of software with it. You'll do a lot of DIY around discovering their edge cases and handling them.
There are a number of other problems but I think they're lesser in scale, like the lack of a standard way to deliver and use extensions. It's hard to do safe file access. Bats is a pretty good testing framework but no one (including the author of this script) seems to use it. But what I don't really see are many advantages over a regular scripting language, for something like this. The fact that "bash is already installed" is only an advantage as long as your script isn't going to do anything interesting; once it starts doing interesting things, it needs to worry about all kinds of other things being installed.
That said, I do think it's a little unfair to tear the author apart over something that really shouldn't be any more ambitious than a cron job doing "git clone https://config-repo/config.git && config/manage.sh". The fact that they call it an "alternative to Ansible" is a little eyebrow-raising, but I'm sure it is: to them. The question isn't really whether it does what Ansible does, but if you can manage a machine's configuration with a shell script, and of course the answer is yes. But the answer to the question, "Is bash a fine language to write a general-purpose configuration management system in?": the answer is no.
That said, features such as word-splitting are terrible footguns that only exist because of the burden of backwards compatibility. Hackernews, what other languages do you suggest as Bash replacements that doesn't have as many opportunities to shoot yourself on the foot?
This is an area that I never found something I truly liked. A lot of people suggest Python for this kind of task but when I have a small script that is mostly doing pipelines and command substitutions with "$()" then the python version with subprocess is much more verbose.
... I guess pure bash might be better.
while [[ "$#" -gt 0 ]]; do
I do a non-trivial amount of shell scripting, and there is only one warning I disable in shellcheck on a general basis (SC2016).
The author of this script clearly doesn't like quoting where the underlying value is guaranteed not to include special characters (or space), although this is somewhat inconsistent with the fact that he's using quotes in several patterns where it's not needed (eg. left or single operands in bash conditionals, or in variables initialization).
On can consider always quoting a defensive practice. In a fragile language like Bash, I personally think defensive practices are very important, so the first impression I get from a Bash program not adopting them is "not great".
Then come up with something better.
It will need the same or better ease-of-use for non-programmers, should function as a shell (so that their everyday use also teaches them scripting), with the same flexibility that lets people who don't care about typing or other fancy programmer stuff and just need to get work done.
If your first instinct is to say "python", you don't understand the problem.
Bash programming is not easy and full of loads of gotchas.
Useful for running multiple commands but when you start to need to do programming things like conditionals/loops/variables then using an actual programming language makes more sense.
Bash "just works" with some very well understood limitations. Every one of these "better" solutions introduces 10x the complexity with 10x the limitations (always undocumented).
After spending dozens of hours configuring them thanks to terrible documentation, generally taking multiple support tickets (if there's anyone to talk to at all), we get it working. Then, after using it for a period of time, they fall flat on their face in far worse ways than bash.
Granted I'm only managing hundreds of servers, not thousands or more, but more often than not a
for i in {000..199}; do ssh machine$i "some command"; done
Edit: As long as linux/unix maintains the "everything is a (plaintext) file" approach, I have trouble picturing a better solution than text/string based interface like Bash. You can build more front-end complexity on top of it, but you're always going to end up back in the same place.
"[W]e need to stop it with the stringly-typed, footgun-filled madness that is bash scripting" is pretty categorical.
I consider myself an accomplished system engineer and software developer. I use bash all the time. Not writing bash scripts is like saying not to use the terminal, it's just cacophony.
What have we done to deserve better? Have you seen a community of people working on a bash scripting replacement lately? Has it hit HN? Does it have the same confluence of features, convenience, portability, simplicity, and utility that bash scripting has? If so, why aren't we using it yet?
The fact is, we all use bash because it's the best we have for what we really want to do. Bash is what we deserve. 31 years of "good enough".
Ansible is written in Python. You could make the same argument.
One of the things that I love about Ansible's model is that we never need an agent on the host. Once you configure something with Ansible, there's no artifact of the configuration left on the machine.
So I created `apply`[0].
This small bash thing pushes bash scripts called 'units' through ssh to execute through an uploaded 'run' script:
./push units/update units/sshd units/ssh_authorized_keys root@foo.example.com
./push groups/base groups/ruby units/dockerd root@foo.example.com
./apply hosts/foo.example.com hosts/bar.example.com
It was a deceptively simple experience, immediately accessible, trivially enabled literate coding, and overall extremely useful both to set up and maintain those VPSes as well as creating dev environments, or local VMs to test a e.g one-shot unit performing a change or migration.
[0]: https://gitlab.com/adhoc-gti/apply
[1]: https://github.com/lloeki/apply (personal fork)
With an agent, it’s applying all the time and this doesn’t happen.
We actually moved to salt from ansible and we’re happy..
That gets you the standard advantages of agentless setups, including not requiring the runtime of your config management tool to be everywhere, being able to reprovision ephemeral + immutable cloud resources, and being able to centrally report errors, without any more risk of configuration drift or bitrot.
Machines sitting behind VNets, governed by security review boards makes getting agents approved a bit tricky
I still love Ansible, especially for small quick things. As with many tools, though, it's not the only one I reach for any more.
Ideally, I’d like to see a configuration tool that uses Ansible’s SSH push model but without the Python dependency.
There was a pen-test tool designed on these lines once called Mosquito Lisp, which as far as I can see evolved into something called WaspVM: https://github.com/swdunlop/WaspVM I have no idea if it's in use, alive, or even working these days.
Is anyone working on this side of the problem and I've missed it?
Use python over SSH. Super fast to begin, if you can ssh into the server, you can run the equivalent of shell commands (subprocess calls in python) remotely. With python, you can abstract and reuse, with the scriptability of bash, but higher level niceties of the language, libraries and toolchain (linting, formatting, autocompletion, imports).
Can it pull the latest version of your app, build it on your server? Yep. Restart daemons? Update libraries (npm, python, etc)? Update system packages? All work fine.
Can it configure a vanilla server from scratch? Yes, it's helpful to bring in a higher level reusable library like https://github.com/fabtools/fabtools helps with that.
When does it hit limitations? More complex orchestration where you have multiple servers (and variations of them) with configuration to talk to each other over networks. Eventually you get to the kind of setup where having a tool like Ansible / Salt / Puppet / etc. makes sense.
For a basic PHP/python/ruby/node site that's just running on a single cloud server? A declarative config manager would be overkill for me.
This literally describes ansible. In fact, ansible one-ups this because you can specify raw commands if python is not installed remotely [1].
Ansible provides lots of modules, you don't have to use any of them. I have plenty of 1-off ansible playbooks that I don't care about idempotency that are just a bunch of 'cmd' statements. It's a very flexible tool.
1: https://docs.ansible.com/ansible/latest/modules/raw_module.h...
Some of the modules seem pretty pointless as well, like the synchronize files one; why not just use rsync directly? And while you're at it, just SSH to servers from shell! Sure, you would need something similar to the inventory, but that wouldn't be too difficult to come up with...
I think investing time in building a library of idempotent shell scripts instead is the way to go personally.
It's only non-idempotent if the shell command runs every time, and the shell command itself isn't idempotent.
The reason for some of the "pointless" modules is that they integrate into Ansible better. I.e. for synchronize, you can run Ansible with the "--check" option, which won't change anything, but will report back what would be changed if you actually executed it. Running rsync in a shell command doesn't have that option because Ansible has no idea what that command does.
My bigger issue with Ansible is that the YAML is annoying. The looping behavior is frustrating, variable precedence is weird, IDE autocompletion is generally lacking. I wish they had designed a sane way to just write Python code.
GP's description of fabric is slightly wrong - it's not python over ssh, it's ssh in python. You only need python locally, fabric runs shell commands remotely.
https://github.com/ploxiln/fab-classic
We decided that migrating to Fabric 2.0 doesn't worth it.
In my case, I'm using docker-compose.yml files for each machine, and then running:
docker-compose pull
docker-compose up -d
Thanks for pointing this out.
I had a quick glance over this, but for contexts which are defined as remote ssh hosts, I couldn't see where the images are obtained. Are they being built remotely on the host, or are they being downloaded from your local machine over a tunnel or via a docker registry?
My god, no, stop. Funkyness in production equals outages.
For reference, I have an ops background.
I ended up writing my own - and hating that I had to do that - because I couldn't figure out how to use any of the existing options. Their documentation all seems to assume multiple hosts and a separate machine that you run the scripts on.
"Each host periodically fetches the latest version of the inventory to see what roles should it be performing" - that's exactly what I want.
This isn't a limitation of Ansible, you can run playbooks against localhost.
https://www.consul.io/docs/commands/exec
edit: Also, Nomad would be another choice if you're looking at hashi tools.
https://github.com/skx/marionette/
As of this week it can now pull down docker-images from dockhub, which was something I'd been needing to rebuild my own systems. Early days, and it is as much proof-of-concept as anything else, but it seems robust and reliable in my own use.
Of course I need to add "written in go" to advertise properly!
For me it was a learning experience as much as anything else. I'd probably have stuck with ansible, puppet, or similar established project if I didn't want to learn/experiment.
It replaces a simpler alternative I wrote in the past which used ssh/scp to upload files and run commands:
https://github.com/skx/deployr
Right now I have a git-repository containing per-host config. So I can run something like this via cron:
cd /srv/config/$(hostname --fqdn)/
marionette main.in
curl https://aviary.sh/install | sudo bash
A better way to try a new software is to download the sources, check them, build/install/run them under a low user with lowest required access to your system. Even better, do this in a virtual machine.
> curl https://aviary.sh/install | sudo bash
I would really try to avoid this, especially when it's targeted for sysadmins ( I guess DevOps nowadays ).
What's the big deal of having this instead :
> $ wget https://gitlab.com/dchester/aviary.sh/-/archive/1.3.2/aviary...
> ./aviary-install.sh
wget https://gitlab.com/dchester/aviary.sh/-/archive/1.3.2/aviary-install.sh
less aviary-install.sh
./aviary-install.shOn non-production machines the first one is easier.
Maybe, I hope maybe, it's not a real portal that we pass through, but it does seem so.
You could probably keep things simple for longer if you issue everyone with a new fidget spinner monthly. A lot of what I see is just people who can’t resist fiddling with stuff that works until it doesn’t.
cat <<EOF > /etc/motd
"Ever make mistakes in life? Let’s make them birds. Yeah,
they’re birds now."
--Bob Ross
EOFThe greater-than redirector fits that description, as it is normally implemented as an atomic write. Note that it does not say anything about optimization or efficiency. Only that the result remains unchanged.
Web developers have bastardized the term and take it to mean many things it does not because of the definition of HTTP GET, even though no GET operation is ever idempotent in the real world.
This will overwrite every time, which is inefficient.
As a plus, Ansible also notifies you which things have changed. This also enables easy dry runs (check_mode in Ansible).
Also didn't realize it wasn't agentless which is pretty critical imo
Well, POSIX sh, ideally. That way you get free portability to ~everything; it's not a great burden to install bash or dash on ex. NetBSD or AIX, but sh is as close to universally preinstalled as you're going to get.
> Also didn't realize it wasn't agentless which is pretty critical imo
Ouch, yeah that outright disqualifies it for me.
https://github.com/skx/marionette/
Early days, but the primitives are enough to do useful things on my own systems..
Will Aviary be able to solve this?
After a new Ubuntu install you can just pull down your dotfiles and run your install script. Aside from some things like ssh keys, gnome tweaks, etc. - this approach would likely get you 90% of the way to set up and would take ~15 mins.
Arriving at this set up would take a little bit of time, and require periodic maintenance, but given a weekend you could have it set up and tested.
Here[0] is an interesting way to store dot files.
[0] https://www.anand-iyer.com/blog/2018/a-simpler-way-to-manage...
It would be even cooler if I didn't need to do anything to do the box besides install openssh-server. I should be able to:
$ sudo apt-get install aviary
$ aviary install my_server 192.168.1.25 # a fresh ubuntu 18.04 install
The solution to CM is to use immutable infrastructure, and versioned immutable artifacts. With these paradigms, state never drifts, so there is never a need for configuration management at all. Everything becomes stable and predictable and you no longer have to maintain a finicky pet.
But how do you bootstrap your systems, you say? The simplest way possible: make a crappy procedural program that bootstraps the very beginning of your system just enough to push versioned immutable artifacts and run arbitrary commands (essentially just "scp", "reboot", "docker pull", "exec"). With cloud-based systems, you shouldn't need to bootstrap anything at all. Build your versioned system images and containers, deploy them, destroy them, re-deploy them.
No offense meant to Aviary, I'm sure there are still legacy systems that require some CM before they can be abandoned, but I really hope people building new systems will abandon them ASAP.
Imagine if this grew to support all of Ansibles features and modules, but did so in bash.
shudders
Yeah, naming conflict with the debugger. Was a simpler replacement for rerun so the shorter command name made sense to me.