undefined | Better HN

0 pointsscrooched_moose6y ago0 comments

On paper I agree with you, but that has not been our experience in practice.

Bash "just works" with some very well understood limitations. Every one of these "better" solutions introduces 10x the complexity with 10x the limitations (always undocumented).

After spending dozens of hours configuring them thanks to terrible documentation, generally taking multiple support tickets (if there's anyone to talk to at all), we get it working. Then, after using it for a period of time, they fall flat on their face in far worse ways than bash.

Granted I'm only managing hundreds of servers, not thousands or more, but more often than not a

    for i in {000..199}; do ssh machine$i "some command"; done

is more reliable, with better error handling, than all of the other solutions.

Edit: As long as linux/unix maintains the "everything is a (plaintext) file" approach, I have trouble picturing a better solution than text/string based interface like Bash. You can build more front-end complexity on top of it, but you're always going to end up back in the same place.

0 comments

6 comments · 2 top-level

notyourday6y ago· 3 in thread

> Granted I'm only managing hundreds of servers, not thousands or more, but more often than not a

> for i in {000..199}; do ssh machine$i "some command"; done is more reliable, with better error handling, than all of the other solutions.

This is absolutely not the case:

From the top of my head: there is a network error that hits while running this against 5% of machines. What is the state of those servers?

If one is managing more than a few machines, one should absolutely use something like Puppet/Ansible/Salt/Chef with a repo to ensure that nothing is ever lost.

scrooched_mooseOP6y ago

And all of those tools (at least the ones I've looked at) use SSH internally as well and are just as susceptible to network errors, except they tend to fail with less feedback than the raw ssh command.

I never take it on blind faith either:

for i in {000..199}; do ssh machine$i "echo new_setting > /file"; done

for i in {000..199}; do ssh machine$i "grep new_setting /file"; done

for i in {000..199}; do ssh machine$i "reload service"; done

for i in {000..199}; do ssh machine$i "verify reload"; done

As I was taught "never make a change without verifying it". All changes are documented as well.

notyourday6y ago

> SSH internally as well and are just as susceptible to network errors, except they tend to fail with less feedback than the raw ssh command.

Maybe some. Puppet, for example, fetches a manifest and applies it on the server. Failure to apply a manifest triggers re-running the manifest. In small installations it will be asking puppetmater(s). Large installation run masterless, where they fetch the repo and run it. I would be highly surprised if Ansible did not have this mode even if it is not a popular way of doing it.

> for i in {000..199}; do ssh machine$i "echo new_setting > /file"; done > for i in {000..199}; do ssh machine$i "grep new_setting > /file"; done > for i in {000..199}; do ssh machine$i "reload service"; done > for i in {000..199}; do ssh machine$i "verify reload"; done

You are running this sequentially and by hand?! What happens when the server is dead and you need a new one? You provision it by hand and go through the stages to get it into the state where it can accept the same cattle command?

scrooched_mooseOP6y ago

That's fair. Maybe it's worth looking into again.

> You are running this sequentially and by hand?! What happens when the server is dead and you need a new one? You provision it by hand and go through the stages to get it into the state where it can accept the same cattle command?

Depends on the change, but generally something like that. Change + Verify. It really doesn't take much more than 10-15 minutes for any given change. We've been at a stable state for about 2 years so changes are rare at this point, other than our security update cycle.

For rebuilds, we pull periodic disk images (monthly + any major changes) and can stand up a new server in about 20 minutes, almost all of which is hands-off.

1 more reply

llama0526y ago· 1 in thread

I have the complete opposite experience.

Places that have hundreds of bash scripts that they manually run in a loop like your example are normally a one off nightmare of random changes in random places with zero consistency, tons of edge cases, and a lot of failure scenarios.

Ansible is very entry level friendly, we've had devs grok it in 20-30 minutes and write their own playbooks. We've administrated thousands of servers at scale. We can even use it to package immutable servers with Packer and Terraform, with mostly expected results.

I'd choose the latter every time.

In the end of the day it's a collection of python modules that people have invested in that let you write steps as plainly as possible in yaml with some light logic involved.

Why reinvent the wheel?

scrooched_mooseOP6y ago

We do realize that's a downside and are very careful about documentation. We only have 2 admins on the system, and both of us are well aware of everything that has been done.

Our experience was Ansible was amazing until it isn't. It worked for a while, then any failures would leave systems in an unknown state that had to be verified via adhoc testing like this anyway.

Then, when Ansible ran into errors, it was just a pain to get working again. Granted we haven't tried in a couple years.

Funny, because to us, Ansible just feels like reinventing the wheel. It's a more complicated tool for pushing text based changes to a group of servers.

j / k navigate · click thread line to collapse

0 comments

6 comments · 2 top-level

notyourday6y ago· 3 in thread

> Granted I'm only managing hundreds of servers, not thousands or more, but more often than not a

> for i in {000..199}; do ssh machine$i "some command"; done is more reliable, with better error handling, than all of the other solutions.

This is absolutely not the case:

From the top of my head: there is a network error that hits while running this against 5% of machines. What is the state of those servers?

If one is managing more than a few machines, one should absolutely use something like Puppet/Ansible/Salt/Chef with a repo to ensure that nothing is ever lost.

scrooched_mooseOP6y ago

I never take it on blind faith either:

for i in {000..199}; do ssh machine$i "echo new_setting > /file"; done

for i in {000..199}; do ssh machine$i "grep new_setting /file"; done

for i in {000..199}; do ssh machine$i "reload service"; done

for i in {000..199}; do ssh machine$i "verify reload"; done

As I was taught "never make a change without verifying it". All changes are documented as well.

notyourday6y ago

> SSH internally as well and are just as susceptible to network errors, except they tend to fail with less feedback than the raw ssh command.

scrooched_mooseOP6y ago

That's fair. Maybe it's worth looking into again.

For rebuilds, we pull periodic disk images (monthly + any major changes) and can stand up a new server in about 20 minutes, almost all of which is hands-off.

1 more reply

llama0526y ago· 1 in thread

I have the complete opposite experience.

I'd choose the latter every time.

In the end of the day it's a collection of python modules that people have invested in that let you write steps as plainly as possible in yaml with some light logic involved.

Why reinvent the wheel?

scrooched_mooseOP6y ago

We do realize that's a downside and are very careful about documentation. We only have 2 admins on the system, and both of us are well aware of everything that has been done.

Our experience was Ansible was amazing until it isn't. It worked for a while, then any failures would leave systems in an unknown state that had to be verified via adhoc testing like this anyway.

Then, when Ansible ran into errors, it was just a pain to get working again. Granted we haven't tried in a couple years.

Funny, because to us, Ansible just feels like reinventing the wheel. It's a more complicated tool for pushing text based changes to a group of servers.

j / k navigate · click thread line to collapse