The core problems we had with chef were:
• worse than ruby: the chef ruby DSL is like some bastardized, crippled ruby - e.g. ruby_block{}, just uggh
• way too slow & resource intensive: chef itself uses a lot of memory and CPU, has a slow boot time, and does stuff like execute apt for each desired package on each node on each run. this might work fine if you're running on beefy physical/virtual hardware, but not for managing hundreds/thousands of tiny LXC containers that need to be scaled on-demand in seconds
• not self-hosted: chef seemed to have real difficulty closing the loop and being the thing that deployed and configured itself. there's guides online for scripting yourself up to a basic chef setup, but what if you want your chef client to bootstrap with some custom rubygems? back to bash scripting - and then how does that script get on each of your nodes? chef isn't intended to build/deploy itself the way it does the rest of your stuff
We've now transitioned everything to heroku buildpacks + a build server, which create self-contained "slugs" and therefore are self-hosting (i.e. the build server can build itself), and allows us to have a single build/deploy process for everything
"and does stuff like execute apt for each desired package on each node on each run". No you just need to set it up correctly so that it keeps the timestamp of the latest apt-get update, and does not refresh it within the next x hours.
"not self-hosted". Somewhat agreed, although you can easily get around this with the excellent https://github.com/tobami/littlechef
> what if you want your chef client to bootstrap with some custom rubygems?
The build server takes URLs for an app and a buildpack, runs the buildpack to do any dependency fetching/compilation/etc, bundles up the output into a .tar.gz file that contains everything necessary to boot up on the standard LXC image, and uploads it into the distributed storage. Then when we want to boot 1 or 2 or 10 of that app, we just grab the gzipped "slug" and boot it up on an ephemeral LXC containers (i.e. a single base image + temporary overlay file system)
This system can build a rails web app, a node.js app, other services like mysql/redis/nginx, and even the build server itself
So the bootstrapping is a little tricky, but e.g. the process to run a build may go git -> api -> mysql/redis -> worker -> build server -> ceph/s3, and each of those pieces themselves is built/deployed/managed the same way, which we've found to be a huge win for maintainability
You can always drop to regular, plain Ruby, anywhere you like.
For a non-Puppet and non-Chef user, would someone explain why you'd limit your choices to puppet and chef?
I'm not familiar with Ansible, but Fabric seems more like SSH with a for loop. Puppet and Chef are not about automating the typing as much as they are about ensuring consistency. In my Apache example above, if someone goes and changes a vhost file by hand, that file will be replaced with a good one and Apache will be restarted, even if that file was made by a complicated template.
then you don't care if you're running on Solaris,
Debian, or CentOS.
Fabric seems more like SSH with a for loop.
if someone goes and changes a vhost file by hand, that file
will be replaced with a good one and Apache will be restarted,
even if that file was made by a complicated template.
The resource model was reasonably inspired by Puppets, even if other aspects were not.
I look at Fabric as a deployment tool, fine if you like it, but configuration management requires more things on top.
Just a pointer for anyone else that doesn't have much love for either: I found myself happy and productive with Salt. Fast, simple, and lets me do what I needed with less Byzantine setup: http://docs.saltstack.org/en/latest/
There's also Salty Vagrant and Salt Cloud for local and remote provisioning.
The job of an OS not a human admin. "apt-get install puppet" on the clients and "apt-get install puppetmaster" on the puppetmaster. That's about it.
"Puppet was mind-bogglingly slow"
Was it a pause exactly equal to one DNS lookup timeout? The SSL inside puppet used to get all wound up about reverse DNS matching the forward or whatever exactly. You need working DNS to puppet. If DNS is dead you may as well forget debugging puppet until your local DNS is healthy.
Also its possible to do unusual SSL configurations that can make it a bit slow. Vanilla out of the box should be reasonably fast. Starving a virtual image of CPU can make the SSL slow... a virtual 40 MHZ 386 equivalent is not going to do SSL any faster than a physical 40 MHZ 386 used to.
It might be a simple apt-get command, but consider setting up Chef Server. You're suddenly adding the following to your system: Ruby, CouchDB, RabbitMQ, Java, merb-assets, merb-core, merb-helpers, merb-param-protection, merb-slices, thin, solr-jetty. And then maybe libxml-ruby, merb-haml, haml, coderay. (From http://wiki.opscode.com/display/chef/Installing+Chef+Server)
That's a whole heap of stuff and moving parts I wasn't looking for. Compare that to the above mentioned Ansible or cdist on the lightweight end of the spectrum.
The story may be better for puppet, but with chef, it's pretty much "off to the racetrack" to get the latest and greatest Ruby, Chef, and other deps installed.
Compilation is way quicker in 3.0, averages a couple of seconds per node for me. Client side is parallel, not sure how it could be quicker.
It was only a basic setup of a user account, a directory tree, and ufw. It took over a minute for either a first or subsequent run on a clean install of Natty. Considering how much more config I had to add and too little time to dedicate to investigating and speeding it up, I had to put Puppet aside.
Good to know 3.0 is a lot faster. I'll give it another look in future.
FWIW, front page is http://ansible.cc and there's a FAQ there that explains a bit more.
Docs are http://ansible.cc/docs
I chose Puppet.
http://chester.id.au/2012/06/27/a-not-sobrief-aside-on-reign... (scroll down if you don't enjoy my windbaggery).
The short version is: Puppet best fits the way I think about how such systems should work. Despite ostensibly belonging to the same genre of system, the three of them have subtle but very important differences.
(http://chester.id.au/2012/06/27/a-not-sobrief-aside-on-reign...)
There are lots of things I'd like to improve in Leo that involve chucking its current code base (e.g.: back end storage is XML and thus version control hostile; front end is not a web browser; acyclic should get dropped from "rooted, ordered, acyclic graph"), but until that bit of brilliance dawns over the world, I'll continue to use Leo.
* rsync * http * git clone * hg clone
etc. It is pretty extensible and covers my needs. Beyond that it is a little hard to know.
http://steve.org.uk/Software/slaughter/new.html for the upcoming 2.x release.
If you are managing a small application, Chef may feel heavyweight. However, imho, if you can't manage Service-Oriented Architecture w/out a serious configuration management system, be it Chef or Puppet. There are just too many moving parts.
http://puppetlabs.com/blog/ruby-dsl/ http://puppetlabs.com/blog/gsoc-project-ruby-dsl-for-puppet/
I'm getting all riled up just remembering the hell.
Next time I get to pick, I'll use salt.
Things like Salt or Fabric require direct access to the target servers and won't work in my situation. I'm not sure about how Chef works as it wasn't available (popular) when I was setting this up several years ago.
There is a third and fourth solution to the "Encrypted data bags for puppet" problem. The third, my solution, is to never, ever, store AAA in configuration system. EVER! I do store calls to programs and such or even just data files as a program. I admit sometimes the "program" to get certain passwords is something like "backtick cat somefile backtick" but usually I do better. Those AAA programs/repos are handled much more delicately and securely than a "everything goes" config system that everyone can mess with.
The fourth solution is the implied idea that you'd never rotate AAA credentials on a regular basis and never change infrastructure passwords when someone quits, which sounds pretty funny to me. Hey HN, my mysql root password for a month back in 1998 was: (insert something like line noise here)
Another old time puppeteer observation is everyone has a SSL nightmare eventually and even the mighty GOOG can't help you sometimes. Especially on restoral of backups, or replication of live systems, it can get pretty hairy. Also DNS malfunctions can horribly confuse puppet's SSL occasionally. This is something you'll only hear from an old puppeteer not a short experiment like the article.
I've used both (and cfengine) to varying degrees and would have to agree. Simply using a configuration management tool takes an incredible amount of work off your shoulders, you can't go wrong with either.
0 - http://linuxadminshow.com/2012/10/28/episode-4-puppet/
(edit - I said I interviewed the Chef guy, I meant Puppet)
But, instead of trying to fix what is broken, it seems that a lot of people end up reinventing the wheel and writing their own "lightweight" configuration management system.
As someone that wants to switch to such a system, I believe that it will be easier to learn puppet since the existing code base is just enormous.
On the - sides, I don't find it particularly well documented (it feels like reading the MSDN, where things are self-referentials), and it is quite slow.
Puppet, on the surface, has less 'messing about' to do in order to get things running - but my experience was that I encountered a few issues which basically caused me to spend way too much time mucking about rather than working.
Ideally I would like to use Puppet, I feel like it's cleaner and it seems more 'logical' to me at least - but I only have actual experience of Chef working, so make of that what you will!
Once Chef is up and running, it absolutely flies, which is always good!
Some highlights:
• Python based
• Uses SSH key-based auth
• No central server required
• Only requires SSH and a Posix shell on the client
In our experience puppet is easier to setup and start, chef server stack is HUGE compared to just installing a package for puppet master and certs/knife setup is a PITA.
The DSL is pretty similar except for how they track dependencies and general workflow, where chef is `procedural`, puppet internally builds a call graph based on a declarative syntax which can be hard to track down and understand in some situations.
Regarding cookbooks vs manifests, both have tons of modules around, but like with any plugins/modules software the basic stuff in general is cover, but you will need to get your's hand dirty to get things to work your way and not all manifest/cookbooks are good to use at all..
We don't found compilation and install time to be an issue for us, a complete lamp stack install fully customized takes less than 2 minutes per node.
We have a linux background and didn't find ruby hard at all to hack and mess around to solve some of our issues like custom facters and install some rare stuff like nsis :)
gsl https://github.com/imatix/gsl and synctool https://github.com/walterdejong/synctool
The former is used to generate configurations for various classes of systems and the latter pushes them out over ssh. Works very well.
Know of another big hosting company using Puppet and they love it, for about 10k+ servers.
While on the topic of devops, worth checking out:
See "Relative Origins of Puppet, Chef and CFEngine" http://verticalsysadmin.com/blog/uncategorized/relative-orig...
I mainly gleaned this from him correcting himself about puppet all the time and took this to mean he had a harder time picking up puppet.