Ah, now I understand DevOps. Well, the sysadmin who can personally automate his job through programming--this is very productive person.
Why is this server running slowly? The graph shows a spike in disk IO usage. What's the process? The database. What changed at the spike? Adam's production push. Why? The explain plan for the new front page query shows a table scan against a 5m row table.
And done. No sysadmin needed, Adam can now identify and fix his own problems. The management can now watch the graphs of application response time go back down in response to Adam's fix, and everyone's happy.
This isn't theory, it's the example I saw demonstrated at a database convention a few weeks ago. Graphite, logship, nagios, and some development which pulls all of this data together can create some virtually magical systems. Casting devops as a side project for "real" developers limits what could be done to facilitate the creation of real agile development environments.
> Graphite, logship, nagios, and some development which pulls all of this data together can create some virtually magical systems.
Who do you think pulls all that data together? DevOps/Sysadmins.
I build and wrap together the tools that make your job easier. And I take pride in that. I'm also on call so you don't have to be.
Relevant XKCD: https://xkcd.com/705/
It's about carrying out what are traditionally "Operations" tasks (Provisioning, Configuration, Orchestration, Monitoring, ect) using tools and workflows with are traditionally used by Developers (Version Control, Automated Testing, ect)
In large companies this work is often carried out by operations people who have expanded their skill set to include things previously limited to developer roles. In startups this work is often carried out by developers, as in the original rant, who are tasked with maintaining infrastructure as well as developing.
If you see places where production code has to run in WebSphere or Weblogic because it's "enterprise", but want developers working in Tomcat to save money, run away.
This leads to situations where operations starts providing "developer resources" in terms of servers, dba support, maven repositories, etc, that turn into a tar pit where development gets stuck in incomprehensible bureaucracy, and "hack" around the system in order to actually get anything done. And then security or someone else in ops finds some little skunkworks dev system and has a fit because developers don't understand what they're playing with - or worse, they find out about the skunkworks because they get pulled in to fix what developers screwed up, with their own managers howling about who's gonna pay for it...
So to me, the beauty of DevOps is getting dev and ops working together in concert from the start to create a consistent development/production environment, where the same automation can be used to maintain both. This is a win for everyone.
In between putting out fires and occasionally setting up new systems, this is what sysadmins have been doing long before the word "devops" came into being. Anything you had to do twice, write a script for it.
At least that's what I remember from my sysadmin days: automate the shit out of everything so we could spend more time on Usenet, IRC and reading BOFH stories. This of course to counterbalance having to be on call to fix issues at 2am on a Saturday.
If you set up your systems right, they pretty much run themselves.
I've been doing systems contracting for a few years and contracting often turns out to be much better suited for this work than full-time employment (unless you're a company at significant scale).
My typical contract goes like this:
- gain an understanding of their current systems
- document *everything* that matters in their systems
- simplify *everything* that matters in their systems
- organize *everything* that matters in their systems
- secure *everything* that matters in their systems
- automate *everything* that matters in their systems
- monitor *everything* that matters in their systems
- train their engineering team on how to run the systems
The best sysadmins make themselves redundant as quickly as possible.
The best sysadmins put the power of the systems in the hands of the other engineers.
The best sysadmins are well rested because the systems they set up are so damn stable and boring.
How do you know you have a bad sysadmin?
- they are a 'hero' that often rescues the company
- no one understands the systems but them
- they are a bottleneck for deploys
- little is documented
- little is automated
- they work long hours
- they're often frazzled and stressed
For more, see my posts on these topics:
Boring Systems Build Badass Businesses https://devopsu.com/blog/boring-systems-build-badass-busines...
Metrics that Matter for your Systems https://devopsu.com/blog/metrics-that-matter-for-systems/
Even if you think that it entails more stuff, like knowledge about AWS, rackspace APIs, provisioning tools and whatnot, it's not enough. Mostly because in any medium to large company they build tools and processes on top of them. Have fun learning all of that, while you only need to commit your code.
Plus, since I've been for many years a sys admin guy that turned into a developer, I can tell you that there are different skillsets involved, albeit with some overlapping ones.
That's where the devops come in.
I try to make any developer who will listen to reason at all read the Twelve Factor App manifesto.
I have heard too many times that something is complicated then we can automatize it. But automatization has to be debugged, so, if the task is not complex per se, try to find a simpler tool, try to limit your intervention in the config files, try to limit the number of tools etc. But you can make those tradeoff across the whole stack. And suddenly you're balancing between adding a library to the code or a program on the machine, or a function package in the DB. You have a wider view, and you have the pressure to do the right thing because your phone is connected to new relic.
I can code, I do code, but I'm a much better sysadmin, mainly because I've been doing it as a dayjob. You want networking? I'll do that. you want multi gigabyte a second storage arrays? I'll do that. You want monitoring? I'll do that.
You want me to recompile python 3.3, naaa you can do that. I can, but you'll do better at it
From where I came and from my background... RPM/DEB packaging is a SysAdmin job.
And... In a properly built infrastructure, what is recompile python 3.3 if not just repackage it?
If you link to the blog homepage e.g. http://www.ansible.com/blog then in a few weeks time this post on Hacker News will no longer link to the correct content.
We'll soon have a new way for users to suggest better urls, like you did, that is more reliable than a moderator happening to see a comment. I'll announce it as a Tell HN.
I know I came in to DevOps (the word, not the job - I've been doing the job since the mid-1990s) with a strong opinion of what it ought to mean - cross-org cooperation between development and operations. This article has me rethinking that a bit.
It's not quite the red flag that "ninja" and "rockstar" have become, but it definitely makes me wary when I hear it.
Dehaan is pretty much spot on. Knupp's post seemed to first latch onto the idea that devops is about devs doing ops works. To add, I think Knupp should do a rewrite after listening to some devops cafe and food fight podcasts, if he really believes in what he wrote.
It's somewhat an unfair comparison though. If you're the author/creator of ansible you surely have spent a lot of time thinking and understanding what devops means. To be frank, I cannot tell if Knupps post is just link bait to sell his book.
The concept (and value) of the devops mindset & culture is less debatable in 2014 than in the past 5 years. Those who latch onto the totem pole model will simply get left behind while the rest of the industry moves on... or at least that's what they tell me!
I've found this to be the case both in my full-time job (enterprise/healthcare industry) and in my larger side projects. Using some simple tools to make sure both server environments and deployments are uniform/repeatable/known in a good state has given me much more time to focus on actually developing.
Additionally, there are times when certain features (especially surrounding performance and scalability) requires a deeper knowledge of part of your stack. Being a good developer requires you to be a specialist (in your primary language/tools) and generalist (in almost all other areas) at the same time.
This kind of situation, specifically (relative to other work): http://upload.wikimedia.org/wikipedia/commons/b/b8/JevonsPar...
I see all of this as just 'engineering' work, but it's easier to delegate certain things than others. When I ask someone new to configure Postgres for secure remote administration, script or otherwise automate backups, and rough out a schema for some tool we're thinking about, they'll likely accomplish at least a subset those tasks. The remainder is an opportunity for discussion. Someone else might just ask me to build something, and we'll talk about usability when he's done with the physics side of things.
But if I leave, what does that physicist look for in a new hire? The skillset of an 'engineer' is frustratingly broad and vague from a hiring point of view. Getting so-called DevOps out of the way might give another new hire the chance to develop as a programmer, and vice versa. Hopefully one or both of those two will eventually develop into an engineer.
You know what matters? Writing code and making it work. Solving problems. Helping people. If the right way to do that is for me to make a system more reliable, then I'll do my best even if I'm not interested in being "an ops guy". If the best thing for me to do is read machine learning papers, then I'll do that.
The real problem: technology companies have terrible leadership. You know how small towns in the Midwest used to send their genetically unfortunate to San Francisco, where there'd be more social services and a warmer climate? Well, that's what the business world does with its rejects, too. Buses 'em to San Francisco. Except, in tech, they end up making $400k as VPs of HR at startups and get to boss nerds around.
"Agile" won't fix that. Nor will "DevOps". These terms mean too many different things to different people, and ultimately fail to address the real, core issues with technology.
