Ask HN: Best practices for DevOps pager/on-call schedule?

Megacorp sysadmin here - we do on-call for weekly rotations, though technically anyone can get woken up for the service they own. Weekly is easy to schedule, and it lets our boss know who the contact is for the week (since the schedule is on the wiki).

Never page if it's not an absolute dire emergency. One server out of a cluster - Next Business Day. Failed disk - NBD, unless you're out of hot spares.

As much of your work as possible should be automated to fix it without you having to touch anything. Service down? Try restarting it. Still down? Maybe then consider an email or page.

Other stuff

+Monthly or quarterly sync up meetings between all pager people. Doubly so during super critical times for the business to ensure stability.

+Single email list/PDL for the on-call (+ manager) so they can communicate about issues, as well as be cc'd on vendor support tickets (helps with hand offs)

+FAQ for your services so you don't have to wake the DBA or web admin until you know it's really hosed.

+(Sounds silly, but bears mentioning) During pager hand-off, last week's guy and this week's guy should talk about what happened and if there's anything they should know

Have you considered contracting the ops out? I am in the process of developing a managed service for ops, using a lot of automation, config management etc; basically I install a chef agent on the server, and do everything via code I've written, email me if you want to talk about it anthony@makerops.com. Otherwise, are you doing 24x7 rotations? That will be the determinate along with everyones' geo location, on the best way to set up shifts.

Sorry for the brain dump, but here goes, I guess it depends on how large your company is, and the size of your admin team, along with how many emergencies you expect to have (is there is history you can look at? X emergency per month, etc?). We have 4 sysadmins, and we each have a cell phone, where we can communicate via sms in an emergency. We are all on call, but no one is required to answer, so there is no schedule! Our emergency rate is low, one page every 3-4 months (if that). When your emergency rate is low, by stabilizing your environment, and clearing defining what an emergency is, we have a culture where emergencies are really emergencies, like HVAC outage, uplink outages, etc. Personally, I like to be in the loop, even if I'm not helping. This does not happen often, as I said, so there is not really a burden.

First let me give you some advice about "what an emergency is" and "how we are alerted". You need to define what an emergency is in your company, and notify everyone (with clear guidelines on "how to get help"), so that you limit pages to critical issues, post this on an internal wiki (you have a wiki right?). What is really worth getting woken up and coming into the office for? Alerts are issued like this, nagios alerts go to email, these are not generally emergencies, a couple checks do fire sms alerts, so I consider these email alerts issues for the workday, and I do not check these on the weekend. An automated system scans syslog for alerts that might be an emergency (based on prior experience (i.e. db errors about a disk subsystem, etc)), we also have our apps log emergency issues to syslog, and if one is triggered, a sms goes out to the group. We also have any helpdesk tickets (you have a helpdesk right?), with "emergency" in the title issue a page, users know to do this via the "what an emergency is" wiki page.

When a page comes in, if you can take it, you simply issue a sms "ACK" to the group, this tells everyone that you have Accepted this page, and you are the owner. This helps us load balance across everyone's lives. If you need help, you pull in other people as needed. You also issue a sms "All Clear", when the issue is resolved, this will typically go alongside an email to the group with an issue summery.

This entire system does not need to be complex. Start simple and iterate as needed. There also needs to be a process to find out what happened, do we need more monitoring, additional syslog triggers, etc.

ps. our UPS, HVAC, and security systems can issues pages via sms too as needed. I didn't mention this because it highly dependent on our environment. We also use a modem and landline to issues these pages. We have a linux server with qpage [1] running on it, which issues the pages by dialing a landline at a telco. This allows us to issue pages if our network link goes down too.

pss check out my website @ http://sysadmincasts.com/ where I plan to cover issues like this.

[1] http://www.qpage.org/

There are two pieces of advice that I give all of the startups I work with: - No Spurious Alerts - Don't test code in production, ever.

Keep a release schedule, stick to it, do not deviate. If you can't get your stuff tested before the deadline, thats on you and your peers should not suffer. Make sure that all engineer receive alerts via email, its neccesary to "share the pain" so that people get an idea of what mistakes do. Weekly rotation is probably the best thing to do, that way there is a consistent point person for the week.