Using AWS Lambda to call and text you when your servers are down (opens in new tab)

(thisdata.com)

160 pointsnhm9y ago53 comments

53 comments

From a engineering point of view this is really cool, but as an ex-sysadmin I feel that I need to reiterate and emphasize something that is alluded to in the second paragraph.

Too many things can go wrong and you are all around better off outsourcing this to something like Pingdom. You don't have sufficient levels of reliability, you aren't dual homed across twilio and another phone system. Maybe the cause of your outage is that AWS is having issues. Now your site and your monitoring is down.

Much better to outsource to people who obsess over doing this right and making sure they are properly redundant.

nhmOP9y ago

> outsource to people who obsess over doing this right

Completely agree! I often have to fight that "I could just build that myself" mentality, which glosses over the points you made so well.

avitzurel9y ago

This and that!

It's the same as "Twitter clone" with just posting messages with 140 char limit and "build a blog in 15 minutes".

Alerting over a downed website are is sorta like a glacier, there's so much under the surface, if you just see the surface you're missing out.

1. Multiple locations 2. Multiple check intervals 3. SMS/email provider switch on fail 4. Auto recovery of your checkers 5. Multiple providers with a single storage.

melvinmt9y ago

> Now your site and your monitoring is down. Much better to outsource to people who obsess over doing this right and making sure they are properly redundant.

You make valid points about redundancy and levels of reliability but keep in mind that even Pingdom can go down: http://royal.pingdom.com/2016/10/24/ddos-attack-affects-ping...

user59944619y ago

Chances are that pingdom won't be down at the same time that your site is down.

Diversify to avoid cascading failures ;)

imtringued9y ago

With your own solution you will likely encounter the same problems that pingdom faced including this one. The benefit of a service like pingdom is that they already solved those problems for you or if they haven't you don't have to waste time solving them yourself. It's not very efficient if everyone solves the same problems over and over again.

dharma19y ago

Use 2 or more providers. Signing up takes a minute or two and there are free alternatives

IgorPartola9y ago

My favorite issue recently came up with a Django app of mine which was set up to email me when a request errors out. Turns out, when I switched which server it ran on I misconfigured the email settings and one of the errors was caused due to the inability to send an email. Thankfully it only took a few days to figure this out.

teddyh9y ago

We’ve had issues with Pingdom at work. We don’t use them ourselves, but we host web sites, and some customer of ours used Pingdom to monitor their web site hosted on our servers. The customer would complain to us about downtime reported by Pingdom, but we would read the logs and find everything OK, with multiple successful accesses from other people during the time which Pingdom reported our customer’s site as being down. A huge pain.

snom3809y ago

Doesn't services like Pingdom support multiple ping locations? If all of those fail, there's a very high chance there's an actual problem, if not with your server then with your (ISPs) connectivity.

2 more replies

tjholowaychuk9y ago

Yep, plus most engineering time is worth at minimum $60+/h, which would pay for a year or more with most of these services.

vacri9y ago

On the other hand, it's 'set up once and it just keeps chugging along', and isn't Yet Another SaaS To Manage.

Also, if you want a 'proper' ops alerting SaaS, you're looking at something along the lines of $50/user/mo or $15/server/mo, neither of which is trivial.

1 more reply

falcolas9y ago

And I'm sure Lambda will never go down. Right? Right??

(It has. Completely and silently stopped processing against Kinesis queues for a few hours recently. Guess what AWS Step is built on?)

jlgaddis9y ago

Well, sure, of course it will, but I don't think Nick is advocating replacing a complete, full featured monitoring system with this.

It could be very useful to, for example, keep an eye on your monitoring system. At $work, we have a pretty extensive monitoring system that we've built out. We use an external service to watch over the monitoring system, though, to alert us of any issues with it that we haven't otherwise caught.

Besides, like he said, it's "fun" and kinda neat.

cddotdotslash9y ago

Of course it can go down, and you can have CloudWatch alerts to alert you about that. But so can your Nagios server sending pings go down or the fancy SaaS you signed up for.

robinson-wall9y ago

Did you just suggest using a third AWS service to let you know if the second AWS service monitoring your first AWS service goes down?

1 more reply

thisone9y ago

don't worry, the status was green the entire time...

falcolas9y ago

Yes, it was indeed green the entire time. Of course, AWS is almost always green, so long as something is up...

tjholowaychuk9y ago

I wrote Apex Ping (https://apex.sh/ping/) for those who want more features and/or don't want to waste the time to save a few bucks :D.

gingerlime9y ago

Apex ping is great, but I'm still waiting for SMS / Twilio integration (hint, hint, nudge nudge) :)

postila9y ago

I now use okmeter.io and really happy (especially for nginx and Postgres monitoring). They improve it constantly, and installation took just a couple of minutes. SMS/email/slack notifications work great (however, for slack, I needed to put a webhook).

tjholowaychuk9y ago

:D I wish SMS wasn't so awkward, you're pretty much forced to have a credit system since it's so expensive. I'll probably still do it at some point. Makes it awkward for the customer as well if you have to babysit the credits

2 more replies

paps9y ago

Thanks for this. I would use it but you only do HTTP requests, right? It would be great if you could also do a true ICMP ping (like the name suggests!)

tymm9y ago

I wrote something similar in bash and put it into a docker image: https://hub.docker.com/r/simplepush/alerta

Just running this docker image on a server you want to monitor is enough.

Instead of Twilio it uses Simplepush (https://simplepush.io).

cyberferret9y ago

Simplepush looks like a cool service - thanks for the heads up. It seems that it accomplishes the author's main need - that for a constant buzzing which needs to be picked up and dealt with.

EDIT: Just seen that it is Android only! :-/

ubercow9y ago

If you need something similar that works on iOS (and Android), take a look at https://pushover.net/

I use it for some personal automation scripts that might need to get my attention if something goes wrong.

dmourati9y ago

Isn't the better plan to use Lambda instead of your servers?

justinc86879y ago

I use https://aremysitesup.com/ and I've found it really helpful as it one of the few inexpensive services I've found that will CALL me if things are down. SMS is nice, but I use the do-not-disturb feature on my phone in the evenings, and at least on iOS, the only way to punch through that is with a call from a number on my favorites list. This meets that need very well and I've found the service to be quite spot on alerting me (both when I had one instance of things hitting the fan, but also during scheduled maintenance). I'd highly recommend.

intrasight9y ago

The advantage of this type of cloud solution over a one-size-fits-all cloud service like Pingdom (which I use) is flexibility. You can configure cloud agents to perform nearly any task you can envision.

gravypod9y ago

I don't know if everyone knows this but you can make texts using email.

Most providers have SMTP gateways for SMS services. Verizon runs @vtext.com

gtaylor9y ago

Just keep in mind that these aren't incredibly reliable across the board. Others have very low or arbitrary autoban or blacklist policies. I eventually caved and paid Twilio to hassle with SMS logistics for me, rather than deal with the weirdness.

tomascot9y ago

I did this for non-critical ops, it's just a foward to my phone's email address, really simple. The problem is that not every company is reliable or even has that email

social_quotient9y ago

Note: instead of dynamodb for the lookup mentioned at the bottom, maybe consider https://aws.amazon.com/athena/ for an s3 query

illumin89y ago

You could literally just store a .CSV file in S3 with a table that has the on-call schedule in it and run SQL queries against Athena that would be cheap... you'd be querying a few KB, but DynamoDB is probably better for this use case, honestly. Athena is great for scanning huge datasets very quickly.

cagataygurturk9y ago

Route53 Health Checks & SNS can send sms message without any Lambda involved.

theparanoid9y ago

I've used montastic.com for years. 2min setup to fire and forget.

j / k navigate · click thread line to collapse

53 comments

Dobbs9y ago

From a engineering point of view this is really cool, but as an ex-sysadmin I feel that I need to reiterate and emphasize something that is alluded to in the second paragraph.

Much better to outsource to people who obsess over doing this right and making sure they are properly redundant.

nhmOP9y ago

> outsource to people who obsess over doing this right

Completely agree! I often have to fight that "I could just build that myself" mentality, which glosses over the points you made so well.

avitzurel9y ago

This and that!

It's the same as "Twitter clone" with just posting messages with 140 char limit and "build a blog in 15 minutes".

Alerting over a downed website are is sorta like a glacier, there's so much under the surface, if you just see the surface you're missing out.

1. Multiple locations 2. Multiple check intervals 3. SMS/email provider switch on fail 4. Auto recovery of your checkers 5. Multiple providers with a single storage.

melvinmt9y ago

> Now your site and your monitoring is down. Much better to outsource to people who obsess over doing this right and making sure they are properly redundant.

You make valid points about redundancy and levels of reliability but keep in mind that even Pingdom can go down: http://royal.pingdom.com/2016/10/24/ddos-attack-affects-ping...

user59944619y ago

Chances are that pingdom won't be down at the same time that your site is down.

Diversify to avoid cascading failures ;)

imtringued9y ago

dharma19y ago

Use 2 or more providers. Signing up takes a minute or two and there are free alternatives

IgorPartola9y ago

teddyh9y ago

snom3809y ago

Doesn't services like Pingdom support multiple ping locations? If all of those fail, there's a very high chance there's an actual problem, if not with your server then with your (ISPs) connectivity.

2 more replies

tjholowaychuk9y ago

Yep, plus most engineering time is worth at minimum $60+/h, which would pay for a year or more with most of these services.

vacri9y ago

On the other hand, it's 'set up once and it just keeps chugging along', and isn't Yet Another SaaS To Manage.

Also, if you want a 'proper' ops alerting SaaS, you're looking at something along the lines of $50/user/mo or $15/server/mo, neither of which is trivial.

1 more reply

falcolas9y ago

And I'm sure Lambda will never go down. Right? Right??

(It has. Completely and silently stopped processing against Kinesis queues for a few hours recently. Guess what AWS Step is built on?)

jlgaddis9y ago

Well, sure, of course it will, but I don't think Nick is advocating replacing a complete, full featured monitoring system with this.

Besides, like he said, it's "fun" and kinda neat.

cddotdotslash9y ago

Of course it can go down, and you can have CloudWatch alerts to alert you about that. But so can your Nagios server sending pings go down or the fancy SaaS you signed up for.

robinson-wall9y ago

Did you just suggest using a third AWS service to let you know if the second AWS service monitoring your first AWS service goes down?

1 more reply

thisone9y ago

don't worry, the status was green the entire time...

falcolas9y ago

Yes, it was indeed green the entire time. Of course, AWS is almost always green, so long as something is up...

tjholowaychuk9y ago

I wrote Apex Ping (https://apex.sh/ping/) for those who want more features and/or don't want to waste the time to save a few bucks :D.

gingerlime9y ago

Apex ping is great, but I'm still waiting for SMS / Twilio integration (hint, hint, nudge nudge) :)

postila9y ago

tjholowaychuk9y ago

2 more replies

paps9y ago

Thanks for this. I would use it but you only do HTTP requests, right? It would be great if you could also do a true ICMP ping (like the name suggests!)

tymm9y ago

I wrote something similar in bash and put it into a docker image: https://hub.docker.com/r/simplepush/alerta

Just running this docker image on a server you want to monitor is enough.

Instead of Twilio it uses Simplepush (https://simplepush.io).

cyberferret9y ago

Simplepush looks like a cool service - thanks for the heads up. It seems that it accomplishes the author's main need - that for a constant buzzing which needs to be picked up and dealt with.

EDIT: Just seen that it is Android only! :-/

ubercow9y ago

If you need something similar that works on iOS (and Android), take a look at https://pushover.net/

I use it for some personal automation scripts that might need to get my attention if something goes wrong.

dmourati9y ago

Isn't the better plan to use Lambda instead of your servers?

justinc86879y ago

intrasight9y ago

gravypod9y ago

I don't know if everyone knows this but you can make texts using email.

Most providers have SMTP gateways for SMS services. Verizon runs @vtext.com

gtaylor9y ago

tomascot9y ago

I did this for non-critical ops, it's just a foward to my phone's email address, really simple. The problem is that not every company is reliable or even has that email

social_quotient9y ago

Note: instead of dynamodb for the lookup mentioned at the bottom, maybe consider https://aws.amazon.com/athena/ for an s3 query

illumin89y ago

cagataygurturk9y ago

Route53 Health Checks & SNS can send sms message without any Lambda involved.

theparanoid9y ago

I've used montastic.com for years. 2min setup to fire and forget.

j / k navigate · click thread line to collapse