Webhooks Are Magic (opens in new tab)

With Stripe, the process is:

1) Receive a webhook and record the ID in your database. If it's a duplicate, stop.

2) Ask Stripe for the event matching the ID they gave you. If they reject it, stop.

3) Process the event.

It requires a roundtrip, but it's the only sane way to validate that the event you got is real and that you haven't seen it before without trying to validate a signature (lots of other APIs make you do that, though)

Great question!

For Iron.io, all our requests are authenticated using an OAuth token. So only people that know your OAuth token (and thus could use our API anyways) can use your webhook endpoints.

But there are other ways, too. The most common I've seen is to provide an API endpoint to verify events with (Stripe does this). If you use HTTPS to receive the webhook, and verify it with a request over HTTPS to the API you expect it to be coming from, you're ensuring the request is authentic.

For some webhook styles, it doesn't actually matter. Some people use webhooks to just say "Something happened", without actually saying what. In this style, the API is still responsible for the data and authentication, the webhook just says "Hey, wake up, the API has something new for you."

There isn't a standard way of authenticating the incoming notification yet. I like how Mailgun is doing it, to verify the webhook is originating from Mailgun you need to:

  * Concatenate timestamp and token values.
  * Encode the resulting string with the HMAC algorithm (using your API Key as a key and SHA256 digest mode).
  * Compare the resulting hexdigest to the signature.
  * Optionally, you can check if the timestamp is not too far from the current time.

So, you essentially get 3 extra parameters from your webhook, `timestamp`, `token` and `signature`. Obviously, the API key is the shared secret here between your app and Mailgun.

I came across http://blog.iron.io/2013/01/queue-webhook-events-with-ironmq...

This is pretty sweet, but it seems like it might be more useful for the baked-in reliability to be on the remote end.

That is, the side sending the POSTs should notice that it didn't get a 200 reply and retry some number of times with backoff. Also, the POSTs need to be idempotent (include a sequence number or nonce or something) so that if the "200 OK" gets lost and the sending side retries, you can discard the duplicates on the receiving side.

That's of course a lot harder since you have to convince everybody whose service you want to use that this is worth the time/effort :)

That's why logs from the webhook provider are important. :)

Part of the problem has traditionally been that email servers haven't been as easy to "hack" as web servers have. That was a failing on the part of the developers of those servers.

Thankfully now you have things like Qpsmtpd, James, Lamson and Haraka which can do those things for you. But not many people install those servers. I'd love to see that change (for obvious reasons).

The interesting thing about SMTP is that a mail transfer agent is typically built using queues for both outbound and inbound transmission. Those queues help ensure durability of a message. You could implement webhooks with a transmission queue, retrying for a bit, maybe giving up depending on the nature of webhook. The awesome part about SMTP is that there is so much existing infrastructure in place, although some agents aren't as reliable as others at delivery.

Web crowd continues to struggle with the concept of interrupts, what else is new?

I can't speak for other providers, but our practice is to let users set the number of retries they'd like and the space between retries.

We also are close to rolling out error queues, so when a push notification fails, the message is automatically placed into a queue, to be processed at the user's convenience.

I agree, the need for safe retries (as this is a process without user feedback) calls for idempotency, but that's at odds with the need for a webhook to be possible to be triggered multiple times.

One way around this is to use "tickle" webhooks--webhooks that tell the destination that data has changed, but require the destination to pull in that information themselves. Then the webhook call is still idempotent in practice (at worst, you're refreshing data more than you need to, which is still less than polling), but multiple webhook calls can be made.

> There's no (specified) retry mechanism — and POSTs aren't meant to be idempotent.

The semantics of the data exchanged by POST can be such that the request is effectively idempotent even though that's not an HTTP-level expectation (it's not wrong to have a layered-over-HTTP protocol in which methods are safer than HTTP requires, it would be wrong for them to be less safe than HTTP requires -- e.g., non-idempotent GET.)

If you are sending updated notifications over POST with the actual data then needing to be retrieved by a request the other directions, the first POST is effectively idempotent.

If you are sending the content in the POST without a key to prevent duplication, then there is a problem with retries.

Most senders seem to implement retries if they don't get a 200 response.

I love this idea. I've been wanting to do something similar to this, making webhooks even easier to hook into the Iron.io platform. I don't know if it will make it into HUD, but I'm definitely going to be trying to setup some push-button webhook configurations for popular services that support configuring webhooks from an API.

Firefox is throwing a "This Connection is Untrusted" alert on attempting to load resthooks.org via ssl. Someone should probably look into that.

Just don't disable SSL on your server.

http://baudehlo.wordpress.com/2013/09/06/stripe-webhooks-don...

I'm incredibly excited about them.

As far as I'm concerned, webhooks and callback requests are the same. If we were going to get pedantic, I suppose we could say that for a request to be a callback, it would have to be a response to an earlier request (think asynchronous processing) whereas a webhook is an evented request (it wasn't prompted by a previous request).

But being pedantic is lame. Let's just be excited about both!

Relevant: https://twitter.com/paddyforan/statuses/254375310554968064

There was a somewhat-confusing attempt at standardizing that pattern a few weeks ago: http://resthooks.org

Awesome!

John Sheehan, of Runscope.com (which, by the way, is a super rad tool for API developers) did a great roundup on receiving webhooks for testing: http://john-sheehan.com/blog/a-survey-of-the-localhost-proxy...

> This is basically pub/sub messaging, but over the internet?

Its basically hubless pub/sub messaging over HTTP ("...over the internet", sure, but there is lots of much older versions of that); there's also the with-a-hub version, PubSubHubbub.