Security Hole in Sendgrid (opens in new tab)

You should fault Sendgrid as they specifically have a policy NOT to perform this change of email request (from the article).

Sendgrid can also change their systems so that phone support personnel can NOT perform this change or perform this change with approval from a supervisor.

Sendgrid being in the business they are in should also know that they are susceptible to these types of attacks and what they can lead to (many, many systems which can have password requests sent to email addresses).

>I don't really fault Sendgrid for this

They literally handed over a customer's credentials over the phone. What's more, there didn't appear to be any ID verification.

I don't see how that could be anything but a huge, glaring, faultable security hole. I'm a Sendgrid user, and this is pretty scary.

In a nutshell, social engineering can be countered by proper software & processes engineering.

So if social engineering is possible, blame the software architects.

Maybe in extreme cases changing the email of an account might be needed, but there's no excuse a first level rep was able to do it. Least thing, he/she should've been forced by the system to escalate to someone above her, who has a much lesser chance to screw up.

That would be because sendgrid is lame.

I've recently tried to use their service to send emails, club member newsletters. I've never thought much about bulk e-mails before and I thought it was a "solved problem" by now. Sendgrid are well known so they were my first choice.

During initial testing I found both bugs in the API and missing functionality. I've worked over 20 years with IT and yet sendgrid support is easily on the top-3 list of my worst support experiences, a total waste of time.

We ended up using mailgun instead and so far it looks much better.

Are there any web hosting companies that don't rely on the "send a reset link to your email address on file" model of password resets?

You're right, that model is deeply broken if anyone can intercept those emails (as happened in this case), but it seems unfair to single out ChunkHost for criticism.

I don't see how this was specific to ChunkHost, or how anybody else using SendGrid and supporting password resets through email wouldn't be vulnerable to the same attack... Since presumably password reset emails aren't outside SendGrid's target market, it seems reasonable to suggest that this is a problem SendGrid should try to fix.

Would you apply the same logic to your DNS provider?

SendGrid once went over my head and changed my account settings on behalf of a customer of mine, without even consulting me beforehand. I had a customer with a history of reporting mails as spam (activity reports he requested in a webapp). When you report a mail as spam, the address goes into a blacklist to avoid causing sender reputation issues. After the third or so time of having him ask to get the mails again, then mark one of them as spam, I told him I wouldn't be offering that feature to him any longer. He contacted SendGrid about it directly, and the SendGrid rep actually went into my account and added his address to a whitelist to bypass the blacklist, where I intended it to remain.

I thought that was just the strangest thing, and it didn't sit well with me. Accessing a customer's account when they request service is one thing, but making changes on behalf of a stranger you know has no authority over that account? That was when I moved the last of my apps over to Mandrill.

> "... confirms your suspicion that these people convinced one of our representatives to change the email address on file."

This is the part that scares me. Do they not have auditing in the system where the representatives are able to change the email address on file?

And people complain that Google has bad support! At least you can be certain that there's no way an attacker can get the Google support rep on the phone. There aren't any.

This shows why so-called PR-speak is necessary. Email like this should not be in a conversational style because it can easily be misinterpreted.

Text message or phone verification of details only you should know about the account history, payment methods, etc. As others mentioned, a waiting period during which they try to contact using any means s previously authorised for a response. Compare IP addresses and deny logins from strange countries or origins without further verification, etc. Of course, every measure and countermeasure needs to be justified, since there's an implementation and upkeep cost, but... It's possible to be "more certain," that something is legit or not.

You should review the work NearlyFreeSpeech.NET recently did on customizable account recovery options. It's easy the best I've every seen. It works like this:

1) You decide how valuable the account is, the probability that you will lose access to the account, and the probability that the account will be attacked. 2) You selected the required number of recovery actions, from one recovery action to completely unrecoverable. Possible recovery actions include (copied from NFSN):

* You provide a scanned copy of a government-issued photo ID. * You provide a scanned copy of a statement showing both the most recent deposit and a name and address matching one of your accounts. * You complete SMS verification. (SMS must be previously configured.) * You complete 2-factor verification. (2-factor auth must be previously configured.) * You correctly answer your security question. (Security question and answer must be previously configured, below.) * You use an ssh key to create a file with a specific name on one of your sites hosted here. (Must be previously configured, won’t work if account is empty.) * We try and fail to contact you via your currently configured email address. (This one may take a long time.)

As far as I'm concerned, this is the way it should be done. The public details are on their blog: https://blog.nearlyfreespeech.net/2014/02/28/price-cuts-more...

In this case, looping in the original email address on the SendGrid account before changing to a new one would have kept this from happening. SendGrid's support personnel should almost certainly not be able to change an email address without the change being signed off on through the old address first.

I think these services could use timed release. If you have locked yourself out, the timer starts running. If after 30 days no one has denied the request, access is granted. I'm pretty sure this approach would foil most if not all social engineering.

Is AWS any less susceptible to Social Engineering attack of this type?

Specifically--can AWS support staff grant access to AWS accounts, and if so:

what are their criteria for doing so, and what are the policies in place to ensure those criteria are met, and how are those policies audited?

As a TechStars alum, my company was granted $50k in AWS credits, which were tied to my AWS account[1]. When I left the Company, the CEO was able to get the credits moved to a different AWS account that was company owned, without my intervention at all, even though I was the only account owner.

The fact that he could have credits moved out of the account without any kind of verification from me[2], should be cause for concern.

[1] I should have created a new Amazon account for a group email [2] Obviously the credits belong to the company; they weren't mine to use, so I would have authorized the migration.

Bitcoin is like cash. It's far more valuable than credit card data.

I'm coming around to thinking you should get a secret email address for your sensitive accounts. Treat the string of that email address as almost as important as the string of your password. The problem is that you do have to tell it to people who probably don't share that philosophy to the same extreme.

Your logic is a bit weird.

Sendgrid just experienced this major embarrassment and are currently re-training their staff to avoid it again at all costs.

And you're going to move away from them now?

The attacker got SG to change the email on file, so the notification would just be sent to him.

It's that easy. You can use anything you want.

> Or is it that easy to register a domain with a fictitious persona?

Virtually every registrar allows instant update of whois records via a web interface. I can have Bill Gates be the technical contact on my personal domain in about a minute.

SendGrid's policy (as stated in their first email) is that the support people shouldn't be changing account emails in this fashion. Even if SendGrid had 2 factor auth, who's to say the support guy wouldn't have just disabled that?

Sending email is hard and for most people not a thing that provide them with a proper ROI, because among other things...

- You need to make sure that your email servers IPs are not on black lists.

- That you use DKIM properly

- your multipart mime encoding is correct

- bouncing e-mails are handled...

- take care of scaling & operating the servers

- and so on....

You can spent ( and waste ) a lot of time on this... especially if you are new to email sending..

Is sending emails yourself as easy as changing two lines of code in your app? That's what it took to integrate Mandrill for me. If you think that's as hard as sending email yourself, you are wrong.

Yeah, it's not hard: until you want to reach mailboxes at some obscure providers like Yahoo or Outlook.

Yes, SendGrid sends emails, but there is value add on top of that. (e.g. negotiating with hotmail to ensure your emails get through, webhooks, APIs, statistics, templates, etc.).

Yes, sending an email is easy enough. It's all the other stuff we have services like this for.

Deliverability plays an immense part why people rely upon an ESP.

If you use EC2 or the like your IP address or the entire address block could have easily ended up on a spam list so your email will be blocked. I wish SendGrid was only necessary for people who sends lots of mail but the reality is that no cloy provider can guarantee that email from their IP addresses will be delivered.