I'm curious to find out what other entrepreneurs think of this situation, where a partner, once trusted, and for which technical foundation has been built upon, now has shown to be acting in bad faith.
Every once in a while, some scammer will send a phishing text message to one of our phone numbers. Here is an example: """ Your Facebook account has been placed on hold for verification. To avoid account suspension, Please visit: https://opensopstat.com/ """
The message will be relayed to en employees cell phone as is what happens with all txt messages. Now Twilio thinks our account was hacked and someone is sending text phishing text messages from it.
The latest time this happened, the account was immediately suspended by an automated system. They did not communicate to us that this happened or why it happened. I had to fill out a support ticket and wait about 3 hours for a response before I even knew what the problem when was. This happened at night, so no one knew there was even a problem until the next morning when business operations resumed and the phones didn't work.
Its bad enough that they shut down the phone system for my entire company because of their mistake, but in order to get the system back online, I have to go through their ticketing process that is only through e-mail, where it takes hours or days to receive a response. If I want to speak with someone on the phone, which probably would have gotten the problem resolved more immediately, I have to pay $1,500 per month for their phone tech support. Obviously this is an unreasonable amount to pay. I don't need tech support, I just need someone to call, explain the situation to, and have them click a button.
We pay them about $600 a month and have been working with them for over 10 years. I understand their profit margins might be thin? But are they really that thin? And if so, there should be a more reasonable phone option. I don't need to speak with an engineer, I just need to speak with someone who can click a button and unblock the account.
Temporarily, I will re-program the system so that it does not forward text message content to my employees phone numbers. Which is fine. But my bigger problem is what do I do now? If they're willing to shut my system down without even giving me a number to call, what else are they going to do to me in the future?
The way in which they have been so cavalier with me is a red flag. And if I'm being honest, it does make me angry how they are willing to so readily damage my company in such a profound way AUTOMATICALLY without giving me a way to talk with them. I understand they may have a big phishing problem and will need to use automated software to help, but it is very reckless to not have this counter-balanced with a reasonable way for legitimate customers to even contact them after the suspension.
Are there other API-driven VOIP options that I should be considering bearing in mind that it would be expensive to re-write the software to work with another vendor? Or is there some way I should be looking to work things out with them?
What do you guys think?
I might be reading this wrong, but it sounds like you take inbound text messages to one number and then send outbound messages with the same content to employee phone numbers. Is that right? If so, that sounds like you're SENDING the spam messages in addition to receiving them. Regardless, it sounds like customer service needs to be improved, though.
If twilio wants us to start filtering spam messages for these cases, they need to give us the API/tooling.
Or let us pre-register receiving numbers through an opt-in process.
I understand Twilio not wanting to be an open relay but the reported solution is not the way.
If they can’t update the integration with spam filtering, then it seems like they should be using COTS software.
I guess it's unlikely that one can make this consent clear to Twilio, though, and if the messages aren't distinguishable then they might still get seen and reported as spam by the employee, with all the bad reputation that can cause for all involved.
I don't think anyone is arguing that Twilio was wrong to detect the message as spam. It's more that an unexpected side-effect of the customer's current implementation had outsized consequences. Similar unexpected friction with Twilio's anti-spam mechanisms are likely to occur again and they will cause outsized damage to this legitimate business or others if Twilio doesn't change their strategy.
The business case of relaying inbound text messages to an employee's phone number should be a very common one and should not warrant an account suspension.
However, obviously allowing such sms forwarding have to be done with care.
This is a bizarre and almost intentionally obtuse interpretation of the ops problem (albeit literally correct).
If I have built an auto-forward between two endpoints that I control (or at least have permission or authority over) I am not a bad actor in any capacity.
A much more appropriate workflow here would be for Twilio to cross-check this "spam" with inbound spam into twilio itself for some other number his account controls.
Which is to say, if a Twilio account originates a suspect message, first check to see that suspect message was sent, inbound, to it before auto-DoSing an entire business. This shouldn't be too tough, especially since these events probably occur in step with each other.
I agree, though there are a few reasons why Twilio is sensitive to this.
I have almost a decade of experience in working with SMS and building apps for messaging, forwarding etc. In my case, I connected directly with SMS aggregators (who are the entities that actually connect carriers to each other), which is what Twilio does as well, so I've had to deal with many aspects of operating directly in this ecosystem.
For these messages that are being forwarded to other phone numbers, the messages are likely going through Twilio and out to the SMS network and physical carriers. I'm inferring this based on OPs comments which makes it sound like the forwarded messages are going to personal cell phone numbers. Even if there was a way to let Twilio know that those people want to receive those messages, there isn't a way to get the carriers on board with this.
In the US at least, the physical carriers have been standoffish with the virtual carriers like Twilio et. I have close to 10 years building similar things and in a company that connects directly to SMS aggregators just like Twilio does.
It's worth noting that long codes (i.e. traditional phone numbers) and short codes have entirely different cost structures. Since carriers get paid for messages on the latter, that's where they want automated messaging to originate. Since Twilio and others offer automation of long code messaging, they have to be very careful not to look like spam generation or consistently have too large an imbalance (i.e. one number generating far more messages than received). Carriers can and will block numbers (i.e. all SMS traffic from your number will be dropped) and, from my experience, they do it silently and with little recourse.
I should have clarified that this is from Twilio's perspective. They don't know that the recipients actually want to receive these messages, which makes them (by definition) not actual spam.
Many of us use twilio and an account suspension is an undesirable scenario.
“Forward unless spam/problematic” would be very nice.
I can confirm that my system did forward the message without any modifications at all. (my employee knew what it was because he knows all text messages from that number are forwards)
It's getting pretty rediculous
But seriously, it can't take a thread on HN to get a way to get support for a paying customer. There needs to be an 800 number they can call which is staffed with humans empowered to fix problems.
While I have no connection to this thread, I am also using Twilio in production and finding out here that if we ever have a problem there will be nobody answering the phone makes me reconsider what we should be doing to keep our business uptime.
I also have a #critical channel anything can post to that always has alerts enabled on my phone so I don't miss anything important.
It actually works pretty well and costs nothing.
https://gdpr-info.eu/art-22-gdpr/
> The data subject shall have the right not to be subject to a decision based solely on automated processing, including profiling, which produces legal effects concerning him or her or similarly significantly affects him or her.
Quote this and maybe it will get you escalated, but who knows? A lot of companies seem to just ignore GDPR entirely.
If you send abusive texts through Twilio, Twilio could lose it's contracts with carriers. I haven't read through the paperwork Twilio makes you sign, but I'm gonna guess the fact that this could happen is in there. Also it sounds like the account wasn't banned, it's ability to send messages was suspended. Because it was being used to send spam text messages. I'm pretty thankful Twilio has automated systems for that
I agree with the other comments that relaying phishing to internal users is probably what they dislike. There, of course, isn't a good solution beyond using some open platform. Your self-hosted IRC server isn't going to cancel your account because someone sent a phishing link, for example. But, nobody will know how to connect to it anyway. Sigh!
HN crowd looks down on sales reps. But a good sales rep will 1) understand your business 2) will look for ways to match your needs to solutions they offer 3) will be your point of contact when things go wrong.
Sales reps are human beings, treat them well, thank them for their time if you don't buy the fancy/expensive enterprise plan.They'll understand. Most reps are young college grads that are building a network for life.
You could be a solo SaaS creator and cheap because it is a side gig and your don't have the money. But still make an effort to nurture those little young reps. One day they'll make the phone call to somebody that will remove that account suspension. That will save your business.
For a "few" dollars a month, maybe not.
But here the OP said it was around $600/month, so that's a fairly substantial chunk of ongoing spend, just to have zero way to contact someone on the phone on a showstopper problem that needs fixing ASAP.
Contrast with more traditional businesses like PG&E or Comcast. Both hated companies for reasons, but even so they are heads and shoulders above these cloud provider companies (google/twilio/etc) in terms of customer support.
I spend way less than $600 with each (around $100 for my small office) and yet I can immediately reach a human on the phone if there's any problem with my electric or internet service.
The thing is, I actually had this when I first started with them. They were so small at the time, that I had one of the founders on the phone once and he told me all about how he studied cloud computing at MIT. But this was over 10 years ago. After that, they had phone support for a long time.
Last few years they changed over to this way where there is no path to resolving a problem like this, which should be easy to solve, in an appropriate amount of time.
0 - The handler doesn't send out via Pushover any message that contains words we're unlikely to use; Facebook is one, for example. If a message isn't forwarded via push notification, it is emailed to the sysadmin list for one of us to manually look at during daytime hours.
I like the Apple Watch integration as I can interact with urgent notifications just on my watch without my phone handy.
Otherwise, you are contributing to a pattern where HN becomes de facto Tier 1 Customer Service, similar to how Twitter was a few years ago. This is already the case for various Google services [1], but I would hope that we don't want to normalize it for every service.
----------------------------------------
[1] A familiar pattern to most of us – Ask HN: Google suspended my account without warning; Googler escalates internally; problem is solved
I posted to HackerNews because I suspected my issue would be seen and taken seriously if I did that. I could tell the ticketing system people were overlooked and only looking at my issue in a very shallow way.
If this had not worked, I would have started using my Stack-overflow account then Reddit. If that didn't work, I found an address of a lady how works for Twilio nearby, I was going to go knock on her door and see if she could put me in touch with someone.
Luckily I was able to get the problem resolved without those additional steps.
I would like to point out that Twilio's ticketing system works well for complex problems that are not time-sensitive. I had an issue a few months back that I think was probably quite complicated involving bureaucracy and multiple carriers and it was resolved in a few days via their ticketing system which was very cool.
Thanks for taking the time to respond. The fact that you responded to this message at all resolves my initial concern, which had more to do with if I needed to change providers because of a cultural shift within your company. I understand that you are dealing with a very large and complex system that is changing quickly and will have problems, sometimes serious problems sometimes. Customer communication specifically is a very difficult and complex problem to solve at scale.
I actually spoke with one of the founders of Twilio when I first signed up. Evan maybe? He told me about how he studied cloud computing at MIT. This was a very long time ago.
Gregg was able to get the problem resolved within about 30 minutes once I reached out to him. He also provided me with a few solutions to prevent the problem in the future.
I understand the complicated problem that that led to this mistake and I think it is reasonable to make mistakes like this sometimes, especially if you're providers are threatening to suspend you.
The main problem that I would like you to solve is the lack of phone number. There needs to be a way for people to contact the company if there is an account administration emergency like this. Even chat would have been fine.
That being said, I did call sales and could not reach someone. If this is due to covid omicron and if normally I had called sales and would have been able to plead my case and gotten them to connect me with someone, I think that would have been fine and this truly is an edge case.
DISCLAIMER: I do work for TWLO, but on a completely unrelated division. My opinion and this message does not represent my employer in anyway. I'm just shooting the breeze here.
It's like if you had a restaurant and you wanted to use Twilio. You might forward text messages sent to the restaurant number hosted on Twilio to your personal cell phone number.
Well, maybe next time you get somebody that implements a standard.
With that kind of behavior (not letting you speak to anybody, the blocking is understandable), it's clear you shouldn't keep their services. So, you have now an opportunity to do it right, and make the next move cheaper.
There isn't really a standard for the interactive voice processing, SIP is a standard, but it's not quite simple to use it compared to Twilio or similar APIs.
However, most of the providers in this space have fairly simple APIs, so it's like 30 minutes of work to do the integration, maybe a little more if you're also receiving SMS or if URLs are particularly hard to use in your language of choice; if you're adding yet another SMPP vendor, that's faster, if you're adding yet another GSMA SOAP vendor, it's probably longer because you'll have to figure out why your XML doesn't work even though it should. Plus whatever it takes to get the account setup. Plus however long to build a way to choose from multiple providers (this part may be a lot of work!), and however long you want to run with limited traffic to see if the new provider does better/worse/same as the old provider.
But then, if they were only sending texts, migrating wouldn't be expensive.
The retail wireless carriers are really driving a lot of this with recent 10DLC A2P changes. In particular, T-Mobile is waving around threats of $10k fines per message for messages they deem to be in violation of their content rules. (Which obviously prohibit fraud and such, but also somewhat-arbitrarily anything relating to marijuana.) The way it's written T-Mobile will fine Twilio, who is supposed to pass it on, but knows they'll struggle to collect that.
Meanwhile, on my personal cell phone AT&T can't even seem to figure out that when they get a message from a Nexmo number that starts with "ATT Free Msg" that they didn't send, maybe they shouldn't deliver it. As a consumer I'm glad someone is trying to squash these scams, but they're breaking more than a few eggs in the process.
I'd echo the advice to get off the SMS channel for notifications if at all possible, unless you're sending enough and spending enough to have named support contacts. The rules are being written for people sending thousands of messages per day. We serve small businesses who send maybe 100 messages per month, and it's been a mess trying to get carriers to recognize that these businesses exist and need a solution that works for them too.
"You package (#US853121) containing the following products: 1. iPhone 13. Cannot be delivered until outstanding duties have been paid. Current outstanding balance: $1.68. More info <sketchiest website ever>"
I get about 20 of these a day. I've lodged multiple complaints. Like, why can't AT&T solve this problem that AOL solved in 1994?
Due to Greg from Twilio seeing this post and providing me a way to reach out, I was able to get the problem resolved.
He spent about an hour on the phone with me today and provided some more information about the issue. A few highlights:
* Twilio has doubled in size since the beginning of the pandemic * Spamming and phishing through text message has gotten a lot more common very recently.
These two things together caused a sort of novel situation with them having to either auto-ban accounts of ban accounts with only a very shallow look and then not having a way for someone to get the account un-banned in a timely manner.
My initial concern with this post was that something had changed within the company culture where they were willing to cull off "smaller" accounts like mine in the $10,000 a year range by treating them very recklessly so that they only needed to work with very large companies which would be more simple and more profitable. This would mean that I would need to change providers or risk them doing other damaging things in the future that I would not be able to predict.
Based on a few things that Greg said in the conversation, I no longer believe this to be the case for a few reasons:
1) They have people like Greg reaching out to people like me at all. 2) In case Greg was not available the next time something like this happened, he provided me the contact information of some other people who were kind of high up in the company and explained that they would be very concerned that something like this was going on where legitimate customer accounts were being suspended.
This changed my interpretation of the situation because Greg's actions communicated to me that this is a temporary problem having to do with Twilio increasing in size very quickly at the same time spam and phishing became a big problem. They had to scramble to fix a problem with their providers before having a chance to refine their systems to make sure the implementation was done fairly and correctly. It does not seem to be a problem with top-level executives deciding that customers like me don't matter.
I also own a company and am very familiar with how things can get out of hand very quickly when demand increases. Shit hits the fan, then things suck for a while until the work is put in to become more organized. This takes time. And it takes trial and error.
I would expect over time for them to correct their systems and properly service smaller mid-range customers like me.
Mostly I can forget they exist and do other things with my business. But my use case is very vanilla: no outbound automated marketing, its used by only a few people at my specific company, we're not even a tech company -just blue collar stuff.
This year has been different though. I had to verify that I had a real business so that my number didn't get blocked on certain carriers, submitting the paperwork in the right way turned out to be kind of difficult, however I don't think this was something they had control over if I understand the situation correctly. One of the carriers blocked us anyway (They were blocking text messages with links to job info that I was sending to my employees). I used the ticketing system to get that problem fixed and they resolved it in a few days - I was under the impression it was kind of a complex problem too.
They also help us port numbers in from cell phones sometimes. And again the ticketing system is slow but they always get the problems resolved. None of these things being time sensitive, we were perfectly happy.
This type of thing is a difficult business problem for small and mid size businesses to solve. In this particular situation, I had a series of actions I was going to take to put out this particular fire. Normally I wouldn't do something like this, but my back was to the wall: * Call their sales number so I could plead my case to a real person within the company and have them put me in contact with someone * If that person refused - Call again and try with another sales person (This approach did not work because no one was answering the phone for sales, perhaps due to covid omicron?) * Post on HackerNews to see if I could get the attention of one of the brand ambassadors. (This worked, so I did not have to move on to the next steps) * Post on StackOverflow to see if I could get the attention of one of the brand ambassadors * Post on Reddit to see if I could get the attention of one of the brand ambassadors * Use LinkedIn to track down people who worked at Twilio. Use a paid service to get their phone numbers and addresses from their names. Call some of the people. * If no one answered the phone, go by the houses of Twilio employees who lived in my area
Long term, I wasn't sure what I would do because getting all the code switched to another provider would have been a huge hassle and it would have seriously gotten in the way of some of my other business development efforts. So I'm glad that Greg and Jeff reached out and reassured me that they don't intend to run their business this way.
A more difficult problem is Google and Facebook. I have had valuable pages stolen from me on both platforms (ex-employee) And neither company would engage with me. We're talking maybe $100,000 in lost property because of the amount of business the pages would bring in. Someone mentioned in another post on here that people are actually able to use HackerNews to get Google's attention. If I had known this I would have tried that. I knew Twilio might respond because they very often would help me when I posted technical questions on Stackoverflow in the past. I didn't think google and facebook cared as much about their brand because they have a monopoly.
What you are describing is tech support.
No that is customer support - big difference.
Also:
> I don't need tech support, I just need someone to call, explain the situation to, and have them click a button.
What do you think tech support is?
Because what you described worked? They got customer service from two higher ups at Twilio who cared way more about the internet mob than literally everything else happening at their $42,000,000,000 company that is teetering on a confidence crisis based on share price trends, or the multiples that investors are willing to accept from this management team.
Unfortunate outcome though. Automated banning is always frustrating.
Slack is probably a good idea.
Of course we have an internal database and he can look at text messages through our web application too. But these are not push notifications to his phone.
Support with them is significantly better but if I remember correctly pricing is around $1k/mo minimum (which was more than worth it in our case).
Best of luck to you.
All data was lost, number ID's, account ID's all completely different. It took us a LOT of dev hours to update everything, whilst losing some of our customers. Twilio is cheap, fun and dev friendly until they mess up, then you're on your own.
As another small biz, I've had very good experience with Phone.com over the past several years. Prompt and solid tech support the few times I need it (mostly for configuration and 'is there a way to do this peculiar thing?' questions), and mostly just works.
They're trying to offload the problem onto Twilio which then winds up passing that onto their customers.
Of course solving the abuse problem means spending money to cut off the revenue they actually see from the scammers sending texts. They'll never be incentivized to do anything about it unless the government were to make them an offer that they couldn't refuse.
That'll never happen though because the government is bought off by corporate lobbyists, so we will continue to evolve into more and more of a third world scam economy.
If the first Twilio should fix this bug in their system, if the second then they should maybe have some process of setting up employee phone numbers in their system so the shut down process does not happen. At any rate both scenarios should be common enough that they should have a process to handle that.
