This morning I received a notice that all my services with Hetzner have been locked with no explanation and no direction for contacting a human. Below is the full email I received.
What can I do?
--- Dear Mr ______
Unfortunately we have had to lock all services you have with us due to violations of our Terms and Conditions (https://www.hetzner.com/rechtliches/agb/) and/or System Policies (https://www.hetzner.com/rechtliches/system-policies/).
We will not be accepting any more orders from you, and your account will be cancelled to the next possible cancellation date, as per our Terms and Conditions.
This decision is final and cannot be appealed.
Kind regards
Your Hetzner Online Team
> No lead time (not even a paltry 24 hours) to find an alternative service provider.
> No mention or clarification that any data will be available for download in some fashion. Imagine relying on them for anything truly critical and being insta-banned. You can't operate a tech company on a server like that.
> No apology for having to do this. Using the word "unfortunately" doesn't count. They're giving a human being a shitty day: the least they can do is playact at being a tad sympathetic.
> No explanation of any wrongdoing or a reason for why this is needed. Even a simple "Due to legal requirements" or "excess resource usage" might help.
> No way to contact anybody if there's any error or outstanding business issues.
> A display of real arrogance by using the word "final" and "cannot be appealed" in the message.
> Addressing themselves as the "Your Hetzner Online Team" rather than a specific individual. If a human made a decision, they should own responsibility for it. If a human didn't make that decision and it was some algorithm, there's no way it shouldn't be appealable.
I have used OVH in the past, but these days I would definitely think twice before using budget providers like these.
This was a dev instance, maybe I was hacked, cannot rule that out. But was not expecting an outright ban with no details on what has happened.
Did you call them? What did they tell you on the phone?
I've been with them since 2004 with currently 100+ servers and cloud instances among my companies. Yes, they employ one trigger-happy young sysadmin, who can be quite stubborn, too. But in all the years, they never took any action completely without reason. Like we might disagree about the weight of my mistake, but there always was one.
If I had to guess, you were working on Crypto or used torrents. They insta-ban for some protocols. Also, if you connect to too many unroutable IPs, they will create an "abuse" case and disconnect the offending IP from their network.
If true, I'd check all configured email addresses. They let you configure different addresses for support/bills etc. and will send warnings only to certain addresses.
Hetzner is usually good at revolving issues.
If you don't pay a bill, they eventually will block incoming traffic from the web. They are still reachable from inside hetzner network and they will unblock traffic as soon it's paid.
If the BSI finds Ports that shouldn't be open to the public, they will forward the mail to you and won't take actions.
If you disturb their network due to misconfiguration, they will block you, demand an explanation within 24 or 48 hours and unblock you, if they find it plausible.
If you call them with technical issues - in my experience - you typically want to prepare logs, traceroutes etc. because they will know enough to provide guidance on how to resolve it.
I have a single configured email address on which I received my welcome email on July 15 and "Server Locking" email today.
Looking into Hetzner dashboard, it seems they did not delete my instance, just turned it off and banned my IP so I cannot ssh into it. There is an option to request unblocking which I will request soon and which wants me to answer "What caused this problem?" and "How do you plan to correct this problem and prevent for the future?".
This was a development instance: running docker, postgres, SchemaSpy, some service emulators, node, vscode and accessed the services through ssh port-forwarding.
It seems there is an "Abuse" incident linked to the blocking of my IP but I only see the incident ID, no additional details.
This was a dev instance, I did not think about making it airtight. I do not rule out that someone broke into it and violated their terms (this happens with production systems and I am definitely a worse engineer than people there). If this happened, I am happy they locked it down but I wish they informed their users in these cases: I had git ssh keys and other secrets there which I proactively revoked and more information on the incident would definitely have helped choose the right course of action.
I have quite a bit of rep with Hetzner, so they didn't outright nuke me, but I once got an abuse email because I was running an IPFS daemon, and the reference IPFS implementation allows RFC1918 IPs and GCNAT on discovery announcements... so dialing into nowhere a lot upset the router.
With the new no-ip-at-all option you can set up a Network and set up an extra instance as NAT as you would with a home network. That should cut down on issues like that.
I know that obviously there's no obligation to share etc, but I can't help but feel like if they truly weren't up to anything sketchy they would be more forthcoming?
They stated in no uncertain terms they're running a remote VSCode instance and nothing else. I've done something similar back in the day just to have a consistent environment (target was a hacked up, cheap, used Android phone I left at home, but same idea -- remote IDE for one reason or another).
These posts aren't that crazy when you think about the level of abuse a service like this probably gets and how few dollars per account are spent from the bulk of users. I've been banned from Digital Ocean before without any crypto mining or heavy workloads or failing to pay bills or hosting porn or any of the other sorts of things you might expect. My best guess is that they flagged the fact I was using a privacy card as an indicator of potential eventual fraud. Or else a new fraud model took into account age and didn't take into account that there was no way for a new account to ever become old enough before being flagged, or some other sort of "data-driven" blunder. No biggie though, there are plenty of operators willing to actually accept my money for their services.
Anything bad has to be because the user, usually the person with the least power and information in the system, did something wrong.
The many other humans behind the technology -- the coders, architects, testers, managers, executives, lawyers, 3rd parties -- can never go wrong. They must have studied the entire set of probable situations and devised just and fair solutions to every situation. Therefore, they should not be held accountable though they typically have both more power and more information. They're incapable of making mistakes.
That’s how I read all of these when people don’t add more details.
> Online Dispute Resolution in accordance with Art. 14, para 1 of the EU Online Dispute Resolution Regulations
Online dispute resolution in accordance with Article 14, Paragraph 1 of the ODR-VO (Online Dispute Resolution Regulations): The European Commission has established a platform for online dispute resolution (ODR). You can visit the platform at http://ec.europa.eu/consumers/odr.
>This decision is final and cannot be appealed.
This translates directly to "FU! We do not want your business."
Right now, none of us know what your code was doing. Portscanning the entire internet? Botnet C&C? Got hacked because something that was forseeably your fault?
Put some details in so that your complaint and theirs don't have the same amount of evidence.
* A data centre got flooded
* Another data centre caught fire
* One provider went bankrupt
* One provider discontinued the product with a week's notice
* Another provider terminated the account because they thought the account was fraudulent
You cannot build your business based on the assumption that one provider will always be there for you and at the price level you are comfortable with. At some point somehing will happen, and when it does you're screwed if you don't have a plan for switching providers.
There have been a lot of similar reports about Hetzner competitors, so it seems one just has to maintain off-site backups and be prepared to randomly jump ship. There are lots of reports of this in the DigitalOcean sub-reddit.
As to the cause, I've gotten caught up in things like this before... no so much from cloud providers but from other e-commerce vendors and even on-line banks. I've had some luck writing paper letters, not going away without an answer on Twitter, and filing government complaints.
The general gist is that like like spam is a problem for email, other types of fraud are a problem for cloud providers and merchants. They're turning to some of the same kinds of tools that are used against spam... with the same mediocre results. I've taken a lot of time to get under their skin and get to the root cause. I've been successful about half the time, and the reasons are usually lackluster:
- You used a VPN when you signed up years ago - The bank the issued your credit card (the first 8 digits) matches a lot of other fraud events (this is particularly the case with gift cards, over the counter debit cards, and virtual cards... though I've had the same problem with major brick and mortar banks.) - You had account activity that doesn't match normal hours for your time zone. - I ran an ad blocker, which also messed up some CAPTCHA/JavaScript thing - I have "load images" disabled on my email client, so it looked like I wasn't opening mail from them. - Other fraud occurred from a similar IP address.
Often they use plugins from commercial anti-fraud companies, much like Facebook or Google ad plug-ins. These companies look at information from lots of places and try to identify patterns among accounts that later are reported as fraudulent. We use one of them where I work. It's about as effective as a spam filter, meaning it catches most but has both false positives and false negatives. You can tune it to be more or less aggressive.
Depending on where you are in the world, you may have more rights to dig into it than Americans do. Also, if you used a promo code, you might ping the advertiser and let them know as this hurts their brand as well.
I hope this helps.
I'm not sure hetzners policies but for example if your code is utilizing certain ports and traffic types that they might have limits on?
The response from them is very flippant and robotic though. It may be an automated action but I'd be curious to hear your experience with the "human" you get in touch with.
Edit: as for the decision being final this is usually just to deter bad actors. I've had some issues with a compromised server when colocating who said the same. This was a pain to prove, they did overturn it but I imagine it had something to do with the higher fees being paid to them.
Paypal closed a account of mine (business) while keeping another (private). Amazon closed several tries to sell there, only worked with an incorporated company to sell the book of my wife.
Without Amazon you can't sell a book (fiction, no massive social media following).
Both companies of course, no mention of the reason, just a link to their TOS and this vague speak.
On top of that we need a way to applay to an external arbirter for companies that have more than 10% market share.
I blanked that out and they didn't accept it until I showed them the national police website where they showed how to do this and recommended to always do that.
I only used them for a while, eventually I mixed to scaleway which was cheaper and doesn't need any invasive info. I've been a happy customer there for years. I even run an IRC bouncer there without any issues, which many such providers specifically forbid (eg OVH)
