HTP ("Hack The Planet") is a group that likes to break into things. Another (unnamed) group of people impersonated a third group of people ("ac1db1tch3z") and tried to cause trouble for HTP.
The impersonators located HTP by examining one of HTP's botnets (a collection of compromised computers that are used to launch things like denial of service attacks). Botnets have to receive instructions (e.g., targets to attack) from somewhere, so it's likely that the impersonators followed the path taken by commands to the botnet, and found the network(s) that HTP uses to organize themselves.
HTP realized this, and wanted to get back at the impersonators. They found out that the impersonators used an IRC channel (chat room) hosted on a network called SwiftIRC. If HTP could break into SwiftIRC (which is hosted on Linode), they could cause all sorts of trouble for the impersonators. So HTP decided to break into Linode, so they could break into SwiftIRC, so they could break into the group of impersonators.
To break into Linode, HTP broke into their domain name registar (name.com). They planned to secretly take control of linode.com, and replace it with a version of linode.com would look and feel and work correctly, but had one additional feature -- it would collect the login information that people typed in. HTP probably hoped to gain the login for SwiftIRC directly, or collect the logins for Linode admins and obtain SwiftIRC's login from there.
But, before they enacted the domain takeover (a maneuver that would likely be somewhat difficult to employ without being noticed), an HTP member discovered a new vulnerability in ColdFusion, the server software used by Linode. The ability to discover a new exploit on demand implies a high level of skill within the group. Using this exploit, HTP obtained direct access to Linode. They proceeded to gain access to SwiftIRC, as well as other sites hosted on Linode, including a well-known security site, nmap.org
The FBI apparently had a mole in HTP, and they alerted Linode that HTP had access to nmap.org. This posed a bit of a problem for HTP: if it became public knowledge that they had obtained access to Linode, then perhaps they wouldn't have time to go after the impersonators using their newfound access to SwiftIRC. So, HTP tried to strong-arm Linode into staying quiet until May 1st. HTP had obtained the customer information and credit cards of all the Linode customers. HTP threatened to widely publish all this sensitive information if Linode didn't stay quiet. If Linode complied, then HTP would just delete all the info.
Linode, though, was forced by the FBI to announce that they'd been broken into. HTP told Linode to just publicly acknowledge that HTP was the group that broke into Linode, and they'd delete the sensitive info. Linode did so (https://blog.linode.com/2013/04/16/security-incident-update/).
HTP conducted an internal investigation to determine which group member(s) were working with the FBI. HTP broke into the mole's computer and turned on their webcam, and saw an FBI employee looking over the shoulder of the mole. They kicked the mole out of the group, so the FBI doesn't have access to HTP anymore.
(Remember, this is the story according to HTP.)
Here's hoping the FBI "causes trouble" for the lot of them. Breaking into other people's stuff is not cool. If I leave my door open by mistake, yes, that makes me a bit absent minded, or foolish, but it does not give anyone the right to wander into my house.
Just a personal opinion.
" Recognizing their situation, we instead told them that if they acknowledged HTP in their analysis, we'd go ahead and shred their customer data anyway."
Do they honestly, for a single second, think that any LEA, corporation, or, well, anyone would believe that once the information was compromised, that there was no putting the genie back in the bottle? Also - I suspect there are probably disclosure laws that had to be followed by Linode anyways.
Comply, and trust the honor of black hat hackers?
Or refuse, and have customers' data appearing on FTP and torrent sites within the day?
As a Linode customer, I would rather them choose the latter. Not to try and sweep the incident under the rug (to your point about disclosure) but to prevent the data from being scraped by groups who exist solely for extracting credit card details from releases by groups like HTP (note the reference to "carders" in the article) and then being sold.
(According to Linode's own post, the CC data were encrypted, meaning that it should be intractable to actually extract usable CC numbers from the data. But why would Linode not accept those terms, even if they believed that HTP were lying? At least it would give them until 1 May to get their house in order.)
Also makes you wonder, if there are holes like this, how many more holes like this are there? Especially if this is a pattern across the system.
Highlights: 1900+ days uptime on a sparc box somewhere in sourceforge.net, root on ICANN, root on Debian repositories..
- HTP claims to have{, had} access to name.com, which Linode currently uses. This access enables an unauthorized party to update authoritative nameservers for your domain; i.e., if you host at Amazon, very likely your authoritative nameservers are Route 53 on your account. HTP would not have access to modify the zone directly through the registrar and would instead have to hijack the entire domain with a working, completely-transferred zone on their own nameservers. For this to go down entirely unnoticed is extraordinarily difficult. I won't say impossible, but damned close without a copy of the zone in hand and with Linode running AXFR disabled (you should be too). There are subzones of linode.com; they wouldn't have gotten them all, and it would have been noticed within minutes.
- In order to attack SwiftIRC, to get back at some script kiddies DoS attacking them after their last release (because you know, that's a good target to burn registrar access on and all), HTP decided to backdoor SwiftIRC via their nameservers which are hosted at Linode. That's not the same as the registrar nameservers discussed above, but is instead the DNS data actually stored on a Linode on SwiftIRC's account. They do not hint what they were going to do with it once they had hijacked the nameservers, and I will not theorize. I could guess, though.
- Before utilizing their registrar access (from the first bullet point) to hijack the linode.com zone and intercept manager logins silently by redirecting traffic via DNS -- also fairly difficult to pull off without a good linode.com certificate in hand, in terms of keeping the TLS session non-suspicious to a browser -- they instead discovered a zero-day in ColdFusion (Linode's stack) and got in that way. That's much quieter and much more likely to not be noticed. If we take the FBI's actions at HTP's word, the FBI was the only reason Linode was made aware of this outside of HTP's control; a DNS hijack would have been immediately noticed by Linode administrators.
- Knowing what I know (let's leave it at that), a successful exploit on Linode's ColdFusion stack entails a database of Linodes, DNS, credit cards, e-mails, addresses, and keys to decrypt the actual card numbers, and a lot more data. You have to decide whether to take HTP at their word that they deleted credit cards. Consider your credit card and all prior credit cards compromised if it were in the system before April.
- The access that HTP obtained does not, full stop, lead to root on Linode instances without at least one shutdown job or change of root password job showing up in your Linode's history that you did not ask for. Your Linode's root password is not stored in any Linode system aside from on your Linode itself. Your LISH password, as they say, is, and according to them is stored in plaintext; if you see things on your Linode's console (located under the Remote Access tab) that you did not type, that access was used upon you. If not, it wasn't. If you used the same root password on your Linode that you did for your LISH password, consider that password compromised. I'm suspicious of the claim that they rooted all those (assuming) customers without any of them noticing their Linode being rebooted to apply the new root password to allow HTP in, and I would read that as "potential access" instead of "access". They probably bounced some nmap.org servers to reset their root passwords -- a Linode system requirement -- without fyodor noticing. Which is interesting for a couple reasons.
- Also, the access they obtained does not lead to root on the Linode host fleet itself, unless they are holding back some extra access they obtained such as a shared password between the ColdFusion stack and administrator credentials for Linode systems, which I consider unlikely for a couple reasons. With several days to get familiar with the architecture, HTP could have used their database write access to do things on the hosts, but it's a fairly limited set of things. Dumping Linode's database is bad, but root on their hosts is far, far worse, and by indications, I don't think they got it.
- How does this relate to the Bitcoin hacks of yesteryear, you ask? The Bitcoin hackers probably got in the exact same way -- Linode hinted at a compromised admin credential, which is close enough to do everything HTP was able to do -- then shut down and reset root passwords on the Bitcoin Linodes they were after, which then gave them filesystem access.
So ends clarifications, thus begins conclusions:
- PAY ATTENTION WHEN YOUR SERVERS ARE REBOOTED WITHOUT YOUR COMMAND.
- PAY ATTENTION WHEN YOUR SERVERS ARE REBOOTED WITHOUT YOUR COMMAND.
- Linode added a feature that shoots you an e-mail when your Linode is manipulated in any way via jobs, such as resetting your Linode's root password (a la Bitcoin/HTP hacks). It's depressing they had to do this, but pay attention if you get the mail. External monitoring like Nagios that pages you when your server goes down is also a good idea, as long as it is hosted at another provider.
- EDIT: After reading the zine, yet again, /CFIDE is the vector. There's no excuse for not hiding your administrative tools, generally the soft underbelly of the whole smash, from the Internet. None. It's one rewrite in nginx. Match /CFIDE<anything> from the public, redirect to /. Done.
- EDIT: Again, after looking at how trivial the exploit was, it's probably time to reconsider using Adobe ColdFusion from a business continuity standpoint. Half Linode's fault for not hiding /CFIDE, half Adobe's fault for the engineering missteps that lead to this capability for a remote attacker. We should be just as hard on ColdFusion as we are on Rails.
- SwiftIRC is a den of inquity, up there with EFnet; if you run a hosting provider, think twice about permitting SwiftIRC anywhere near you. To reiterate that, Linode was a casualty of someone going after SwiftIRC. Delink their nodes, cancel them, and kick them to the curb if you're interested in preserving your business. Not worth the money. Same with damned near all the IRC networks except OFTC and, to a far lesser extent, Freenode. There will always be targets but harboring SwiftIRC is probably a malicious-actor magnet.
- Registrars (and CAs, though that's outside this discussion) are the weak point in the entire system. This is not the first time they have been shown to be so. Linode could be Fort Knox of digital security but if name.com falls over, it's all over; that's entirely outside of Linode's, and your, control. Currently, the registrar market is heavily profit-centric and, personally, I think people spend far too little on a domain in the general case. I would happily pay a registrar a lot more money -- hundreds a year or more -- if their offering were run competently, as it is fairly obvious name.com isn't. Compare your hosting bill to your registrar bill; what's wrong with that picture?
- HTP is apparently fairly easy to troll into using valuable access for vengeance purposes. Shameful target selection and a burn of a good hack just to root SwiftIRC. That's like pissing in the ocean for a good time.
- Linode got railroaded here and the general reaction by folks is a little overdone. You know that's true when even the hackers' overview of the hack specifically calls out people bitching about Linode security on Twitter. All it takes is one zero-day, and you will all be hit by one in your career, so cut Linode a little slack.
Unfortunately, there's not much slack left to cut. Linode pulled that line taught with the last major breach of their system (and their subsequent opaque response).
It's clear Linode's security practices leave a _lot_ to be desired, and their response to this incident, while better than the last, didn't go anywhere near far enough in terms of transparency (as well as demonstrating some fundamental gaps in their understanding of the current state of infosec).
If they had access to the database, it may have been possible to delete malicious jobs from people's histories. Even if the user had email notifications turned on, an attacker with full access to the database could have turned it off temporarily (just flip a boolean flag).
This whole thing really makes me wonder how many hacks and deals behind the scenes are going unnoticed.
what's stopping the bad guys from just proxying dns queries they don't care about to the original NS?
with this sort of trickery you could get a "domain control validated" https certificate too!
My usual suspicion is that in general, the volume of DNS traffic should give you pause before you start putting custom code in the path of answering a query. Clearly it's possible -- Route 53 is built upon that very notion -- and I suppose in this scenario it's feasible.
Don't forget every Linode has a hostname under linode.com. I think splicing yourself in and running a conditional on every query would overwhelm whatever you point the firehose at and you'd have to plan accordingly. All it would take would be to add a couple hundred milliseconds of latency to the average DNS query (even before the inevitable carpetbombing of p99 latency) and a competent high-traffic administrator is going to start looking around.
I don't understand why HSTS didn't allow some sort of pinning, or the ability to specify a certain kind flag in the certificate is required (like EV).
No, nearly ideal use. Its like a strategic nuclear weapon. You don't use it sneakily, that's the opposite of the whole point. Always intimidate as publicly as possible and in the tech community messing with linode is about as public as it gets.
The other part is showing off, its like declaring "we have access to them all but we don't care about name.com". Maybe they do...
> All it takes is one zero-day, and you will all be hit by one in your career, so cut Linode a little slack.
Actually, the biggest lesson I'm taking away from this is to never trust any one piece of software, and to always have multiple lines of defense/alerting sitting on independent software stacks.
I wouldn't bet on that.
> There will always be targets but harboring SwiftIRC is probably a malicious-actor magnet.
Isn't that the same logic Everydns/Dyn Inc. used when they censored Wikileaks?
> All it takes is one zero-day, and you will all be hit by one in your career, so cut Linode a little slack.
True dat.
I don't need to wager, and can speak with authority based on what I know (which I'd prefer to leave vague). There are two vectors into a Linode's filesystem from the perspective of an internal attacker: having root on the Xen host or gaining a login on the Linode. Knocking over the database and Web server gives you neither unless the person reused their account's password as their root password, in which case it's behind a cryptographic hash and subject to the typical rules there. If you own the database, you do have LISH access which gives you the equivalent of a VGA console; if someone left that console logged in, it's a vector as well.
The only vector HTP would have had in the general case would be bouncing the Linode. It's a fairly sufficient air gap, in a way.
Don't we also have to take them at their word that they even had decrypted CC's? I haven't seen any proof yet that they obtained the decrypted private key, just their statement saying they grabbed in-memory keys.
Domains are jacked and proxied a lot more than people know. The hackers have custom tools (rather than Squid + BIND etc.) that perform these tasks and keep them hidden.
Even better, you could host spoofed DNS for a number of Linode's on a single small 128-256MB virtual machine. The infrastructure required is tiny. Definitely possible, definitely happens all the time.
Keep up with CVE's, don't provide a wide attack area (so lock down interfaces to your machine and don't expose much to the world), and keep blast radii as small as possible (so even if your machine does get owned, you can possibly restrict it so it doesn't automatically mean they gain access to other systems in your network.)
Oh, and model the threats to your network/application. Make sure you're securing against the right threat. As an example, anti-malware is wholly ineffective against social engineering - maybe it's more productive to train employees and make sure that each employee doesn't have total access to all privileged systems.
Damned near every exploitable security problem in an application can be traced back to this one simple rule. XSS is a common one. The ones that don't conform to the rule are rare and magical, like unicorns, and usually involve something exotic like hardware side channels. If you take input it will be abused, so plan accordingly. That occasionally manifests in unexpected ways like timing attacks, in which case an attacker repeatedly sends you lots of carefully-chosen input to deduce something based on how long it takes your code to reply, much like cracking a safe. It's a basic attack on a string compare, which is O(n). This simple code is vulnerable:
def price_multiplier(form):
if form.input == 'super-sekrit-coupon-code':
print >>browser, "100% discount applied!"
return Decimal('0.0')
print >>browser, "Sorry, that's an invalid coupon!"
return Decimal('1.0')
Securing an infrastructure is another story, but the rule is applicable with the occasional clarification to everything there, too. There are domains of trust even within your own organization; malicious employees will try to harm you, as well, and you get little to no warning of those. Do your interns need root on machines containing customer data? Employees are users too.
I just want to know one thing: did or didn't they leak the credit cards?
Does it really matter? If your card was used with Linode, it should have been blocked by now anyway.
:)
The only real solution is either to not depend on any single network, or to make it clear that you will simply kill anyone who troubles you. Or just remain innocuous enough that nobody will care to.
It does have me worried however.
There are plenty of CAs out there who will give out a (non-EV) cert to anyone who can receive emails to webmaster@example.com.
And Linode isn't saying anything new
> We identified which users on HTP were involved with the FBI, and promptly gained access to one of their cams.
Not sure what they mean here. FBI camera? User's laptop camera? Either way this also seems far-fetched.
If everything they said was actually true it's very impressive.
It would be nice if we had internet role models. IRC is full of low-life degenerates who perpetuate the vitriol that reinforces this way of life as an acceptable pastime. If there were well respected hackers who spoke publicly against this kind of behavior it might make some people think twice. (Unfortunately, most well respected hackers used to be these kids before they got real jobs)
HN is full of individuals who try to take the high road, versus the kind of anonymous internet idiocy that exists in nearly every forum and chatroom. I love this about HN. I wish more of the internet took it as an example.
Really? You can't think of a terrorist definition that uses, you know terror to coerce people into doing things? While you may not like these guys, labeling it "terrorism" is counterproductive. If you stretch terrorism to include anything that involves the potential for someone to feel fear about anything, you can classify nearly everything as terrorism.
I know hundreds of people who dedicate their lives to the drama and bullshit that is spawned solely by being in an IRC channel. If it went away, these people might just find something productive or positive to do with their lives.
If someone has a beef with society, there are plenty of honest and constructive ways of "sticking it to the man" without resorting to violating people's property.
People say this all the time, but always fail to give examples of what sort of activities they think would be a constructive alternative. Would you mind sharing what you envision as alternative activities?
Best of all, don't have anything worth stealing. Don't keep the credit cards on your servers, parrot them through a vendor. Don't keep user credentials on your system, us OAuth, Fb or Google auth. If you've got nothing valuable to steal, they'll likely not break in.
But then again, if the Fed's want in, they'll just pull your box from the rack. Don't forget that you can literally freeze a DIMM and dump it and all the encryption keys to another mobo. So, as is the CIA's policy, if you don't want anyone to know something, don't let it touch a computer. ;)
The best thing is to keep your eye on the culture of the developers and how seriously they take security - for instance the Ruby on Rails developers ignored exploits/reports until people blew them wide open. Now, if some other hacker had known about that before the disclosure, they could have owned any RoR sites.
From my experience, Django seems to be the best, and has not had any unfixed vulnerabilities for a while (though, due to it's complexity, it's completely possible that 0days exist). However, if I'm running Django sites and some do get owned, I can tell my boss/client/self/whatever that I did everything possible to prevent it happening.
There is no such thing as 100% secure, however, it's fairly reasonable to be hardened to all but the most dedicated crackers.
With an attack like HTP's, there's no fucking way anyone could have been expected to prevent, without running their entire own infrastructure, because they owned registras, Linode's LISH shell (so they get near-physical access to your Linode), and various other crap. If your boss were to fire you for getting owned in this attack, despite it preeetty much being zero of your own fault, they would be in the wrong (unless you have the resources to not depend on anyone).