I get him speaking out for them about the hosting having been free, but Akamai is now the CDN that got bullied into kicking someone of their service against their own will.
Terrible PR, and that mud will stick in tech circles. Akamai folds under pressure.
I know it's a crude comparison, but we don't negotiate with terrorists for a reason.
DDoS mitigation is fundamentally a problem of "who has more bandwidth?" - if the attacker has more bandwidth than you (and how much bandwidth you have depends on "to where" ) - it's over.
The problem is that the economics, right now, are heavily tipped in favor of the attacker.
I would bet money that the attack was truly epic
Sounds pretty epic.
Is there no way to stop such attacks by coordinating with your upstream ISP, or with the sources of the traffic? Why do backbones allow it to be carried? Is the problem that these attacks are too many different individual streams to identify and filter?
It seems like there ought to be some way to hierarchically punt the problem to network operators. "Your network is contributing 10 gigabytes per second to this DDOS attack. Identify the sources and shut them down." - times each identifiable traffic stream.
Is there no way to capture a list of all IPs involved in the traffic and quickly distribute a "shut off this device" request to the origin network? Maybe a good-faith collaboration of different ISPs could result in quick shut-downs. Or if networks don't cooperate, then the next-nearest border does it for them (and if they don't like the policies under which their neighbor suspends the traffic, they can sign up to do it themselves). Imagine something like a mini automated DMCA type request. "This IP is DDOSing me", signed by the operator of a reputable network, having the effect of suppressing origin traffic from the IP when received by a reputable network. (Any abuse of the mechanism causes the network to lose its privilege to participate, and DMCA requests related to that network fall upon its neighbors.) Perhaps the suppression would be destination-limited so as not to be vulnerable to too much abuse of the mechanism.
5575 is a BGP extension that says "for packets from x to y, do z". Assuming a router knows on which if its input ports such packets arrive (and during a DDoS it doesn't have to wait long for the next packet), it can disseminate the flow specification towards the actual source(s) quickly, so the packets can be dropped quite far from the DDoS target, in the ideal case as soon as it reaches an honest ISP.
Egress filtering should kill much of the spoofed-origin traffic and this much of the rest — if deployed.
I'd love to know why 5575 isn't deployed. Memory concerns maybe?
> Is there no way to capture a list of all IPs involved in the traffic and quickly distribute a "shut off this device" request to the origin network?
Now hackers have a new attack vector.
Plus you get the problem that you are usually seeing DDOS traffic from innocent bystanders, if the ISP is shutting of a compromised home router, then their customer will not have internet for some length of time.
> Why do backbones allow it to be carried?
Because they get paid for any traffic passing through. The more traffic they have the more profit they earn.
Seems like it not only was one of the most expensive attacks so far (and by far the biggest one to have ever hit Prolexic, according to them), it also made little use of reflection of amplification making it much harder to mitigate.
> to the point where it was impacting (or was about to impact) other Akamai customers.
That's exactly the scale it had reached, and Akamai provided free service to Krebs, which was nice of them but only to the extent that it wasn't significantly impacting customers.
I... wonder if anything different would have been done for a paying customer. I mean, if the attack was big enough to take down other customers, and if Akamai had the choice between kicking one customer and all customers being down?
Asking for a friend. /s
Definitely. The lesson I'd take from this is that Akamai isn't serious about DDOS protection.
For me, buying DDOS protection is something like buying insurance. I don't expect to need it, but if the worst happens, I expect them to stick with me. The way I measure insurance providers is by asking friends how it was when they had a claim.
It strikes me as especially bad that they're doing it in the moment. It'd be bad enough if they said, "Sorry, Brian, this is too big a distraction; you've got 90 days to find a new home." But that they're dropping him in the middle of an attack? That means I can't trust Akamai.
>For me, buying DDOS protection is something like buying insurance. I don't expect to need it, but if the worst happens, I expect them to stick with me. The way I measure insurance providers is by asking friends how it was when they had a claim.
DDoS protection isn't insurance, Krebs gets attacked 24/7. Only an utter moron would be willing to sell Krebs DDoS insurance.
>That means I can't trust Akamai.
Which means nothing at all in a world without alternatives, hosts capable of tanking attacks like that number at two or less. But I get the impression you're not looking to spend hundreds of thousands of dollars a year on DDoS protection anyway.
They were hosting it pro bono. He never paid them enough to do anything. And yet...
> Only an utter moron would be willing to sell Krebs DDoS insurance.
But a smart person would cover him for free as a way of proving that they could handle the worst the DDoSsers gave out. To prove that they stick by their customers.
Most people who buy insurance never really use it. So what are they buying? A feeling of safety. Just think about the various insurance company slogans that come to mind.
That being said, I definitely agree with your thoughts on insurance.
So, if he had paid one cent (thus being a paying customer), you could extrapolate?
I don't see how the price is in any way relevant here. They promised to protect him, and they failed to do so. Claiming afterwards that the premium was too low isn't the way this works.
Also, I doubt that it actually was "for free". He may not have paid in money, but likely in the form of (at the time positive) PR, for example.
If Akamai can't provide their service for free, then they shouldn't provide their service for free.
A CDN's whole business is resilience, which in this case makes them the bodyguard, not a bystander.
Whatever your opinion of Cloudflare, it seems clear to me that Matthew Prince keenly understands this, hence him reaching out and offering to step in and get Krebs back online.
tldr; If Akamai can't do the one job they exist to do in the face of an (albeit well armed) assailant, then they're the problem, not Krebs.
The biggest DDOS ever, and Akamai dumps the client rather than defend it.
However they spin it, doesn't look good.
Frank Leighton needs to make a statement about this, immediately, and he better pull the Rabbit of Caerbannog out of his hat.
The site must be constantly under attack, and it must cost Akamai a fortune in real $$ all year long. And, when their service is performing well, nobody is talking about it, so there must be very little positive PR.
If someone is ready to foot a 620 Gbps bandwith bill all year long, I am pretty sure Akamai will be more than happy and able to scale up further.
Too bad there is no always a cleaner/smarter solution than pure bandwidth and $$ to fight those attacks.
I already know that Akamai is expensive and not particularly good at it. They are a CDN who is trying to make some money on the side with unused bandwidth, and will protect their CDN business if it comes down to it.
They don't invest in active defense. In fact, I know folks who actually had to block malware being served from Akamai-owned IPs!
I suspect that is why they gave service to Krebs for free in the first place--they need the marketing.
Kind of like the police protects gangsters from getting shot by other gangsters, but you would really like them not to do that, so that the gangsters can just shoot each other.
In this case, Brian Krebs tried to convince Cloudflare to kick off the booter sites, so they are unprotected, and can DDoS each other. Cloudflare didn't put any effort into that idea, and now he's apparently angry that he didn't get through to them.
https://webcache.googleusercontent.com/search?q=cache:kaymYs...
I would bet things would be a fair bit easier for them if they agreed to take things down which most people don't like, but from my position they are taking a very principaled stand for free speech. Are people on hn actually arguing we want more censorship on more places on the web?
If this is a CloudFlare Vs Akamai attack Krebs isn't saying, but I would put dollars to doughnuts it is.
I don't see why being pro bono would matter in an established company?
You've gotta get the bandwidth to your filtering servers before you can filter it. DDoS mitigation, as I understand it, is first and foremost a matter of having more capacity than the attacker.
https://web.archive.org/web/20151115154842/http://krebsonsec...
Thanks for giving a link to this post!
http://webcache.googleusercontent.com/search?q=cache:kaymYsb...
(it's the "strip=1" parameter in the URL)
Plus, threatening to kill Krebs' wife.
But since we're talking about ethics here: Sure, now that the hacker faces 30 years in prison, he's not short of probably sincere apologies. I could really believe that he now has changed his view and accepted his guilt. It makes me ponder the thought of if I were Krebs' to not only feel sorry for the guy but (if it were legally possible) to dismiss the charges against him.
Consider the much more likely outcome of the hackers' plan: That it worked. Would the hacker had the same sense of guilt then? Or the same sense of forgiveness as Krebs or me seems to have? Maybe. We'd never know unless he did. It's more likely, he would have enjoyed Krebs' ruined live. Maybe even continued to threaten his wife and family. Just for the fun of it.
Because doing that in the anonymity of the web makes it easy to misbehave in ways no one ever would in front of the public eye and even less in the eye of his family and friends.
It's not difficult to open a bank account in your spouse's name without his/her knowledge, and use that account to do things he/she wouldn't do.
When I was in middle school and the internet was still fairly new (we had just gotten it) a classmate of mine hatched a terrible plan to get rid of a teacher he hated. He waited until the teacher was out sick one day then during the substitute's typical teaching pattern of having us "read these 3 chapters, answer questions then keep your head down until class is over" he jumped on the teacher's computer. After reassuring the substitute that he was allowed to he tried desperately to find child porn. His plan was to save it into semi hidden folders onto the computer then later on turn the teacher in for having child porn.
Fortunately this classmate wasn't able to find any and eventually gave up. But I've always remembered his plan. It's terrifyingly believable that if someone managed to get into your computer and download child porn, there is likely little recourse or way to prove you did not do it.
here is him talking about it: https://www.youtube.com/watch?v=CzdFOpRTvyU
and for the existing IOT devices, are they the same thing, or were different exploits used for different devices?
Again, the speculation that it is IoT devices is unfortunately just that. However massive compromise of internet connected embedded device is not new: http://internetcensus2012.bitbucket.org/paper.html
ISPs are uniquely situated to stop this kind of ddos because the traffic originates from IPs they don't own. The traffic has a spoofed from address. And as a rule, the ISP.should only need to send traffic out of a neighborhood from the block of IPs that is assigned to that neighborhood. You can put a filter on every switch or even every interface allowing only traffic from the IP or IPs on the other side of the link to send traffic. A company like Comcast could make it default part of account setup scripts. If everyone did that, these would disappear over night.
Further, technologists tend to be pretty good at solving problems. I know this isn't the ISPs problem, but it is a flaw in the network, I'm simply wondering if anyone is attempting to solve this problem at the network level rather than simply building bigger caching services to protect those that pay for protection.
Then again, I could be reading into this too much, and the computing part has always been a bottleneck at backbone level.
Sure, this costs Akamai money they don't want to spend, but is such an attack noteworthy? Eh.
It is, just about, possible to perform actions on every packet in a 10Gb stream on an x86 machine. You have to use a userspace stack, handle packets across multiple cores, and be VERY careful with what you are doing so you don't do cache misses. At 10Gb/s you're talking only a few hundred clock cycles per packet - anything that doesn't work as planned causes massive backlog.
Now try serving (dynamic) HTTP to that.
Essentially, it comes down to the fact that getting packets from point a to point b requires a lot of cooperation, and cooperation is difficult. Yes, yes, if you bought me the fiber, I could build you a 665 gigabit network, on the kind of money that a nerd could come up with, (not counting the fiber) but interconnecting that network with other people's networks? yeah, that's gonna cost you. Settlement-free peering is a thing, but it is really difficult to set up and maintain those relationships.
:-)
"Before everyone beats up on Akamai/Prolexic too much, they were providing me service pro bono. So, as I said, I don't fault them at all."
Let us not permit companies to co-opt language for their benefit.
If it was genuinely pro bono ( lit: for the public good ) then they would have taken all steps possible to keep the site online since the public good was served more by having Mr Krebs online than not.
However, in this case they were hosting him free-of-charge because it was good publicity for them. That's a very different scenario.
Besides, pro bono isn't literally "for the public good", it is literally "for good".
Finally - that is a ridiculous standard to hold everything categorized as "pro bono" to. Law firms oftentimes take on cases/clients that can't afford their services, pro bono. Because they call it pro bono, does that necessitate that said law firm should continue to fight all pro bono cases in court until either A. they win or B. they go bankrupt? Of course not.
Law firms (and other professional services firms) call it "pro bono" when they use their specific skill-set to provide their services to those (e.g. the indigent) who couldn't otherwise afford them.
In that example, it's the fact that the indigent can get access to quality legal representation which is itself considered the "public good".
Kind of like a reverse wild-wild-west evolution, where the previously carefully cultivated academic and company site presence, gradually degenerates into misclick-hell? And the non-technical, non-IT savvy masses, in a bid to escape this all, end up in a facebook-style future where media is curated and presented for consumption (or perhaps in future, facebook-type entities end up with their own wild-wild-west hell)?
I have a strange feeling that we are seeing the decline of a city/civilisation; once you used to feel safe walking out at night, knew everybody in the neighbourhood, could leave your doors unlocked... and now, you don't dare to go down the lane to the left in case you pick up a nasty virus, and if you hear a knock on the door at night/email from DHL, you don't dare to even look through the peephole/preview the JPG!
I think prevention should be emphasized. If there wasn't so much garbage plugged into the Internet, there wouldn't be huge botnets to send DDoSes. There are few groups that scan the Internet for vulnerable systems, and rather than compromise them, send notices to the ISPs. In Canada, the CCIRC does this. But they only check IP blocks assigned to Canadian ISPs and enterprises.
Plus, why do so many ISPs still allow spoofing of IPs? It isn't 1999 anymore.
We should start a grass roots group to talk to everyone they meet, and get people to update their OSes, devices, and get rid of crap.
Which pretty much illustrates the worst-case outcome: spam and trolling rendered completely worthless.
I'm not sure how well this would work outside of the U.S. though. Not everyone is as litigious as Americans are.
I'm personally amazed that people don't get hacked more often TBH... I can't think of any instances where non-technical people have been pwned in my own life.
I personally have a pa55word that I use for sites I don't trust, but the accounts never seem to fall or even falter. It's amazing really.
> I'm personally amazed that people don't get hacked more often
... the machines participating in the DDOS are (almost entirely) hacked, yes?
Peak Internet:
http://blog.kozubik.com/john_kozubik/2010/12/peak-internet.h...
Perhaps he should re-post his blog articles everywhere: Facebook, flickr, tumbler, watpad, wordpress, various feedback forums, etc.
Combat a DDoS attack with a DPD (distributed publishing defense - just made that up)
It could work with Facebook Instant Articles, he may even be better off using it since they source the advertising and have been out trying to poach and source quality content.
Real men mirror.
Krebsonsecurity deserves to be on git and use something like Jekyll. Mirror it instantly in a hundred different places.
http://quotes.yourdictionary.com/author/linus-torvalds/19029...
To some, the implication would will be "they couldn't handle it" so why should I trust the DDOS they are heavily promoting on their site?
At minimum they should comment on the situation, at best restore his service and learn how deal with high profile clients.
They just don't want to provide it for free.
And pretty much no one can afford to use their services @ 655Gb/s for that long unless they had billions of $.
Krebs also hasn't publicly criticized OVH like he has CloudFlare, so I could see that working out well. Would be great press for them, too.
But more specifically, whoever launched the attack cost them that money.
Also, ha:
PING krebsonsecurity.com (127.0.0.1): 56 data bytes
https://twitter.com/briankrebs/status/779144394360381440
@ 123 IN SOA ns1.prolexic.net. hostmaster.prolexic.com. 2016092204 86400 900 1209600 3600
@ 900 IN NS ns1.prolexic.net.
@ 900 IN NS ns2.prolexic.net.
*@ 300 IN A 127.0.0.1
@ 300 IN MX 10 smtp.krebsonsecurity.com.
@ 300 IN TXT "v=spf1 ip4:... ip4:... ip6:... a mx ?all"
m 300 IN CNAME krebsonsecurity.mobify.me.
smtp 900 IN A 198.251.81.28
*www 300 IN A 127.0.0.1This could trick the computers that make up the botnet to either attack themselves on the public interface (more resource-intensive than trying to DDoS your own loopback), or even better, their ISP's resolvers (it would force the ISP to do something about it).
Reminds me of: https://twitter.com/troyhunt/status/716408697266679808
Hosting static blogs is really easy on IPFS (and if you absolutely can't live without comments: use disqus) but the URL's are cryptic and you either need a public IPFS gateway to access the site - which could get DDoS'ed - or run your own.
Another alternative is ZeroNet but you still need to run the client to access the site.
If the URLs are cryptic, you can use dns to make them look nicer. Take a look at the TXT record for ipfs.io, as well as the TXT record for _dnslink.ipld.io
Both of those websites are hosted through ipfs and have A (or CNAME) records pointing to our gateways. You can also access this locally if you happen to be running an ipfs daemon at http://localhost:8080/ipns/ipfs.io
I've recently seen a ~200 Gbit/s hit us.
Does anyone have good resources around mitigation? I was looking at the BGP flowspec but was hopefully that someone might have come across other tactics?
Consumer bandwidth is increasing.
Also, medium is bad. Everyone now thinks if you publish something on medium, the writing is suddenly a masterpiece.
1: Take out the bootstrap nodes. These are several nodes that bootstrap a new client into the DHT system. BitTorrent, Inc. keeps a couple such nodes. On first boot, the client registers it's DHT address and collects a few from the bootstrapping node. The client could then can traverse the network itself. By knocking out these nodes, newly started clients now have to browse the whole IP space for possible DHT clients, which is not feasible.
2: Attack the peers themselves. A malicious program could traverse the network searching for DHT peers in the same way. At first, it would only collect a large number of DHT addresses and their corresponding nodes. Once a sufficient mass is gained, each is targeted with a low level DDOS to knock them offline to further requests. Most of these peers will be homes and local ISPs, which can't effectively deal with DDOS traffic themselves. Others trying to connect to a down client will eventually remove them from their own address space for later queries.
3: Poison DHT peers. This is probably the hardest, but once complete could poison an entire network with a switch. On each of your compromised Bot machines, you make a valid DHT node. Make a LOT of these (like a Botnet). For the most part, participate correctly with the DHT network. Collect as many valid/real DHT user and content addresses as you can and host them in your nodes. When it's time to attack, prevent these valid DHT addresses from resolving on inquiry. Even better, make them go in the wrong direction and infinitely pass around requests to other poisoned bots in your ring to prevent resolution but not hang the process. This is especially useful for content attacks because it attacks the content addresses themselves.
1b. When you have 10 million nodes like torrents, you can go searching random IPs. As long as many nodes bind to the same ports.
2. Sure, if you have comparable bandwidth to the entire network you can take it down. But that's a lot harder than overwhelming a single target. Nobody can send 20mbps each to millions of IPs.
3. This is the method that takes the least resources, but pretty good countermeasures can be made.
It doesn't; you're using compromised machines to initiate the attacks, which is free to you.
(Of course, in the very special case of Krebs, the people he is reporting on frequently are the owners of the botnets, who can of course use their own botnets freely.)
I'd guess the DDoSer is jumping with joy over this news actually, because now the DDoSer can advertise his service with "I DDoSed Krebs so hard Akamai had to drop him!"
We already see publishing through FB Instant Articles etc. moving in that land on top of the current internet, to combat these types of firehose attacks, the only solution may be to take authentication one level deeper into the connection level.
That of course sounds good to security agencies as that's the end of anonymity online.
Additionally, in situations that don't make heavy use of amplification (where egress filtering doesn't help much), the way it's usually accomplished is by compromising a bunch of hosts - home computers, routers, etc, and assembling a botnet. In those cases, if your device is compromised, it would authenticate as you anyway, so such a scheme would solve nothing.
But if you had authenticated access, you could find exactly which C&C server controlled that botnet node and then who controlled that C&C server right? All these attacks depend on some form if amplification - if only to go from C&C servers to botnet. If just being on the network required authentication, you could trace back network connections and ID the controller even if attack was by a botnet.
https://twitter.com/briankrebs/status/779111614226239488 https://twitter.com/briankrebs/status/779062433902170112
http://webcache.googleusercontent.com/search?q=cache:0uf9RIu...
And it's honorable he wants to meet Fly in person, recognizing him as a human being. I haven't read it yet but I'm assuming the reference to 12-step hints that Fly's having some post alcohol binge regrets.
I'm sure alcohol makes it easier to hurt other human beings, which is why violent people are often drunk. I'd be ashamed of myself if I woke up realizing that I'd spent my life actively trying to harm other human beings for money, feeling no remorse until Karma (here defined as law enforcement officials) finally caught up with me.
Or is it that they actually can hold it off but it costs too much money?
Akamai may also have the capacity, but bandwidth is not free.
So if the capacity of your system is X Gbps, then it will start to have problems if the attacker sends X + 1 Gbps. And will probably be completely unreachable if the attacker sends X * 2 Gbps.
The best thing is that access ISPs need to implement BCP38 (https://tools.ietf.org/html/bcp38). And shutdown all open recursive DNS servers. It would be great if Microsoft didn't ship such a retarded DNS server too. I would say that most ISPs do not do this.
NTP really should be replaced with something better. There are still large numbers of NTP amplification attacks going on. The big issue with NTP today, is that by default ntpd in daemon mode, is also a NTP server and responds to NTP requests. And so many of the two bit home routers run ntpd.
But the reality is, that no one is even reporting DDoSes right now. I work at an ISP, and I haven't seen a DDoS report in the past year. We pro-actively scan for open DNS and open NTP services. But many DDoS attacks just use regular HTTP/HTTPS, are hard to detect at the individual network connection level. Do you think Akamai sent out a single notice to any ISPs, saying "The following X IPs are sending excessive traffic to site Y, and are suspected to be part of a botnet"?
And investigation is difficult because attacking nodes might be in different countries, in some of which DDOS attacks are not illegal.
Maybe it is time to start building international firewalls to protect local infrastructure?
It's a pity Akamai booted him off; on the one hand, I can understand that it would significantly impact on their SLAs to other customers, but on the other hand it's a shame they don't have a lower impact network to re-host him on, and use this as a learning lesson on how to better mitigate such DDoSs...
"Before everyone beats up on Akamai/Prolexic too much, they were providing me service pro bono. So, as I said, I don't fault them at all."
It's also useful to point out that Krebs' hasn't been the only target as half a dozen other large targets were attacked http://www.webhostingtalk.com/showthread.php?t=1599694
Of course, maybe the goal is to deny someone ad revenue, but that seems awfully low-status for such a high-profile attack: "Yeah, we really got 'em! Denied 'em AD REVENUE for a whole week!"
> This botnet with 145607 cameras/dvr (1-30Mbps per IP) is able to send >1.5Tbps DDoS. Type: tcp/ack, tcp/ack+psh, tcp/syn.
https://twitter.com/olesovhcom/status/779297257199964160
This is much higher than the Akamai attack on Krebs too. Welcome to the wonderful side-effects of the totally insecure firmware of IoT...
But no, they'll drop this client which had to have continually given good referrals.
http://webcache.googleusercontent.com/search?q=cache:kaymYsb...