That doesn't seem right to me.
If you're buying residential it's safe to assume you're purchasing a burst-able speed and downloads/uploads aren't going to sustain that for hours on end.
Much in the same way that a Utility Company probably isn't gathering enough water for everyone to be maxing their pipes 24/7, residential ISP's don't operate under the assumption that every consumer will have maxed their connection at the same time, and they shouldn't have too. Smart traffic shaping and peering arrangements gives ISP's room to compete.
If consumers could afford a dedicated 35x15 connection they'd have a T1. When you're buying copper it's safe to assume it's going to be over saturated.
The real problem is ISP's for the most part don't compete. They zone of sections for eachother and rack in as much dough as they can. Consumers aren't informed about the quality of different types of connections vs others, and buy based off the cheapest Mbps down/up they can get.
That's not what's happening, though. What's happening is that the ISPs have not provisioned enough capacity to serve the actual aggregate demand at peak times. The hydraulic analogy would be if water pressure dropped every day between 0800 and 0900 because everyone was having their morning shower. That would not be acceptable to customers, and nor is the situation with ISPs.
Cable providers have a large amount of capacity provisioned to their customers and are paid a fortune by their subscribers and municipalities, using the same network for data and cable services. They just want you to use their services as opposed to a competitor. Not only that they have turned the tables forcing content providers to pay them to deliver bits.
With respect the economics of a small ISP reselling transit aren't comparable to this situation.
http://www.npr.org/blogs/money/2012/01/06/144737864/forget-s...
funny how the blau.de "3.5G" connection is wildly faster and less latent than the cable modem connection. actually it's kinda sad: 5GB for €10/month?
Since this is HN: yes I realize that there has been substantial work subsequently, and that cacti/munin/nagios etc.. are more commonly used.
Netflix pays Level 3 and Cogent to connect them to Comcast's network. Comcast claims that only the Level 3 connection is saturated, and that Netflix is sending all their bandwidth over Level 3 because it's cheaper for them.
Doesn't Level 3 buy a contract from Comcast that says "we get to send this much data per month"? If Netflix (or Level 3) tries to push more data through that, which causes congestion, it seems like a contractual fact that Comcast is either holding up their end of the bargain or not.
If that's true, and assuming that Comcast is transferring the contractually agreed upon part, isn't this actually Netflix and Level 3's fault? Isn't it reasonable to assume that Netflix would need to either use another entry point into Comcast's network, or build one?
If Netflix/Level 3 congest one entry point, so movies stream slowly for Comcast users, it doesn't seem like Comcast is being unneutral, in the sense of packet inspection and routing based on content/source.
The problem seems upstream from the last mile networks, and it's very unclear to me whose fault it is, and if this even has anything to do with net neutrality at all (it doesn't seem to).
Please correct me where I'm wrong, I'm sure I am!
However, you must remember that every single packet that Level 3 sends to the Comcast network is a packet that was requested BY A COMCAST CUSTOMER. Level 3 isn't just deciding to send a bunch of traffic over Comcasts network, they are sending the data requested by Comcast customers to that network.
This is an important distinction, and is usually part of all of these 'settlement-free' peering arrangements. In agreeing to peer with Level 3, I am sure Comcast has an agreement that they will not send any traffic to Comcast's network that is not actually destined for a Comcast customer. Now, if Level 3 were sending traffic to Comcast and expecting Comcast to route, for free, that traffic to another 3rd part network, that would NOT be kosher, and Comcast could fairly ask for either money or for Level 3 to stop sending that traffic to them.
Level 3 is ONLY sending the traffic to Comcast that Comcast is requesting; it is THEIR customers who are creating this demand for Level 3's customer's content. Comcast told their customers that for $X per month, they would get Y amount of internet bandwidth, which will in almost all cases be traffic originating from a non-Comcast network; that is just the nature of the internet. Even though Level 3 is willing to send the data that Comcast's customers want to the Comcast network for free (settlement free), Comcast is instead refusing to accept all the traffic that their own customers are requesting and demanding additional payment to provide the service they already sold and are paid for.
This still misses the mark.
Netflix creates a service on Level3 that Comcast customers want. So Netflix/Level3 are responsible for making a product that people want to use, and Comcast customers are responsible for wanting to use that product and increasing bandwidth use. Neither Level3 nor Comcast want to pay more money just because Netflix and Comcast customers want to use more tubes, so both Level3 and Comcast wash their hands of any responsibility for the networks they both run.
In a very real sense, both Level3 and Comcast are equally responsible for the increase in traffic [in this case]. Yet neither seem to accept their share in the cost. If it's because it's hard for Level3 to prove, they should put up some money to shore up the connection, and then write a public letter to Comcast customers telling them how Comcast is shorting them and to get angry.
(Sadly, many customers have no alternative provider, so their only option would be to quit the internet, and nobody's going to do that)
Level 3 doesn't buy the connection from Comcast. Comcast buys a connection from Level 3. Comcast needs to do this because they're selling a connection to the "Internet" to their customers, not a connection to the servers Comcast owns, and to other Comcast customers.
Comcast's paying customers are requesting data from the Internet, data that Comcast doesn't have on its network. To get there those bits travel over Level 3's (or Cogent, or another major provider) network. The issue at hand is not one of Level 3 (or Netflix) trying to PUSH data onto Comcast's network. It is one of Comcast's customers trying to PULL data from Level 3, over a connection that Comcast refuses to upgrade.
If you had two truly distinct cable options, say Comcast and TascCom. Your friends with Comcast say Netflix sux, low quality, lots of problems. Your friends with TascCom say netflix is awesome, great HD content, too bad there's only like 8 3D movies! You'll sign up for TascCom. Comcast's board will get angry they're losing customers, and fix their interconnects. But it's all a dream, there is no TascCom, there's only Comcast.
Disclaimer: The blog I'm about to post was linked from Comcast's blog. They claim he is independent, I know nothing about him.
"Netflix could use multiple providers to connect to ISPs and could also use third party CDNs like Akamai, EdgeCast and Limelight, who are already connected to ISPs, to deliver their traffic. In fact, this is how Netflix delivered 100% of their traffic for many, many years, using third-party CDNs. Netflix likes to make it sound like there is only one way to deliver videos on the Internet when in fact, there are multiple ways."[1]
It would seem to me that if Comcast is using all the bandwidth they've purchased from Level 3 than it's a simple reality for them that they'll need to increase that bandwidth or they'll see reduced service. I don't see why this is explicitly either side's problem. It seems like it would be great if they could figure out a mutually beneficial situation, but I guess I fail to see the moral hazard. If Comcast doesn't want to upgrade their network who cares? Barring, of course, the monopoly argument, I see no moral hazard here.
I also see no reason that this has anything to do with net neutrality. It seems like Netflix is simply so big they have to act more like an ISP than they used to.
[1] http://blog.streamingmedia.com/2014/03/netflix-level-3-telli...
Only, I'm sure, because it's untenable to do that. Yet. Cable execs really, really like their model of charging people for channels; it lets them extract maximum revenue. If they can do that with the Internet, they will. It may be difficult to do that on the consumer side yet, but what they're doing with Netflix is basically a complicated way to sneak more Comcast charges on to their customer's bills. In this case, via the Netfilx bill.
EDIT: I understand it's an emotional issue but it's true. You can only send packets and hope others will talk to you.
Comcast is connected to the internet through 'settlement-free' peering. Comcast can command this because they have grown large enough to trade access to their customer base for access to the transport network's data. Comcast would like to sell direct access to their network to the content providers, but would like to see more money for the service that the content providers are willing to pay.
Comcast has allowed (through inaction of normal peering upgrades) the interconnections to Level-3/Cogent to become saturated. The primary data on those interconnections is data that competes with the normal content that Comcast is used to delivering to their customers. Cogent and Level-3 offer less expensive bandwidth than their competitors. This is because of their lack of incumbent telephony models, and being dedicated fiber bit pushers.
Comcast is claiming that it is too expensive for them to deliver the service levels to their customers that their customers have contracted. Comcast is demanding that the companies providing the data (not the transit network delivering the data) must pay to have their traffic delivered the last mile. Simultaneously to crying that their internal networks are saturated and require upgrade, the MSOs are offering products that allow their customers to use the local provider's network (for locally sourced content) without incurring billing penalties or usage caps.
(veteran employee of a transit network involved)
The exception to this is if Comcast wants to charge rent to the internet to have access to Comcast customers. Then it is in Comcast's corporate interest to let peer connections saturate and then charge upstream content providers rent to remove the congestion.
A pretty ballsy move for a company (who's assets are stretched out over thousands of miles and in plain sight) to attempt to extort the entire planet. ;)
I like the cut of your jib.
What's happening here is that Level3 and other transit providers are starting to see their industry be squeezed by the big ISPs. If you have 5 ISPs that serve 90% of the customers in the US, and 10 service providers that generate 90% of the bandwidth, why do transit providers even exist? I think they see their market share decreasing significantly as more of their customers follow in Netflix's footsteps, so they're trying to pile on and start a grass-roots outrage like Netflix did. Problem there is that most people haven't heard of Level3, so it makes it more difficult to get people behind them.
And you are completely right about the whole Netflix issue being Netflix and Level3/Cogent's problem. Level3 and Cogent have business models based on selling transit acquired through settlement-free peering. Settlement-free peering assumes that bandwidth usage is roughly symmetric: when it's not symmetric, the side sending more bandwidth has to pay. Level3's position here seems a bit hypocritical: if they had a customer that was routinely sending data at a 20:1 ratio, they would charge that customer for sending more bandwidth. But they're expecting the big ISPs not to do the same to them?
If peering is untenable because it leads to outcomes like the one we have now (Netflix) where it is fundamentally unfair to one party (assuming that is the case) or the other, then we're all screwed. There might not be a fair solution that still manages to resemble the Internet.
In one alternative, the Comcasts and the AT&Ts are smited and no longer exist, and Level 3 goes into the consumer internet business, connecting things end-to-end. That's not something we could trust them with. Same if Comcast replaces Level 3.
In another, Netflix continues to pay the extortion (if it is that) which we have little doubt that Comcast will continue to ratchet up the price for. Or Comcast has to shoulder that burden alone (and can't charge customers extra for doing so, without people screaming "net neutrality!").
Or maybe services like Netflix just can't exist in such an environment. Also a bad outcome.
Or god help us, internet infrastructure is nationalized, and the same people who manage our roads and traffic lights take over.
Am I blowing this out of proportion?
Are you talking about on this graph?
http://blog.level3.com/wp-content/uploads/2014/05/route_info...
Because all the numbers I see are showing about a 5:1 or 6:1 imbalance.
> ...transit providers are starting to see their industry be squeezed by the big ISPs.
No.
> ...why do transit providers even exist?
Because it's untenable (and inefficient) for Comcast to build a separate fiber network to every service provider (e.g. to start providing a service on your version of the internet, you'd have to build your own network connection to Comcast, and then again to Verizon, and again to AT&T, and again to every other ISP on the planet).
Take another look at the L3 network map from the blog post:
http://blog.level3.com/wp-content/uploads/2014/05/network_ma...
If you think transit providers are purely middlemen for the sake of middlemen then you're basically saying that every service provider should build out a network of that same size in order to reach every ISP.
> Settlement-free peering assumes that bandwidth usage is roughly symmetric
No. It assumes that the benefit to each network is roughly symmetric. Put another way, if Netflix videos only degrade on Comcast's network, Comcast's goodwill with its own customers will increase as a result of increasing transit/peering capacity. To a normal company, that's a tangible benefit to the agreement enough to make it worthwhile.
Comcast empirically does not care about the goodwill of its customers, which is not surprising given its monopoly of local markets. If customers had alternatives to Comcast and it became known that Netflix worked on one ISP and not the other, customers would switch ISPs en masse.
I've never understood why this is - especially in a case like this, where the sending side is sending because (the customers of) the receiving side explicitly asked for it.
Does it assume that?
Since about the time of the web browser, we've had consumer-focused ISPs. Consumer traffic is mostly small requests for large replies. But as the article suggests, Level3 has a lot of settlement-free peering with consumer ISPs. My belief is that being settlement free isn't about equality of packets, but equality of demand. E.g., I have a lot of servers made to serve consumers; you have a lot of consumers wanting access to my servers; let's just split the costs.
In any case, given that those Comcast packets are mostly requesting all those Level3 packets, it seems much fairer to charge Comcast disproportionately, as its their customers who are creating the demand.
Even if you want to privileged "sending" as being more charging worthy since you control what you send— in this case it's Comcast customers that have requested those bits, hosts connected to the level3 network are providing them.
Networks like comcast have built out imbalanced networks which likely wouldn't exist without their access to monopoly infrastructure. Practically anyone they peer with except other consumer broadband monopolists is going to be unbalanced.
Comcast is playing the monopoly card against what is an oligopoly of backbone providers. Comcast isn't strategically as well positioned as their actions would indicate.
It is one thing for Comcast to have degraded Netflix service. It is a different thing to present their customers with access to only half of the Internet .
Obviously Cogent would be willing to cut off their own nose to spite their face, but Level3 plays a much more sane game.
Hardball by coordinating efforts of the backbone providers is somewhat thwarted by the coordinated efforts of the MSO's, some with their own backbones (VZ, AT&T). Cutting off Comcast would also invalidate a number of (government) customer contracts that require service 'to the full Internet'.
Whenever their performance drops towards part of the internet, their value proposition gets lower. You can view the value chain as
(residential customer) <-> Comcast <-> Level 3 <-> (hosting customer)
Level 3 and equivalents, by cutting off non-performant interconnects will hurt their own customers, of course. However, they are just speeding up the same effect being applied softly over time, by Comcast and equivalents. A softly failing interconnect moves revenue over to last mile ISPs, thus permanently lowering Level 3's value proposition. By failing the interconnect sharply, instead of softly, they actually improve the steady state of the system, in general (good side effect), and in particular (the real objective).
Comcast cripples customers internet services to extort money from companies connecting Comcast to the internet for free.
The current system is pretty straight forward.
Level 3: Server(Who Level 3 Charges) -> Level 3 -> Comcast -> End User
Comcast: End User(Who Comcast Charges) -> Comcast -> Level 3 -> Server
Now Comcast wants to charge everyone in the chain that isn't them. They are attempting to do this by degrading (or at the very least not upgrading) their services for their customer.
The techno-libertarians that try to defend this nonsense are going to sink the whole ship and make every pay to receive even less.
But though they are a large ISP they are not dominant, and their peering with Level3 is actually quite good... which is why I tunnel most of my connection to my server (the route to which happens to be through Level3) so I can get usable internet.
It really sucks to be able to download an iso from some server in the US at 2 MB/s, but seeing small Imgur gifs load frame by frame, or Github repos cloning at 20 kB/s.
Telefonica in Spain might be a reasonable guess. Ono the cable company isn't huge.
I think Telecom Italian is dominant in Italy and there is virtually no cable at all so that would be my guess.
* http://www.golem.de/news/level-3-sechs-grosse-internetprovid...
* http://www.welt.de/wirtschaft/article117963991/Bei-Europas-T...