Is it a simple fix like Level 3 & Netflix claim? If so, what is Verizon's advantage to continue the slow Netflix. Netflix is really slow for Verizon users, it is very obvious for all their subscribers.
This is the increasingly common case of an ISP wanting to have it's cake and eat it too. It's an analogy of course, but it's a good one.
Verizon has either a monopoly or a near monopoly on the last mile from POPs to it's customers. In other words customers have no choice or little choice but Verizon.
And these customers pay Verizon for some amount of bandwidth and expect Verizon to make a good faith effort to deliver that bandwidth. The Terms & Conditions always say "up to X Mbps" because Verizon can't control network congestion once the data is off of Verizon's network. It might be that once the data leaves their network it has to go to Australia which has notoriously poor internet as a result of being kind-of out of the way. Or maybe it heads to Africa and starts making cell tower hops in which case it's going to be abysmal.
But in this case Verizon isn't making a good-faith effort to deliver the bandwidth that their customers are paying for to them. What they want is to extract extra money from someone who also provides a service to their customers, namely Netflix or Level3. Why do they want to do this? It's hard to say for sure. Trying to make extra profits, trying to keep Netflix slow and encourage people to subscribe to Verizon cable TV, spite? We can't know.
But what is clear is that this isn't the fault of Netflix or Level3. Level3 has said they will upgrade their side of the connection for free which would reduce congestion and Netflix has offered to locate servers on the Verizon network for free, which would reduce the Verizon/Level3 network congestion.
But Verizon refuses to accept either of those offers. Why? I'll venture a guess: it's not because they have the best interests of their customers in mind. If they did, they would have upgraded already.
To me it seems like they're abusing monopoly(ish) power.
They're not necessarily doing this "for free", but as a part of the service their customers have already bought and paid for. It's not doing anything extra, it's normal operations, it's keeping the promise the business made to the customer, which is the opposite of what Verizon is doing in this case.
http://knowmore.washingtonpost.com/2014/04/25/this-hilarious...
One way to see the conflict is that it is about what status verizon have - are they considered an end user isp or is it a transit provider? L3 is in the right if they should be considered the former, and Verizon is right if they should be the latter.
EDIT: John Oliver also talks quite eloquently about this as well: https://www.youtube.com/watch?v=fpbOEoRrHyU
That's what I would call a mortal wound to Verizon's arguments.
Verizon's argument is clear:
> Netflix chose to attempt to deliver that traffic to Verizon through a few third-party transit providers with limited capacity over connections specifically to be used only for balanced traffic flows
Level3 hasn't refuted this point (the imbalance of the traffic flow). If the traffic is not balanced, the contract between Verizon and level3 probably has clauses requiring Level3 to pay Verizon for the imbalance (in this case, since Level3 is pushing more data, they would be the ones paying). That is the real problem, most likely, and the technical argument merely serves to confuse
No. That's not how the internet works. Nobody PUSHES data. People PULL data.
Verizon has customers who pay for internet access. Those customers make REQUESTS to Netflix for data and Netflix RESPONDS with data.
They happen to be streaming movies which use a lot of bandwidth, but on average it's about 3Mbps per concurrent stream. That is WELL below the 25/3 or 50/5 or 100/10 that Verizon advertises for purchase.
If someone were pushing data it would be called a DoS or DDoS. An attack is when someone sends unrequested data to try and break your network.
But this data isn't unrequested. Verizon's customers have requested it from Netflix as they are within their rights to do since they have literally paid for it.
This is Verizon wanting to bill their customers once and then their customers' vendors as well. Double billing for a single service is a neat trick if you can pull it off. But it tends not to engender goodwill.
For example, suppose there are 5 separate networks, run by different groups: A, B, C, D, and E. C connects to A,B and D, and D connects to C and E. In this case, the only way networks A and B can connect to D or E is through the connection between C and D.
Now, if in this arrangement, the D or E networks are sending/requesting a lot of traffic to/from networks A and B, while C is rarely requesting data from D or E, then network C might have a fair argument to make that D is abusing their balanced-flow peering agreement. Basically, they are being asked to be a middle-man for all the traffic going between networks that are NOT theirs. They might ask for compensation for doing this job (carrying traffic between two other networks).
However, this is NOT what Verizon does. All of the traffic between level 3 and the Verizon network is bound FOR A VERIZON CUSTOMER. They are not a middle-man, they are the end destination. Now, if Level-3 is sending data over those congested level-3/verizon gateways that is bound for, say, AT&T's customers, then they would have an issue and could fairly demand payment.
That is not the case, however.
Verizon's argument would be a lot stronger if they didn't attempt to say that Netflix chose balanced flow contracts. They are saying that Netflix went shopping for backbone providers and chose the ones that had balanced traffic flow agreements to deliver video across, which is just silly.
Such rhetorical overreaching is a common debating error, and can ruin an otherwise good argument. Verizon may or may not have balanced flow agreements, I don't know what the contracts are, but now I don't trust Verizon to honestly report on them based on their imputation of Netflix's motivations.
Oh, please. If that's really what's stopping this from being fixed, then I would be hard pressed to find a better illustration of the need for regulating "last mile" providers as public utilities. We're talking about an investment that is not even rounding error in Verizon's accounting, to connect Level 3's network to a Verizon network that, by Verizon's own admission, has plenty of excess capacity that is not being used. So the whole idea that, if the traffic isn't balanced, Verizon is somehow incurring huge extra costs that they need to be compensated for, is obviously bogus, which means their contractual terms are nothing less than an attempt to use their privileged position to extract huge monopoly rents in exchange for no added value whatsoever.
"Netflix chose to attempt to deliver that traffic to Verizon through a few third-party transit providers with limited capacity over connections specifically to be used only for balanced traffic flows"
But isn't that a misstatement of fact? CDNs like Level3 do nothing but transfer data, usually media files, onto consumer ISP networks when those ISP's customers' request them.
In what dream world are Akamai and Level3 receiving as much data from Verizon's consumer network as they are sending? Why would any ISP expect a link with a CDN to have a balanced traffic flow?
But it is Verizon customers, who request the traffic and Verizon is not providing the best quality, because they don't want to upgrade their own equipment and the peering with e.g. Level3. In fact, it should be Verizon paying to Level3.
Verizon looks at it as if Netflix and Level 3 are pushing the Data, everyone else clearly sees that they are pulling it.
That's the way the Internet works - since consumers are generally prohibited from running servers (and even if they did, they wouldn't operate at the same scale), traffic from consumers is always going to be orders of magnitude less than traffic to consumers.
All of the data that Verizon is receiving from Level 3 is in response to a request that a Verizon customer has issued. Because the size of a request is much smaller than the size of the payload (response), the traffic will always have this pattern.
Do you have any evidence supporting that Level 3's contracts with Verizon require them to pay? This is the first I've ever heard that implied, and it would be very surprising, since it would be incredibly foolish on Level 3's part (they know that the traffic will always be shaped like this).
Verizon wants to bill it's customers for the bandwidth that they use. And then for some of that traffic, if it's from Netflix, they also want to bill Netflix.
Verizon is the company which is selling asymmetric internet plans so it should come as no surprise to Verizon that their customers are requesting more data than they're sending.
But they use this asymmetry to suggest that what's happening between Verizon and Level3 isn't business as usual (which it is) and that they should be able to double bill.
Verizon is doing a decent job of spinning but if you really look at what's going on it becomes clear that they're really torturing definitions to make their arguments.
I'm a Verizon subscriber and I get apoplectic when Netflix stutters, because I know exactly what's happening. But I also don't know what's the exact right model for peering points, and I think net settlement might be a better model.
>Q8: Comcast says that Level 3 sends it 5 times the traffic that Comcast sends Level 3. Is that true? If it is true, why shouldn't Level 3 pay for the traffic it sends to Comcast?
Comcast goes on to argue that Level3 pushes 5X more data onto the Comcast network than Comcast pushes onto the Level3 network. What I find ridiculous about the whole claim is that every single one of Comcast's residential internet packages has 5X higher download speeds than upload speed. (e.g. 105Mbps/20Mbps, 50Mbps/10Mbps, 20Mbps/4Mbps)
Comcast provides internet speeds that force customers to download 5x faster than they can upload and somehow Comcast is surprised that their customers are pulling down 5X more data then they're pushing to Level3?
[0] http://corporate.comcast.com/comcast-voices/20-qs-with-accur...
Regardless, if Level3 keeps insisting this is a peering point, and then demonstrating that it's more like a transit to ISP link. Which ISPs pay for.
I actually don't know. Is that legitimate?
For transit providers, asymmetry = Paid.
For local ISPs, they are typically willing to asymmetrically peer because they'd otherwise have to pay transit.
Since verizon is a T1 transit provider, it doesn't want to give it's service away for free.
I think the fair answer in this dispute depends on whether L3 and Cogent are forcing Verizon to provide transit for free or are they just delivering the data the local network for the customer (like a local ISP peering arrangement).
I'm not sure which is going on. If L3 is just dumping Netflix packets on the nearest Verizon peering location, well then Verizon is providing them a transit service. If L3 carries the data all the way to the local network, then it's not transit at all.
I'm not sure what is actually happening. I'd imagine a mix of both.
From Wikipedia:
"However, the most common definition of a tier 1 network is a network that can reach every other network on the Internet without purchasing IP transit or paying settlements."
Not really, for the consumer ISP side of its business Verizon absorbs more traffic than it produces by definition: consumers don't typically produce much data outside of p2p.
Furthermore, Netflix has offered putting server inside Verizon's network which makes this argument moot.
VZ just wants to double-bill.
But honestly, it couldn't be done. Not with the upload restrictions at the end of the pipe.
I'm not getting why I'm being down voted for posting the cached version when I was getting the 404.
edit: this one seems a bit better:
http://webcache.googleusercontent.com/search?q=cache:DBHDyx7...
But, assuming they really did need to throttle traffic in order to protect their own network. For one it wouldn't make sense to control your throttling by running your routers at 100% capacity. But also any throttling should be equal among all of their peers and not selectively chosen based on Verizon's business interests.
By definition, the only way they can be 'crushed' is if they're not using the funds they get from their customers for maintenance of their network.
Even with Net Neutrality it's fully within an ISPs right to say X GB/mo for $Y and throttle or block over-use. The problem is an ISP deciding that, regardless of what is going on in the network, regardless of what a particular user has used or is trying to use, traffic to Netflix will be slower than traffic to Vudu.
Verizon, Comcast, et al have plenty of available tools to prevent congestion without violating Net Neutrality.
They're intentionally trying to confuse the issue with usage, because it's an easier PR sell.
With comcast my data cap is 300GB a month, I should be able to use 300GB of whatever I want Youtube, Netflix, hell even torrents or file sharing. Doesnt matter where it comes from I'm paying for 300GB of data at X speed. How hard is that to get?
Netflix, on the other hand, has paid good money for a very fast and reliable connection all the way to the Level3/Comcast PoP.
For yet another analogy in this thread - if your server was instead sitting in a data center with an open 10GE ethernet port or two, right next to a Comcast router, yes, they should just install the extra NICs and wire the things up without for the price of supplies and installation.
Think about your own ISP. I pay $80 for service, but to get the same bandwidth shared between 2 ISPs I'd have to pay $120 plus now I'm doubling my hardware costs (routers to each ISP).
Or perhaps they could ask Level 3 (who are clearly pissed at Verizon) to make the deal for them. These edge providers already have business with each other (peering), so it should be relatively simple for Level 3 to purchase some bandwidth from another large provider and then route some of their packets (e.g. the Netflix ones) through there.
In any case, the edge providers have good reason to collaborate on thwarting Verizon on this, so why don't they?
But Level 3 claims it would be very inexpensive for Verizon to increase capacity there. Just a few port cards and a few thousand bucks for a major city like L.A. They even offered to pay Verizon's costs.
This is Verizon wanting to double-bill for traffic. Plain and simple.
> Verizon wants them or their transit provider to pay more to "ensure a level of capacity that accommodates their volume of traffic".
That's Verizon's job. To go out and get by any and all means necessary enough bandwidth to satisfy their customers, namely the residential customers who seem intent on using Netflix. At no point do they have a moral imperative to hold their customers hostage to extort money out of someone else that their customers are trying to access. It's already been paid for!
All of your comments keep ignoring or misunderstanding what peering is. It is supposed to be roughly equal data traffic for both sides. Once it is lopsided in one direction, it is no longer peering.
It doesn't matter who requested what, the fact that L3 traffic is consuming a majority of the bandwidth now is the issue, and Verizon wants L3 to pay to fix that.
The haggling over semantics here doesn't really change the substance. Ok, "Netflix" isn't saturating the interconnection.. "customers using Netflix" are saturating the interconnection. And it would still reportedly cost Verizon very little to fix it.
I think that points the finger pretty strongly at Verizon.
If we do that, we find that physical link speeds of any variety have improved by many many magnitudes over. As this statement suggests, 10GBit is absolute jellybean in the networking world. Yet, last mile bandwidth available to customers has stagnated. Imagine a perfectly paved autobahn, and here we are discussing if we should be able to drive 20 mph on it, and who should pay for the extra damage that will cause to the surface. Incredulous.
There are some regulatory systems, where in a goldilockes moment, these basic truths are recognized and acted upon. Consider the EU drastically cutting roaming charges. What was so blatantly obvious that even the senile, crooked decision makers at the EU couldn't miss it? All the providers already roam over the internet, at basically zero cost for both parties if you consider that speeds are limited by the over-the-air interface at either end. Yet they kept charging customers the price for a specially erected microwave link from 1960.
now i just torrent everything so i can watch uninterrupted when i want. I consider i am still legal since i'm paying for both subscriptions yet and they pay the content. if not, not my problem.
but that raises a question... why the RIAA/MPAA is not using its bullying powers to harass ISP like verizon?
as i just showed, they are the reason lots of people are going back to torrents that do not make money to the studios like streaming services does. Why do they rather sue john does instead?
Because it presupposes that whether or not in fact, streaming via Netflix is profitable or not, the studios didn't have to be dragged kicking and screaming to allow it at all.
I wonder if Netflix or Verizon would lose more customers...
They should continue to allow access, but they should also keep saying when the problem is with the network provider. Customers love Netflix (NPS of 54) more than Verizon (NPS of 32), so they'll be more inclined to believe Netflix.
But after reading this... am I the only one that thinks peering agreements are done bass ackwards? If Verizon's customers are _requesting_ Netflix access, shouldn't Verizon pay Level3, not the other way around?
The providers that Netflix does not use do not experience the same problem.
This needs to be put in front of the FCC since it is the bare minimum language they should be able to understand.
