In an extraordinarily long, nearly 6000-word blog post dubbed “Jammed,” persistent telecom critic and net neutrality proponent Susan Crawford argues that newly published data from M-Lab proves that virtually all major US ISPs engaged in a conspiracy against Netflix, to the detriment of consumers.
As has been common for arguments made by net neutrality activists, and by Ms. Crawford in particular, this latest argument includes no shortage of factual misinformation about technical matters, about the history of the internet and regulatory policy, and about the economics of internet service. In making her argument, Ms. Crawford also cites an OECD report that, in fact, resoundingly rejects her assertions that access networks (or “eyeball networks,” as she calls them) deserve blame for peering disputes, and that consumers and the development of the internet have suffered due to those disputes, as well as rejecting the aggressive, heavy-handed, interventionist internet regulation she espouses. (See exemplary excerpts from the OECD report, at the end of this post.)
Like many net neutrality activists, Ms. Crawford appears so consumed by religious fervor in the belief that major ISPs such as Comcast, Time Warner Cable, Verizon and AT&T are evil, that she seems simply unable to process factual information or make logical inferences without coloring them to fit her worldview of those ISPs as villains doing harm to hapless consumers and heroic content providers, alike.
Consider, for example, a claim Ms. Crawford makes regarding the impact of an apparently inadequate interconnection between Cogent and Verizon on customer connections:
Just after the handoff from Cogent, Verizon loses many of the packets, often causing the data transfer to fail.
As anybody with even a basic understanding of networking would know, however, the packet loss in such a case would have to occur prior to the Cogent handoff, not after it. The purported problem is that the connection between Cogent and Verizon is insufficient to carry all the traffic Cogent has to send to Verizon, so Cogent can’t hand all of it off to Verizon. In this situation, it’s not that Cogent hands the traffic off to Verizon, and then Verizon drops much of it. It’s that Cogent would have to drop much of the traffic before it ever got to Verizon.
Think of it like a funnel being used to pour gas from a can into an empty gas tank. If you keep pouring gas into the funnel faster than it can pass through the narrow end, some of the gas spills out from the top of the funnel, before it can get into the tank. It’s not that the excess gas goes through the funnel, into the tank, and then gets lost from the tank!
In her zeal to see Verizon as a villain, however, Ms. Crawford is not satisfied to simply argue that Verizon should be blamed for the packet loss. Rather, she adopts a clearly erroneous view that it is Verizon who actually drops the packets, because it fits her worldview of ISP villany for Verizon to be the party actually dropping the packets, and that worldview contorts her perception of facts and logic.
Unfortunately, this contortion of facts and logic is not limited to an occasional minor point here and there, but pervades the entirety of Ms. Crawford’s argument, including her primary thesis. Ms. Crawford’s sense of logic and evidence is overwhelmed by her pre-ordained presumptions that access network ISPs such as Comcast, TWC, Verizon and AT&T are villains, engaged in conspiracy against consumers and content provider heroes like Netflix.
Ms. Crawford tells us we should believe that when Netflix switched a substantial portion of its traffic delivery to transit-provider Cogent, all of the major access network ISPs immediately, and simultaneously, decided to engage in anti-competitive business practices to punish Netflix by restricting their traffic from Cogent (and also catching smaller Cogent customers “in the crossfire”).
As evidence, Ms. Crawford cites data showing the simultaneous and dramatic decline in performance of Cogent traffic delivery to the major ISPs, coincident with Cogent’s agreement with Netflix.
Perhaps recognizing that Cogent is well-known within the industry for its lack of reliability (Ms. Crawford herself observes that Cogent’s business strategy is to gain customers by substantially undercutting competitors on price), Ms. Crawford attempts to disprove the possibility that Cogent (or Netflix) might bear any responsibility:
One possibility, of course, would be that the problem is with Cogent’s network. But that is not the case. We know this because the M-Lab data also show that Cablevision users in NYC did not experience problems with traffic coming from Cogent between June 2013 and March 2014. Cablevision maintained parity and high throughput for both Cogent and Internap throughout this period.
And why was Cablevision different, according to Ms. Crawford?
Why would Cablevision, which is not available in Comcast or Time Warner Cable territory, act differently? Because, unlike Comcast, it has to compete. More than half of Cablevision’s territory overlaps with that of Verizon FiOS; only 7% of Comcast’s and 11% of Time Warner Cable’s does. And so Cablevision (but not Comcast or TWC) makes every effort to ensure that Netflix works well for its customers, including by allowing Netflix to bring its content inside its network—via Netflix’s “OpenConnect” content delivery network—at no cost. Cablevision, unlike Comcast or Time Warner, is at risk of losing its customers to Verizon’s fiber services.
Ms. Crawford is certainly correct that Cablevision, with roughly half of its territory subject to competition with Verizon FiOS, has far more competitive pressure as an ISP than Comcast or Time Warner Cable. But Ms. Crawford somehow misses the fact that two other ISPs she accuses, Verizon and AT&T, face much more competition than even Cablevision. In fact, both of their systems (FiOS and U-Verse, respectively) upgraded to compete with cable ISPs, face competition in essentially 100% of their footprint—by Ms. Crawford’s logic, Verizon and AT&T should have even more competitive motivation than Cablevision to eliminate problems!
In reality, there is a much simpler explanation for the Cablevision divergence—a divergence that, in fact, doesn’t just include Cablevision, but also Cox, a large ISP that, like Comcast and TWC, faces much less competition than Cablevision.
As Ms. Crawford herself notes, by spring of 2013, Cablevision (and Cox) had already joined Netflix’s OpenConnect CDN program. As a result, the vast majority of Netflix content delivered to subscribers on Cablevision’s and Cox’s networks would have been served from that CDN, rather than via transit through any third-party. Even if Netflix actually routed all of their transit to Cablevision and Cox through Cogent, that traffic would have been a tiny fraction of the traffic routed to other providers.
So all that the divergence actually tells us is that Netflix and Cogent suddenly started routing a tremendous amount of traffic to non-upgraded Cogent interconnections with many major ISPs (each of which had, coincidentally, declined to join Netflix’s OpenConnect program and grant Netflix privileged access to their networks).
Needless to say, this is not normal practice.
Normally, a transit provider would ensure that they have upgraded all interconnect points to be affected before unleashing a torrent of new traffic on those interconnect points. And normally, a content provider would validate the quality of a new transit provider before re-routing a huge amount of traffic through them.
Here, neither of those things happened.
What’s more, even though Netflix’s own data demonstrates that each of the ISPs Ms. Crawford accuses was successfully carrying more Netflix traffic prior to the sudden switch to Cogent, Netflix persisted in routing traffic through Cogent, long after it was obvious that Cogent was unable to successfully deliver the traffic.
The correct logical conclusion from this is simple: Netflix intentionally redirected traffic to inadequate Cogent interconnects. Presumably, Netflix did this in order to pressure those ISPs to rethink their position on OpenConnect, and grant Netflix special privileges, unavailable to other content providers. As we have explained previously, the standoff between Netflix and major ISPs was indeed a game of chicken between Netflix and ISPs, but it was a game of chicken that Netflix initiated and controlled, not the ISPs.
And for whatever large sums of money Netflix was willing to shift to Cogent for this huge shift in transit, Cogent was apparently perfectly willing to throw its own other customers under the bus, knowing that they would be “caught in the crossfire,” as Ms. Crawford says.
Not surprisingly, this much simpler and logically consistent explanation is precisely what independent transit analyst Dan Rayburn concluded from the same M-Lab data Ms. Crawford cites, particularly noting that:
While some may want to take this report as a smoking gun that ISPs are causing congestion, they may forget, not understand, or purposely leave out, the fact that large content providers control the delivery of their traffic and can AVOID congestion. A recent MIT study “Measuring Internet congestion: A preliminary report” pointed out the fact that the ISPs singled out in this report have multiple alternative paths to reach them. The report states that, “Congestion at interconnection points does not appear to be widespread. Apart from specific issues such as Netflix traffic, our measurements reveal only occasional points of congestion where ISPs interconnect. We typically see two or three links congested for a given ISP, perhaps for one or two hours a day, which is not surprising in even a well-engineered network, since traffic growth continues in general, and new capacity must be added from time to time as paths become overloaded.”
The simple reality is that Netflix was not interested ensuring quality. Quite to the contrary, Netflix was flexing its own market power muscles, and was specifically ensuring poor quality—presumably believing that, by doing so, they could bring more consumer and political pressure to bear on the ISPs than they would bring on their own business. Netflix believed they had the upper hand, and they were not the least bit reluctant to try to exploit that.
(If Susan Crawford is looking for anti-competitive business practices, she would do well to look at content providers like Netflix, for it is content providers who have actually demonstrated a repeated willingness to interfere with open internet principles, while the alleged threat from ISPs remains almost entirely based on conjecture of what ISPs might do.)
Ironically, while net neutrality proponents like Ms. Crawford have recently taken to exhorting support for Title II regulation of ISPs with the argument that any payment to access network ISPs for terminating internet traffic is some kind of abomination, they strangely ignore the fact that, under Title II, regulation of phone service actually imposes government-mandated termination fees for phone traffic: fees that Title II regulation holds to be important for ensuring sufficient capital to build and maintain capacity in those networks terminating traffic, and fees which the OECD observed are some five orders of magnitude (or a factor of approximately 100,000) greater than the typical cost of privately negotiated paid peering of internet traffic.
But as John Oliver amply demonstrated, it doesn’t require an accurate understanding of the facts, or sound logic, for net neutrality proponents to win popular support. All that’s needed is a hand-waving (or in John Oliver’s case, comical) argument that sounds plausible to a public not savvy to the technical details, and which doesn’t require much encouragement to buy into conspiracy theories of bad behavior by ISPs that they are already inclined to dislike and distrust.
For those who are interested, below are some example excerpts from OECD Digital Economy Papers No. 207: Internet Traffic Exchange, cited by Ms. Crawford, demonstrating that the OECD has actually adamantly rejected her claims of access network ISP misconduct and market failure, as well as her proposed aggressively interventionist regulatory approach.
For those who aren’t quite so interested, try reading just the bolded passages, to see how the OECD concludes that the current system of negotiated peering has actually benefited consumers by improving quality and reducing costs. (Emphasis added.)
The contrast between the results observed in the Internet market and comparable markets for exchange of traditional circuit-switched voice (time-domain multiplexed, or TDM) traffic is striking. […] Stated in terms of an equivalent per-minute price for delivery of voice traffic, this is less than USD 0.0000008, five orders of magnitude less than wholesale rates for services providing comparable functions in TDM markets. The reasons for this performance include the efficiency of packet-switched technology, competition in Internet markets, and the flexibility of routing arrangements among Internet networks. The market has also benefitted from the policy environment, in which governments have refrained, in most OECD countries, from regulation of the market for Internet traffic exchange.
[…]
In a market based on voluntary agreements, it is inevitable that parties will not always agree. In the Internet, where less than 1% of the possible bilateral arrangements are actually in effect, this is not a cause for concern. Occasionally disputes rise to the attention of policy makers, and more rarely may actually lead to some disruption. While there must be some limit to the amount of disruption that a private dispute can be allowed to cause, governments should generally resist the temptation to intervene.
[…]
As noted above, the structural evolution of the Internet has led to realignment of the roles of Internet participants, including creators of content, online distributors of content, CDNs, backbone networks, and access networks. This has led to negotiation of new agreements for Internet traffic exchange. As agreements among online service and content providers, CDNs, and access networks are negotiated, a balance is being struck on the extent to which quality-enhancing resources will be brought to bear, who will provide those resources, and on what terms. In the process, the market for Internet traffic exchange is generating answers to many of the questions raised in recent debates over network neutrality.
The report reviews these market developments, as well as some disputes which have arisen within the last year, including those between Cogent and Orange in France, and between Level 3 and Comcast in the United States. In general, the market appears to be developing in an orderly way, with the outcomes falling within a relatively limited range that appears to be reasonable. The best course for regulators at this point may be to monitor and observe this process, as the NRAs in these two cases have chosen to do. It is not clear how intervention in favour of any one party or group (content providers, CDNs, access networks) would improve the outcome.
[…]
When there is disagreement between commercial parties, there is an incentive for one or the other to seek intervention by regulatory authorities. Any real possibility of such intervention will overhang commercial negotiations. In 2005, for example, Level 3 notified Cogent [Editorial: both “good guys” by Ms. Crawford’s logic] of its intent to end their peering agreement. When the notice period specified in the contract expired, Level 3 terminated the arrangement. Because many of Cogent’s customers had no alternative path to Level 3, their access to the subset of Level 3’s customers who were single-homed was cut off. In the United States, Cogent made appeals to the FCC and Congress for intervention. Eventually a new agreement was reached. It is not clear to what extent the new terms reflected only commercial considerations and to what extent they were influenced by the visibility of being fought out in the press and in front of potential regulators.
It is clear that a form of implicit regulation can be applied by allowing market participants to believe that intervention is possible. In a few cases, such as in an emerging economy where the market is not well established and the incumbent has a protected position, a certain amount of informal discussion may be beneficial. In well-functioning Internet markets, though, allowing the possibility of intervention to influence parties’ negotiating positions runs the risk of distorting the outcome in ways that are not beneficial. For this reason, it may be advisable for the regulator, having drawn a bright line, to stick to it with a certain amount of clarity. To be sure, there must be some limit to the amount of disruption a private dispute can be allowed to cause, just as there is with a labour dispute, for example. But if this range of outcomes is too narrow, or applied inconsistently, it will have negative consequences for the market.
[…]
Given the rapid growth of CDNs over the past few years, there is no market evidence that access providers have been able to prevent the delivery of services on this basis or to extract unreasonable terms. On the contrary, some observers, noting the concentration of large online service providers, have wondered whether their countervailing market position may be too strong. In many cases, parties have agreed to exchange traffic between CDNs and access networks on a peering basis. Where payment has been made, it has evidently not been large enough to slow the growth of the CDN segment. Remembering that transit payments on the Internet are very small compared to traditional inter-carrier charges, it appears unlikely that any amount a CDN might agree to pay would materially affect the competition between online video services and the proprietary services of the access network. If a distortion of this kind should develop, intervention can be considered where necessary. At present, it appears that voluntary agreements between CDNs and other parties have helped to reduce costs and improve the quality of online content delivery.