“Under Weight of #Superbowl…@Aereo Collapses”

Posted By on Feb 3, 2014 | 0 comments


On Saturday, after Aereo had closed their flagship NY region to new signups, we posed the question of whether there could be snow in the Superbowl forecast for Aereo users.

If not snow, we now know there was at least inclement weather for Aereo users last night.

A small sampling of twitter traffic Sunday evening:

‏

‏

A few Aereo users reported good results, but complaints were widespread, and weren’t limited to users in the NY region.

So what happened?

Well, there was little or nothing in the way of reports of users who were blocked altogether from viewing, so it doesn’t appear that Aereo ran out of antennas to assign to users.  There are at least three other critical resources that could have caused problems for Aereo, however: 1) transcoding, 2) disk I/O, and 3) internet connectivity bandwidth.

I tested Aereo’s service during the Superbowl, myself, in the New York region, scheduling recordings of the Superbowl and the shows immediately before and after it.  With an account that is allowed two antennas, I also scheduled simultaneous recordings for various programs airing opposite the Superbowl, on the local NBC and CBS affiliates.

Trying to watch the Superbowl live on my computer was an exercise in frustration, with frequent artifacts and buffering.  The player eventually recommended that I switch the quality to “Auto” (from “High”), which I did.  If that reduced the buffering, it was only by a slight amount, so after making that change, I simply had frequent buffering and frequently very poor quality, even during low-motion scenes that shouldn’t require a high bitrate.  Occasionally with quality set to high, error artifacts would be visible.  (These kind of artifacts occur when a portion of the stream data is lost.)

My luck with other programs, and using my iPhone to view, was no better. I attempted to watch 60 Minutes on my phone, but it would alternate between playing and buffering a few times, and then stop altogether. The picture above is a screenshot of what happened at one point when I tried to restart the session.

I knew it had to be an issue with Aereo, but to be certain there wasn’t a broader problem with my internet connection, I ran a test using Speedtest.net: 59.6 Mbps down, 16.9 Mbps up. Clearly no general issue with my internet connection. Attempting to use Aereo’s own speed check tool was futile. It simply never responded.

I went back to check on the recordings, today.  It appears that all the recordings were made, and sampling them, they appear to have recorded properly, with just an occasional error artifact present in the recording.

Frequent buffering on high quality

Frequent buffering on high.

Prompted to switch to auto quality

Prompted to switch to auto quality.

Still buffering regularly on auto

Still buffering regularly on auto.

Auto quality often very poor.

Auto quality often very poor, even with little motion.

Error artifacts.

Error artifacts.

Speedtest.net results, during Superbowl.

Speedtest.net results, during Superbowl.

So what does this tell us?

Aereo hasn’t publicly documented the details of their system, but we know that they use individual antennas, individual transcoding, individual disk storage, and individual streaming.  My expectation would be that Aereo is storing a single encoding of programs to disk at high quality, and then performing a second round of transcoding when streaming, for any playback setting other than high quality.

That there were few complaints of failure to tune altogether, together with my anecdotal test of recordings, indicates that they didn’t run into a hard limit on simultaneous streams, like not having enough antennas.  Based on the recording quality, it would appear that they didn’t have any major problems with ingest transcoding capacity, or disk ingest capacity.  (The occasional error artifacts observed in the recordings could be either a momentary transcoder error, or an uncorrected disk write failure, but these artifacts were rare enough that I couldn’t classify it as a major issue.)

The problems with live streaming, during the Superbowl, could conceivably have included some problems keeping up with disk reads, or output transcoding.  However, that there seems to have been very little problem with disk writes suggests there most likely wasn’t a major disk read bandwidth issue.  (It’s at least plausible that Aereo could have specifically customized their system to prioritize disk writes over reads, for precisely these situations.  But from the totality of the symptoms, it still seems unlikely that disk read bandwidth was the primary problem)  The presence of significant problems even when streaming high quality suggests that output transcoding was not the primary problem, or at least not the only major problem.

Which leaves internet bandwidth.  Based on how widespread the issues were, and my own verification that it wasn’t a local issue for me, it seems safe to conclude that demand overran Aereo’s outbound internet bandwidth—and it looks as if this was probably the primary issue impacting service last night, during the Superbowl.

If so, the problem is directly related to the individualized reception, storage and streaming model Aereo has adopted, to avoid the need to license retransmission.  As we noted yesterday, the need to stream to each user from their own recorded copy of the program means that Aereo cannot make use of conventional CDNs (content delivery networks).  Rather, they have to serve every stream directly from their own facility.

While a conventional internet video provider can scale streaming capacity with demand fairly easily, via their CDN partners, Aereo needs to have sufficient capacity to transmit directly from their facilities.  What’s more, because each stream has to be sent directly from their facilities, their quality of service is more at risk from transit congestion (as their streams travel through the internet cloud to the ISPs serving their end users) than conventional providers using CDNs.

The question that remains is whether this is a problem Aereo can easily get its hand around.  Is it merely a growing pain that they lacked sufficient bandwidth?  Or do these resources represent enough of a cost that increasing them would threaten Aereo’s business model, and its ability to sustain current price points?  And to what degree did congestion outside of their premises—presumably an even harder problem for them to solve—play a role in the service quality issues last night?

I don’t expect that Aereo will comment publicly on these issues anytime soon, so for now, we probably just need to wait and see if Aereo’s quality and consistency under heavy load improves, in the future.

For more background on Aereo see our Aereo in a Nutshell article.  Or for comprehensive information on Aereo litigation, see our Understanding Aereo page.