Khirman & Son

At the recent CoolTech Club meeting we enjoyed an excellent presentation by William Norton , Equinix co-founder and Chief Technical Liaison. Bill presented his calculations on cost of digitized media distribution over various Internet architectures (presentation – full paper) .

Apparently, convenient Content Delivery Network (CDN – Akamai and others) architecture has a quite significant price tag – about $0.24 per full-format movie. Some cost improvement can be made by deploying content servers in few major peering points managed by Equinix ( surprise ! ;-)  - cost decrease to about $0.17 per movie. This is certainly improvement over reported $1 spend by Netflix for snail-mail delivery, but still to high for mass adoption (Also , it questionable if existing Internet infrastructure can survive significant volume of video content).

Alternative P2P ( peer-2-peer) delivery architecture cost to content owner/distributor significantly less – about $0.0018 – two order of magnitude over the best conventional architecture! However, the catch is that delivery expenses are not disappeared – it just passed on weak shoulders of ISPs and Internet Backbone. It is not surprise that some ISPs are pressured to deploy P2P “profiling” equipment – futile (and probably - illegal) attempt to suppress P2P content distribution across their networks. However, it is obvious that any long-term solution has to take in consideration not only profitability of the distribution business, but profitability of consumer ISPs businesses as well.

In our traditional after-party brainstorm, we “designed” a few architectures to achieve this goal – from “topology aware P2P” to massive set of “localized” peering points. The massive peering idea, as described by Bill Norton, called for large number (about 80,000) small peering points that interconnect all local ISPs. Those peering points serve as excellent location for content providers/distributors to deploy their “cache”/replication servers, “pushing” video content to the location as close as possible to the end user. Indeed, heavy video content will travel on short distance only, using cheap or free interconnect, releasing Internet backbone from pressure of high-volume multimedia traffic. This model has a lot advantages to ISPs and content distributors – cheap or free inter-domain traffic results in fix-cost service. This model is advantageous from end-user perspective – low delay, high quality of service over reliable last-mile connect, potential ability to see high-def content in streamable ( no-waiting) mode and so on…

Unfortunately, this architecture will not fly….Why ????

Humans are not perfect ………Caching works well when majority of users are asking for the same content. However, outside of North Korean “one party, one leader, one movie” domain, people probably have a quite diversified interests. If large portion of traffic generated by dissimilar content, then caching architecture may became to expensive to implement. Let’s investigate if it will be the case for large catalog of video content.

BitTorrent offers us an excellent model for investigation. Let’s consider [in]famous “The Pirate Bay” (TPB) – probably largest repository of pirated video content.  You can find virtually every US movie in this repository, as well as popular international ones, often translated into 2-3 major languages. As a pirated content is available for “free” , TPB usage trends can serve as a good model for flat-priced or add-supported legal subscription service with a large movie catalog ( comparable to 90,000 titles of Netflix). (note – many popular US serials are not part of ThePirateBay catalog – it will be a topic of separate investigation.)

I took a “scrape” snapshot from one of TPB “trackers” - http://tracker.prq.to/  ( TPB uses about 10 different trackers, but PRQ is seems to be the most popular). At the moment of the measurement, PRQ tracker coordinated distribution of over 400,000 torrents ( unique contents files) with almost 3.7m cumulative number of peers. Certainly, some peers may participate in few torrents, but usually no more then in 2-3 simultaneously. So we can safely assume that collected data represents about million unique users. Quite a good sampling population for statistical analysis!

I ranked torrents in accordance with number of participating peers and , assuming that bandwidth is proportional to number of peers,  calculated cumulative distribution of bandwidth as function of content popularity.

This graph describes relationship between movie/torrent popularity and traffic wasted for its delivery. Horizontal axis measure movie/torrent “rank” – its position in the list of torrents sorted by decreased popularity. Vertical axis represents portion of total bandwidth used by torrents ranked from 1 to the given one. For example, accordingly to the graph, 50,000 most popular torrents/movies are consuming about 70% of total traffic.

The same graph with logarithmic X scale gives us a close look on the “most popular content” :

A few immediate observations:

• The classical 20/80 rule is still strong – 25% (100,000) most popular content uses 80% of traffic .
• 10,000 most popular torrents ( or 2.5% of content library) uses just under 45% of traffic
• 1,000 most popular torrents ( or 0.25%) use almost 20% of traffic.

And now a bad news for CDN/caching architecture – lets calculate a storage size required for significant ( 80%) drop in the traffic use. Accordingly to the graph above, we need enough storage to cache 100,000 (25%) most popular movies.

Decent DVD-quality DivX-compressed movie takes about 1.4G. 100k movies require at least 140 terabytes of high-speed storage.  In NetApp terms it will cost about $500k or more per piece.... And we’ll need such storage to be allocated in each of 80,000 peering point locations – or about $40Billions  in total….Even for much smaller Netflix catalog, we’ll get about $10Billions for storage only. Not a rocket science, but still quite significant investment….

Limited caching for top 0.25% of content seems to be more reasonable – about $100M for tflix-size catalog. But it saves less then 20% of traffic – quite limited results to justify such significant infrastructure investment….

Conclusion: useful CDN/caching/peering solution will break any reasonable budged..

Probably “topology aware P2P” is a more reasonable model, but it is another story to discuss ….