One of the most daunting technological challenges we face today is scaling up this old internet of ours to meet the burgeoning consumer demand for bandwidth-intensive real-time applications such as telecommuting, cloud computing, and streaming media.

And as internet video continues to trend from short-form/long-tail/low quality content towards long-form/short-tail/high quality (premium) content (i.e. from YouTube to hulu to TV/films on embedded hardware), exploding consumer demand could bring things to a head even more quickly than currently anticipated.

Research firm Nemertes has taken the issue to heart.  Citing independent research from the University of Calgary,  the firm has published several reports predicting that the public (Tier-1 peered) internet will cease to be able to meet bandwidth demand by 2012.    These reports note that the three largest US content providers (Microsoft, Google and Yahoo) have already “built out dedicated infrastructures.”  As Nemertes puts it, “the oligarchy, in other words, is devolving into individual city-states”  as companies begin “investing in technologies to accelerate traffic to their sites ahead of that on the regular Internet.”

In fact, Google has been buying ‘dark’ (unused) transcontinental fiber, and Microsoft is pretty much betting the farm on Azure – and smaller companies recognize the need to invest in workarounds as well.  Lacking the resources of a Google or Microsoft to implement their own backbones, though, they rely instead on 3rd party  CDNs (content delivery networks) such as Level 3, Limewire, and Akamai.  Through both network optimization/caching software and the brute-force deployment of redundant content-caching servers deployed at strategically positioned and geographically diverse locations along the edge of the internet, CDNs are designed to circumvent the increasingly messy core of the Tier 1 internet to provide the higher bandwidth, lower latency and increased scalability required to meet the challenges of the future.

While the third-party CDN vendor market is relatively mature (industry leader Akamai now maintains over 34,000 servers located in 70 countries), several large internet service and hardware players have recently opted to ‘roll their own’: in June of 2008, AT&T announced it was building out a new CDN using software licensed by several smaller firms (ExtendMedia, Qumu and Stratacache), several months ago I received a press release from Verizon announcing a new CDN partnership with UK CDN firm Velocix, and most recently comes news of Cisco building a CDN.   Why are these companies deciding to make such a sunbstantial capex investment during such challenging economic times?  There are several reasons:

• On a technical level, the synergy of an integrated CDN/last mile solution offers potential performance advantages an external wholesaled CDN would have difficulty matching – and as the last mile becomes faster due to increased presence of fiber (i.e. FIOS) and/or Docsis3.0, an integrated ISP/CDN solution makes even more sense.
• Unlike AT&T, Verizon is also planning to leverage their new proprietary CDN on the content owner side as well, having already contracted directly with Starz Entertainment to offer exclusive Starz content to Verizon customers (in this way, the in-house CDN could very well resuscitate the largely failed ‘walled-garden’ model).
• Lastly (and most importantly), to the extent an ISP is able to build a demonstrably better mousetrap on their own, the quality of that company’s proprietary CDN could well become a primary competitive differentiating factor driving subscription growth – especially if, as expected, long-form internet video usage continues to grow.

Are 3rd party CDN wholesalers Akamai and Limelight losing sleep?  Probably not – deploying an effective CDN is a heavy lift (AT&T is spending $70M on theirs), and not an option for your neighborhood (non-tiered) ISP.

In Conclusion Google’s and Microsoft’s new proprietary data links…  3rd party CDNs… Verizon’s and AT&T’s new proprietary CDNs…  Regardless of how it all plays out, what’s clear is that one way or the other, the internet landscape will look profoundly different five years out – the current public net-neutral internet as we know it could become merely the least expensive and lowest-performing rung of a multi-tiered system including numerous high-performance proprietary “internets” designed for more demanding tasks such as getting you that real time HD internet video stream or workplace desktop session.   An analogy: as these competing private ‘Fedex’ internets emerge, today’s public internet could come to seen as the public post office, more suited to delivering bulk mailings (i.e. email spam) and bills.

In a way, it’s inevitable:  internet technology was created (and to a large extent is still managed) by governmental agencies and research institutions – but usage of that technology has long since been thoroughly commercialized.   It only follows then, that the technology itself – the ‘plumbing’ of the internet – will become commercialized as well.

So while the Nermertes reports are alarming and propose nothing less than a ground-up redesign of the internet (largely based on outspoken internet pioneer John Day’s book “Patterns in Network Architecture“), I feel they give short shrift to the role private industry will play.

The internet was never designed to do the kinds of things it’s doing today – that it’s scaled up to the extent it already has is something too many of us take for granted.  But while the future is bright, it’s a future that includes daunting technical challenges  requiring a partnership between the public and private sectors.