I promise to do a better job in 2015 of posting to this blog more frequently. In the meantime you can find me on Twitter, where I will be posting and sharing content relevant to streaming media technologies. I am @zambelli24fps.
I’ve written an article for The Guardian’s media network blog about current trends in TV content distribution, how they’re affecting our viewing habits, and how ultimately those will affect the future of live linear TV. You can read it here:
This is actually the second article I’ve written for The Guardian. The first one, published a year ago, reminisced about the early days of Internet video streaming, how far we’ve come and how much more we still have to grow:
The Olympics are upon us – again! Even though the ancient Greeks set the Olympic schedule 4 years apart (in fact, the literal definition of an Olympiad is “4 years”), the modern Olympics since 1992 have been only 2 years apart due to the interleaving of the Summer and Winter games. In 2008 I got involved in my first Olympics streaming project (while at Microsoft, for NBC), and here we are 6 years later and I am now involved in my 4th consecutive Olympics project, this time with iStreamPlanet and again for NBC. Incidentally, it is also iStreamPlanet’s 4th consecutive Olympics, but this time it’s a particularly special occasion. This time we’re encoding it with our own software.
What makes the Olympics stand out in the world of live video streaming is their formidable combination of volume and (in)frequency. Whereas most live sporting events take place on a weekly or monthly basis and wield modest viewership numbers, the Olympic Games are recurring 2-year events that can feature up to 30-40 concurrent live streams and attract millions of viewers over the course of 2 weeks. That means that anybody wishing to prepare for such an event must plan for 2 weeks of high volume operations (including capex and staffing), but then also be ready to give it all up after the 2 weeks of excitement are over. Such need for extreme elasticity and scalability makes these Olympic projects perfect candidates for cloud computing – and cloud encoding in particular.
Another fascinating aspect of Olympics streaming is that it happens just sporadically enough to truly show how far streaming technology has advanced since the previous Olympics. Every Olympics streaming project pushes the envelope of streaming media technology, making each and every Olympics feel like a brand new baptism of fire. When I was involved in the 2008 Beijing Olympics, the live video streaming technology was still Windows Media based and streaming to mobile devices seemed like an extravagant idea. The live video was compressed with VC-1 codec and it peaked at 592×336 resolution (600 kbps), which at the time (before live adaptive streaming) was deemed about as high as we could realistically go without risking last-mile delivery issues. We couldn’t even dream of SD 480p streaming, let alone HD 720p streaming.
Fast forward 2 years to Vancouver 2010 Olympic Games – and the streaming technology had already advanced by leaps and bounds. By February 2010 Microsoft had officially launched Smooth Streaming, its HTTP-based adaptive streaming technology, and 720p HD streaming video was now officially a reality. iStreamPlanet did the live encoding for those Olympics, using Inlet/Cisco encoders and VC-1 codec, and it was quite possibly the best live video streaming the world had seen at that point. Microsoft’s subsequent 2012 London Olympic efforts iterated on those Vancouver foundations by replacing VC-1 with H.264 and for the first time introducing Windows Azure’s cloud computing potential on the origin/services side. Which brings us to 2014 Sochi.
iStreamPlanet’s involvement in these 2014 Olympics consists of:
- acquiring IP-based video feeds from NBC/OBS
- scheduling Aventus channels in our internal CMS
- encoding live multi-bitrate video in Aventus for delivery to Windows Azure Media Services
- inserting ad markers into live streams to enable downstream ad insertion scenarios
The live Olympic video feeds are delivered to us over a private fiber IP connection as 20-25 Mbps 1080i H.264-compressed MPEG-2 Transport Streams. Aventus, our cloud-based video encoding software, then ingests those MPEG transport streams and transcodes them to 7 different video bitrates before publishing them to Windows Azure Media Services entry points.
Aventus “talks” to WAMS live publishing points using Smooth Streaming as its transfer protocol, but WAMS then uses its Dynamic Packaging feature (popularly known as Dynamux) to re-multiplex the Smooth streams to Apple HLS and Adobe HDS formats for delivery to end users. The live content is pulled from WAMS origins by Akamai CDN and distributed throughout their network to viewers playing back NBC Olympics content on Windows, MacOS, iOS, Android and Windows Phone devices.
The Smooth/HLS/HDS live streams are compressed using H.264 and AAC codecs for video and audio, respectively. Audio is encoded as 56 kbps stereo HE-AAC v1, whereas video is delivered as H.264 in multiple bitrates/resolutions:
|Bit Rate (kbps)||Resolution|
This somewhat odd encoding profile (where particular resolutions are duplicated) was specifically created in order to minimize the number of client-side resolution changes, but otherwise follows fairly standard bitrate progressions.
Another interesting aspect of the Aventus Olympic workflow is ad insertion. Aventus receives REST-based API calls for ad insertion from ad operators via its Video CMS, which it converts to XML-formatted SCTE-35 cue messages and inserts into Smooth Streaming sparse tracks. Those ad markers are then converted to HLS and HDS ad markers by WAMS which downstream video players can interpret and take appropriate actions.
The most groundbreaking thing about these Olympics is that they are the first Olympics to be streamed (acquired, encoded, packaged, delivered) entirely in the cloud. If you’re in the U.S., I hope you’re enjoying watching the Sochi Olympics online. If not… well, there’s always Rio 2016.
As I glance at my blog I realize it’s been 7 months since my last post, which is probably as good of a sign as any that I should probably finally tell you a little bit about what I’ve been up to since I left Microsoft.
iStreamPlanet, for most of its 13-year history, has been predominantly a streaming service company, focused on providing the best-in-breed streaming media services for live events and live linear channels: live signal ingest, video encoding (live and VOD), origin server hosting, DRM packaging and licensing, video workflow automation, Xbox app development, etc. A little over a year ago, iStreamPlanet decided to diverge and invest (thanks to its Series A investors) into something brand new, aiming to reduce the complexity of live video delivery workflow through software development – and that’s how Aventus was born.
Aventus officially marks iStreamPlanet’s foray into the live video encoder market. Aventus is a software-based live video encoder designed to run specifically in virtualized environments (currently Microsoft Hyper-V) on x86-64 commodity hardware and focused exclusively on IP-based video delivery scenarios. It’s different from most other live video encoders in that:
- It supports only IP-based signal ingest (e.g. M2TS/UDP, RTMP) and therefore isn’t burdened with legacy support of video capture hardware devices
- It’s focused primarily on over-the-top delivery: it outputs only modern, HTTP-based adaptive streaming protocols such as Microsoft Smooth Streaming, Apple HTTP Live Streaming, Adobe HDS and (eventually) MPEG-DASH
- It’s designed from ground up to work in virtualized environments, allowing it to take advantage of grid computing and to easily scale resources according to need, deployable in either private data centers or public clouds
- It’s entirely CPU-based, which means that it can run on any x86-64 commodity hardware thus significantly reducing capital costs for customers looking to build out their live encoding capacity
Last 9 months have been a thrilling experience and I look forward to being able to announce more details about Aventus v1.0 at the upcoming IBC Show in Amsterdam next month.
Last but not least… We are hiring! We are currently looking for talented C++ and C# developers with experience in digital media (codecs, formats, DSPs) who are passionate about streaming media and who would enjoy working on a v1 live video encoder product. Our software engineering team is located in Redmond, Washington. If you are interested, please contact me by e-mail and include your resume and/or LinkedIn profile link.
OK, so maybe it was a shorter break from blogging than I expected. As it turns out the world does not stop when I change jobs.
The media world today is abuzz with news of H.265/HEVC approval by the ITU. In case you’ve been hiding from NAB/IBC/SM events for the past two years – or if you’re a WebM hermit – I will have you know that H.265 is the successor standard to H.264, aka MPEG-4 AVC. As was the case with its predecessor it is the product of years of collaboration between the ISO/IEC Moving Picture Experts Group (MPEG) and the International Telecommunications Union (ITU) Video Coding Experts Group (VCEG). The new video coding standard is important because it promises bandwidth savings of about 40-45% for the same quality as H.264. In a world where video is increasingly being delivered over-the-top and bandwidth is not free – that kind of savings is a big deal.
What most media reports seem to have focused on is the potential effect that H.265 will have on bringing us closer to 4K video resolution in OTT delivery. Most reports speculate that H.265 will allow 4K video to be delivered over the Internet at bit rates between 20 and 30 Mbps. In comparison, my friend Bob Cowherd recently theorized on his blog that 4K delivery using the current H.264 video standard would require about 45 Mbps to deliver 4K video OTT.
While I think the relative difference between those two estimates is in the ballpark of the 40% bandwidth savings that H.265 promises, I actually think that both estimates are somewhat pessimistic. Given the current state of video streaming technology, I think we’ll actually be able to deliver 4K video at lower bit rates when the time comes for 4K streaming.
A common mistake that most people dealing with lossy video compression seem to make is to assume that the ratio between bit rate (bps) and picture size (pixels/sec) remains proportional and fixed as the values of both axis change. I don’t think that’s the case. I believe that the relationship between bit rate and picture size is not linear, but closer to a power function that looks like this:
In other words, I believe that as the pixel count gets higher a DCT-based video codec requires fewer bits to maintain the same level of visual quality. Here’s why:
- The size of a 16×16 macroblock, which is the smallest unit of DCT-based compression used in contemporary codecs such as H.264 and VC-1, grows smaller relative to the total size of the video image as the image resolution grows higher. For example, in a 320×180 video the 16×16 macroblock represents 0.444% of the total image size, whereas in a 1920×1080 video the 16×16 macroblock represents only 0.0123% of the total image. A badly compressed macroblock in a 320×180 frame would therefore be more objectionable than a badly compressed macroblock in a 1920×1080 frame.
- As many studies have shown, the law of diminishing returns applies to video/image resolution too. If you sit at a fixed distance from your video display device eventually you will no longer be able to distinguish the difference between 720p, 1080p and 4K resolutions due to your eye’s inability to resolve tiny pixels from a certain distance. Ipso facto, as the video resolution goes up your eyes become less likely to distinguish compression artifacts too – which means the video compression can afford to get sloppier.
- Historically the bit rates used for OTT video delivery and streaming have been much lower than those used in broadcasting, consumer electronics and physical media. For example, digital broadcast HDTV typically averages ~19 Mbps for video (in CBR mode), while most Blu-ray 1080p videos average ~15-20 Mbps (in 2-pass VBR mode). Those kinds of bit rates are possible because those delivery channels have the luxury of either dedicated bandwidth or high-capacity physical media. However, in the OTT and streaming world video bit rate has always been shortchanged in comparison. Most 720p30 video streaming today, whether live or on-demand, is encoded at average 2.5-3.5 Mbps (depending on complexity and frame rate). 1080p30 video, when available, is usually streamed at 5-6 Mbps. Whereas Blu-ray tries to give us movies at a quality level approaching visual transparency, streaming/OTT is completely driven by the economics of bandwidth and consequently only gives us video at the minimum bit rate required to make the video look generally acceptable (and worthy of its HD moniker). To put it bluntly, streaming video is not yet a videophile’s medium.
So taking those factors into consideration, what kind of bandwidth should we expect for 4K video OTT delivery? If 1080p video is currently being widely streamed online using H.264 compression at 6 Mbps, then 4K (4096×2304) video could probably be delivered at bit rates around 18-20 Mbps using the same codec at similar quality levels. Again, remember, we’re not comparing Blu-ray quality levels here – we’re comparing 2013 OTT quality levels which are “good enough” but not ideal. If we switch from H.264 to H.265 compression we could probably expect OTT delivery of 4K video at bit rates closer to 12-15 Mbps (assuming H.265’s 40% efficiency improvements do indeed come true). I should note that those estimates are only applicable to 24-30 fps video. If the dream of 4K OTT video also carries an implication of high frame rates – e.g. 48 to 120 fps – then the bandwidth requirements would certainly go up accordingly too. But if the goal is simply to stream a 4K version of “Lawrence of Arabia” into your home at 24 fps, that dream might be closer to reality than you think.
One last thing: In his report about H.265 Ryan Lawler writes that “nearly every video publisher has standardized [H.264] after the release of the iPad and several other connected devices. It seems crazy now, but once upon a time, Apple’s adoption of H.264 and insistence on HTML5-based video players was controversial – especially since most video before the iPad was encoded in VP6 to play through Adobe’s proprietary Flash player.” Not so fast, Ryan. While Apple does deserve credit for backing H.264 against alternatives, they were hardly the pioneers of H.264 web streaming. H.264 was already a mandatory part of the HD-DVD and Blu-ray specifications when those formats launched in 2006 as symbols of the new HD video movement. Adobe added H.264 support to Flash 9 (“Moviestar”) in December 2007. Microsoft added H.264 support to Silverlight 3 and Windows 7 in July 2009. The Apple iPad did not launch until April 2010, which was also the same month Steve Jobs posted his infamous “Thoughts on Flash” blog post. So while Apple certainly did contribute to H.264’s success, they were hardly the controversial H.264 advocate Ryan makes them out to be. H.264 was already widely accepted at that point and its success was simply a matter of time.
As we leave 2012 behind us and turn our attention to 2013, the time has come for me to share some exciting personal news.
After 10 years at Microsoft I have recently decided to leave the company to pursue other career opportunities. It wasn’t an easy decision, but I felt it was time for a change. On January 1st I officially joined iStreamPlanet Co. as a Principal Video Specialist, where I will be focusing on building cloud-based live encoding workflows.
iStreamPlanet was founded in Las Vegas in 2000 by Mio Babic as a digital media services provider and has been a trusted Microsoft partner for over a decade. Since its humble Windows Media streaming beginnings iStreamPlanet has built a worldwide reputation in providing premium video workflow services and products for large scale live streaming events. They were Microsoft’s go-to partner for bringing the 2010 Vancouver Olympics to U.S. and Canada, and have also been responsible for online delivery of many other high-profile events such as NBC Sunday Night Football, Wimbledon, U.S. Open, Roland Garros, Tour de France, TNT NBA Overtime, PGA Tour, NASCAR and UFC.
At Streaming Media West last October iStreamPlanet announced and demoed for the first time Aventus, its new cloud-based automated video workflow platform for delivering live events and live linear channels. This is the product that I will be working on at iStreamPlanet for the foreseeable future. I will remain in the Seattle area, working out of iStreamPlanet’s Redmond office.
As I focus on my new job I will take a short break from blogging for the next few months. When I eventually return I will likely redesign this blog to reflect my new professional responsibilities and interests. Until then, I wish everyone all the best in 2013!
It’s been a busy summer with most of the team focused on Windows Azure Media Services, but I’d like to take a moment to highlight a few other Media Platform releases of the past few months:
November 7, 2012
The RTW version of the framework contains support for Smooth Streaming (via Smooth Streaming Client SDK) and advanced playback heuristics; closed captioning (SMPTE-TT, TTML); advertising (VAST, MAST, VPAID); DVR style playback; and robust skinning and styling.
October 26, 2012
The release version of the Windows 8 SDK contains support for x86, x64 and ARM processor architectures; on-demand and live playback with seek/DVR function; support for H.264, VC-1, AAC, DD+ and WMA Pro codecs; multiple audio languages; track (bitrate) selection; offline playback; text and sparse tracks; closed captioning; trick play; and PlayReady DRM content protection and key rotation.
This release of the PlayReady Client SDK supports online scenarios such as progressive download and streaming, as well as offline scenarios such as download-to-own, rental and subscription.
October 24, 2012
The new version fixes a number of issues related to PlayReady protection, and adds support for transmuxing Dolby Digital Plus audio from MP4 to Smooth Streaming file format.
June 11, 2012
Smooth Streaming Client for Silverlight and Windows Phone version 2.0 has been released to Microsoft Download Center. Download it here.
The latest version of SSME adds support for linear ad insertion and playback, DRM key rotation, updated cache plug-in model and fragment download APIs.
Player Framework for Silverlight and Windows Phone version 2.7 has been released to Codeplex. Download it here.
This release of Player Framework adds support for SSME 2.0 and Silverlight progressive download video trick play, and fixes various bugs.
I’ve been holding back this post for about two weeks now, wanting to make sure the service got rolled out to all major Windows Azure datacenters first, but here it is now:
Windows Azure Media Services Preview has been launched! The new media-focused cloud services recently announced at NAB have been deployed to Windows Azure datacenters in US-East, North Europe, West Europe, East Asia, Southeast Asia, with more datacenters to follow in the weeks to come.
Anyone interested in cloud-based media services can request access by following these instructions, but keep in mind all requests have to be approved by the WAMS product team as this is still a Preview release.
The Preview release of Media Services supports:
- Encoding with Windows Azure Media Encoder (aka Expression Encoder 4 SP2)
- Format conversion (remuxing) from MP4 to Smooth Streaming and from Smooth Streaming to Apple HLS
- Content Protection with Microsoft PlayReady
- On-Demand Smooth Streaming and HTTP Live Streaming
- Scalable delivery via Azure CDN or 3rd party CDN
For a full list of features in this release, check out the Media Services Feature Status: Preview Release.
Documentation and Support
How-To Guides: http://aka.ms/WAMSHowTo
MSDN Documentation: http://aka.ms/WAMSMSDN
Release Notes: http://aka.ms/WAMSNotes
Windows Azure Media Services Forum: http://social.msdn.microsoft.com/Forums/en-US/MediaServices
The long awaited 1.0 release of IIS Transform Manager (which for a while there seemed stuck in GMail-like alpha/beta development) finally hit the Web last week! You can download it and read more about its features here: http://www.iis.net/download/transformmanager
John Deutscher has written a nice overview of IIS Transform Manager 1.0 on his blog. Since we announced Windows Azure Media Services one of the most commonly asked questions has been “Will Transform Manager be supported in WAMS?” The answer is not a straightforward “yes” or “no” because many on-premises server concepts don’t translate directly to the cloud. The goal of Transform Manager is to automate transcoding, transmuxing and DRM packaging of media, and those will be exactly the core services provided by Windows Azure Media Services. So one could argue that Windows Azure Media Services will be a big, amped-up, cloud-based version of Transform Manager. On the other hand, concepts like “local watch folders” and “HPC clusters” don’t really exist in the cloud so workflows will need to be adapted to the cloud paradigm. Bottom line: all the key features of Transform Manager such as media transcoding, MP4–>Smooth transmuxing, Smooth–>HLS transmuxing, and DRM encryption will be available in Windows Azure Media Services too. Microsoft Expression Encoder will be available natively in WAMS under the new name Windows Azure Media Encoder, but WAMS will also offer the possibility of using 3rd party encoders.
The release of Windows 8 Release Preview was announced earlier today, so accordingly we also released some Windows 8 related updates to the Microsoft Media Platform. Smooth Streaming Client SDK for Windows 8 Metro Apps has been refreshed to Beta 1. This version of the Smooth Streaming Client includes many improvements, supports multiple platforms (x86, x64 and ARM) and enables the following features in the <video> tag and MediaElement object:
- On-demand Playback (Play, Pause, Stop, Seek)
- Live Playback (Play)
- VC-1 and H.264 codec support
- TrickPlay (New in Beta refresh)
- Slow motion (New in Beta refresh)
- Content Protection w/ PlayReady integration
Beta 1 release notes, including breaking changes and known issues, can be found here.
MMP Player Framework for Windows 8 Metro has also been updated to support Windows 8 Release Preview and the latest Smooth Streaming Client for Win8. The new version, known as Preview 3, can be downloaded here. Preview 3 focuses on advertising scenarios by adding support for VAST, MAST and VPAID standards, and dynamic ad insertion.
This week at the annual National Association of Broadcasters (NAB) Show in Las Vegas we made two big announcements:
“New cloud-based Windows Azure Media Services is designed to make creating, managing and delivering media to any device easier than ever by offering a comprehensive set of ready-to-use first- and third-party media technologies. […] Taking advantage of the worldwide Windows Azure cloud infrastructure, Windows Azure Media Services gives content providers and media partners the nearly unlimited capacity of the cloud to cost-effectively handle a huge volume of digital media and make it available in the format that customers want, when they want it. Windows Azure Media Servicesâ€™ ready-to-use services allow customers to simplify the creation of complex media workflows built on Microsoft Media Platform and third-party technologies.”
In addition to Windows Azure Media Services, we announced in detail our support for the MPEG-DASH standard across the Microsoft Media Platform:
Finally, in my last blog post I mentioned the Smooth Streaming Client Porting Kit, a C++ source code package for implementing Smooth Streaming in TVs, STBs and other embedded devices. Here are more details about SSPK licensing: