Released this month:

• Expression Studio 4, which of course includes Expression Encoder 4.
• IIS Media Services 4.0 Beta
• Smooth Streaming Format SDK 1.0 Beta 2
• IIS Smooth Streaming Client 1.0, also known as Smooth Streaming Media Element (SSME), formerly known as IIS Smooth Streaming Player Development Kit
• Silverlight Media Framework 2.0 RC1

Before I dive into more details about those products, I’d also like to highlight a few case studies we recently published, all of them focused on the Silverlight-based 2010 Winter Olympics experiences built for NBC, CTV, NRK and France TV:

• NBC Universal (United States)
• CTV (Canada)
• NRK (Norway)
• France TV (France)

Some of the statistics are quite impressive, such as the number of peak concurrent users (181 thousand Americans, 134 thousand Canadians), total amount of video consumed (7.2 million hours in Canada), or the average minutes viewed over the 2 weeks of Olympics (nearly 2 hours per unique user in Canada). If you’re interested in the business side of video streaming, the value proposition of Silverlight and Smooth Streaming, and monetization data – I recommend you check out these case studies.

Expression Encoder 4

First change you might notice is that the pay version of the product is now called Expression Encoder 4 Pro, to differentiate it from the free version available for download. The good news is that the basic free version now includes Smooth Streaming (VOD) encoding, but you’ll still need to shell out $49.95 to get:

• H.264/AAC encoding (now courtesy of MainConcept H.264 SDK)
• Additional decoders for input types (MPEG2, MPEG/TS Splitter, Dolby Digital AC3, MP4 and H.264/AAC)
• Live IIS Smooth Streaming support
• Unlimited screen capture
• Digital Rights Management (PlayReady) integration

In case you missed it hidden there in the middle of the list, let me repeat: Expression Encoder 4 Pro supports LIVE Smooth Streaming! That was by far the feature most frequently requested by users, and the Encoder team listened and delivered. Live encoding in EE4 works and scales remarkably well, but I wouldn’t recommend trying to run it on any computer with less than 8 CPU cores. Video encoding is a CPU intensive process and when you have to encode multiple bitrates in realtime there’s no such thing as too much CPU. With 2 cores you can probably manage to churn out 1 bitrate up to SD resolution; with 4 cores about 2 bitrates up to SD; with 8 cores about 3-4 bitrates up to SD or 720p; with 16 cores about 7-8 bitrates up to 720p, etc.

The question I expect I’ll get asked most frequently about EE4 is: “OK, now that EE4 can encode Live Smooth Streaming for $49.95 on any PC hardware, why would I pay thousands of dollars for professional products like Digital Rapids Stream, Inlet Spinnaker or Envivio 4Caster?” My answer would go something like this: EE4, while fully capable of encoding Live Smooth Streaming, is an entry-level encoding product. It’s to professional encoding products what Windows Movie Maker is to Adobe Premiere Pro. Is it good enough for encoding a high school basketball game once a week? Absolutely. But would I use Expression Encoder to deliver 2 weeks of Olympics live video to hundreds of thousands of viewers around the world? Probably not. To use yet another analogy: Anybody can buy a stock PC, install Windows Server 2008 on it and call it a server. But does that make industry standard HP ProLiant servers obsolete? Hardly. Companies like Inlet, DR and Envivio have invested years and millions in building reliable professional encoding products which are designed and tested to run 24/7 in broadcast-type environments. So when you pay top dollar for their products, you’re not so much paying for raw codecs as you’re paying for guaranteed uptime and professional support. They cost a lot of money because they do what they do very well.

IIS Media Services 4.0 Beta

Though only a beta, this new release of IIS MS 4.0 delivers one particularly awesome feature: it can deliver Smooth Streaming H.264/AAC content to Apple “iDevices” such as the iPhone and iPad. How does it do that? Both formats support H.264 video and AAC audio; Smooth Streaming is based on MP4 (ISO Base Media) file format, while Apple Live HTTP Streaming is based on MPEG-2 TS file format. Smooth Streaming tends to use short GOP chunks (2 seconds), while Apple HTTP streaming uses long GOP (10 second) chunks. Therefore, converting between the 2 formats merely requires transmuxing A/V streams from one format to another, and this is exactly what IISMS 4.0 does: it dynamically transmuxes Smooth Streaming format into Apple’s Live HTTP Streaming format. No re-encoding.

Associated with this release are also 2 other IIS media releases:

• Smooth Streaming Format SDK 1.0 Beta 2 – provides documentation, tools, and samples you can use to multiplex video and audio bitstreams into on-demand and live Smooth Streaming output. In addition, the SDK supports the encryption of content using Microsoft PlayReady DRM. Note that this SDK doesn’t include video/audio codecs – it expects already compressed A/V samples.
• Transform Manager 1.0 Alpha – provides simple integrated video encoding and batch conversion of video files to the IIS Smooth Streaming format and the MPEG-2 TS streaming format supported by Apple devices.

For more information check out the following blog posts:
http://blogs.iis.net/chriskno/archive/2010/06/10/iis-media-services-4-0-beta-is-now-available.aspx
http://blogs.iis.net/chriskno/archive/2009/12/01/faqs-on-using-iis-smooth-streaming-with-the-apple-iphone.aspx
http://learn.iis.net/page.aspx/854/live-smooth-streaming-for-iis-7—apple-http-live-streaming/

Smooth Streaming Client 1.0 and Silverlight Media Framework 2.0 RC1

SMF (popularly called “Smurf”) 2.0 introduces a whole new, more modular architecture with an extensive plug-in API for 3rd-party developers and partners. It also enables developers to selectively exclude libraries that aren’t needed in their projects in order to avoid unnecessary file size increases. The v2 framework will include plug-ins for Timed Text (DFXP), URL frame linking, a metadata framework, support for the Microsoft Silverlight Analytics Framework, support for multiple audio tracks, improved bitrate monitoring, support for Silverlight 4 global styling, a JavaScript API, a logging plug-in, and a host of other new features. It also takes advantage of the new Managed Extensibility Framework (MEF) in Silverlight 4.

The Smooth Streaming component of SMF is known as the Smooth Streaming Media Element (pronounced “Smee”, like that pirate from Peter Pan). SSME previously shipped as beta under the name “Smooth Streaming Player Development Kit” but has since been renamed to just Smooth Streaming Client. The 1.0 release (compatible with SMF 2.0) comes with rich online documentation and code samples.

A year ago we began working with NBC and CTV on putting together the 2010 Vancouver Winter Olympics video site, later adding Norway’s NRK as another customer. We teamed up with a number of partners to get the job done:

• iStreamPlanet – video encoding services and hosting operations
• Akamai – edge content delivery
• Vertigo – Silverlight player design and development
• Delta Tre – content management systems, ad insertion, realtime data integration, player development, site integration
• Inlet – live video encoders
• Rhozet, Anystream (Grab Networks) - offline video (VOD) encoders
• Switch Communications – datacenter hosting
• SouthWorks.net – Rough Cut Editor design and development
• DoubleClick – ad serving
• Signiant – content distribution software
• Conviva – usage tracking and analytics
• Wintellect – origin health monitoring and maintenance
• Intel, Dell – server hardware

As you can tell just from the list of partners, this was a hugely complex project. Despite the Winter Olympics being smaller in scale than Summer Olympics, I’d estimate that this project was about 2-3 times more complex than the Beijing 2008 Olympics project due to the additional technical challenges we decided to take on in order to raise the bar in online video streaming.

In order to reduce some of the complexity we also made an early decision to deliver everything exclusively in Smooth Streaming and Silverlight, without a WMS/WMP fallback option. Though I’m sure some critics will be quick to assert such a decision was meant to force greater Silverlight adoption, the truth is less political and more practical: Trying to encode all videos for both Windows Media and Smooth Streaming (let alone additional formats such as MP4/H.264) would’ve probably doubled or tripled the cost and severely impacted the amount of functionality we were able to add to the client.

If you follow Smooth Streaming developments, you’ve probably heard of the Smooth Streaming Player Development Kit and the Silverlight Media Framework. Both of these frameworks and their underlying Smooth Streaming Media Element (SSME) were in fact designed for the Olympics project and first put to test on NBC Sunday Night Football 6 months ago. And while on the client side the Olympics player is actually quite similar to the SNF player (aside from the much improved rew/ffwd/slo-mo features and the lack of multi-camera angles), the chief difference between the Olympics and SNF is actually on the backend. One word: automation.

From a video operations standpoint, SNF was very much a manual operation. We created publishing points and started encoders by hand – and then stopped them 4 hours later. When you’re doing only one game a week, you can afford to do that with just a few people. But when you have to run 20-30 events per day, as many as dozen of those simultaneously, for 2 weeks straight – it’s absolutely unthinkable to try to run everything by hand. You’d need an army of engineers just to keep things running smoothly.

Enter iStreamPlanet. Our encoding service partner took it upon themselves to build a fully automated live video encoding service for the Olympics. This involved: turning on H.264 multicast decoders (our source streams arrive from Vancouver as H.264 multicast streams over dedicated OC-12) and tuning them to the right channels; routing the decoded video to the available Inlet Spinnaker HD encoders; creating multiple publishing points on IIS origin servers; starting publishing points and encoders; stopping encoders and publishing points; moving VOD archives to expected locations. Having such a service allowed Delta Tre, our CMS provider, to remotely schedule events without any human involvement required. For a young technology such as Smooth Streaming this is a big deal because it proves it’s possible to scale Smooth Streaming to large professional broadcast environments.

Another huge development and step forward for this project was the creation of the Rough Cut Editor, a soon to be publicly available “light touch” editor for Smooth Streams. The RCE allows editing of Smooth Streaming sources, both on-demand and live (!), without any re-encoding whatsoever. The concept is remarkably simple: if a Smooth Streaming manifest is like a playlist of video/audio fragments which live in the cloud, then editing, merging and appending multiple Smooth Streaming sources should be as simple as re-arranging entries in a manifest. Since Smooth Streaming video is just a series of fragments and each 2-sec fragment must be downloaded separately, it’s completely irrelevant whether fragments are downloaded/played in a sequential or non-sequential order – the end result plays equally smooth either way. This is something that hasn’t been possible with Windows Media or any other streaming media technology until now. It’s a total game-changer.

Remember, even though the Olympics Games are an international event, the International Olympics Committe (IOC) sells Olympic broadcast rights exclusively to 1 broadcaster per country – and only for that country. While that practice obviously originated for television broadcasts, the IOC still hasn’t gotten around to changing the rules for online streaming, which unfortunately means that access to Olympic video content on NBC, CTV and NRK sites is restricted to American, Canadian and Norwegian users only, respectively. No, they won’t check your passport, but they will check your IP address. :)

So if you’re lucky enough to live in the United States, Canada or Norway, you can check out the Silverlight and Smooth Streaming powered video coverage of the 2010 Vancouver Winter Olympics by visiting the following sites:

• United States:  http://www.nbcolympics.com/video/
• Canada:  http://www.ctvolympics.ca/video/index.html
• Norway:  http://www.nrkol.no/video/index.html

You will, of course, need Silverlight 3 installed to view any of the video content. If in doubt whether you have the latest version of Silverlight, visit http://www.microsoft.com/silverlight/get-started/install/default.aspx. Silverlight 3 works on Windows and Mac OS (Intel only); and with Internet Explorer, Firefox, Safari and (still unofficially) Chrome browsers. If you’re a Linux user, you might be pleased to know that the Novell Moonlight team has been working frantically to get Moonlight 3.0 ready in time for the Winter Olympics. It’s still just a preview release so your mileage may vary, but please give it a shot – you can download Moonlight 3.0 Preview 2 here. I recommend reading the prerelease notes for Moonlight team’s recommendations on browser builds.

Everyone should also be aware than not all broadcasters have chosen to make the same amount of Olympics content available to their viewers. While CTV’s and NRK’s coverage is pretty much unlimited and requires no special sign-up process, NBC has chosen to restrict access to “premium” content only to cable/satellite/IPTV customers. First time you try to access such content you will be asked to sign in with your Comcast/DirectTV/Cox/DISH/TimeWarner/etc account. Many more providers are supported, but be prepared to possibly jump through some hoops to prove that you’re an NBC affiliate customer. In addition to the sign-up procedure, NBC is also withholding much of the live online coverage, preferring to show the popular events on primetime television first. Don’t shoot the messenger.

If you experience any issues with Silverlight or content access, there’s a good FAQ over on the NBC website: http://www.nbcolympics.com/faq/index.html.

Silverlight 3 Beta

A mere 5 months after releasing Silverlight 2 RTW, we have now made Silverlight 3 Beta available to the public. Check out http://silverlight.net/getstarted/silverlight3/default.aspx for the full list of new features and download links. As usual, Scott Guthrie offers some great insight on Silverlight 3 in his MIX Keynote and Channel 9 video.

The most interesting new media features in Silverlight 3 are:

• Native H.264 video, AAC audio, and MP4 file playback support
  • Take your MP4-contained H.264/AAC encoded files, put them on a Web server and play them directly in Siverlight using progressive download!
• Extensible media format support
  • Support for raw A/V bitstream playback allows codec and media developers to write custom decoders and format parsers using C#, VB or any other .NET language. Will you be the first to write an Ogg Vorbis or FLAC decoder for Silverlight?
• GPU accelerated video scaling
  • Stretching the video to full screen can now be offloaded entirely to the video card, freeing up the CPU and enabling smooth video playback.
• Advanced media logging
  • Log playback usage to Windows Media Services and IIS7 Media Services, like with the good old Windows Media Player.
• Custom effects / Pixel shaders
  • Apply post-processing effects to your video by writing custom effects using the same HLSL pixel shader code that works in Direct3D and WPF today.
• Perspective 3D transforms
  • Spin and rotate your video around all 3 axis – X, Y and Z. Video collage? How about a video cube?

I’ll be blogging in the near future in more depth about the details of our H.264/AAC/MP4 support in particular.

IIS Media Services 3.0 / Live Smooth Streaming

Just a short month after announcing the availability of Smooth Streaming for On-Demand Video beta, the IIS Media team announced the availability of IIS Media Services 3.0 beta – featuring Live Smooth Streaming. That’s right, with IIS7 you will soon be able to deliver Smooth Streaming video for both on-demand and live!

Inlet Technologies has simultaneously announced they will be the first to add Smooth Streaming support to their line of live and VoD encoding products. Besides Inlet, we are currently working with a number of encoding ISVs on enabling them to add Smooth Streaming support to their products.

Akamai Technologies announced the wide commercial availability of their AdaptiveEdge Streaming service based on IIS Smooth Streaming. Besides Akamai, we are currently working with all the major CDNs on enabling Smooth Streaming support in their networks. As with the encoding ISVs, our goal is to build a rich Smooth Streaming ecosystem to be available to customers by the time Silverlight 3 ships.

Besides the newly redesigned home page, the IIS Media team has also put up a great working example of how Smooth Streaming works.

NBC Winter Olympics 2010 – Vancouver

During the MIX 2009 Keynote, Perkins Miller, Senior VP of Digital Media for NBC Universal, announced that NBC Universal has chosen to deliver the NBC Winter Olympics 2010 using Microsoft Silverlight. Watch the MIX Keynote to see his announcement.

Here are the details I can share at this point:

• All video content, both live and on-demand, will be delivered using Smooth Streaming
• The live video player will feature DVR-like capabilities (pause, rewind, seek and slo-mo of live video)
• Video quality will go up to true 720p HD

March Madness

CBS Sports has launched a Silverlight-based March Madness video player that lets you watch all NCAA Basketball Tournament games live. Visit http://mmod.ncaa.com/video to launch the March Madness video player. If you are using Internet Explorer on Windows, the default player will actually be an old-school WMP player, so you’ll need to click on the HQ Player button to launch the new Silverlight player.

The live video for the tournament is being streamed using Windows Media Services. Obviously, we couldn’t use Smooth Streaming because the server technology is still in beta and the encoders aren’t yet commercially available. But CBS did the next best thing! All live streams are available in 4 video quality levels:

Video codec used is VC-1 Advanced Profile. Audio codec used is WMA Professional at 44.1 kHz 16-bit stereo.

All March Madness games are being encoded by MLB.com’s encoding facilities using Inlet Spinnaker 7000 encoders. The Spinnakers were configured based on my own recommendations in order to provide maximum quality at all bitrates.

The March Madness Silverlight player uses preroll ad download statistics to estimate available client bandwidth and tries to make an appropriate first choice of bitrate level. Of course none of this would be necessary with Smooth Streaming, but we really tried to make the best of the Windows Media Streaming experience anyway. The player also has built-in heuristics to detect quality-of-service issues, such as frequent rebuffering or low frame rate rendering, at which point it can suggest to the user to choose a lower bitrate. Users can manually switch between available bitrates using the “-” and “+” buttons in the button of the player UI.

Traditional Streaming

Let’s start by first taking a look at RTSP as an example of a traditional streaming protocol. RTSP is defined as a stateful protocol. This means that from the first time a client connects to the streaming server until the time it disconnects from the streaming server, the server keeps track of a client’s state. The client communicates its state to the server by issuing it commands such as PLAY, PAUSE or TEARDOWN (the first two are obvious; the last one is used to disconnect from the server and close the streaming session).

Once a session between the client and the server has been established, the server begins sending down the media as a steady stream of small packets (the format of these packets is known as RTP). The size of a typical RTP packet is 1452 bytes, which means that in a video stream encoded at 1 Mbps each packet only carries roughly about 11 msec of video. In RTSP the packets can be trasmitted either over UDP or TCP transports – the latter is preferred in cases where firewalls or proxies block UDP packets, but can also lead to increased latency (TCP packets get re-sent until received).

RTSP is an example of a traditional streaming protocol

An HTTP protocol, on the other hand, is known as a stateless protocol. If an HTTP client requests some data, the server will respond by sending down the data, but it won’t remember the client or its state. Every HTTP request is handled as a completely standalone one-time session.

Windows Media Services supports streaming over both RTSP and HTTP. Now, you may ask yourself, “But if HTTP is a stateless protocol, how can it be used for streaming?” WMS uses a modified version of HTTP known as MS-WMSP, which uses standard HTTP for transfer of data and messages but also maintains session states, thus effectively turning it into a streaming protocol like RTSP/TCP. Windows Media Services has also supported RTSP streaming since 2003 (9 Series), over both UDP and TCP. Its implementation of the protocol is publicly documented as MS-RTSP.

The important things to remember about traditional streaming protocols like RTSP and WMS-HTTP is that:

1. The server sends the data packets to the client at a real-time rate only - that is the bit rate at which the media is encoded (i.e. a 500 kbps encoded video is streamed to the client at approx. 500 kbps)
2. The server only sends ahead enough data packets to fill the client buffer. The client buffer is typically between 1 and 10 seconds (WMP and Silverlight default buffer length is 5 seconds). This means that if you pause a streamed video and wait 10 minutes – still only ~5 seconds of video will have downloaded to your client in that time.

Progressive Download

The other most common form of media delivery on the Web today is progressive download. Progressive download is nothing more than just a plain, ordinary file download from an HTTP web server like IIS or Apache. It is supported by Silverlight, Flash, WMP, and nearly every other media player and platform under the sun. The term “progressive” likely stems from the fact that most player clients allow the media file to be played back while the download is still in progress - before the entire file has been fully written to disk (typically to the browser cache). Clients that support the HTTP 1.1 specification can also seek to positions in the media file that haven’t been downloaded yet by performing byte range requests to the web server (assuming it also supports HTTP 1.1).

The most popular video sharing websites on the Web today almost exclusively use progressive download:  YouTube, Vimeo, MySpace, MSN Soapbox – and even the rather misnamed Silverlight Streaming Service. (See my previous blog posts for a list of reasons why HTTP download is becoming increasingly popular in the online delivery of media.)

Unlike streaming servers which rarely send more than 10 seconds of media data to the client at a time, HTTP web servers keep the data flowing until the download is complete. This leads to the user experience that we have by now grown very accustomed to thanks to YouTube – if you pause a YouTube video at the beginning of playback and wait, eventually the entire video will have downloaded to your browser cache, allowing you to smoothly play the whole video without any hiccups. There is a downside to this behavior as well – if 30 seconds into a fully downloaded 10 minute video you decide you don’t like it and quit the video, both you and your content provider have just wasted 9:30 minutes worth of bandwidth. In an effort to mitigate this problem, IIS7 Media Pack 1.0 provides a cool feature called Bit Rate Throttling which allows content providers to throttle the download bitrate in order to reduce costs. But that’s another story…

Adaptive Streaming

Speaking of misnomers… Here’s another one: Adaptive Streaming. Guess what? It’s not really streaming in the classic sense at all.

Adaptive streaming is really a hybrid delivery method. It acts like streaming but is in fact based on HTTP progressive download. A more technically accurate name for adaptive streaming might be “A Series of Progressive Downloads of Variable Sized Video Fragments,” but even the most determined marketing experts would have a hard time selling that one. :)

A very important thing to remember about adaptive streaming is that there doesn’t really exist a standard implementation for it today because it’s just an advanced download concept, rather than a new protocol. This is why we talk about both Microsoft Smooth Streaming and Move Networks Adaptive Stream as examples of adaptive streaming, even though they use mutually incompatible codecs, formats and encryption schemes. They both rely on HTTP as the transport protocol and perform the media download as a long series of very small progressive downloads, rather than one big progressive download.

In a prototypical adaptive streaming implementation, the video/audio source is cut up into many short segments (“chunks”) and encoded to the desired delivery format. Chunks are typically 2-4 seconds in length. On the video codec level this typically means that each chunk is cut along video GOP boundaries (each chunk starts with a key frame) and has no dependencies on past or future chunks/GOPs. This allows every chunk to later be decoded completely independently from other chunks.

The encoded chunks are then hosted on a regular HTTP web server. A client requests the chunks from the web server in a linear fashion and downloads them using plain HTTP progressive download. As the chunks are downloaded to the client, the client plays back the sequence of chunks in linear order. Because the chunks were carefully encoded without any gaps or overlaps between them, the chunks play back as a seamless video.

The “adaptive” part of the solution comes into play when the video/audio source is encoded at multiple bitrates, generating multiple chunks of various sizes for each 2-4 seconds of video. The client now has the option to choose between chunks of different sizes. Because web servers usually deliver data as fast as network bandwidth allows them, the client can easily estimate user bandwidth and decide to download bigger or smaller chunks ahead of time. The size of the playback/download buffer is fully customizable.

Adaptive streaming is a hybrid media delivery method

Adaptive streaming, like other forms of HTTP delivery, offers the following advantages to the content provider:

• It’s cheaper to deploy because adaptive streaming can utilize any generic HTTP caches/proxies and doesn’t require specialized servers at every node
• It offers better scalability and reach, reducing “last mile” issues because it can dynamically adapt to inferior network conditions as it gets closer to the user’s home
• It lets the audience adapt to the content, rather than requiring the content providers to guess which bitrates are most likely to be accessible to their audience

It also offers the following benefits for the end user:

• Fast start-up and seek times because start-up/seeking can be initiated on the lowest bitrate before moving up to a higher bitrate
• No buffering, no disconnects, no playback stutter (as long as the user meets the minimum bitrate requirement)
• Seamless bitrate switching based on network conditions and CPU capabilities
• A generally consistent, smooth playback experience

Microsoft Adaptive Streaming Prototype: NBC Olympics 2008

We first prototyped an implementation of HTTP-based adaptive streaming as part of the NBC Olympics 2008 project. In order to deliver the desired level of quality in a short period of time, we took the most basic adaptive streaming implementation approach. We had NBC’s Digital Rapids and Anystream encoders produce multiple WMV files of different bitrates/resolutions for each source. The encoders didn’t employ any new encoding tricks but merely followed strict encoding guidelines (closed GOP, fixed length GOP, VC-1 entry point headers) which ensured exact frame alignment across the various bitrates of the same video. Then we ran the WMV files through a post-processing tool which physically split each WMV file into thousands of 2-second chunks (files). The rest of the solution consisted of simply uploading the chunks to Limelight’s origin web servers (running Apache) and then building a Silverlight player that would download the chunks and play them in sequence. Simple!

The good news: Our implementation worked great for the end users. We were able to offer a better-than-WMS streaming experience while using just simple HTTP download!

The bad news: CDN operators lost many hours (days?) managing millions of tiny files in their systems. Imagine: if every 2-seconds of video is split into a separate file and this is repeated for 5 available bitrates, you end up with 150 files for every minute of video. That’s 13,500 files for a 90-minute soccer game!

So despite NBC Olympics being a huge success for Silverlight and HTTP-based adaptive streaming, it quickly became apparent we had to go back to the drawing board on this one.

At Last! Smooth Streaming!

The IIS Media team soon took charge of turning the NBC Olympics adaptive streaming solution into a real server product. Its official name – IIS7 Smooth Streaming, an extension for Internet Information Services 7.0.

The IIS Media team redesigned the content creation and delivery aspect of the prototype solution in order to fix the file management issues while still keeping all the advantages of the original solution. The new design eschewed the one-file-per-chunk approach in favor of a single contiguous file for each encoded bitrate. The file format of choice: MPEG-4.

Smooth Streaming server uses the MPEG-4 Part 14 (ISO/IEC 14496-12) file format as its disk (storage) and wire (transport) format. Specifically, the Smooth Streaming specification defines each chunk/GOP as an MPEG-4 Movie Fragment (moof) and stores it within a contiguous MP4 file for easy random access. One MP4 file is expected per each bitrate. When the client requests a specific source time segment from the IIS server, the server dynamically finds the appropriate Movie Fragment box within the contiguous MP4 file and sends it over the wire as a standalone file, thus ensuring full cacheability downstream.

In other words, with Smooth Streaming file chunks are created virtually upon client request, but the actual video is stored on disk as a single full-length file per encoded bitrate.

Smooth Streaming Availability

Smooth Streaming server support will ship as part of the next edition of IIS7 Media Pack, a free download for Windows Server 2008. A technology preview of IIS7 Smooth Streaming Server is already available now through Akamai. They are calling this service Akamai AdaptiveEdge Streaming for Microsoft Silverlight. A demo of the service is available at http://www.smoothhd.com.

On the content creation end, creation of on-demand Smooth Streaming-compatible video is already possible with the latest Expression Encoder 2 SP1. Note that you’ll need to purchase the full version of Expression Encoder 2 in order to get Smooth Streaming encoding support – it’s not included in the “Express” trial version. As a helper tool for encoding to multiple-bitrate formats such as Smooth Streaming, I recommend my Smooth Streaming Calculator. (More about Smooth Streaming encoding with Expression Encoder to come soon.)

In addition, we are already working with a number of encoding ISVs on enabling support for the Smooth Streaming format in their professional encoding products.

Smooth Streaming Playback:

You probably already know that Silverlight 2 supports playback of Smooth Streaming sources (if you don’t, go to http://www.smoothhd.com). But how does it do it?

Despite popular belief, Silverlight doesn’t actually feature native support for any particular adaptive streaming technology – Microsoft’s, Netflix’s or Move Networks’ for example. Smooth Streaming support in Silverlight is implemented via the MediaStreamSource API. This API allows developers to implement their own media transport methods (instead of relying on MediaElement‘s native transport methods) while still leveraging Silverlight’s native decoders and renderers. In other words, Silverlight support for Smooth Streaming is provided entirely in .NET code: the parsing of the MPEG-4 file format, the HTTP download, the bitrate switching heuristics, etc. This allows developers to modify and fine-tune the client adaptive streaming code as needed, instead of waiting for the next Silverlight release and hoping it magically fixes every customer scenario.

The most challenging part of Smooth Streaming Silverlight client development is the heuristics module which determines when and how to switch bitrates. Elementary stream switching functionality requires the ability to swiftly adapt to changing network conditions while never falling too far behind, but that’s often not enough to deliver a great experience. One must also consider: What if the user has enough bandwidth but doesn’t have enough CPU power to consume the high bitrates/resolutions? What happens when the video is paused or hidden in the background (i.e. minimized browser window)? What if the resolution of the best available video stream is actually larger than the screen resolution, thus wasting bandwidth? How large should the download buffer window be? How does one ensure seamless rollover to new media assets such as ads? As any web application developer will tell, you there’s much more to building a good player than just setting a source URL for the media element.

Fortunately for those who prefer not to write such code from scratch, there are already two options available for adding Smooth Streaming support to your Silverlight application:

1. Expression Encoder 2 SP1 templates
   Every Silverlight 2 player template included with Expression Encoder 2 SP1 includes a ready Smooth Streaming module as well as complete source code (which can be modified and used freely). The Smooth Streaming object (named AdaptiveStreaming.dll) can be easily integrated into any Silverlight project. See James Clarke’s blog for additional Expression Encoder tips & tricks.
2. Open Video Player (OVP)
   The Akamai-led Open Video Player Initiative is an open-source community project that strives to provide a best-of-breed video player platform for Silverlight and Flash. The Silverlight version of the Open Video Player provides integrated support for Smooth Streaming playback, and is in fact the video player used by Akamai on SmoothHD.com and many of their customer sites.

  These templates provide great out-of-the-box Smooth Streaming experiences while also allowing developers to continue innovating and fine-tuning the client code.

In my next blog post:  The Smooth Streaming Format

Let’s begin by explaining the 2 video player user interfaces and the plugins that power each.

User Interface

The NBC Olympics video player is available in 2 flavors: Standard and Enhanced. The Standard player UI is what you get when you first launch the video player. The Enhanced player UI is what you get when you click on the “Enhanced” button in the lower right corner of the Standard player.

The Standard player has a video rectangle of size 592×336 (roughly a 16:9 aspect ratio) and can be experienced with either WMP or Silverlight plugins. As explained in earlier posts, if you are running Windows OS + Internet Explorer or Firefox browser + WMP9 or better (ideally WMP11), you can choose to use the WMP plugin instead of the Silverlight plugin to view video by choosing “Watch without Plugin” when prompted to install Silverlight. The video streams available in the Standard player are identical regardless of whether you’re using WMP or Silverlight. The bitrate of those streams never exceeds 650 kbps in the Standard player.

The Enhanced player is only available to those who have installed the Silverlight plugin. It provides a more interactive experience and features a larger video window, as well as higher resolution and higher bitrate video streams (for some content). The video rectangle is 848×480 (also roughly 16:9 aspect ratio).

Video and Audio Codecs

All video on the NBC Olympics website is encoded as VC-1 Advanced Profile in CBR mode at various bitrates (described below).

All audio is encoded as WMA 10 Professional audio at 48 kbps, 44.1 kHz, stereo. The special Low Bitrate (LBR) mode of the WMA Professional codec offers improved fidelity over the more commonly used WMA Standard codec and is comparable with HE-AAC quality.

Content Categories

The content is generally divided into 2 categories: Live/Rewind and Highlights/Encore.

Live video (and its archived counterpart Rewind) is encoded on site in Beijing, then beamed back to New York and distributed to homes via CDNs. It comes in 2 bitrates and sizes:

• 592×336 at 600 kbps
• 320×176 at 300 kbps

The reason why higher bitrates aren’t offered for Live streams is because NBC’s link from Beijing to New York has a fixed bandwidth and needs to be able to sustain many simultaneous live streams (1 Mbps per event, and there can be as many as 30 events happening at the same time). In addition, delivering more than 1 Mbps of video around the world without losing packets all over the place or running into last-mile bottlenecks – is still incredibly difficult even in 2008.

Highlights/Encore video is content produced and encoded by NBC in New York. It typically features highlights, previews, recaps, interviews – so generally anything that’s not a full rewind of an event. It comes in 4 bitrates:

• 320×176 at 350 kbps
• 424×240 at 600 kbps
• 592×336 at 1050 kbps
• 848×480 at 1450 kbps

As mentioned above, the higher bitrates are only available in the Silverlight-exclusive Enhanced player interface. The Standard player is only able to consume the first 2 lower-bitrate streams.

In addition to all the bitrates and resolutions mentioned above, all content is available for thumbnail-sized Picture-In-Picture viewing. PiP video is always encoded as 128×96 at 50 kbps and half the source framerate.

This means that the minimum bandwidth needed to view the highest quality video + PiP is 1550 kbps (1450 video + 48 audio + 50 PiP) in perfect conditions. In reality, you probably need at least 100 kbps overhead on top of that in order to compensate for Internet unreliability.

Much of the press coverage of the NBC Olympics website has referred to the video content as being ”HD quality.” The definition of “HD” for television has always been pretty clear: you need at least 1280×720 to call something “HD.” Unfortunately, the definition of HD video on the web has been far more ambiguous. It’s the YouTube effect. Once you get used to watching 320×240 poorly compressed video for so long, anything above that suddenly starts looking like Digital Cinema. :) Whether or not you choose to think of 848×480 video as HD is up to you. I personally wouldn’t, but then again – it’s my job to be nitpicky about video quality.

Streaming Methods

Finally, there’s the actual delivery of the content. Two basic methods of streaming are used on the NBC Olympics website.

All Live video – regardless of which plugin is consuming it – is streamed via WMS HTTP streaming protocol from Windows Server 2008 servers running Windows Media Services. The same streaming method is also used for all delivery to the WMP plugin. If you’re using the WMP plugin, you always get the WMS stream, regardless of content type.

As mentioned before, the Silverlight-powered Enhanced player has several features that make it a superior experience to the Standard player and WMP plugin. One of them is its ability to seamlessly switch between streams of different bitrates and resolutions during playback to dynamically match the user’s bandwidth and CPU power. This feature, often referred to generically as Adaptive Streaming, is something that Microsoft developed for NBC based on Silverlight 2′s MediaStreamSource interface. NBC’s website does not utilize the Move Networks adaptive streaming technology, as has been widely rumored. Silverlight 2 supports hooks to multiple adaptive streaming approaches, including Move’s - but in this particular case Microsoft provided the solution.

The easiest way to recognize that you’re watching an adaptively streamed video while in the Enhanced player is by seeking to another point in the video. If the player is using adaptive streaming, you will see the video start up very quickly without a buffering notification and the resolution will briefly drop. After a few seconds the blurry video will get sharper, and then sharper again… and then sharper again, your bandwidth allowing, of course.

Summary – Setting expectations

Here’s the best quality you can expect for NBC Olympics video, as well as minimum requirements:

• Live/Rewind content:
  • Either plugin: 592×336 at 650 kbps
• Highlights/Encore content:
  • WMP plugin:  424×240 at 650 kbps
  • Silverlight plugin and Enhanced player:  848×480 at 1500 kbps

http://silverlight.net/forums/p/22318/80644.aspx

Microsoft’s Tom Taylor has provided some context and explanation for the controversial design decision.

The NBCO website specifically lists the plug-ins required for experiencing all sections of the website: http://www.nbcolympics.com/pluginsneeded.html

As you can see, only Flash is a requirement for access to the main (non-video) site. To access the video content on the website, Silverlight 2 Beta 2 plug-in or Windows Media Player are required. The various Silverlight supported platforms were outlined in my previous post, but what about WMP support?

WMP “fallback” mode was a key part of the NBC Olympics player design from the very start. Because the Silverlight 1 and 2 media pipeline is  built on top of the Windows Media format (ASF) and codecs (VC-1, WMV8, WMV7, WMA9 Standard & Pro, MP3), any media content produced for Silverlight is also backwards compatible with WMP – so it only made sense to re-use the same streams for WMP as a “fallback” option in case certain users didn’t wish to install Silverlight 2 (after all, it is a beta) or simply couldn’t install it due to non-admin restrictions or due to being on an unsupported Windows OS.

Now, I know some may immediately ask, “Well, if you wanted to provide a fallback option, why not just use Flash as an alternative?” All business politics aside, it’s important to understand that creating all the content in duplicate (and trust me, there’s A LOT of content being produced for these Olympic games) would’ve been extremely inefficient with regards to both time and cost. Not only would all content need to be encoded twice, but the bandwidth of NBC’s direct link from Beijing to New York would need to be doubled, and NBC would need to deploy twice the number of encoders and servers, etc, etc, etc. Supporting Flash would’ve also doubled the engineering cost of designing, implementing and testing the video player application. Speaking from a purely engineering perspective, I can say that getting this project off the ground and to this final stage was an incredibly ambitious undertaking with just Windows Media alone. Supporting a whole additional set of formats, codecs and RIA technology would’ve been nothing short of impossible.

But back to WMP support…

At the moment WMP support is limited to Windows OS only, and Internet Explorer and Firefox browsers only. The good news is that any Windows OS running WMP9 or later ought to work – and that includes even ancient Windows 98SE, Millennium and 2000 systems. However, if using the WMP fallback mode, it is definitely recommended that you use WMP11 (XP, Vista and WS2008) for optimal video and audio playback quality. If you are using WMP9 and WMP10, upon visiting the NBC Olympics video player page for the first time you may be prompted with an ActiveX security dialog asking you to install “wvc1dmo.cab” or “Windows Media Audio Codec.” It is safe to install these updates – and actually required to make the NBCO video player work with WMP9/WMP10. If you don’t get any security dialog prompt, but can’t see any video either – your browser’s security settings might be blocking the ActiveX install prompts. In that case you can install the necessary video codec update manually from http://support.microsoft.com/kb/942423.

It has also been suggested that Mac PPC users might be able to get the WMP fallback solution working for them by installing Flip4Mac WMV. Unfortunately, at the time of this writing that solution is not working, partially due to an apparent incompatibility between the NBCO player Javascript code and Flip4Mac/Safari/Firefox. If this changes at any point, I’ll make sure to post about it immediately.

Finally, for those who do have a choice of installing Silverlight instead of using WMP, what advantage does Silverlight bring to the table? For starters, all WMP-targeted video streams are limited to 650 kbps, whereas the Silverlight plug-in can take advantage of higher-bitrate and higher-resolution video streams, all the way up to 1500 kbps. Furthermore, all WMP playback is single-bitrate only with no dynamic/adaptive stream-switching capability. The Silverlight-based player, on the other hand, can use adaptive streaming (dynamic bitrate switching) for most NBCO content that’s not Live or Rewind. So the short answer to the question of what advantage Silverlight has over WMP is: better video quality and more reliable streaming methods.

Anyone living in the U.S. can watch live and archived streaming video of nearly every Olympic event for free by visiting:

http://www.nbcolympics.com/video/index.html

The reason that only United States residents have access to NBC’s video streams is because NBC owns the rights to broadcasts of the 2008 Summer Olympics only for the United States. The International Olympic Committee sells broadcasts rights to only one broadcaster per country, so if you’re in Canada – you have to watch CBC; if you’re in the UK – you have to watch the BBC, and so on. I’m sure NBC wouldn’t have minded selling ads globally if the IOC had let them. ;)

The NBC Olympics website doesn’t require (as some news reports and blogs have stated) Microsoft Silverlight to run, but is optimized for it. This is even stated on the NBC Olympics website. The Silverlight 2 Beta 2 plugin needed for the optimal (and intended) rich web experience is a minimal download and a very light-weight install. And despite being dubbed a “Beta 2,” it’s actually easily one of the most stable Microsoft products I’ve seen in years. Here’s the list of Silverlight 2 supported operating systems and browsers:

• Windows Vista: Internet Explorer 7 or better, Firefox 1.5 or better
• Windows XP SP2 or SP3: Internet Explorer 6 or better, Firefox 1.5 or better
• Mac OS X 10.4.8+ (Intel only): Firefox 1.5 or better, Safari 2 or better

Microsoft’s Silverlight website also mentions the following platforms as being supported by Silverlight 2, but I guess NBC isn’t explicitly supporting them, so proceed at your own risk: Windows 2000 w/ IE6, Windows Server 2003 w/ IE6+ or Firefox 1.5+. And if Silverlight 2 works on Vista, one would also assume it works on Windows Server 2008 too.

A list of Frequently Asked Questions is also available on the NBC Olympics Video website.

Finally, here’s a few articles that describe the project and showcase its highlights:

http://news.cnet.com/8301-13860_3-10002909-56.html
http://seattlepi.nwsource.com/business/373888_msftoly07.html?source=mypi

MSDN’s Channel 9 also has a 20-minute interview with fellow Silverlight evangelist Eric Schmidt who talks about the various features of the NBCO video player and what it took to pull off this massive effort:

http://channel9.msdn.com/shows/Continuum/Building-NBCOlympicscom-with-Silverlight/