Service Pack 1 for Expression Encoder 4 will be released on January 31, 2011. The encoder update will be available for download at:

http://www.microsoft.com/Expression/service-packs/Encoder.aspx

This is a significant update to EE4, boasting an impressive list of new features:

The most significant one, as I mentioned in my previous post, is GPU-accelerated H.264 encoding. It is powered by NVIDIA CUDA technology via the MainConcept H.264 CUDA SDK. The Expression Encoder team has put together an excellent white paper describing the new feature and best practices. A separate report will be published later this week with performance data and comparisons of recommended PC configurations.

Update (1/27/2011): An interesting note about the H.264 GPU acceleration feature:  When using the EE4 GUI application, GPU-accelerated H.264 encoding will be enabled by default (assuming your hardware is CUDA 1.1 compatible) unless you decide to disable it via the Tools|Options configuration dialog. However, when using the EE4 SDK, GPU acceleration will be disabled by default, meaning that any apps that you built on the EE4 SDK will need to be recompiled with the GPU acceleration option explicitly enabled if you want to take advantage of the new feature. Otherwise they will just default to the old CPU-based H.264 encoding method.

After PDC 2010 in November which focused on new web technologies (e.g. IE9, HTML5, WP7), we hosted a special event in December called Silverlight Firestarter which focused exclusively on Silverlight. Scott Guthrie announced some of the new features going into Silverlight 5. Among the many features, the ones most interesting to media developers were:

• Support for GPU-accelerated H.264 decoding, intended to make H.264 video perform better on low powered computers and increase battery life on laptops and netbooks.
• Variable speed playback, also known as “trick play”, which will allow you to fast forward and rewind videos as well as play them back at up to 2x speed while maintaining the audio pitch. This is a feature particularly useful for e-learning, as well as suffering through boring corporate training videos. :) Note that some aspects of trick play, such as rew/ffwd, have already been implemented in the Silverlight Media Framework for Smooth Streaming videos.
• OS power scheme awareness. This will allow you to watch long videos in Silverlight without worrying about your computer going to sleep during the season finale of “Dexter”.
• Remote control support for media scenarios. I presume this will include support for Media Center remote controls.
• Advanced DRM management. I don’t know much about this, but I’m guessing it has to do with supporting seamless key rotations.

Other noteworthy SL5 features include 64-bit support and improved GPU-accelerated graphics APIs.

Around the same time we shipped Windows Phone 7 which uses Silverlight as one of its two app development platforms (the other one being XNA). WP7 support ended up being one of the main features of the Smooth Streaming Client 1.5 (aka SSME) too. Vishal Sood wrote a great post about encoding video for Windows Phone 7. The biggest difference between encoding for WP7 versus the desktop is WP7 video driver’s current lack of support for mid-stream resolution changes. What this effectively means is that all streams in a Smooth Streaming asset delivered to WP7 must be encoded at the same resolution. We hope to fix this in future WP7 updates, but for now we have to live with it.

2010 also saw the release of IIS Media Services 4.0. The most exciting new feature of this release is support for dynamic repackaging and delivery of Smooth Streaming assets (VOD and live) to Apple iOS devices such as the iPhone and iPad. Apple iOS devices support HTTP Live Streaming, an HTTP adaptive streaming technology developed by Apple and based on the MPEG-2 Transport Stream file format. Because both Microsoft and Apple support H.264/AAC codecs, it is possible to repackage (remux) Smooth Streaming H.264/AAC assets into Apple HTTP Live assets by simply changing the container formats, without a need for re-encoding the video and audio streams. The IIS Media Services 4.0 release also supports live DVR features, AES encryption, archiving and B-frame encoding (H.264 Main/High Profile) for iPhone4 and iPad.

Apple has documented some of their own encoding best practices for iOS devices, though frankly, they seem a little on the conservative side. I’ve seen many online reports of iPhone4/iPad being able to handle H.264 High Profile at 1280×720, so it seems strange that Apple would limit their recommendations to just Main Profile at 640×360 (especially considering iPad’s display resolution is 1024×768). If you have experience with streaming to iOS devices and would like to share your encoding settings and feedback, please leave a comment. Even better – if you work for Apple and your focus is HTTP Live Streaming, send me an e-mail.

IIS Media Services 4.0 also includes support for low-latency streaming which can bring down the typical Live Smooth Streaming end-to-end latency to under 2 seconds. This requires the latest Smooth Streaming Client version and a low-latency compatible realtime video encoder, such as the ones made by Allegro DVT, Digital Rapids, Elemental, Envivio and Inlet. (Did I forget your company? Let me know.)

Though not officially a part of the IIS Media Services package, the IIS Media team also recently released IIS Transform Manager 1.0 alpha, another IIS extension which allows you to automate your transcoding/transmuxing workflow. The most recent public release is just an alpha, but shows a lot of promise. It features integration with Microsoft Expression Encoder (for transcoding to WMV, MP4 and Smooth) and offline transmuxing (repackaging) of Smooth VOD assets into Apple HTTP adaptive assets. The final release, coming soon, will also feature support for distributed transcoding, DRM packaging tasks and – my favorite – MP4-to-Smooth tasks. The last one is particularly exciting because it will allow you to encode to Smooth Streaming using any H.264 encoder regardless of its lack of Smooth Streaming file support. For example, you’ll be able to encode videos with the awesome open-source x264 encoder and automatically convert its MP4 output files into Smooth Streaming assets as a background task.

Completing the slew of recent media releases are:

• Silverlight Media Framework 2.3
  • New SMF features include support for WP7 and SSME 1.5.
• Silverlight Rough Cut Editor 1.0 SP1
  • If you have never seen RCE in action, do yourself a favor and check out my colleague Jason Suess’ excellent session from last year’s MIX. It’s an incredible tool which any broadcaster or content publisher can benefit from.
• Native Extensions 1.0 for Microsoft Silverlight
  • While not necessarily directly related to media, NESL provides features in out-of-browser Silverlight apps that aren’t available in the browser, such as integration with Windows sensors, access to portable devices, webcam capture/encoding/archiving, speech recognition and Windows 7 taskbar integration.

Last, but certainly not the least, stay tuned for the upcoming Microsoft Expression Encoder 4 service pack which will enable GPU-accelerated H.264 encoding using NVIDIA CUDA technology. Expression Encoder Pro’s H.264 compression engine is powered by MainConcept, and the new acceleration feature will be powered by MainConcept’s CUDA H.264 Encoder SDK. SP1 is scheduled for release in Q1 2011.

Silverlight 3 Beta

A mere 5 months after releasing Silverlight 2 RTW, we have now made Silverlight 3 Beta available to the public. Check out http://silverlight.net/getstarted/silverlight3/default.aspx for the full list of new features and download links. As usual, Scott Guthrie offers some great insight on Silverlight 3 in his MIX Keynote and Channel 9 video.

The most interesting new media features in Silverlight 3 are:

• Native H.264 video, AAC audio, and MP4 file playback support
  • Take your MP4-contained H.264/AAC encoded files, put them on a Web server and play them directly in Siverlight using progressive download!
• Extensible media format support
  • Support for raw A/V bitstream playback allows codec and media developers to write custom decoders and format parsers using C#, VB or any other .NET language. Will you be the first to write an Ogg Vorbis or FLAC decoder for Silverlight?
• GPU accelerated video scaling
  • Stretching the video to full screen can now be offloaded entirely to the video card, freeing up the CPU and enabling smooth video playback.
• Advanced media logging
  • Log playback usage to Windows Media Services and IIS7 Media Services, like with the good old Windows Media Player.
• Custom effects / Pixel shaders
  • Apply post-processing effects to your video by writing custom effects using the same HLSL pixel shader code that works in Direct3D and WPF today.
• Perspective 3D transforms
  • Spin and rotate your video around all 3 axis – X, Y and Z. Video collage? How about a video cube?

I’ll be blogging in the near future in more depth about the details of our H.264/AAC/MP4 support in particular.

IIS Media Services 3.0 / Live Smooth Streaming

Just a short month after announcing the availability of Smooth Streaming for On-Demand Video beta, the IIS Media team announced the availability of IIS Media Services 3.0 beta – featuring Live Smooth Streaming. That’s right, with IIS7 you will soon be able to deliver Smooth Streaming video for both on-demand and live!

Inlet Technologies has simultaneously announced they will be the first to add Smooth Streaming support to their line of live and VoD encoding products. Besides Inlet, we are currently working with a number of encoding ISVs on enabling them to add Smooth Streaming support to their products.

Akamai Technologies announced the wide commercial availability of their AdaptiveEdge Streaming service based on IIS Smooth Streaming. Besides Akamai, we are currently working with all the major CDNs on enabling Smooth Streaming support in their networks. As with the encoding ISVs, our goal is to build a rich Smooth Streaming ecosystem to be available to customers by the time Silverlight 3 ships.

Besides the newly redesigned home page, the IIS Media team has also put up a great working example of how Smooth Streaming works.

NBC Winter Olympics 2010 – Vancouver

During the MIX 2009 Keynote, Perkins Miller, Senior VP of Digital Media for NBC Universal, announced that NBC Universal has chosen to deliver the NBC Winter Olympics 2010 using Microsoft Silverlight. Watch the MIX Keynote to see his announcement.

Here are the details I can share at this point:

• All video content, both live and on-demand, will be delivered using Smooth Streaming
• The live video player will feature DVR-like capabilities (pause, rewind, seek and slo-mo of live video)
• Video quality will go up to true 720p HD

March Madness

CBS Sports has launched a Silverlight-based March Madness video player that lets you watch all NCAA Basketball Tournament games live. Visit http://mmod.ncaa.com/video to launch the March Madness video player. If you are using Internet Explorer on Windows, the default player will actually be an old-school WMP player, so you’ll need to click on the HQ Player button to launch the new Silverlight player.

The live video for the tournament is being streamed using Windows Media Services. Obviously, we couldn’t use Smooth Streaming because the server technology is still in beta and the encoders aren’t yet commercially available. But CBS did the next best thing! All live streams are available in 4 video quality levels:

Video codec used is VC-1 Advanced Profile. Audio codec used is WMA Professional at 44.1 kHz 16-bit stereo.

All March Madness games are being encoded by MLB.com’s encoding facilities using Inlet Spinnaker 7000 encoders. The Spinnakers were configured based on my own recommendations in order to provide maximum quality at all bitrates.

The March Madness Silverlight player uses preroll ad download statistics to estimate available client bandwidth and tries to make an appropriate first choice of bitrate level. Of course none of this would be necessary with Smooth Streaming, but we really tried to make the best of the Windows Media Streaming experience anyway. The player also has built-in heuristics to detect quality-of-service issues, such as frequent rebuffering or low frame rate rendering, at which point it can suggest to the user to choose a lower bitrate. Users can manually switch between available bitrates using the “-” and “+” buttons in the button of the player UI.

If you’re interested in the world of online video streaming (and I guess you are since you’re already reading this blog) and plan on visiting MIX 2009, I suggest you check out the following few sessions while there:

What’s New in Microsoft Silverlight 3
Speaker:  Joe Stegman (Microsoft)
Date/Time:  Wed, March 18, 11:30 AM – 12:15 PM
Location:  Lando 4204

Microsoft Silverlight Media End-to-End
Speaker:  Alex Zambelli (Microsoft)
Date/Time:  Wed, March 18, 2:15 PM – 3:30 PM
Location:  Delfino 4105

Creating Media Content for Microsoft Silverlight Using Microsoft Expression Encoder
Speaker:  James Clarke (Microsoft)
Date/Time:  Wed, March 18, 4:00 PM – 5:15 PM
Location:  Delfino 4105

Delivering Media with Microsoft Internet Information Services 7 (IIS) Media Services and Microsoft Silverlight
Speakers:  John Bocharov (Microsoft), John Bishop (Inlet)
Date/Time:  Thu, March 19, 1:00 PM – 2:15 PM
Location:  Delfino 4105

For a complete list of sessions, see https://content.visitmix.com/2009/sessions/.

To register for MIX 2009, visit http://2009.visitmix.com/Registration/.

Hope to see you there!

http://alexzambelli.com/blog/smooth-streaming-faq/

In this post we’ll take a closer look at how Smooth Streaming works: format, server, and client.

Smooth Streaming Format

Smooth Streaming is the first Microsoft media format in over a decade to use a file format other than ASF. It is based on the ISO/IEC 14496-12 ISO Base Media File Format specification, better known as the MP4 file specification. Why MP4 and not ASF? Well, there are several reasons:

• MP4 is a lightweight container format with less overhead than ASF
• MP4 is easier to parse in managed (.NET) code than ASF
• MP4 is based on a widely used standard, making 3rd party adoption and support more straightforward
• MP4 was architected with H.264 video codec support in mind, and we’re counting on H.264 support in Smooth Streaming and Silverlight 3 (ASF can also contain H.264 video, but it’s not as straightforward as with MP4)
• MP4 was designed to natively support payload fragmentation within the file

There are actually 2 parts to the Smooth Streaming format: the wire format, and the disk file format. In Smooth Streaming a video is recorded in full length to the disk as a single file (one file per encoded bitrate), but it’s transfered to the client as a series of small file chunks. The wire format defines the structure of the chunks that get sent by IIS to the client, whereas the file format defines the structure of the contiguous file on disk. Fortunately, the MP4 specification allows MP4 to be internally organized as a series of fragments, which means that in Smooth Streaming the wire format is a direct subset of the file format.

What are these MP4 “fragments” that I speak of? The basic unit of an MP4 file is called a “box.” These boxes can contain both data and metadata. The MP4 specification allows for various ways to organize data and metadata boxes within a file. In most media scenarios it is considered useful to have the metadata written before the data so that a player client application can have more information about the video/audio it’s about to play before it plays it. However, in live streaming scenarios it is often not possible to write the metadata upfront about the whole data stream because it’s simply not fully known yet. Furthermore, less upfront metadata means less overhead, which can lead to shorter startup times. For these reasons the MP4 ISO Base Media File Format specification was designed to allow MP4 boxes to be organized in a fragmented manner, where the file can be written “as you go” as a series of short metadata/data box pairs, rather than one long metadata/data pair. The Smooth Streaming file format heavily leverages this aspect of the MP4 file specification, to the point where at Microsoft we often interchangeably refer to Smooth Streaming files as “Fragmented MP4 files” or “(f)MP4.”

Here is a high-level overview of what a Smooth Streaming file looks like on the inside:

Smooth Streaming File Format

In a nutshell, the file starts with file-level metadata (‘moov‘) which generically describes the file, but the bulk of the payload is actually contained in the fragment boxes which also carry more accurate fragment-level metadata (‘moof‘) and media data (‘mdat‘). (The diagram only shows 2 fragments, but a typical Smooth Streaming file has a fragment per every 2 seconds of video/audio.) Closing the file is a ‘mfra‘ index box which allows easy and accurate seeking within the file.

When a Silverlight client requests a video time slice from the IIS Smooth Streaming server, the server simply seeks to the approriate starting fragment in the MP4 file and then lifts the fragment out of the file and sends it over the wire to the client. This is why we refer to the fragments as the “wire format.” This technique greatly enhances the efficiency of the IIS server because it requires no remuxing or rewriting overhead.

Here is what an MP4 fragment looks like in more detail:

Smooth Streaming Wire Format

We say that the Smooth Streaming format is based on the MP4 file format because even though we’re following the ISO specification, we specify our own box organization schema and some custom boxes. In order to differentiate Smooth Streaming files from “vanilla” MP4 files, we use new file extensions: *.ismv (video+audio) and *.isma (audio only). I keep forgetting to ask the IIS Media team what the acronyms exactly stand for, but my best guess would be “IIS Smooth Streaming Media Video (Audio)”.

Smooth Streaming Media Assets

A typical Smooth Streaming media asset therefore consists of the following files:

• MP4 files containing video/audio
  • *.ismv – contains video and audio, or only video
    • 1 ISMV file per encoded video bitrate
  • *.isma - contains only audio
    • In videos with audio, the audio track can be muxed into an ISMV file instead of a separate ISMA file
• Server manifest file
  • *.ism
  • Describes the relationships between media tracks, bitrates and files on disk
  • Only used by the IIS Smooth Streaming server – not by client
• Client manifest file
  • *.ismc
  • Describes to the client the available streams, codecs used, bitrates encoded, video resolutions, markers, captions, etc.
  • It’s the first file delivered to the client

Both manifest file formats are based on XML. The server manifest file format is based specifically on the SMIL 2.0 XML format specification.

A folder containing a single Smooth Streaming media asset might look something like this:

A folder containing a Smooth Streaming media asset

In this particular case the audio track is contained in the NBA_3000000.ismv file.

Smooth Streaming Manifest Files

The Smooth Streaming Wire/File Format specification defines the manifest XML language as well as the MP4 box structure. Because the manifests are based on XML they are highly extensible. Among the features already included in the current Smooth Streaming format specification is support for:

• VC-1, WMA, H.264 and AAC codecs
• Text streams
• Multi-language audio tracks
• Alternate video and audio tracks (i.e. multiple camera angles, director’s commentary, etc.)
• Multiple hardware profiles (i.e. same bitrates targeted at different playback devices)
• Script commands, markers/chapters, captions
• Client manifest Gzip compression
• URL obfuscation
• Live encoding and streaming

For an example of a Smooth Streaming On-Demand Server Manifest file, see here.

For an example of a Smooth Streaming Client Manifest file, see here.

Smooth Streaming Playback: Bringing It All Home

Microsoft’s adaptive streaming prototype (used for NBC Olympics 2008) relied on physically chopping up long video files into small file chunks. In order to retrieve the chunks for the web server, the player client simply needed to download files in a logical sequence: 00001.vid, 00002.vid, 00003.vid, etc.

As I’ve explained in this and previous posts, Smooth Streaming uses a more sophisticated file format and server design. The videos are no longer split up into thousands of file chunks, but are instead “virtually” split up into fragments (typically 1 fragment per video GOP) and stored within a single contiguous MP4 file. This implies two significant changes in server and client design too:

1. The server must be able to translate URL requests into exact byte range offsets within the MP4 file, and
2. The client can request chunks in a more developer-friendly manner, such as by timecode instead of by index number

The first thing a Silverlight client requests from the Smooth Streaming server is the *.ismc client manifest. The manifest tells it which codecs were used  to compress the content (so that the Silverlight runtime can initialize the correct decoder and build the playback pipeline), which bitrates and resolutions are available, and a list of all the available chunks and either their start times or durations.

With IIS7 Smooth Streaming, a client is expected to request fragments in the form of RESTful URLs:

http://video.foo.com/NBA.ism/QualityLevels(400000)/Fragments(video=610275114)

http://video.foo.com/NBA.ism/QualityLevels(64000)/Fragments(audio=631931065)

The values passed in the URL represent encoded bitrate (i.e. 400000) and the fragment start offset (i.e. 610275114) expressed in an agreed-upon time unit (usually 100 ns). These values are known from the client manifest.

Upon receiving a request like this, the IIS7 Smooth Streaming component looks up the quality level (bitrate) in the corresponding *.ism server manifest and maps it to a physical *.ismv or *.isma file on disk. It then goes and reads the appropriate MP4 file, and based on its ‘tfra’ index box figures out which fragment box (‘moof’ + ‘mdat’) corresponds to the requested start time offset. It then extracts the said fragment box and sends it over the wire to the client as a standalone file. This is a particularly important part of the overall design because the sent fragment/file can now be automatically cached further down the network, potentially saving the origin server from sending the same fragment/file again to another client requesting the same RESTful URL.

As you can see, requesting chunks of video/audio from the server is easy. But what about dynamic bitrate switching that makes adaptive streaming so effective? This part of the Smooth Streaming experience is implemented entirely in client-side Silverlight application code – the server plays no part in the bitrate switching process. The client-side code looks at chunk download times, buffer fullness, rendered frame rates, and other factors – and based on them decides when to request higher or lower bitrates from the server. Remember, if during the encoding process we ensure that all bitrates of the same source are perfectly frame aligned (same length GOPs, no dropped frames), then switching between bitrates is completely seamless – and Smooth.

In my next blog post: Encoding For Smooth Streaming

Anyway, here’s the big news:

We will be adding support for H.264 and AAC-LC decoding to the next version of Silverlight (post v2). This is in response to the loud and clear customer demand for H.264/AAC that we’ve been hearing since Silverlight 1.0, the general convergence of the video industry around H.264, and a continuation of Microsoft’s own investment into the MPEG-4 standard.

Here’s the official Microsoft press release, framed as Q&A with Silverlight’s Scott Guthrie:

http://www.microsoft.com/presspass/features/2008/sep08/09-09silverlight.mspx

The Q&A does a very good job of answering some of the questions regarding the seemingly complex relationship between H.264, VC-1 and Windows Media, but one answer in particular bears repeating:

Addition of H.264 support in Silverlight does not mark a departure from VC-1 or Windows Media, but instead serves to enhance and expand the existing video/audio format ecosystem. The idea is to give users more choice and allow Silverlight to adapt to their existing workflows, rather than forcing it the other way around. I’ve seen too many people over the past few years get bogged down in “VC-1 vs H.264 codec wars” and I always found such obsessions to be very counterproductive. Both codecs are efficient enough to deliver excellent video quality at similar bitrates, so the question of which one to use should really be answered with “whichever one best fits your workflow and project constraints.” By adding H.264 support to Silverlight, we hope to get to a point where Silverlight is codec agnostic and customers can spend their valuable time focusing on end-to-end media delivery.

We will be showing a technology preview of H.264/AAC playback at IBC in Amsterdam (September 12-16), for which we partnered with Inlet Technologies to produce the demo content. The exact techhnical details of “what” and “how” are still being worked out, but one thing I can tell for sure is that MP4 file progressive download will definitely be supported.

A few links to blogosphere coverage of this announcement:

• http://blog.streamingmedia.com/the_business_of_online_vi/2008/09/microsoft-silve.html
• http://timheuer.com/blog/archive/2008/09/09/silverlight-to-support-h264-and-aac.aspx
• http://realworldvideocompression.com/2008/09/h264-support-for-silverlight/