Bob Cowherd from Vertigo Software has put together a nice blog post about one of the cool new features going into SMF 2.5: stereoscopic 3D video support. We will ship SMF with a sample red-cyan anaglyph plugin, while NVIDIA will provide a plugin for their 3D Vision active shutter solution.
A bit of shameless promotion:
On Thursday, March 24th, I’ll be participating in a roundtable web event titled “Transcoding Methodology 101″ hosted by Streaming Media. Joining me will be Kevin Louden (Telestream), Charlie Good (Wowza) and Jon Robbins (Rhozet). The live event is scheduled to start at 11 am Pacific Time.
You can register for the web event here:
Microsoft’s annual web developer/designer conference MIX is taking place the same week as the annual NAB show this year – also in Las Vegas. MIX 2011 is taking place April 12-14 at the Mandalay Bay hotel, while NAB 2011 is happening April 11-14 at the Las Vegas Convention Center. Microsoft’s media platform will be represented at both events, and I will be present at both too.
Here is the current list of MIX sessions:
I will be presenting a session titled “Introducing Microsoft Media Platform” in which I will give an overview of Microsoft’s media technologies and frameworks and recent releases. Other noteworthy media related sessions will be:
“Introducing MMP Content Manager” by Steven R. Woodward (Microsoft)
“MMP Video Editor” by Jason Suess (Microsoft)
“MMP Player Framework: Past, Present, Future” by Tim Greenfield (Vertigo Software)
“Introducing MMP Audience Insight” by Eric Schmidt (Microsoft)
“Behind the Scenes of Channel 9 Live at MIX” by Nic Fillingham (Microsoft)
“5 Things You Need To Know To Start Using <video> and <audio> Today” by Nigel Parker (Microsoft)
It’s not too late to register for MIX 2011! Visit http://live.visitmix.com/Registration and register today!
If you are new to IIS Media Services and Smooth Streaming, IIS.net/media is the website for you. However, finding relevant technology and deployment information on IIS.net can be somewhat cumbersome since it’s a wiki-styled site and a lot of the pertinent information scattered around employee blogs. It’s an issue we’re aware of and actively working on improving.
So in the meantime, I highly recommend using the IIS.net Media Content Map to assist you in finding useful IISMS deployment how-to docs:
Just received this news…
Service Pack 1 for Expression Encoder 4 will be released on January 31, 2011. The encoder update will be available for download at:
This is a significant update to EE4, boasting an impressive list of new features:
|Feature||Expression Encoder 4||Expression Encoder 4 Pro|
|Live template support||
|Live screen capture||
|Improved deinterlacing options||
|DRM content key||
|Windows Phone 7 presets||
The most significant one, as I mentioned in my previous post, is GPU-accelerated H.264 encoding. It is powered by NVIDIA CUDA technology via the MainConcept H.264 CUDA SDK. The Expression Encoder team has put together an excellent white paper describing the new feature and best practices. A separate report will be published later this week with performance data and comparisons of recommended PC configurations.
Update (1/27/2011): An interesting note about the H.264 GPU acceleration feature: When using the EE4 GUI application, GPU-accelerated H.264 encoding will be enabled by default (assuming your hardware is CUDA 1.1 compatible) unless you decide to disable it via the Tools|Options configuration dialog. However, when using the EE4 SDK, GPU acceleration will be disabled by default, meaning that any apps that you built on the EE4 SDK will need to be recompiled with the GPU acceleration option explicitly enabled if you want to take advantage of the new feature. Otherwise they will just default to the old CPU-based H.264 encoding method.
It’s been too long since my last post, so I thought I’d get warmed up and back into blogging form by posting something about recent releases on the Microsoft media platform.
After PDC 2010 in November which focused on new web technologies (e.g. IE9, HTML5, WP7), we hosted a special event in December called Silverlight Firestarter which focused exclusively on Silverlight. Scott Guthrie announced some of the new features going into Silverlight 5. Among the many features, the ones most interesting to media developers were:
- Support for GPU-accelerated H.264 decoding, intended to make H.264 video perform better on low powered computers and increase battery life on laptops and netbooks.
- Variable speed playback, also known as “trick play”, which will allow you to fast forward and rewind videos as well as play them back at up to 2x speed while maintaining the audio pitch. This is a feature particularly useful for e-learning, as well as suffering through boring corporate training videos. :) Note that some aspects of trick play, such as rew/ffwd, have already been implemented in the Silverlight Media Framework for Smooth Streaming videos.
- OS power scheme awareness. This will allow you to watch long videos in Silverlight without worrying about your computer going to sleep during the season finale of “Dexter”.
- Remote control support for media scenarios. I presume this will include support for Media Center remote controls.
- Advanced DRM management. I don’t know much about this, but I’m guessing it has to do with supporting seamless key rotations.
Other noteworthy SL5 features include 64-bit support and improved GPU-accelerated graphics APIs.
Around the same time we shipped Windows Phone 7 which uses Silverlight as one of its two app development platforms (the other one being XNA). WP7 support ended up being one of the main features of the Smooth Streaming Client 1.5 (aka SSME) too. Vishal Sood wrote a great post about encoding video for Windows Phone 7. The biggest difference between encoding for WP7 versus the desktop is WP7 video driver’s current lack of support for mid-stream resolution changes. What this effectively means is that all streams in a Smooth Streaming asset delivered to WP7 must be encoded at the same resolution. We hope to fix this in future WP7 updates, but for now we have to live with it.
2010 also saw the release of IIS Media Services 4.0. The most exciting new feature of this release is support for dynamic repackaging and delivery of Smooth Streaming assets (VOD and live) to Apple iOS devices such as the iPhone and iPad. Apple iOS devices support HTTP Live Streaming, an HTTP adaptive streaming technology developed by Apple and based on the MPEG-2 Transport Stream file format. Because both Microsoft and Apple support H.264/AAC codecs, it is possible to repackage (remux) Smooth Streaming H.264/AAC assets into Apple HTTP Live assets by simply changing the container formats, without a need for re-encoding the video and audio streams. The IIS Media Services 4.0 release also supports live DVR features, AES encryption, archiving and B-frame encoding (H.264 Main/High Profile) for iPhone4 and iPad.
Apple has documented some of their own encoding best practices for iOS devices, though frankly, they seem a little on the conservative side. I’ve seen many online reports of iPhone4/iPad being able to handle H.264 High Profile at 1280×720, so it seems strange that Apple would limit their recommendations to just Main Profile at 640×360 (especially considering iPad’s display resolution is 1024×768). If you have experience with streaming to iOS devices and would like to share your encoding settings and feedback, please leave a comment. Even better – if you work for Apple and your focus is HTTP Live Streaming, send me an e-mail.
IIS Media Services 4.0 also includes support for low-latency streaming which can bring down the typical Live Smooth Streaming end-to-end latency to under 2 seconds. This requires the latest Smooth Streaming Client version and a low-latency compatible realtime video encoder, such as the ones made by Allegro DVT, Digital Rapids, Elemental, Envivio and Inlet. (Did I forget your company? Let me know.)
Though not officially a part of the IIS Media Services package, the IIS Media team also recently released IIS Transform Manager 1.0 alpha, another IIS extension which allows you to automate your transcoding/transmuxing workflow. The most recent public release is just an alpha, but shows a lot of promise. It features integration with Microsoft Expression Encoder (for transcoding to WMV, MP4 and Smooth) and offline transmuxing (repackaging) of Smooth VOD assets into Apple HTTP adaptive assets. The final release, coming soon, will also feature support for distributed transcoding, DRM packaging tasks and – my favorite – MP4-to-Smooth tasks. The last one is particularly exciting because it will allow you to encode to Smooth Streaming using any H.264 encoder regardless of its lack of Smooth Streaming file support. For example, you’ll be able to encode videos with the awesome open-source x264 encoder and automatically convert its MP4 output files into Smooth Streaming assets as a background task.
Completing the slew of recent media releases are:
- Silverlight Media Framework 2.3
- New SMF features include support for WP7 and SSME 1.5.
- Silverlight Rough Cut Editor 1.0 SP1
- Native Extensions 1.0 for Microsoft Silverlight
- While not necessarily directly related to media, NESL provides features in out-of-browser Silverlight apps that aren’t available in the browser, such as integration with Windows sensors, access to portable devices, webcam capture/encoding/archiving, speech recognition and Windows 7 taskbar integration.
Last, but certainly not the least, stay tuned for the upcoming Microsoft Expression Encoder 4 service pack which will enable GPU-accelerated H.264 encoding using NVIDIA CUDA technology. Expression Encoder Pro’s H.264 compression engine is powered by MainConcept, and the new acceleration feature will be powered by MainConcept’s CUDA H.264 Encoder SDK. SP1 is scheduled for release in Q1 2011.
On July 3rd at 7:30 pm ET, you will be able to experience for the first time a live streaming event in 3D using just Silverlight and NVIDIA 3D Vision! We have teamed up with Turner Sports, NVIDIA and iStreamPlanet to bring you the NASCAR Spring Cup Series Coke Zero 400 race from Daytona, Florida live to your computer - in 3D! The TNT RaceBuddy 3D website is available now at http://www.nascar.com/racebuddy3d.
After we showed at NAB back in April that Silverlight and Smooth Streaming could be used to deliver 3D video to anaglyph glasses and external 3D-ready TVs, I also started talking to NVIDIA about the prospect of integrating Silverlight with their 3D Vision technology which utilizes active shutter glasses. As it turned out, NVIDIA was already 2 steps ahead of me and had already made significant progress getting 3D Vision to work with Silverlight, so when the opportunity came along to do a 3D streaming project with Turner Sports we all immediately saw it as a chance to do something innovative and cool. With our trusted partner iStreamPlanet on-board to do the player development and video delivery, the stage was set for building a cutting-edge 3D video experience.
NVIDIA first integrated support for 3D Vision into the latest Silverlight Media Framework which iStreamPlanet then used as the foundation for a rich 3D video player complete with an interactive leaderboard, pre-roll ads and companion ads. But iStreamPlanet didn’t just stop there – they also built stereoscopic play controls that work in 3D mode and seamlessly blend with the 3D video, which is a non-trivial task when dealing with stereoscopic presentations.
While we won’t be able to claim the title of first 3D event streamed over the Internet, there are a few “firsts” which still make this an important milestone event in video streaming:
- This will be the first time you’ll be able to watch true 3D video inside a browser with just a common browser plugin (Silverlight) and no need for a standalone 3D player. (By “true 3D” I mean “discrete stereo images delivered to each eye at 60 Hz”. In other words, no anaglyph or interlaced passive.)
- If you’re familiar with 3D Vision, you’ll know that so far it’s only been possible to use it in full screen mode. Well, this will be the first time you’ll be able to watch 3D video using NVIDIA 3D Vision in windowed mode too!
- This will be the first publicly broadcast live 3D event using HTTP-based adaptive streaming technology, namely Smooth Streaming
Turner Broadcasting will produce the 3D video in Daytona and beam it to iStreamPlanet in Las Vegas via satellite as 1080i side-by-side formatted “frame compatible” video. Using the side-by-side format will allow iStreamPlanet to leverage their existing live video workflow to deliver 3D video to Silverlight clients without requiring any changes to the workflow or the encoders. The frame-compatible 1080i video will feed into the Inlet Spinnaker HD encoders which will transcode it into the Smooth Streaming format at 6 bitrates/resolutions, with VC-1 and WMA Pro as the video and audio codecs, respectively. The transcoded steresocopic video will preserve the side-by-side formatting of the source.
Unlike with traditional Smooth Streaming broadcasts, the minimum bitrate/resolution for this 3D event will be slightly raised to provide sufficient quality even at half-resolution per eye. Therefore, the minimum resolution/bitrate will be 480×224 @ 600 kbps, while the top bitrate/resolution will remain the typical 1280×720 @ 3.45 Mbps. This means that at its highest quality you’ll be able to see a 640×720 image in each eye. While I’m well aware that’s not full 720p quality per eye, do keep in mind that we’re only just getting started with 3D Internet streaming – so this is only the beggining and it can only get better from here! Rome wasn’t built in a day. ;)
For a minimum 3D experience you will need at least 700 kbps of bandwidth (600 kbps video + 48 kbps audio + overhead) but if that’s all the bandwidth you’ve got I recommend that you watch the video in windowed mode. If you plan on enjoying the full screen 3D experience, I do recommend at least 3.5 Mbps of bandwidth or otherwise you might find yourself somewhat disappointed.
The TNT RaceBuddy 3D site is now available if you’d like to test drive the 3D player and ensure your computer meets the minimum system requirements for Saturday’s race. If you have an NVIDIA 3D Vision-enabled PC running Windows 7 or Vista, you will need the latest GeForce (or Quadro) and 3D Vision drivers installed in order to enjoy the RaceBuddy 3D experience. Needless to say, you will also need the latest version of Silverlight 4.
If you don’t have NVIDIA 3D Vision but have a nice 3D-ready TV hooked up to your PC or Mac, you’ll be happy to know that you’ll still be able to enjoy RaceBuddy 3D. Just set your desktop size to 1280×720, then fire up the RaceBuddy 3D player and tell it you have a 3DTV. When playback starts, send the player to full screen and it will show you the raw side-by-side video in 720p. All you have to do then is configure your 3DTV for a side-by-side source (left field first) and you’ll be good to go!
Note: Though NVIDIA and iStreamPlanet have successfully completed this first build of a 3D Vision-enabled Silverlight player, we are still working on figuring out how to best make that code available to all Silverlight developers, so stay tuned to my blog for more details on that in the future.
Now available: Expression Encoder 4, IIS Media Services 4.0 Beta, SSME 1.0, SMF 2.0 RC1, Olympics case studies
While most of the world is on a collective vacation watching the World Cup, media teams at Microsoft have been busy at work. And watching the World Cup. :)
Released this month:
- Expression Studio 4, which of course includes Expression Encoder 4.
- IIS Media Services 4.0 Beta
- Smooth Streaming Format SDK 1.0 Beta 2
- IIS Smooth Streaming Client 1.0, also known as Smooth Streaming Media Element (SSME), formerly known as IIS Smooth Streaming Player Development Kit
- Silverlight Media Framework 2.0 RC1
Before I dive into more details about those products, I’d also like to highlight a few case studies we recently published, all of them focused on the Silverlight-based 2010 Winter Olympics experiences built for NBC, CTV, NRK and France TV:
Some of the statistics are quite impressive, such as the number of peak concurrent users (181 thousand Americans, 134 thousand Canadians), total amount of video consumed (7.2 million hours in Canada), or the average minutes viewed over the 2 weeks of Olympics (nearly 2 hours per unique user in Canada). If you’re interested in the business side of video streaming, the value proposition of Silverlight and Smooth Streaming, and monetization data – I recommend you check out these case studies.
Expression Encoder 4
First change you might notice is that the pay version of the product is now called Expression Encoder 4 Pro, to differentiate it from the free version available for download. The good news is that the basic free version now includes Smooth Streaming (VOD) encoding, but you’ll still need to shell out $49.95 to get:
- H.264/AAC encoding (now courtesy of MainConcept H.264 SDK)
- Additional decoders for input types (MPEG2, MPEG/TS Splitter, Dolby Digital AC3, MP4 and H.264/AAC)
- Live IIS Smooth Streaming support
- Unlimited screen capture
- Digital Rights Management (PlayReady) integration
In case you missed it hidden there in the middle of the list, let me repeat: Expression Encoder 4 Pro supports LIVE Smooth Streaming! That was by far the feature most frequently requested by users, and the Encoder team listened and delivered. Live encoding in EE4 works and scales remarkably well, but I wouldn’t recommend trying to run it on any computer with less than 8 CPU cores. Video encoding is a CPU intensive process and when you have to encode multiple bitrates in realtime there’s no such thing as too much CPU. With 2 cores you can probably manage to churn out 1 bitrate up to SD resolution; with 4 cores about 2 bitrates up to SD; with 8 cores about 3-4 bitrates up to SD or 720p; with 16 cores about 7-8 bitrates up to 720p, etc.
The question I expect I’ll get asked most frequently about EE4 is: “OK, now that EE4 can encode Live Smooth Streaming for $49.95 on any PC hardware, why would I pay thousands of dollars for professional products like Digital Rapids Stream, Inlet Spinnaker or Envivio 4Caster?” My answer would go something like this: EE4, while fully capable of encoding Live Smooth Streaming, is an entry-level encoding product. It’s to professional encoding products what Windows Movie Maker is to Adobe Premiere Pro. Is it good enough for encoding a high school basketball game once a week? Absolutely. But would I use Expression Encoder to deliver 2 weeks of Olympics live video to hundreds of thousands of viewers around the world? Probably not. To use yet another analogy: Anybody can buy a stock PC, install Windows Server 2008 on it and call it a server. But does that make industry standard HP ProLiant servers obsolete? Hardly. Companies like Inlet, DR and Envivio have invested years and millions in building reliable professional encoding products which are designed and tested to run 24/7 in broadcast-type environments. So when you pay top dollar for their products, you’re not so much paying for raw codecs as you’re paying for guaranteed uptime and professional support. They cost a lot of money because they do what they do very well.
IIS Media Services 4.0 Beta
Though only a beta, this new release of IIS MS 4.0 delivers one particularly awesome feature: it can deliver Smooth Streaming H.264/AAC content to Apple “iDevices” such as the iPhone and iPad. How does it do that? Both formats support H.264 video and AAC audio; Smooth Streaming is based on MP4 (ISO Base Media) file format, while Apple Live HTTP Streaming is based on MPEG-2 TS file format. Smooth Streaming tends to use short GOP chunks (2 seconds), while Apple HTTP streaming uses long GOP (10 second) chunks. Therefore, converting between the 2 formats merely requires transmuxing A/V streams from one format to another, and this is exactly what IISMS 4.0 does: it dynamically transmuxes Smooth Streaming format into Apple’s Live HTTP Streaming format. No re-encoding.
Associated with this release are also 2 other IIS media releases:
- Smooth Streaming Format SDK 1.0 Beta 2 – provides documentation, tools, and samples you can use to multiplex video and audio bitstreams into on-demand and live Smooth Streaming output. In addition, the SDK supports the encryption of content using Microsoft PlayReady DRM. Note that this SDK doesn’t include video/audio codecs – it expects already compressed A/V samples.
- Transform Manager 1.0 Alpha – provides simple integrated video encoding and batch conversion of video files to the IIS Smooth Streaming format and the MPEG-2 TS streaming format supported by Apple devices.
For more information check out the following blog posts:
Smooth Streaming Client 1.0 and Silverlight Media Framework 2.0 RC1
The Smooth Streaming component of SMF is known as the Smooth Streaming Media Element (pronounced “Smee”, like that pirate from Peter Pan). SSME previously shipped as beta under the name “Smooth Streaming Player Development Kit” but has since been renamed to just Smooth Streaming Client. The 1.0 release (compatible with SMF 2.0) comes with rich online documentation and code samples.
NAB (National Association of Broadcasters) Show 2010 wrapped up yesterday in Las Vegas. Among the Silverlight and Smooth Streaming announcements and demos at this year’s NAB were:
- Silverlight availability on Broadcom and Intel systems-on-a-chip (SoC) solutions, which will enable set-top-box manufacturers to run Silverlight applications on their devices
- IIS Media Services 4.0 with support for Smooth Streaming over multicast networks, transmuxing support for similar HTTP-based adaptive streaming formats such as the one for Apple iPhone/iPad, low latency delivery, and Smooth Streaming H.264 DRM (PlayReady AES) support – beta availability in Q3 2010
- Silverlight Media Framework 2 with a new, more modular architecture featuring an extensive plug-in API for 3rd-party developers and partners
- Silverlight Analytics Framework for integrating business intelligence analytics into Silverlight applications
- Expression Encoder 4 with support for Live Smooth Streaming
Among other news, we’ve announced that Silverlight plug-in adoption has reached 60% globally. On Tuesday we officially launched Visual Studio 2010 and on Thursday we released Silverlight 4, a mere 3 years since our original release of Silverlight 1.0! Remember, Visual Studio 2010 lets you develop Silverlight applications not just for the desktop, but also for the upcoming Windows Phone 7 devices. Make sure to install the latest Silverlight 4 runtime here: http://www.silverlight.net/getstarted/
Full coverage of Microsoft’s Silverlight announcements at NAB 2010: http://team.silverlight.net/announcement/microsoft-silverlight-recap-at-nab-2010/
The demo that generated a lot of buzz at the Microsoft booth and one that’s very close to my heart was the demo of Live 3D HD video streaming powered by IIS Smooth Streaming and Silverlight. We partnered with TVN Group, Inlet Technologies, Level(3) Communications and THX to deliver a compelling demo showing that Smooth Streaming can be used to efficiently deliver stereoscopic 3D video over the Internet to multiple screens and display technologies.
TVN Group set up a stereoscopic camera rig in a radio station in Hannover, Germany (the radio station was chosen because it operates 24/7, allowing us to have an active live feed in Las Vegas even when it’s nighttime in Germany). They converted the two 1080i feeds into a single 1080i25 side-by-side feed which was then fed into an Inlet Spinnaker 7100 encoder. The encoder produced Smooth Streaming output at 4 quality levels: bottom one at 400×224 @ 500 kbps, top one at 1280×720 @ 3 Mbps. The streams were published to an IIS7 origin server on Level3′s network and carried by Level3 to their edge servers in the United States where we consumed them at our NAB booth (over the open Internet; we didn’t use a dedicated connection).
At our booth we demonstrated 2 playback scenarios:
- Our regular demo pods were running a Silverlight player application (built on Silverlight Media Framework) inside the browser and were connected to conventional LCD displays. Using pixel shader effects in Silverlight (the code for which I hope to make available in the near future) we dynamically converted the side-by-side stereoscopic video into a red-cyan anaglyph image, viewable using widely available anaglyph glasses such as these. We also demonstrated Silverlight’s ability to display only a 2D “mono” version of the stream (left-eye only view), as well as the ability to dynamically switch between different anaglyph methods (monochrome and color).
- In our living room area, we had another PC running the same Silverlight application out-of-browser as well as THX Media Director software. This PC was connected via HDMI to a 3D-ready 120Hz Panasonic plasma display (THX certified) equipped with active shutter glasses. With a click of a button, the Silverlight application would switch to side-by-side full screen view and send a command to the THX software which in turn would inject appropriate 3D metadata into the HDMI output stream, instructing the HDMI 1.4a compatible TV to automatically switch to frame-compatible side-by-side 3D mode. The TV would then deliver the video as frame sequential 120Hz video to the active shutter glasses, creating a superior 3D video experience.
We didn’t make any announcement at NAB regarding support for 3D video yet, but our demo (hopefully) showed that Smooth Streaming is a great way to deliver stereoscopic 3D video over the Internet to a variety of displays while fully utilizing the power and scalability of HTTP adaptive streaming.
Two videos of me talking about the 3D demo:
Remember NBC Sunday Night Football?
Microsoft has just released a detailed case study of the SNF project describing the partners, products, tools and workflow used to deliver the final solution. You can check it out here:
I believe a similar case study is currently underway for the NBC/CTV/NRK Vancouver Olympics project, but it’ll probably be a few months before it’s released.
If you’re planning or considering attending MIX this year, I will be doing a workshop at MIX on March 14th titled “Media Processing Workflow”:
“Managing the media production pipeline for a site like NBC Olympics or NFL Football can be challenging. From real time video feeds and smooth streaming, to asset management, cutting-edge video players, and multiple media formats, tools, and codecs; you need to plan carefully to ensure that your process runs like a well-oiled machine. Come learn the best practices for setting up your media production workflow, developed in some of the most demanding media sites on the Web.”
It’s not too late to sign up for it, even if you’ve already registered for the conference.
Register for MIX here: http://live.visitmix.com/Registration