Just a quick update. We got a Smooth Streaming video working with Nginx+PHP and the recent release of Silverlight Media Framework as the player.
We'll give you all the details during the week as soon as we can commit the changes to Mp4 Explorer in codeplex.
Meanwhile, check the demo at http://www.aldentorres.com/video/planet-51-psa-1-diversity/ and let us know your comments. And yes, we are working hard towards open source all our codes.
Sunday, November 29, 2009
Friday, July 10, 2009
Fragmented MP4 and Smooth Streaming
This time we want to talk about the physical format of video files used in the Smooth Streaming technology and at the end we will present you a nice tool to create H264 smooth streaming files. We recommend the Alex Zambelli post Smooth Streaming Architecture for a general introduction.
The idea of a fragmented MP4 is a key point inside the whole Smooth Streaming concept because this technology is all about sending small chunks of video fragments to clients. Different to other technologies, the client is entirely responsible for apply the heuristic and adapt to the current network and hardware conditions. This heuristic determines which chunk to download at a given moment and present it to the client in the next few seconds. As you can see these chunks needs to be stored in some place inside the server(s). They can be stored using tiny files per chunk or even using a database, but there is no necessity for such complication, the objective is diminish as much as possible any source of latency between the request and the actual delivery of the chunk.
Microsoft decided to solve the problem using an already available format that fit perfectly to this situation, fragmented MP4 or ISO/IEC 14496-12 Annex parts A.8 and C.7. We don't see a widespread support for this feature in the players, but this will change in the near future.
Normally a regular MP4 file has all the time-sample-metadata related information inside the Movie Box (moov). Without this piece of information is impossible to interpret the video file. The Movie Box is commonly located at the beginning or at the end. The Media Data Container Box (mdat) contains the actual data of video and audio samples.
The problem with this scheme is that the moov box can't be calculated and written until the end of the video creation. This is why when we use some programs to create an MP4, the result can't be used as is for http progressive download. The problem is that the moov box has been written at the end of the file and the player, in the case of a progressive download, needs to wait the entire download to read the moov and start playing the file. There are some tools to fix this; they simply swap the mdat and moov boxes and re-compute all the time-sample related information in order to write the moov at the beginning of the file.
There is nothing wrong here, as long as we have a standard computer with a lot of space, memory and CPU power. What happens in the case of a small recording device? If there is some crash during the recording before it writes the Movie Box, the data is unusable. Another scenario is that for long movies, the Movie Box can be greater than the available memory or compute it at the end can be very expensive.
The solution is use the fragmented MP4 structure. This type of MP4 is created writing the moov box always at the beginning. This moov box contains only the minimal information like the codec used and some metadata, also there is an indication that the file is fragmented (the Movie Extends Box or mvex). The rest of the file is composed by fragments that contain the necessary time-sample related information. In this way we can create the file, piece by piece and in the case of crash we can use the entire file up to the last fragment. After the video is closed, it is possible to write a final box (Movie Fragment Random Access Box or mfra) with useful information for random access. This box is not really necessary but players can benefit from it if random access is a must, as is the case for Smooth Streaming.
Let’s talk now about the fragments. One fragment is composed by one Movie Fragment Box (moof) and the associated data box (mdat). Each fragment can contain data for one or more tracks.
There is a recommendation for how to interleave the audio and video tracks in a fragmented MP4 file (see ISO/IEC 14496-12 Annex part C.7) but a close inspection to the implementation of Smooth Streaming reveals that we can have information for only one track in each fragment. This is not problematic, since the use of the fragmented MP4 file structure in this technology is only a matter of convenience. If you use Mp4 Explorer and open the video from IIS Smooth Streaming Beta Sample Content, you can see that all the fragments for each track are placed together.
Now we can have a better idea of how Smooth Streaming uses the fragmented MP4 structure. There is a direct correspondence between each chunk requested and each fragment inside the MP4. Each chunk request can be represented as:
Note that inside the server there is a fragmented MP4 file for each bitrate exposed. The fragments must be perfect aligned in time between the different files. We don’t know how exactly the path is for each request in IIS is, but this is what we use in our solution.
smooth tool
The next step is choosing the right tool to create the files that we want to use in a smooth streaming server. Microsoft Expression Encoder 2 SP1 is a complete program to create this files. The problem is that there is only support for the VC-1 codec and lot of people wants to use the H264 codec for video and AAC for audio.
For the latest case you can use our tool smooth.exe to create such files, this tool is included inside our multiplatform solution for smooth streaming CM Smooth.NET. The actual transcoding is performed by ffmpeg and we use our core library CM Mp4.NET for the creation of the fragmented files. This tool runs on Windows and Linux (using Mono) and the result is a set of files that can be used directly on any smooth streaming enabled server. You can use our multiplatform solution if you want to deploy Smooth Streaming in a Linux server. Remember, the videos are encoded using H264, in order to play these videos you need the latest Silverlight 3.
The idea of a fragmented MP4 is a key point inside the whole Smooth Streaming concept because this technology is all about sending small chunks of video fragments to clients. Different to other technologies, the client is entirely responsible for apply the heuristic and adapt to the current network and hardware conditions. This heuristic determines which chunk to download at a given moment and present it to the client in the next few seconds. As you can see these chunks needs to be stored in some place inside the server(s). They can be stored using tiny files per chunk or even using a database, but there is no necessity for such complication, the objective is diminish as much as possible any source of latency between the request and the actual delivery of the chunk.
Microsoft decided to solve the problem using an already available format that fit perfectly to this situation, fragmented MP4 or ISO/IEC 14496-12 Annex parts A.8 and C.7. We don't see a widespread support for this feature in the players, but this will change in the near future.
Normally a regular MP4 file has all the time-sample-metadata related information inside the Movie Box (moov). Without this piece of information is impossible to interpret the video file. The Movie Box is commonly located at the beginning or at the end. The Media Data Container Box (mdat) contains the actual data of video and audio samples.
The problem with this scheme is that the moov box can't be calculated and written until the end of the video creation. This is why when we use some programs to create an MP4, the result can't be used as is for http progressive download. The problem is that the moov box has been written at the end of the file and the player, in the case of a progressive download, needs to wait the entire download to read the moov and start playing the file. There are some tools to fix this; they simply swap the mdat and moov boxes and re-compute all the time-sample related information in order to write the moov at the beginning of the file.
There is nothing wrong here, as long as we have a standard computer with a lot of space, memory and CPU power. What happens in the case of a small recording device? If there is some crash during the recording before it writes the Movie Box, the data is unusable. Another scenario is that for long movies, the Movie Box can be greater than the available memory or compute it at the end can be very expensive.
The solution is use the fragmented MP4 structure. This type of MP4 is created writing the moov box always at the beginning. This moov box contains only the minimal information like the codec used and some metadata, also there is an indication that the file is fragmented (the Movie Extends Box or mvex). The rest of the file is composed by fragments that contain the necessary time-sample related information. In this way we can create the file, piece by piece and in the case of crash we can use the entire file up to the last fragment. After the video is closed, it is possible to write a final box (Movie Fragment Random Access Box or mfra) with useful information for random access. This box is not really necessary but players can benefit from it if random access is a must, as is the case for Smooth Streaming.
Let’s talk now about the fragments. One fragment is composed by one Movie Fragment Box (moof) and the associated data box (mdat). Each fragment can contain data for one or more tracks.
There is a recommendation for how to interleave the audio and video tracks in a fragmented MP4 file (see ISO/IEC 14496-12 Annex part C.7) but a close inspection to the implementation of Smooth Streaming reveals that we can have information for only one track in each fragment. This is not problematic, since the use of the fragmented MP4 file structure in this technology is only a matter of convenience. If you use Mp4 Explorer and open the video from IIS Smooth Streaming Beta Sample Content, you can see that all the fragments for each track are placed together.
Now we can have a better idea of how Smooth Streaming uses the fragmented MP4 structure. There is a direct correspondence between each chunk requested and each fragment inside the MP4. Each chunk request can be represented as:
Note that inside the server there is a fragmented MP4 file for each bitrate exposed. The fragments must be perfect aligned in time between the different files. We don’t know how exactly the path is for each request in IIS is, but this is what we use in our solution.
smooth tool
The next step is choosing the right tool to create the files that we want to use in a smooth streaming server. Microsoft Expression Encoder 2 SP1 is a complete program to create this files. The problem is that there is only support for the VC-1 codec and lot of people wants to use the H264 codec for video and AAC for audio.
For the latest case you can use our tool smooth.exe to create such files, this tool is included inside our multiplatform solution for smooth streaming CM Smooth.NET. The actual transcoding is performed by ffmpeg and we use our core library CM Mp4.NET for the creation of the fragmented files. This tool runs on Windows and Linux (using Mono) and the result is a set of files that can be used directly on any smooth streaming enabled server. You can use our multiplatform solution if you want to deploy Smooth Streaming in a Linux server. Remember, the videos are encoded using H264, in order to play these videos you need the latest Silverlight 3.
Thursday, June 18, 2009
Details about our Multiplatform Adaptive Smooth Streaming Solution
As promised, we want to give you more details about our solution for a multiplatform adaptive smooth streaming. Multiplatform is very important for us. In every technology project we are involve, there is a strong belief that Linux based servers are more reliable for critical time operations. Well, maybe this is not entirely true for every situation, but this is the main reason we don't want to be tied to the Windows platform.
We decided to implement our products in .NET using C# as much as possible and we knew the challenges about deploying .NET bases solutions in the Linux OS. Ok, Mono to the rescue. Mono has been around for quite a long time and one of us inside the team, has been following the project since the beginning. After an in-depth analysis we know we can move forward and use Mono with confidence. Actually, we see Linux+Apache+Mono as our most common platform to deploy.
Our multiplatform adaptive smooth streaming solution is implemented as a regular ASP.NET http handler. You can use Windows+IIS or Linux+Apache+Mono. In this way, we can focus in the main problem rather than spending time in minor tasks to support multiples http servers and particular native implementations. For the mp4 file manipulation, we use our CM MP4.NET core library (Mp4 Explorer is a live example of how to use this library).
Here is a high-level overview of the solution:
Let's talk about the performance. Performance is a key point here, because media streaming is one of the most resource intense task in the internet. Fortunately, smooth streaming has a lot of advantages over other technologies in this area. Smooth streaming consists in a series of HTTP downloads of small chunks (View Alex Zambelli post Smooth Streaming Architecture). These chunks can be cached along the edge of the network, including the original server. The result is that the major stress is moved from our ASP.NET http handler to the cache solution right in front of our handler. Our tests point us that you can reach a big audience with our solution.
Visit http://www.cmstream.net/cm-smoothnet-demos.aspx for a general demo with videos coded with H264 and ACC (you need Silverlight 3) with a server running CentOS in a small VPS. We rely only in the standard fragmented MP4 structure (ISO/IEC 14496-12 Anex parts A.8 and C.7) and we support both files encoded by our tools or by Expression Encoder SP1.
We decided to implement our products in .NET using C# as much as possible and we knew the challenges about deploying .NET bases solutions in the Linux OS. Ok, Mono to the rescue. Mono has been around for quite a long time and one of us inside the team, has been following the project since the beginning. After an in-depth analysis we know we can move forward and use Mono with confidence. Actually, we see Linux+Apache+Mono as our most common platform to deploy.
Our multiplatform adaptive smooth streaming solution is implemented as a regular ASP.NET http handler. You can use Windows+IIS or Linux+Apache+Mono. In this way, we can focus in the main problem rather than spending time in minor tasks to support multiples http servers and particular native implementations. For the mp4 file manipulation, we use our CM MP4.NET core library (Mp4 Explorer is a live example of how to use this library).
Here is a high-level overview of the solution:
Let's talk about the performance. Performance is a key point here, because media streaming is one of the most resource intense task in the internet. Fortunately, smooth streaming has a lot of advantages over other technologies in this area. Smooth streaming consists in a series of HTTP downloads of small chunks (View Alex Zambelli post Smooth Streaming Architecture). These chunks can be cached along the edge of the network, including the original server. The result is that the major stress is moved from our ASP.NET http handler to the cache solution right in front of our handler. Our tests point us that you can reach a big audience with our solution.
Visit http://www.cmstream.net/cm-smoothnet-demos.aspx for a general demo with videos coded with H264 and ACC (you need Silverlight 3) with a server running CentOS in a small VPS. We rely only in the standard fragmented MP4 structure (ISO/IEC 14496-12 Anex parts A.8 and C.7) and we support both files encoded by our tools or by Expression Encoder SP1.
Monday, June 8, 2009
Multiplatform Adaptive Video Streaming
We are proud to announce the release of our solution for multiplatform adaptive video streaming. We are very excited with this project because we see a lot of potential now and in the near future. This is an alternative solution to the Microsoft's Smooth Streaming in those places where is necessary to use open platforms like Linux.
Advantages of Adaptive HTTP streaming:
- Cheaper deployment due to the use of generic HTTP caches/proxies.
- Better scalability and reach. It can dynamically adapt to inferior network conditions as it gets closer to the final audience's computers.
- Audience adapts to the content, rather than content providers reduce bitrates what is most likely to be accessible to their audience.
It also offers the following benefits for the end user:
- Fast start-up and seek times.
- No buffering, no disconnects, no playback stutter (user needs to meet the minimum bitrate requirement).
- Seamless bitrate switching based on network conditions and CPU capabilities.
- A generally consistent, smooth playback experience.
Areas that can benefit from this solution:
- Small and mid size content providers (nation-wide provider's requirements are different).
- Video websites.
- Open platforms like Linux for video services.
- Home media center systems.
- Monitor and surveillance products.
Since we are using the same format from Smooth Streaming, integrators can use their favorite player with support for Adaptive Streaming. One option is the Open Video Player Initiative, an open-source community project. Microsoft Expression Encoder 2 SP1 includes a few player templates ready for Smooth Streaming.
We support the following combinations of codecs: H264 with AAC and WVC1 with WmaPro. For H264 we provide a tool to create videos with the appropriate structure. Microsoft Expression Encoder 2 SP1 can be used for WVC1. (H264 video compression requires Silverlight 3 Beta).
You can see a demo of this solution in http://www.cmstream.net/cm-smoothnet-demos.aspx. This is a small VPS running CentOS.
We are planning to release more technical details but you can write us your questions and comments.
Wednesday, June 3, 2009
Mp4 Explorer
This is our first blog post and we think we should dedicate this space to our first public project: Mp4 Explorer, current release Mp4 Explorer 20090802.
This tool has a kind of interesting story. We started the development of multimedia protocol libraries (SDP, RTSP, RTP) and at that time, only native tools for MP4 (ISO/IEC 14496-12) file inspection were available. Since our development is based on the .NET Framework, mainly using the C# language, we were forced to do something about it.
We are fans of the renewed (and fun) architectural ideas for developing applications and while keeping a pragmatic approach we said: "Why not to use the CompositeWPF (Prism V2) framework in this tool in order to develop an extensible application? In this way, our hope is not to have the necessity of a full reengineer later this year".
Soon, we realized that they are many variations and special atoms that were not covered by the standard. This was the principal reason to release the project as an open source project inside CodePlex. We see CodePlex as a natural place for any open source .NET Framework project.
Now, let’s talk what Mp4 Explorer it’s all about.
This tool can help you to perform a full inspection of the boxes (atoms) inside the ISO MP4 file. No video preview, no file editing, no tracks manipulation, at least for now. It is intended to be used by technical people who are working with the ISO MP4 file structure.
The usefulness of this tool was proven in our multiplatform Adaptive Streaming solution (another post for this later). Everybody can read the details about the Microsoft Smooth Streaming and their use of the fragmented MP4 file format in the Alex Zambelli post (Smooth Streaming Architecture). Despite this excellent technical explanation, the true is that it is more or less difficult to find a tool to work with fragmented MP4 files. With Mp4 Explorer and the video samples available from Microsoft, we were able to understand the details and later, we succeed in creating a proper fragmented MP4 file using H.264 for video and AAC for audio (we call this files internally as "smoothed files"). We know that these files don’t comply with the ISO standard but in the process, we learned that the differences can be addressed without any problems.
We want to say you that we are actively looking for ways to bring more support to this project inside the open source world. Your are welcome to join us in this effort.
This tool has a kind of interesting story. We started the development of multimedia protocol libraries (SDP, RTSP, RTP) and at that time, only native tools for MP4 (ISO/IEC 14496-12) file inspection were available. Since our development is based on the .NET Framework, mainly using the C# language, we were forced to do something about it.
We are fans of the renewed (and fun) architectural ideas for developing applications and while keeping a pragmatic approach we said: "Why not to use the CompositeWPF (Prism V2) framework in this tool in order to develop an extensible application? In this way, our hope is not to have the necessity of a full reengineer later this year".
Soon, we realized that they are many variations and special atoms that were not covered by the standard. This was the principal reason to release the project as an open source project inside CodePlex. We see CodePlex as a natural place for any open source .NET Framework project.
Now, let’s talk what Mp4 Explorer it’s all about.
This tool can help you to perform a full inspection of the boxes (atoms) inside the ISO MP4 file. No video preview, no file editing, no tracks manipulation, at least for now. It is intended to be used by technical people who are working with the ISO MP4 file structure.
The usefulness of this tool was proven in our multiplatform Adaptive Streaming solution (another post for this later). Everybody can read the details about the Microsoft Smooth Streaming and their use of the fragmented MP4 file format in the Alex Zambelli post (Smooth Streaming Architecture). Despite this excellent technical explanation, the true is that it is more or less difficult to find a tool to work with fragmented MP4 files. With Mp4 Explorer and the video samples available from Microsoft, we were able to understand the details and later, we succeed in creating a proper fragmented MP4 file using H.264 for video and AAC for audio (we call this files internally as "smoothed files"). We know that these files don’t comply with the ISO standard but in the process, we learned that the differences can be addressed without any problems.
We want to say you that we are actively looking for ways to bring more support to this project inside the open source world. Your are welcome to join us in this effort.
Subscribe to:
Posts (Atom)
