Transcoding support!

[brabebhin]

Every now and again it comes a question on how to use transcoding with this library.
And every single time we tell people to use the winRT APIs.
While it is true that covers most of the usual transcoding needs, it is still not as powerful as FFMPEG.

I have looked over the sample here
https://ffmpeg.org/doxygen/trunk/transcoding_8c_source.html

It shouldn't be too difficult to wrap this into a reusable class and adapt it to use winRT/UWP streams instead of file paths.

This is the way I see it:

1. It is separated from the decoding/real time playback loop, so the existing code stays untouched, even if this might mean some limited duplicated stuff (like stream demuxing and filter graph). This will allow us to independently evolve the 2 features without disturbing the other. After all, our capacity to test features is not unlimited.
2. It takes an IRandomAccessStream as input, a filter graph definition for the actual transcoding (just like in the sample) and an IRandomAccessStream as output.
3. Progress report.
4. The interface will be synchronous, users will have to handle threading inside the app (like dispatching progress report events to the proper thread). FFmpeg will do threading as it sees fit.
5. Exceptions thrown in case of errors -> users will need to clean up.
6. The output stream will not be closed automatically by the transcoder, it is up to the user to do that.
7. The user will need to provide the filter graph, so we do not need to creating complex data structures to support applications.

What do you think @lukasf ?

[lukasf]

Interesting proposal. Some thoughts from my side:

• Indeed it looks like the coding part could be done within reasonable amount of time, after looking at the sample code
• If added, the transcoding codebase should be completely separate from our FFmpegMediaSource.
• Also allowing file names as source/target for Win32/WinUI desktop apps should be considered
• Filter graph seems to be optional. The sample code you linked just adds null filter for video and audio, which will just pass on the decoded samples to the encoder without change. But it would be a good idea to allow users to optionally provide a filter graph description per stream. Of course, just as a standard ffmpeg filter graph string, no custom structures.
• Source and target formats should also use standard ffmpeg codec strings, no custom enum we'd have to maintain
• The API would have to be flexible to cover different scenarios. Like for a file with multiple streams, it should be possible to configure the target container format plus the stream format and optionally filter graph for each stream. So the whole process would be a two-step operation: First you open the source file. On the result object, you can iterate over the streams and configure the target formats. Then you start the transcoding, providing the target stream/file.
• It should also be possible to passthrough streams instead of re-encode
• It should be possible to drop streams
• It might be possible to specify a delay per stream, to re-sync streams
• I think the opening of source and actual transcoding calls should be async.
• Progress events can come from any thread, no guarantees there.
• What about decoding errors? Would it be desirable to have a "skip errors" feature? If added, would we need to repeat frames to "fill up" decode error gaps? That would probably need investigation. But it's surely not mandatory for an initial version.
• We'd need a separate FFmpeg build, which includes all the encoders. I did already work on that when experimenting with ARM64 desktop builds. But compiling ffmpeg with the x264/x265 encoders required some ffmpeg patches, which I have lost, unfortunately. Plus, the encoders are very outdated as of now. So this is an area that would also require some work, besides the actual coding. Having x264, x265 and some av1 encoder is absolutely mandatory.
• While software encoders such as x264/x265 are without doubt the best quality encoders, we'd need to be prepared that people will ask for hardware encoding sooner or later. More work waiting around the corner ^^

I should also add that I will be on a long summer vacation. I probably wouldn't have time to work on any of this until end of august.

[brabebhin]

Hi @lukasf

All very good points.
I ended up designing the interface using IAsyncActionWithProgress. This allows async and progress in one go, so it should be fine.
The code in the sample is unfortunately a little obsolete, especially around chanel layouts. Seems to be on a pre 4.0 FFMPEG. So I am going to have to do some digging into that >.>

I completely forgot about the ffmpeg build with the decoders. This is an excellent point, thank you for bringing it up.

I envision the following milestones for this:

1. Allow passthrough (essentially remuxing) as in the sample
2. Allow per stream manipulation
3. H264/265/Av1
4. Hardware acceleration

Enjoy your summer vacation. I doubt I will be ready by the time you're back anyway. I will probably take one myself towards the end of July. Somewhere cold. I was thinking Antarctica or Norway.

[lukasf]

Interesting, i never used IAsyncActionWithProgress, almost forgot that it is there. Downside might be that it is not so flexible, like if we wanted to provide additional data (number of bytes written, etc) then we cannot easily add that.

Only recently I found out that the channel layouts APIs were deprecated in ffmpeg 5.1. They added completely new APIs. Somehow this went unnoticed on my side when I switched to 5.1. I already started woking on this a bit on a ffmpeg 6 branch. You should directly go for the new APIs ofc if you work on transcoding apis.

Wow polar regions, sounds interesting! We are probably heading to Sweden, but not so far north, so it should still feel like summer...

[lukasf]

I have some good and some bad news for you. I just managed to build ffmpeg with x264 and x265. The issues I had back then (requiring patches) seem to be resolved in latest ffmpeg. Not sure yet how to make this configurable, so it could take a while to make a PR.

But during this, I noticed that both x264 and x265 are GPL. So the two most important encoders would require devs to open source the app. I guess this will be a deal breaker for many or most devs interested in the transcoding feature.

SVT-AV1 encoder is BSD. But it is a multitude slower than x264 and x265 on the encoder side (decoding is fast). AV1 is mainly for content producers and distributors, but rarely used by end users, due to the performance characteristics.

libvpx is BSD as well, which allows VP1 encoding. Not sure how performance is on that one.

There is also Cisco openh264 encoder (LGPL).

[brabebhin]

So I've been thinking, what if we continue to relay on winRT for muxing, but we add some additional features in our decider pipeline that could prove useful for transcoding

• per stream filtering
• ability to drop a stream from the source
  -ability to add external files as additional streams (e. G. For audio or subs)
• specifying audio delay

These could potentially be useful for rendering as well

[lukasf]

I am also not sure how to continue. Having the transcode API would of course still allow for remuxing and file stream modifications. Transcoding to AV1 is also possible. People could include x264/x265, if they do not mind sharing their code. Theoretically, you can also buy commercial licenses for x264 and x265, then you don't have to share the code (I read somewhere that costs are about 0.10-0.20$ per device per year. But it's probably not very attractive for apps, especially if you want to provide trial versions).

I also like the idea to allow more control over streams and decoders, similar to what you wrote. You open a file and inspect its streams. You can configure the decoder and settings for each stream (FFmpeg has lots of options on codec level, which we currently cannot really use), and possibly add more streams from other files. Finally, you create the source from your selection.

[brabebhin]

For my own selfish use case, since I'll be open sourcing my app, I couldn't care less about having to open source my code. But it does become difficult for other users. Technically, people already pay for those licenses when they buy windows or GPUs. It would probably be easier to relay on WinRT.

Then it hit me, we don't really demux those subtitles to media samples that the winRT transcoder can mux either.

[lukasf]

Yes the license and patent thing is a benefit when using the WinRT APIs. Quality wise, the codecs are surely inferior to x254/x265 (and some people might not want to use anything else). But most people probably don't care that much. They just want the file to have the right format and right size.

We could pass subtiles down the chain passthrough style. The APIs are all there. Text subs in non-standard formats are decoded into SSA by FFmpeg, so I think we can passthrough more or less any kind of subtitle. I guess it's much easier than adding completely new transcode APIs.

External subtitles might be a bit more difficult to handle, since they are not part of our normal decoding loop. And external audio is even more difficult to get right. It would also have to work in normal playback, and I think I had quite some trouble with getting seek behavior right, when I did some experimental implementation of that a few years ago.

[brabebhin]

I think the winRT transcoder should support HEVC and x264. I see there are methods to create encoding profiles for these. I think the downside is that you need the extension installed.