Render subtitles to directx surface

[brabebhin]

Hi @lukasf

Have you ever managed to successfully render subtitles from the Mediaplayer object in frame server mode?

I have been trying to implement a MediaPlayerElement alternative in winUI and i seem unable to render subtitles. I keep getting an argument exception complaining about invalid parameters, but there is no useful info in the exception.

https://docs.microsoft.com/en-us/uwp/api/windows.media.playback.mediaplayer.rendersubtitlestosurface?view=winrt-22000

Thanks.

[lukasf]

Hi @brabebhin, no, I never really tried to get FrameServer mode working. I heard too often that people had problems with that. I was hoping that someone would come up with a working solution, but that does not seem to be the case. I am also following that issue on WinUI github.

[brabebhin]

Truth to be told, I am getting a bit impatient with how Microsoft is handling media APIs. It's like the media team has been disbanded or something.
I have been using frame server mode for years now, it works decently well for rendering video + video effects, I haven't encountered any big issues yet. It is just the subtitles that seem completely broken.

I am looking into alternatives, maybe some lower level media foundation APIs...

[brabebhin]

Right, so here's my crazy idea.
What if we can integrate our media stream source with the IMFPlayer?

I am going to assume that most of our winRT objects are somehow convertible to the MF interfaces. I will start out based on the samples found around the interwebz.

I'm kinda tired of waiting for Microsoft to fix their bugs ^^

[brabebhin]

It seems the most complicated thing is working around Microsoft's extremely annoying C++/CX/winRT UWP/Desktop dance that basically cuts in half the available APIs just for the lulz.

[lukasf]

I am pretty sure that subtitles are completely handled inside UWP API code, not in MF. Same is probably true for a lot of MediaPlayer functionality. As base it uses MF, but MF is pretty low level, so UWP builts a lot on top of that. That is why I said, we'd need to do custom subtitle rendering using libass. MF does not have anything to help with subtitles.

The MF APIs are indeed very limited in UWP. A full player might only be possible to implement using full Desktop MF APIs.

[brabebhin]

I actually found how UWP MediaPlayer probably works, with the help of IMFMediaEngineEx.

I also learned that the frame server mode is the natural way of functioning for this interface, so it feels weird that the other poster had issues with it (i tried some of his scenarios in my own implementation and it seems to work correctly).

But yeah, I haven't found anything about subtitles yet. I also had a look at libass, it seems the way to do it is via a filter. That might be problematic for performance.

In the bigger picture, I am not sure how much of a life UWP still has. It seems winUI 3 is still somehow bound to winRT and UWP but not the UWP XAML framework. Perhaps implementing a full desktop player might be helpful, but integrating it nicely with MediaStreamSource seems to be quite the pain.

[lukasf]

I don't think that there will be a problem when doing subtitles as filter. The frequency is so low that it should not matter performance wise. We just need to transfer the decoded bitmap to a texture, then rendering will be fast until the next sub is decoded. All big desktop players do it like that, and there is no issue with performance. And I think more or less all of them use libass, either directly or indirectly through ffmpeg filters.

I am also unsure about the future of Windows development. App model and UWP XAML is officially dead. WinUI is not nearly there: Performance is poor, Control and API set is incomplete, lots of bugs and speed of bug resolving is really slow. They expect community to do most of their work, which is not really happening (big surprise). I painfully know how long it took to get the UWP XAML framework mature, so it's no surprise that WinUI is still pretty beta. It will probably take a year or two to get WinUI into a good shape, if they work on it seriously. But I don't even know how serious they are about it.

[brabebhin]

Well if we fix subtitle rendering, then the frame server mode should allow us to create a MediaPlayerElement for winUI and by extension any desktop app (with or without winUI since our api will be universal).

I am basically working on such a MediaPlayerElement, the only problem is subtitles rendering...

I will now focus on libass. However google isn't too friendly on documents on how to build libass for winRT.

[brabebhin]

Hi @lukasf

I am going to need a bit of guidance on this libass business.
I noticed that some of the libs that we link against are not built before building ffmpeg, and I assume this is also the case for libass, judging on this guide (which is not for Windows but it is the most complete of all)

https://trac.ffmpeg.org/wiki/CompilationGuide/Ubuntu

However, when I try to do this, I get

ERROR: libass >= 0.11.0 not found using pkg-config

I placed the libass source code in its "libass" folder under Libs.

[lukasf]

All libs must be built before building ffmpeg. So first you must figure out how to create a static Windows UWP build using MSVC compiler for the lib. This is the most difficult part. Once you have a .lib file, you can continue integration. I will try to write a short guide on the steps required.

[lukasf]

I have created a guide on how to add new libs: #261

I have also looked into libass a bit. The latest version of libass talks about font providers for UWP. So it seems that we are not the first ones who want to compile for UWP. The lib uses linux "autotools" build chain, which I don't know, and on short search I have not found out if and how it can be switched to use MSVC. There is also zero guidance about the lib build system, which is surprising. Most libs have at least some basic documentation on the build system in use, the prerequisites and some example build calls with parameters. You probably need to install autotools im MSYS2 and continue from there.

I think I would just try to contact the devs and ask how to create UWP builds. Looks like someone has done it, so there has to be a way. Either create an issue on Github, or contact them on their IRC channel shown on github.

Second option: There is a fork that has VS build files. It is outdated, but if you cannot get the official one to compile, it could be sufficient (0.13.0 > 0.11.0). Not sure though if it will work for UWP. Only the latest official lib has these UWP changes.

[brabebhin]

I did ask the devs. They provided some general hints but nothing concrete. I was also quite surprised that there's literally no documentation on how to build this. The library itself also seems to have dependency on others.

I am starting to think that maybe the idea to do this with win2D isn't that crazy after all. In comparison to libass.

[brabebhin]

Alright, so I did give rendering subtitles with win2D a try, and it is indeed do-able. I managed to get simple SRT rendering done quite fast. So we might just go down this route instead. Seems more fun than bumbling in the dark with libass builds.

[brabebhin]

HI @lukasf
It's been a while.
I have made a POC on sub rendering with win2D, you can find it on branch subtitle-renderer

I am fairly confident I got the styling of cues mostly right.
What I am not so sure about is the positioning of the text for ASS/SSA. I currently hard coded everything to be aligned center bottom, but this is not always the case.

What do you think?

[lukasf]

I was travelling through greece for a pretty long time with my family. Next week I will finally return, then I will have time to take a look at this. I will report back in a week or two.

Getting sizing and positioning right is probably difficult, especially making it work with different combinations of UI and video resolution. And also sometimes subtitles are made for a video file of a specific resolution and then also added to a video file of a different resolution (and maybe even different aspect ratio). Handling such corner cases in a proper way surely is a non-trivial task.

[brabebhin]

Enjoy your vacation 🥳

[sudongg]

Is there any progress?
It would be a major update to actually render libass subtitles to dx surfaces.

[brabebhin]

No. Since MediaPlayerElement is now available in Windows App SDK, the chances of implementing this is getting lower.