Upgrade FFmpeg libraries to 5.0 #5051

nekodex · 2022-02-24T23:30:31Z

Making this to track issues that prevent upgrading of FFmpeg to 5.0.

As part of the osx-arm64/FFmpeg library replacement work in #5043, I also attempted to update FFmpeg to 5.0 but came across issues that caused test failures. I've pushed the work with the updated libraries for 5.0 to a branch here.

It all works with the exception of the following tests in TestSceneVideo:

~~TestJumpBackAfterEndOfPlayback (on osx-arm64, osx-x86_64, win-x86_64)~~
Fixed thanks to @Opelkuh's patch below
Tests fail for webm (TestJumpForward, TestJumpBack, TestJumpBackAfterEndOfPlayback) (on osx-x86_64)
Seems related to videotoolbox VP9 hardware acceleration (as per this ffmpeg issue, limited to intel igpus?), tests pass with hardware acceleration disabled.

The text was updated successfully, but these errors were encountered:

peppy · 2022-02-25T03:00:05Z

Going to cc @Opelkuh as they seem to have the most knowledge around ffmpeg of those that have worked on it here.

Opelkuh · 2022-02-26T00:15:54Z

I got TestJumpBackAfterEndOfPlayback working with 5.0 with the following patch (+ the changes in your branch):

Index: osu.Framework/Graphics/Video/VideoDecoder.cs
<+>UTF-8
===================================================================
diff --git a/osu.Framework/Graphics/Video/VideoDecoder.cs b/osu.Framework/Graphics/Video/VideoDecoder.cs
--- a/osu.Framework/Graphics/Video/VideoDecoder.cs	(revision abe410f850c600c98b0787bba4c3a8de16a32c67)
+++ b/osu.Framework/Graphics/Video/VideoDecoder.cs	(date 1645826975338)
@@ -264,8 +264,9 @@
                 return 0;
 
             var span = new Span<byte>(bufferPtr, bufferSize);
+            int read = decoder.videoStream.Read(span);
 
-            return decoder.videoStream.Read(span);
+            return read == 0 ? AGffmpeg.AVERROR_EOF : read;
         }
 
         [MonoPInvokeCallback(typeof(avio_alloc_context_seek))]

which resolves this part of the breaking changes:

custom AVIOContext instances may no longer return 0 on EOF, they should return AVERROR_EOF or another appropriate error code

As for TestJumpForward and TestJumpBack, I unfortunately don't have any HW with macOS so I can't debug it, but it's possibly related to HW decoding as FFmpeg 5.0 added VP9 support for Video Toolbox (the Apple HW decoder/encoder API).

peppy · 2022-02-28T06:46:33Z

@nekodex were the macOS failures only when running headless, or also via visual tests?

nekodex · 2022-03-02T08:51:19Z

visual tests

nekodex · 2022-03-09T08:42:50Z

After a bit of further digging, I came across this ffmpeg issue, which seems to confirm the issue lies with videotoolbox VP9 hardware acceleration (on intel igpus?)... and yeah, when testing the visual tests with hardware acceleration disabled, the remaining affected tests (webm) pass without issue.

peppy · 2022-03-09T08:59:39Z

Does the fallback flow we have implemented mean this is a non-issue during actual user runtime?

nekodex · 2022-03-09T11:29:25Z

Not sure which flow you're referencing... this fallback logic from #4839?

osu-framework/osu.Framework/Graphics/Video/VideoDecoder.cs

Lines 353 to 356 in 1cd2ff3

    
           // default to the first codec that we found with no HW devices. 
        
           // The first codec is what FFmpeg's `avcodec_find_decoder` would return so this way we'll automatically fallback to that. 
        
           if (firstCodec != null) 
        
               codecs.Add((firstCodec, AVHWDeviceType.AV_HWDEVICE_TYPE_NONE));

If so, it doesn't seem to handle this case as the HW device initializes correctly... but then fails during decode (for VP9/webm at least)

Opelkuh · 2022-03-10T01:02:40Z

As @nekodex said, right now there's only a fallback in cases when a HW-accelerated codec can't be created or opened. This should prevent it from trying to use codecs that aren't supported by the device but any errors after the decoder starts don't disable HW decoding. Related: #4857

Could you please post some logs from the failing visual tests? Just so we know what to potentially look out for in #4857. Unless it's not even logging anything... I tried to repro it in GitHub CI but it passed all tests.

nekodex · 2022-03-10T06:21:30Z

mp4 (playback works, despite hwaccel initialisation returned error messages):

[runtime] 2022-03-10 05:56:34 [debug]: MouseDownEvent(Left) handled by BasicScrollContainer.
[runtime] 2022-03-10 05:56:34 [verbose]: 💨 Video(TestSceneVideo) JumpForward(mp4)
[runtime] 2022-03-10 05:56:34 [debug]: ClickEvent(Left) handled by "JumpForward(mp4)" LabelStep.
[runtime] 2022-03-10 05:56:34 [debug]: MouseClick handled by "JumpForward(mp4)" LabelStep.
[runtime] 2022-03-10 05:56:34 [verbose]: 🔸 Step #68 Reset clock
[runtime] 2022-03-10 05:56:34 [verbose]: 🔸 Step #69 load .mp4 video
[runtime] 2022-03-10 05:56:34 [verbose]: Successfully opened hardware video decoder context AV_HWDEVICE_TYPE_VIDEOTOOLBOX for codec h264
[runtime] 2022-03-10 05:56:34 [verbose]: Successfully initialized decoder: h264
[h264 @ 0x7fea92622b80] Failed setup for format videotoolbox_vld: hwaccel initialisation returned error.
[runtime] 2022-03-10 05:56:34 [verbose]: 🔸 Step #70 Wait for video to load
[runtime] 2022-03-10 05:56:34 [verbose]: 🔸 Step #71 Reset clock
[runtime] 2022-03-10 05:56:35 [verbose]: 🔸 Step #72 Wait for decode start
[runtime] 2022-03-10 05:56:35 [verbose]: 🔸 Step #73 Jump ahead by 10 seconds
[runtime] 2022-03-10 05:56:35 [verbose]: Video too far out of sync (416.6666666666667), seeking to 10197.008754
[runtime] 2022-03-10 05:56:35 [verbose]: Video too far out of sync (450), seeking to 10202.567357
[h264 @ 0x7fea92622b80] Failed setup for format videotoolbox_vld: hwaccel initialisation returned error.
[h264 @ 0x7fea92622b80] Failed setup for format videotoolbox_vld: hwaccel initialisation returned error.
[runtime] 2022-03-10 05:56:35 [verbose]: 🔸 Step #74 Video seeked

webm (shows first frame, but then doesn't play/decode beyond that):

[runtime] 2022-03-10 05:56:59 [debug]: MouseDownEvent(Left) handled by BasicScrollContainer.
[runtime] 2022-03-10 05:56:59 [verbose]: 💨 Video(TestSceneVideo) JumpForward(webm)
[runtime] 2022-03-10 05:56:59 [debug]: ClickEvent(Left) handled by "JumpForward(webm)" LabelStep.
[runtime] 2022-03-10 05:56:59 [debug]: MouseClick handled by "JumpForward(webm)" LabelStep.
[runtime] 2022-03-10 05:56:59 [verbose]: 🔸 Step #84 Reset clock
[runtime] 2022-03-10 05:56:59 [verbose]: 🔸 Step #85 load .webm video
[runtime] 2022-03-10 05:56:59 [verbose]: Successfully opened hardware video decoder context AV_HWDEVICE_TYPE_VIDEOTOOLBOX for codec vp9
[runtime] 2022-03-10 05:56:59 [verbose]: Successfully initialized decoder: vp9
[runtime] 2022-03-10 05:56:59 [verbose]: 🔸 Step #86 Wait for video to load
[runtime] 2022-03-10 05:57:00 [verbose]: 🔸 Step #87 Reset clock
[runtime] 2022-03-10 05:57:00 [verbose]: 🔸 Step #88 Wait for decode start
[runtime] 2022-03-10 05:57:10 [verbose]: 💥 Failed (on attempt 50)
[runtime] 2022-03-10 05:57:10 [verbose]: ⏳ Currently loading components (0)
[runtime] 2022-03-10 05:57:10 [verbose]: 🧵 Task schedulers
[runtime] 2022-03-10 05:57:10 [verbose]: LoadComponentsAsync (standard) concurrency:4 running:0 pending:0
[runtime] 2022-03-10 05:57:10 [verbose]: LoadComponentsAsync (long load) concurrency:4 running:0 pending:0
[runtime] 2022-03-10 05:57:10 [verbose]: 🎱 Thread pool
[runtime] 2022-03-10 05:57:10 [verbose]: worker:          min 12     max 32,767 available 32,766
[runtime] 2022-03-10 05:57:10 [verbose]: completion:      min 12     max 1,000  available 1,000

nekodex · 2022-03-10T08:41:47Z

Also, to confirm I tested playback of the same files using ffmpeg+ffplay and received similar results - which is to say the .mp4 with hwaccel enabled throws the same hwaccel initialisation returned error but decodes fine, whilst the .webm doesn't throw any errors... yet fails to decode.

mp4/h264:

$ ./ffmpeg -hwaccel videotoolbox -i sample-video.mp4 -vcodec rawvideo -acodec copy -f matroska - | ./ffplay -i -
ffmpeg version 5.0 Copyright (c) 2000-2022 the FFmpeg developers
  built with Apple clang version 13.0.0 (clang-1300.0.29.30)
  configuration: --disable-doc --disable-debug --enable-shared --arch=x86_64 --prefix=build-x86_64 --libdir=build-x86_64/lib
  libavutil      57. 17.100 / 57. 17.100
  libavcodec     59. 18.100 / 59. 18.100
  libavformat    59. 16.100 / 59. 16.100
  libavdevice    59.  4.100 / 59.  4.100
  libavfilter     8. 24.100 /  8. 24.100
  libswscale      6.  4.100 /  6.  4.100
  libswresample   4.  3.100 /  4.  3.100
ffplay version 5.0 Copyright (c) 2003-2022 the FFmpeg developers
  built with Apple clang version 13.0.0 (clang-1300.0.29.30)
  configuration: --disable-doc --disable-debug --enable-shared --arch=x86_64 --prefix=build-x86_64 --libdir=build-x86_64/lib
  libavutil      57. 17.100 / 57. 17.100
  libavcodec     59. 18.100 / 59. 18.100
  libavformat    59. 16.100 / 59. 16.100
  libavdevice    59.  4.100 / 59.  4.100
  libavfilter     8. 24.100 /  8. 24.100
  libswscale      6.  4.100 /  6.  4.100
  libswresample   4.  3.100 /  4.  3.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'sample-video.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf58.29.100
  Duration: 00:01:05.05, start: 0.000000, bitrate: 49 kb/s
  Stream #0:0[0x1](und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(progressive), 480x270 [SAR 1:1 DAR 16:9], 43 kb/s, 60 fps, 60 tbr, 15360 tbn (default)
    Metadata:
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> rawvideo (native))
Press [q] to stop, [?] for help
[h264 @ 0x7fcd09404940] Failed setup for format videotoolbox_vld: hwaccel initialisation returned error.
Output #0, matroska, to 'pipe:':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf59.16.100
  Stream #0:0(und): Video: rawvideo (I420 / 0x30323449), yuv420p(progressive), 480x270 [SAR 1:1 DAR 16:9], q=2-31, 93312 kb/s, 60 fps, 1k tbn (default)
    Metadata:
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
      encoder         : Lavc59.18.100 rawvideo
Input #0, matroska,webm, from 'pipe:':1kB time=00:00:00.00 bitrate=N/A speed=   0x    
  Metadata:
    COMPATIBLE_BRANDS: isomiso2avc1mp41
    MAJOR_BRAND     : isom
    MINOR_VERSION   : 512
    ENCODER         : Lavf59.16.100
  Duration: N/A, start: 0.000000, bitrate: N/A
  Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p(progressive), 480x270, SAR 1:1 DAR 16:9, 60 fps, 60 tbr, 1k tbn (default)
    Metadata:
      HANDLER_NAME    : VideoHandler
      VENDOR_ID       : [0][0][0][0]
      ENCODER         : Lavc59.18.100 rawvideo

webm/vp9:

$ ./ffmpeg -hwaccel videotoolbox -i sample-video.webm -vcodec rawvideo -acodec copy -f matroska - | ./ffplay -i - 
ffmpeg version 5.0 Copyright (c) 2000-2022 the FFmpeg developers
  built with Apple clang version 13.0.0 (clang-1300.0.29.30)
  configuration: --disable-doc --disable-debug --enable-shared --arch=x86_64 --prefix=build-x86_64 --libdir=build-x86_64/lib
  libavutil      57. 17.100 / 57. 17.100
  libavcodec     59. 18.100 / 59. 18.100
  libavformat    59. 16.100 / 59. 16.100
  libavdevice    59.  4.100 / 59.  4.100
  libavfilter     8. 24.100 /  8. 24.100
  libswscale      6.  4.100 /  6.  4.100
  libswresample   4.  3.100 /  4.  3.100
Input #0, matroska,webm, from 'sample-video.webm':
  Metadata:
    ENCODER         : Lavf58.29.100
  Duration: 00:01:05.05, start: 0.000000, bitrate: 33 kb/s
  Stream #0:0: Video: vp9 (Profile 0), yuv420p(tv), 480x270, SAR 1:1 DAR 16:9, 60 fps, 60 tbr, 1k tbn (default)
    Metadata:
      DURATION        : 00:01:05.049000000
ffplay version 5.0 Copyright (c) 2003-2022 the FFmpeg developers
  built with Apple clang version 13.0.0 (clang-1300.0.29.30)
  configuration: --disable-doc --disable-debug --enable-shared --arch=x86_64 --prefix=build-x86_64 --libdir=build-x86_64/lib
  libavutil      57. 17.100 / 57. 17.100
  libavcodec     59. 18.100 / 59. 18.100
  libavformat    59. 16.100 / 59. 16.100
  libavdevice    59.  4.100 / 59.  4.100
  libavfilter     8. 24.100 /  8. 24.100
  libswscale      6.  4.100 /  6.  4.100
  libswresample   4.  3.100 /  4.  3.100
Stream mapping:
  Stream #0:0 -> #0:0 (vp9 (native) -> rawvideo (native))
Press [q] to stop, [?] for help

Furthermore, checking playback with ffmpeg+ffplay from 4.3.3 and, well, h264 fails to decode at all with hwaccel whilst vp9 works fine. Go figure. 🤷

$ ./ffmpeg -hwaccel videotoolbox -i sample-video.mp4 -vcodec rawvideo -acodec copy -f matroska - | ./ffplay -i -
ffmpeg version 4.3.3 Copyright (c) 2000-2021 the FFmpeg developers
  built with Apple clang version 13.0.0 (clang-1300.0.29.30)
  configuration: --disable-doc --disable-debug --enable-shared --arch=x86_64 --prefix=build-x86_64 --libdir=build-x86_64/lib
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libswscale      5.  7.100 /  5.  7.100
  libswresample   3.  7.100 /  3.  7.100
ffplay version 4.3.3 Copyright (c) 2003-2021 the FFmpeg developers
  built with Apple clang version 13.0.0 (clang-1300.0.29.30)
  configuration: --disable-doc --disable-debug --enable-shared --arch=x86_64 --prefix=build-x86_64 --libdir=build-x86_64/lib
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libswscale      5.  7.100 /  5.  7.100
  libswresample   3.  7.100 /  3.  7.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'sample-video.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf58.29.100
  Duration: 00:01:05.05, start: 0.000000, bitrate: 49 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 480x270 [SAR 1:1 DAR 16:9], 43 kb/s, 60 fps, 60 tbr, 15360 tbn, 120 tbc (default)
    Metadata:
      handler_name    : VideoHandler
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> rawvideo (native))
Press [q] to stop, [?] for help
Error creating Videotoolbox decoder.
videotoolbox hwaccel requested for input stream #0:0, but cannot be initialized.
[h264 @ 0x7fe5bb016200] decode_slice_header error
[h264 @ 0x7fe5bb016200] no frame!
Error creating Videotoolbox decoder.
videotoolbox hwaccel requested for input stream #0:0, but cannot be initialized.
[h264 @ 0x7fe5bb019000] decode_slice_header error
[h264 @ 0x7fe5bb019000] no frame!
Error creating Videotoolbox decoder.
videotoolbox hwaccel requested for input stream #0:0, but cannot be initialized.
[h264 @ 0x7fe5bb019600] decode_slice_header error
[h264 @ 0x7fe5bb019600] no frame!
Error creating Videotoolbox decoder.
videotoolbox hwaccel requested for input stream #0:0, but cannot be initialized.
[h264 @ 0x7fe5bb019c00] decode_slice_header error
[h264 @ 0x7fe5bb019c00] no frame!
Error creating Videotoolbox decoder.
videotoolbox hwaccel requested for input stream #0:0, but cannot be initialized.
[h264 @ 0x7fe5bb01a200] decode_slice_header error
[h264 @ 0x7fe5bb01a200] no frame!
Error creating Videotoolbox decoder.
videotoolbox hwaccel requested for input stream #0:0, but cannot be initialized.
[h264 @ 0x7fe5bb01a800] decode_slice_header error
[h264 @ 0x7fe5bb01a800] no frame!
Error creating Videotoolbox decoder.
videotoolbox hwaccel requested for input stream #0:0, but cannot be initialized.
[h264 @ 0x7fe5bb01ae00] decode_slice_header error
[h264 @ 0x7fe5bb01ae00] no frame!
Error creating Videotoolbox decoder.
videotoolbox hwaccel requested for input stream #0:0, but cannot be initialized.
[h264 @ 0x7fe5bb01b400] decode_slice_header error
[h264 @ 0x7fe5bb01b400] no frame!

<repeat ad infinitum>

FreezyLemon · 2023-08-19T11:13:05Z

If the information in that ffmpeg issue is to be believed:

After discussion with Apple this is due to the fact that they require VP9 alt-ref frames to be packaged as superframes.

~~then there is a possibility of being able to work around this. FFmpeg includes a bitstream filter (BSF) that can "repack" alt-ref frames as superframes before decoding.~~

I don't have access to an affected machine, but someone else could test this with a newer FFmpeg version (preferrably 6.0). The WebM video in osu.Framework.Tests/Resources/Videos could be used for this.

~~Sanity check (this should hang):~~
~~ffmpeg -hide_banner -hwaccel videotoolbox -i sample-video.webm -f null -~~

~~Apply the BSF before decoding (this might work):~~
~~ffmpeg -hide_banner -i sample-video.webm -c:v copy -bsf:v vp9_superframe -f webm - | ffmpeg -hide_banner -f webm -hwaccel videotoolbox -i pipe: -f null -~~

A BSF like this should be lightweight. It could increase memory usage slightly because it holds more frames in memory, although that can likely be tweaked. Implementing it will be a bit of work though, since it doesn't look like BSFs are used anywhere so far.

EDIT: This has been confirmed to not work. I might have another idea. Will post again if I figure it out

peppy · 2023-08-21T11:02:25Z

cc @FreezyLemon , not sure if you're also looking to bring the libraries up to date as part of #5974.

FreezyLemon · 2023-08-21T11:22:17Z

cc @FreezyLemon , not sure if you're also looking to bring the libraries up to date as part of #5974.

I'd like to, but the VP9 decoding bug is a bit of a dealbreaker. It could be worked around by disabling the hwaccel in this special case, but that's not great. Things seem to be moving in the ffmpeg ticket though, so maybe we can patch the issue soon-ish. If there's no good resolution, disabling hwaccel will always be an option too.

In the meantime, the changes proposed in #5974 are basically version-independent. So I don't see a big reason to wait for this to be resolved.

longnguyen2004 · 2023-10-24T16:05:19Z

If we're going to update, can we add Opus support? It's currently the most efficient audio codec, and can allow us to drop bitrate to 128kbps (or 96 even), and still have good enough quality for mapping, which will help combat the ever-growing space requirement for beatmaps.

bdach · 2023-10-24T16:12:11Z

@longnguyen2004 we do not use FFmpeg for audio decode, so i don't know what this has to do with this issue

nekodex added the area:libraries label Feb 24, 2022

frenzibyte mentioned this issue Apr 2, 2022

Video-related crashes on some maps due to apparently negative duration ppy/osu#17345

Closed

peppy mentioned this issue Aug 19, 2023

Consider adding AV1 support to the VideoDecoder #5971

Open

FreezyLemon mentioned this issue Feb 2, 2024

FFmpeg Linux binaries: Include version in file name #6153

Merged

FreezyLemon linked a pull request Apr 19, 2024 that will close this issue

FFmpeg: Update FFmpeg.AutoGen to 7.0.0 #6256

Open

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade FFmpeg libraries to 5.0 #5051

Upgrade FFmpeg libraries to 5.0 #5051

nekodex commented Feb 24, 2022 •

edited

Loading

peppy commented Feb 25, 2022

Opelkuh commented Feb 26, 2022

peppy commented Feb 28, 2022

nekodex commented Mar 2, 2022

nekodex commented Mar 9, 2022

peppy commented Mar 9, 2022

nekodex commented Mar 9, 2022

Opelkuh commented Mar 10, 2022

nekodex commented Mar 10, 2022

nekodex commented Mar 10, 2022

FreezyLemon commented Aug 19, 2023 •

edited

Loading

peppy commented Aug 21, 2023 •

edited

Loading

FreezyLemon commented Aug 21, 2023

longnguyen2004 commented Oct 24, 2023

bdach commented Oct 24, 2023

Upgrade FFmpeg libraries to 5.0 #5051

Upgrade FFmpeg libraries to 5.0 #5051

Comments

nekodex commented Feb 24, 2022 • edited Loading

peppy commented Feb 25, 2022

Opelkuh commented Feb 26, 2022

peppy commented Feb 28, 2022

nekodex commented Mar 2, 2022

nekodex commented Mar 9, 2022

peppy commented Mar 9, 2022

nekodex commented Mar 9, 2022

Opelkuh commented Mar 10, 2022

nekodex commented Mar 10, 2022

nekodex commented Mar 10, 2022

FreezyLemon commented Aug 19, 2023 • edited Loading

peppy commented Aug 21, 2023 • edited Loading

FreezyLemon commented Aug 21, 2023

longnguyen2004 commented Oct 24, 2023

bdach commented Oct 24, 2023

nekodex commented Feb 24, 2022 •

edited

Loading

FreezyLemon commented Aug 19, 2023 •

edited

Loading

peppy commented Aug 21, 2023 •

edited

Loading