Provide hooks for track customization

[defagos]

As a developer integrating Pillarbox I want to be able to reliably override the default track selection.

Acceptance criteria

• It is possible to force audio and subtitle selection based on the really available options for an item.
• Overriding works in playlists containing several items.
• The default initial selection of an item can be altered (also in playlists) but the user should be able to change it afterwards.

Hints

• We should likely not have a customization hook per item but rather let apps customize MediaSelector instead. If an app needs to override the selection logic it won't be per item but likely based on app-managed settings.

Tasks

• Add hook or similar mechanism to override media selection options at startup.
• Ensure the behavior with playlists is fine.
• Add an example to the showcase (muted playback with always visible subtitles, e.g.).
• Write unit tests with player already loaded with an item and setup for language selection afterwards.
• Write unit tests with player empty, setup for language selection and loaded with an item afterwards.
• Document selection issue reported to Apple (FB13070742) as known issue.
• Check that there are not leaks (in particular the transition demo seems to leak. Maybe because of sheet presented from sheet?). Leak on iOS 17 beta, no issue on iOS 16
• Expose MediaSelector for customization? Not yet
• Fix following issue:
  • Override subtitles and / or audio to ja / it in demo PlaybackView.
  • Play some content without these languages (but others) and attempt to change subtitles afterwards. Selection does not work.

[defagos]

We can likely use the following to specify a preferred language before playback starts, e.g. for subtitles:

let criteria = AVPlayerMediaSelectionCriteria(preferredLanguages: ["en"], preferredMediaCharacteristics: nil)
queuePlayer.setMediaSelectionCriteria(criteria, forMediaCharacteristic: .legible)

(AVPlayerMediaSelectionCriteria can also be constructed with principalMediaCharacteristics).

It might be enough to expose this API so that apps can force a language before playback starts.

We should also likely provide an API which does not use MediaAccessibility when setting subtitles. Using this API apps could force subtitles on a player instance, e.g. when displayed in a news feed with sound muted. The same player could then be shared in full-screen where the interactive MediaAccessibility selection would take place. When reverting the player back to the feed the subtitles could then be forced again using the same API.

I guess this API should be implemented as follows:

• Set selection criteria as above.
• Enable automatic selection but do not call MediaAccessibility.

[defagos]

Use cases

Here are a few use cases with nice behavior we should likely expect from a user experience perspective. The test setup is as follows:

• Test streams: The Morning Show (trailers from seasons 1 & 2). These are extremely well packaged streams with a lot of features (AD, SDH, original version flag, keyframes). They conform to what we expect regarding autoselection as well.
• Preferred device languages: FR, EN

No overrides

Setup:

• Custom player
• No overrides
• No accessibility settings

Expected result:

• Audio starts with 1st device language (FR).
• Audio can be changed. Reopening the player starts with FR again.
• Subtitles can be freely selected (other language, off, auto) and are persisted automatically.

Audio override

Setup:

• Custom player
• Override audio with DE
• No accessibility settings

Expected result:

• Audio starts with DE.
• Audio can be changed. Reopening the player starts with DE again. New audio choice is persisted for the player session (append to audible criteria).
• Subtitles can be freely selected (other language, off, auto) and are persisted automatically (appened to legible MA).

Subtitle override

Setup:

• Custom player
• Override subtitles with DE
• No accessibility settings

Expected result:

• Subtitles starts with DE.
• Audio can be changed. Reopening the player starts with DE again. New audio choice is persisted for the player session (append to audible criteria).
• Subtitles can be freely selected (other language, off, auto) and are persisted automatically (append to legible MA and clear the legible criteria).

🚧 Learnings:

• We must write a UT for this case: Start with override then select .off / .auto. Must deliver .off / .auto as selected value.

No override in playlist

Setup:

• Playlist player, two episodes loaded
• No overrides
• No accessibility settings

Expected result:

• Audio starts with 1st device language (FR).
• Audio can be changed. Reopening the player starts with FR again. Moving to the next item preserves audio selection.
• Subtitles can be freely selected (other language, off, auto) and are persisted automatically. Moving to the next item preserves subtitle selection.

Audio override in playlist

Setup:

• Playlist player, two episodes loaded
• Override audio with DE
• No accessibility settings

Expected result:

• Audio starts with DE.
• Audio can be changed. Reopening the player starts with DE again. Moving to the next item preserves audio selection.
• Subtitles can be freely selected (other language, off, auto) and are persisted automatically. Moving to the next item preserves subtitle selection.

Subtitle override in playlist

Setup:

• Playlist player, two episodes loaded
• Override subtitles with DE
• No accessibility settings

Expected result:

• Subtitles starts with DE.
• Audio can be changed. Reopening the player starts with DE again. New audio choice is persisted for the player session (append to audible criteria). Moving to the next item preserves audio selection.
• Subtitles can be freely selected (other language, off, auto) and are persisted automatically (append to legible MA and clear the legible criteria). Moving to the next item preserves subtitle selection.

Use case TODO 1

Playlist with 3 items, Morning Show, Stream S with only Romansh and Klingon audio, Morning Show again. Probably need this intermediate test stream to be generated.

• Play Morning Show 1, select ES.
• Play S, select Romansh.
• Play Morning Show next, ES should likely be used.

Use case TODO 2

Playlist with 3 items, Morning Show, Stream S with only Romansh and Klingon audio, Morning Show again. Probably need this intermediate test stream to be generated.

• Play Morning Show 1, default language FR.
• Play S, select Romansh.
• Play Morning Show next, FR should likely be used.

TODO: Chinese subtitles should be preserved between items

• Playlist with WWDC and Morning Show
• Pick Chinese Simplified subtitles on Morning Show
• Switch to WWDC item. Subtitles should be Chinese but currently not the case

Hints:

• No issue if we use the two Morning Show streams. Not with some languages or depending on the order of the items. Packaging issue? Bug?
• We should study what happens with a simple AVQueuePlayer loaded with these items and the system standard player UI.
• We should use the locale identifier instead of the language code.

[defagos]

A remark about selection:

• For legible renditions we use Media Accessibility to append recent language user choices. This is persisted at the system level and restored between playback sessions.
• For audible renditions there is no system mechanism but we can still persist audio choices during a playback session (most notably between playlist items) by appending language choices to the current media selection. This provides for a smoother experience, especially when playing several episodes of the same TV series with similar renditions available.

[defagos]

The DEFAULT attribute is still used a bit by AVPlayer, most notably when there are no media selection criteria explicitly set. According to AVPlayerMediaSelectionCriteria documentation:

If no option in a media selection group with any of the preferred languages is available, the default option in the group will be considered the best match.

If there is namely no AUTOSELECT attribute set on options, the one with the DEFAULT one is selected. This means that

#EXTM3U

#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="bipbop_audio",LANGUAGE="eng",NAME="BipBop Audio 1",DEFAULT=YES
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="bipbop_audio",LANGUAGE="fre",NAME="BipBop Audio 2",DEFAULT=NO,URI="alternate_audio_aac/prog_index.m3u8"
...

selects English automatically by default, whereas

#EXTM3U

#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="bipbop_audio",LANGUAGE="eng",NAME="BipBop Audio 1",DEFAULT=NO
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="bipbop_audio",LANGUAGE="fre",NAME="BipBop Audio 2",DEFAULT=YES,URI="alternate_audio_aac/prog_index.m3u8"
...

selects French by default. This can be easily tested by intercepting the Apple Bip Bop 16:9 basic example playlist with Charles and editing the corresponding response accordingly.

[defagos]

A few things about AVPlayer.setMediaSelectionCriteria(_:forMediaCharacteristic:):

• Setting criteria to nil resets to default system-based values. The system takes into account language and accessibility preferences.
• Setting criteria to an AVPlayerMediaSelectionCriteria instance which has either its preferred language, resp. characteristic parameter nil is not equivalent as using default system-based preferences. In this case nil means none.

The above is true for audible and legible characteristics, and for all usual accessibility behaviors (AD, SDH).

[defagos]

Automatic selection has been documented on the dedicated thread.

[defagos]

The MediaSelector protocol now provides a way to implement custom selectors. If there is a need in the future we could add an AVMediaCharacteristic to MediaSelector-conforming classes dictionary to our player configuration, letting developers write and register their own selectors if they are not happy with our default implementations.

Since implementing a selector can be quite tricky it is likely better to wait until there is a real need before we expose this API.