Automatic subtitle selection

[defagos]

As a user I want automatic subtitle selection to be available. Automatic selection must take care of selecting the most appropriate subtitle selection based on the current audio language.

Acceptance criteria

• An API is available to enable automatic subtitle selection.
• A corresponding option is available in the demo.

Documentation

• Team discussion.

Tasks

• Understand what automatic subtitle selection actually is.
• Understand possibly related APIs, most notably preferred settings and automatic behavior.
• Generate new test streams if needed (e.g. stream with DEFAULT for some language or with AUTOSELECT not available for all languages).
• Add API to enable automatic subtitle selection.
• Add corresponding option to the demo player menu.
• Check behavior over AirPlay.

[defagos]

I had a look at what is returned by preferredMediaSelection when AD / SDH settings are enabled or disabled but the results are not affected. This property isn't therefore a cheap way to get the subtitles or audio tracks to prefer based on accessibility settings.

[defagos]

We need to clearly identify what Automatic means. After discussions with @amtins and @StaehliJ it appears that our SRG Media Player was likely taking too many decisions and that the definition of this automatic mode should be simpler.

It is likely that:

• This is related to the AUTOSELECT playlist option.
• According to some non-official Akamai documentation automatic mode might select a best match only within AUTOSELECT options.

We should therefore:

• Think about several use cases.
• Tweak streams to see how packaging affects automatic selection, on iOS, tvOS and macOS. Warning: Safari and QuickTime have different behaviors it seems.

We should also check whether automatic mode agrees with the result obtained by calling selectMediaOptionAutomatically(in:) on a player item. Hopefully this is the case and we therefore can obtain identical behavior with a single call.

[defagos]

See comment elsewhere for an important remark about AUTOSELECT behavior and relationship with MediaAccessibility.

[defagos]

Automatic selection requires appliesMediaSelectionCriteriaAutomatically to be set to true (the default). Our implementation will use this default behavior. The rest works correctly provided playlists are properly configured.

[defagos]

Some remarks:

• We cannot create tweak AUTOSELECT with Shaka Packager since this flag is set internally. So the sample stream set was not extended.
• The implementation was done in Add support for subtitles and audio tracks #516. Please refer to this issue for more information. For automatic selection based on accessibility settings please refer to Accessibility for subtitles and audio tracks #504.

[defagos]

We finally better (but probably not completely) understand how Automatic (Recommended) works. Note that for a rendition to be selectable automatically, most notably in this mode, it needs to be flagged with AUTOSELECT=YES in the master playlist.

Here is how Automatic (Recommended) seems to work to the best of our current knowledge:

• For audible rendition selection, the system finds the first language that matches one of the preferred languages defined in the system settings, in order of appearance. If not it uses a default option (the one with DEFAULT=YES is preferred, if any).
• For legible rendition selection, the result depends on the selected audio language:
  • If the selected audio language appears in the preferred languages defined in the system settings, no subtitles will be displayed. The user is supposedly able to understand this language after all.
  • If the selected audio language does not appear in the preferred languages defined in the system settings, the first subtitles matching one of the preferred languages, in order of appearance, are used. Otherwise a default option is used (the one with DEFAULT=YES is preferred, if any).

Note that this automatic selection logic is affected by accessibility settings, though the idea remains the same. It also seems to be applied when starting playback, not when settings are updated afterwards during playback. The criteria are likely those listed in the official documentation.