Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Proposal]: Thumbnails: Add supplementary metadata to getAvailableThumbnailTracks #1605

Open
wants to merge 1 commit into
base: feat/thumbnail-tracks
Choose a base branch
from

Conversation

peaBerberian
Copy link
Collaborator

@peaBerberian peaBerberian commented Dec 13, 2024

Based on #1496

Problem

We're currently trying to provide a complete[1] and easy to-use API for DASH thumbnail tracks in the RxPlayer.

Today the proposal is to have an API called renderThumbnail, to which an application would just provide an HTML element and a timestamp, and the RxPlayer would do all that's necessary to fetch the corresponding thumbnail and display it in the corresponding element.

The API is like so:

rxPlayer.renderThumbnail({ element, time })
  .then(() => console.log("The thumbnail is now rendered in the element"));

This works and seems to me very simple to understand.

Yet, we've known of advanced use cases where an application might not just want to display a single thumbnail for a single position. For example, there's very known examples where an application displays a window of multiple thumbnails at once on the player's UI to facilitate navigation inside the content.

bbbthumbs
Screenshot: this is the interface of a very popular web media player (the official one from the platform for which you installed newpipe instead) where multiple thumbnails are shown at once, under the seek bar. Interestingly, the video is also replaced by a thumbnail in that mode here, I guess to provide a smoother experience when rapidly navigating in the content.

To do that under the solution proposed in #1496, an application could just call renderThumbnail with several element and time values.
Yet for this type of feature, what the interface would want is not really to indicate time values, it actually wants basically a list of distinct thumbnails around/before/after a given position.

By just being able to set a time value, an application is blind on which time value is going to lead to a different thumbnail (i.e. is the thumbnail for the time 11 different than the thumbnail for the time 12? Nobody - but the RxPlayer - knows).

So we have to find a solution for this

[1] By complete, I here mean that we want to be able to handle its complexities inside the RxPlayer to ensure advanced DASH configurations like multi-CDN, retry settings for requests etc., while still allowing all potential use cases for an application.

Solution

In this solution, I experiment with a second thumbnail API, getAvailableThumbnailTracks (it already exists in #1496, but its role there was only to list the various thumbnail qualities, if there are several size for example). As this solution build upon yet stays compatible to #1496, I chose to open this second PR on top of that previous one.

I profit from the fact that most standardized thumbnail implementations I know of (BIF, DASH) seem to follow the principle of having evenly-spaced (in terms of time) thumbnails (though I do see a possibility for that to change, e.g. to have thumbnails corresponding to "important" scenes instead, so our implementation has to be resilient).

So here, what this PR does is to add the following properties (all optional) to a track returned by the getAvailableThumbnailTracks API:

  • start: The initial time the first thumbnail of that track will apply to

  • end: The last time the last thumbnail of that track will apply to.

    ⚠️ For live contents (which I guess should be the main type of contents relying on this, BIF thumbnails are good enough for VoD contents), for now that end will announce the future expected end of the Period if already known (for example thanks to a Period@end or a SegmentTemplate@endNumber attribute) - and undefined if unknown.
    In this current configuration, an application has to also request another API, like getLivePosition to know the maximum position it can currently load thumbnails from

  • thumbnailsPerSegment: Individual thumbnails may be technically part of "segments" containing multiple consecutive thumbnails each.
    thumbnailsPerSegment is the number of thumbnails each of those segments contain.

    For example you could have stored on the server a segment which is a grid of 2 x 3 (2 horizontal rows and * 3 vertical columns) thumbnails, which the RxPlayer will load at once then "cut" the right way when calling renderThumbnail. In that example, thumbnailsPerSegment would be set to 6 (2*3).

    Note that the last segment of a content may contain less thumbnails as announced here depending on the duration of the content.

  • segmentDuration: The "duration" (in seconds) each segments of thumbnails applies to (with the exception of the last one, which just fills until end)

Then, an application should have all information needed to calculate a time which correspond to a different thumbnail.

Though this solution lead to a minor issue: by letting application make the time operation themselves with start, end, segmentDuration and so on, there's a risk of rounding errors leading to a time which does not correspond to the thumbnail wanted but the one before or after. To me, we could just indicate in our API documentation to application developers that they should be extra careful and may add an epsilon (or even choose a time in the "middle" of thumbnails each time) if they want that type of thumbnail list feature.

Thoughts?

@peaBerberian peaBerberian changed the title Thumbnails: Add supplementary metadata to getAvailableThumbnailTracks [Proposal]: Thumbnails: Add supplementary metadata to getAvailableThumbnailTracks Dec 13, 2024
@peaBerberian peaBerberian added thumbnails Relative to image thumbnails proposal This Pull Request or Issue is only a proposal for a change with the expectation of a debate on it labels Dec 13, 2024
@peaBerberian peaBerberian force-pushed the feat/thumbnail-tracks-more-metadata branch from cd12b1f to 95c5d6d Compare December 13, 2024 16:21
@peaBerberian
Copy link
Collaborator Author

peaBerberian commented Dec 16, 2024

Getting the timestamp of the latest available thumbnail in a live content is a hard thing to get in any case in the current solution:

/**
 * For the current content, returns the position in seconds that will
 * correspond to the currently last reachable thumbnail, or `undefined` if
 * unknown.
 *
 * That position may then be passed to the `rxPlayer.renderThumbnail()` method.
 *
 * @returns {number|undefined}
 */
function getLastThumbnailTime() {
  const livePosition = rxPlayer.getLivePosition() ?? undefined;
  const periods = rxPlayer.getAvailablePeriods();

  let lastSeekablePeriod;
  if (livePosition === undefined) {
    lastSeekablePeriod = periods[periods.length - 1];
  } else {
    // Most applications will not let users seek further than live edge even when possible
    // (e.g. when ads are pre-anounced in the Manifest)
    // So here find the last Period that exists before the live edge
    for (let i = periods.length - 1; i >= 0; i--) {
      if (periods[i].start >= livePosition) {
        lastSeekablePeriod = periods[i];
        break;
      }
    }
  }

  if (lastSeekablePeriod === undefined) {
    return;
  }

  // Just select the first thumbnail track for the last Period.
  const metadata = rxPlayer.getAvailableThumbnailTracks({
    periodId: lastSeekablePeriod.id,
  })[0];
  if (
    metadata === undefined ||
    metadata.start === undefined ||
    metadata.segmentDuration === undefined
  ) {
    return;
  }

  const maximumPosition = rxPlayer.getMaximumPosition() ?? undefined;
  if (maximumPosition === undefined) {
    return;
  }

  /**
   * Seconds at the end of the content for which a thumbnail has not yet been
   * generated.
   */
  const secondsWithoutThumbnailYet =
    (maximumPosition - metadata.start) % metadata.segmentDuration;

  /**
   * Position that will lead to the last available thumbnail being requested.
   */
  const maxThumbnailTime =
    Math.min(maximumPosition - secondsWithoutThumbnailYet, metadata.end ?? Infinity) -
    // To securize against rounding and/or precizion errors, we take a timestamp
    // at the middle of the thumbnail
    (metadata.segmentDuration / (metadata.thumbnailsPerSegment ?? 1)) / 2;

  if (livePosition !== undefined && livePosition < maxThumbnailTime) {
    return livePosition;
  }
  return maxThumbnailTime;
}

Even I, spending most of my time on core player matters, had trouble writing that (e.g. we have to potentially consider the live position several time and all that).

@peaBerberian peaBerberian force-pushed the feat/thumbnail-tracks-more-metadata branch from 95c5d6d to 1df2e07 Compare December 16, 2024 12:44
@peaBerberian peaBerberian added the work-in-progress This Pull Request or issue is not finished yet label Dec 16, 2024
@peaBerberian peaBerberian force-pushed the feat/thumbnail-tracks-more-metadata branch 2 times, most recently from 8352bed to 2f9d45b Compare December 16, 2024 16:59
@peaBerberian peaBerberian force-pushed the feat/thumbnail-tracks branch 2 times, most recently from 4b3ddb5 to 3fb82ce Compare December 16, 2024 17:00
@peaBerberian peaBerberian force-pushed the feat/thumbnail-tracks-more-metadata branch from 2f9d45b to 61c392c Compare December 16, 2024 17:00
@peaBerberian peaBerberian removed the work-in-progress This Pull Request or issue is not finished yet label Dec 16, 2024
Based on #1496

Problem
-------

We're currently trying to provide a complete[1] and easy to-use API for
DASH thumbnail tracks in the RxPlayer.

Today the proposal is to have an API called `renderThumbnail`, to which
an application would just provide an HTML element and a timestamp, and
the RxPlayer would do all that's necessary to fetch the corresponding
thumbnail and display it in the corresponding element.

The API is like so:
```js
rxPlayer.renderThumbnail({ element, time })
  .then(() => console.log("The thumbnail is now rendered in the element"));
```

This works and seems to me very simple to understand.

Yet, we've known of advanced use cases where an application might not
just want to display a single thumbnail for a single position. For
example, there's very known examples where an application displays a
window of multiple thumbnails at once on the player's UI to facilitate
navigation inside the content.

To do that under the solution proposed in #1496, an application could
just call `renderThumbnail` with several `element` and `time` values.

Yet for this type of feature, what the interface would want is not really
to indicate a `time` values, it actually wants basically a list of
distinct thumbnails around/before/after a given position.

By just being able to set a `time` value, an application is blind on
which `time` value is going to lead to a different timestamp (i.e. is
the thumbnail for the `time` `11` different than the thumbnail for the
`time` `12`? Nobody - but the RxPlayer - knows).

So we have to find a solution for this

[1] By complete, I here mean that we want to be able to handle its
complexities inside the RxPlayer, to ensure complex DASH situations like
multi-CDN, retry settings for requests and so on while still allowing
all potential use cases for an application.

Solution
--------

In this solution, I experiment with a second thumbnail API,
`getAvailableThumbnailTracks` (it already exists in #1496, but its role
there was only to list the various thumbnail qualities, if there are
several size for example). As this solution build upon yet stays
compatible to #1496, I chose to open this second PR on top of that
previous one.

I profit from the fact that most standardized thumbnail implementations I
know of (BIF, DASH) seem follow the principle of having evenly-spaced
(in terms of time) thumbnails (though I do see a possibility
for that to change, e.g. to have thumbnails corresponding to "important"
scenes instead, so our implementation has to be resilient).

So here, what this commit does is to add the following properties (all
optional) to a track returned by the `getAvailableThumbnailTracks` API:

  - `start`: The initial `time` the first thumbnail of that track will
    apply to

  - `end`: The last `time` the last thumbnail of that track will
    apply to

  - thumbnailsPerSegment: Individual thumbnails may be technically part of
    "segments" containing multiple consecutive thumbnails each.

    `thumbnailsPerSegment` is the number of thumbnails each of those
    segments contain.

    For example you could have stored on the server a segment which is a
    grid of 2 x 3 (2 horizontal rows and * 3 vertical columns)
    thumbnails, which the RxPlayer will load at once then "cut" the right
    way when calling `renderThumbnail`. In that example,
    `thumbnailsPerSegment` would be set to `6` (2*3).

    Note that the last segment of a content may contain less thumbnails as
    anounced here depending on the duration of the content.

  - `segmentDuration`: The "duration" (in seconds) each segments of
    thumbnails applies to (with the exception of the last thumbnail,
    which just fills until `end`)

Then, an application should have all information needed to calculate a
`time` which correspond to a different thumbnail.

Though this solution lead to a minor issue: by letting application make
the `time` operation themselves with `start`, `end`, `segmentDuration`
and so on, there's a risk of rounding errors leading to a `time`
which does not correspond to the thumbnail wanted but the one before or
after. To me, we could just indicate in our API documentation to
application developers that they should be extra careful and may add an
epsilon (or even choose a `time` in the "middle" of thumbnails each time)
if they want that type of thumbnail list feature.

Thoughts?
@peaBerberian peaBerberian force-pushed the feat/thumbnail-tracks-more-metadata branch from 61c392c to 0398cce Compare December 16, 2024 17:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
proposal This Pull Request or Issue is only a proposal for a change with the expectation of a debate on it thumbnails Relative to image thumbnails
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant