Preserve PTS on long running streams

[Bahlinc-Dev]

Describe the feature

There should be a config flag to preserve the source stream pts so the consumer is not forced to reconnect because of pts mismatch (missing packets, source error, etc..), and thus stream interruptions will be handled gracefully by downstream consumers.

Description

Most video players do not care about the starting pts, only that the pts continues to increase. With long running streams the chance of stream errors, or reconnection from source->simple sever, gets higher and higher. If you are recording the the stream, or restreaming the stream, this causes all sorts of downstream issues when the pts suddenly changes. A indicator of this in ffmpeg is the "Non-monotonous DTS" error, which has been reported here before. In #1046 this was attributed to the circular buffer, which might be true, but only because of packet loss. In reality the packet loss can happen anywhere up to the point it's consumed. Currently there is just no way to recover from that.

Something along the lines of:

on_disconnect() {  // or other error
  if (preservePTS) {
     ptsOffset = lastPTS;
  }
}

on_rtp_packet_received(pkt) {
   if (preservePTS) {
     pkt.pts += ptsOffset;
  }
 lastPTS = pkt.pts;
}

Also, the client should be able to keep their connection, even if the source disconnects, while waiting for more rtp packets. (even if there is a gap in time because of source error). The consumer can always specify their own timeout for determining when to "give up" waiting. Similar to how webRTC handles this.

Even if you do not implement a "preserve pts" capability, when there is known packet loss (because of internal processes such as circular buffer overrun etc.), I think the consumers should be force disconnected, because at that point you are sending invalid data anyway..

PS: I think in the pseudo-code above the rtp sequence-number should also be preserved?

[aler9]

Hello, at the moment, when proxying a RTSP stream, the PTS is copied as is from the original packets. I think it's the source responsibility to preserve PTS even when consumers disconnect and reconnect. Adding an offset to PTS after reconnection may not be a good idea since a lot of people wants to preserve PTS from the original source in order to synchronize frames. Personally, after a disconnection i use either the NTP timestamp (that can be computed by using RTCP sender reports) in order to sync frames or i just start a new segment.

Disconnections can always happen, even if we improve the client, therefore a consumer must handle this situation.

the client should be able to keep their connection, even if the source disconnects, while waiting for more rtp packets.

This can be a client improvement, i've opened a dedicated feature request here: bluenviron/gortsplib#162

Even if you do not implement a "preserve pts" capability, when there is known packet loss (because of internal processes such as circular buffer overrun etc.), I think the consumers should be force disconnected, because at that point you are sending invalid data anyway..

Packet loss has been greatly mitigated by the new RTP reordering mechanism that has been introduced some versions ago. Invalid / unordered packets are either discarded or put in a buffer, they're not sent to consumers. When a source gets too much corrupted, the source is disconnected and all the consumers / readers are disconnected too.

[Bahlinc-Dev]

Hi there, thanks for the quick response! For copying the pts from source->dest, I agree that in an ideal world the source stream would resume the pts from last sent pts on a reconnection event for a long running stream, BUT in most cases the source stream is not a custom source that can be modified (e.g. cameras, obs, or even another rtsp-simple-server). In those cases the source generally assumes a stateless connection, and so starts the stream at pts = 0 on connection. However, I do think that is the correct behavior/assumption from the perspective of the producer.

If we do shift pts responsibility to the to the reader/consumer side, and you are distributing the stream to large numbers of consumers, you end up with something like this:

source -> rtsp-simple-server ->
                               rtsp-simple-server 1 -> 1000 consumers
                               rtsp-simple-server 2 -> 1000 consumers
                                ....
                               rtsp-simple-server n -> 1000 consumers

In the above case, you are forcing 1000's of stalls and reconnections if the source has a brief interruption, only because the pts suddenly changes from xxxxxx -> 0.

We are currently using a custom RTMP receiver that receives the source, maintains pts, then proxy's to rtsp-simple-server for all the other great features like api control. I just thought this might be something that makes more sense as a built-in feature...

I can work on a PR that explores this if you are interested!