WebRTC spec should explicitly specify all causes of a PeerConnection-sourced track receiving mute/unmute

[jan-ivar]

In w3c/mediacapture-main#982 (comment) @guidou wrote:

For example: crbug 941740 implements mute on remote tracks reactively based on (lack of) input, violating the WebRTC spec and causing web compat issues.

In this case, Chromium just is applying the model defined in the main spec to remote tracks. The WebRTC spec indicates some cases in which the muted attribute should be set/unset, but AFAICT it does not say anywhere that this overrides the model defined in the original MediaStreamTrack specification. It also does not state a new definition of muted specific for WebRTC tracks and does not even list the muted/unmuted events in its [Event Summary section].

Crbug 941740 has been open for almost 5 years without mention of this, and I'd argue § 9.3 MediaStreamTrack does "state a new definition of muted specific for WebRTC tracks":

I would argue this describes all causes of mute for remote tracks, based on #139. But @guidou is right that it:

1. it does not say anywhere that this overrides the model defined in the original MediaStreamTrack specification.
2. does not list the muted/unmuted events in its Event Summary section

But this seems editorial as evidenced by #139. Both specs needs to clarify that while this spec follows the model in mediacapture-main, each source is responsible for specifying its own mute/unmute behavior, if any.

We did this for constraints, and it only makes sense to do the same here.

[jan-ivar]

Crbug 1295295 is another bug about non-standard unmute that might be relevant.

[alvestrand]

Crbug 1295295 is about unmuting without waiting for the first RTP packet to arrive, so it's not the behavior we're discussing here (rather the opposite, in fact).

[henbos]

My understanding is that when we define new types of sources (e.g. WebRTC remote tracks as opposed to local capture tracks) then the spec that defines the new type of source should also define when the muting/unmuting happens or not.

In the case of webrtc, if mute/unmute should be based on RTP or based on SDP is another discussion worth having, but that's a separate issue from sources having its own definitions

[fippo]

Given the long-standing issue of libWebRTC sending RTCP BYE at inappropriate times (and then reusing a SSRC) but not on stopping a transceiver (which would be observable through a new negotiation) I would remove BYE as mute reason.

[youennf]

the spec that defines the new type of source should also define when the muting/unmuting happens or not.

+1 to that.

+1 also for the spec to describe what implementations are doing, given the spec status.
We need those to get interop.

FWIW, this has a real impact as, for instance, in case of connection drop, a video element will either display the last received video frame (track not muted UA) or will display black (track muted UA).

[jan-ivar]

From yesterday's meeting I heard agreement that webrtc-pc should explicitly specify all causes of PeerConnection-sourced tracks being muted or unmuted, but some disagreement over what those causes are.

Since we have agreement we need to list causes here for event firing to be interoperable, the next steps I see are:

1. Open separate issue/PR on mediacapture-main to clarify each source is responsible for specifying its mute behavior.
2. Work out here the complete list of when mute and unmute events fire on remote tracks.

[jan-ivar]

Here's https://jsfiddle.net/jib1/p6bzhme0/37/ showing what existing implementations fire on remote PC tracks. It opens a connection, and sender-side calls sender.track.stop() at the 3 second mark, and transceiver.stop() at 5.

Firefox:

0.011: audio track event fired
0.011: video track event fired
0.011: sRD done.
0.016: checking
0.016: connected
0.043: video track unmute event fired
0.045: audio track unmute event fired
3.003: Stopped audio track
3.003: Stopped video track
5.006: Stopped audio transceiver
5.006: Stopped video transceiver
5.010: sRD done.
5.010: audio track ended event fired
5.010: video track ended event fired

Safari:

0.016: audio track event fired
0.016: video track event fired
0.016: sRD done.
0.017: audio track unmute event fired
0.017: video track unmute event fired
0.023: checking
0.023: connected
3.001: Stopped audio track
3.001: Stopped video track
5.008: Stopped audio transceiver
5.008: Stopped video transceiver
5.016: audio track ended event fired
5.016: video track ended event fired
5.017: audio track mute event fired
5.017: video track mute event fired
5.017: sRD done.

Chrome:

0.007: audio track event fired
0.007: video track event fired
0.007: audio track unmute event fired
0.007: video track unmute event fired
0.007: sRD done.
0.013: checking
0.017: connected
3.005: Stopped audio track
3.006: Stopped video track
4.196: video track mute event fired
5.003: Stopped audio transceiver
5.003: Stopped video transceiver
5.010: new
5.013: audio track ended event fired
5.013: video track ended event fired
5.014: sRD done.

Takeaways:

• Firefox appears to follow webrtc-pc's § 9.3 MediaStreamTrack
• Safari deviates by
  • unmuting after 1 ms, likely ahead of RTP reception, compared to ~32 ms in Firefox (webkit 209242)
  • muting tracks on stopped transceiver
• Chrome deviates by
  • unmuting ahead of sRD (crbug 1295295)
  • muting (only!) video track ~1 second after sender.track.stop() (crbug 941740)

I did not try severing the network connection, but based on the above I would guess Chrome would mute the video track only?

Are there other causes of mute/unmute we should consider?

[dontcallmedom-bot]

This issue had an associated resolution in WebRTC March 26 2024 meeting – 26 March 2024 (Clarify each source is responsible for specifying mute/unmute/ended and constraints behavior):

RESOLUTION: close webrtc-pc#2942 as reviewed and webrtc-pc#2915, open an issue on timing of unmuting on RTC reception

[jan-ivar]

Closing per above and #2942 (comment).