fix/make dtls fragment stay within mtu size range

[pnts-se]

closes #1100

This PR will make DTLS fragment stay within MTU size range. Currently that's not always the case, e.g. on large certificate size.

The solution is to stop calling BIO_get_mem_data() manually for pending out going data, and instead rely on a callback function, set with BIO_set_callback_ex().

Packet capture during usage when running a worker build from this branch:

[ibc]

Amazing. Thanks a lot.

However, probably it will take several days before we can review and merge this PR (super busy days after summer and lot of pending work).

[pnts-se]

However, probably it will take several days before we can review and merge this PR

Yes of course, I'm in no hurry. I will try to find something else mediasoup-related to help out with.

As for this PR: I'm just beginning to learn the code base and I'm not sure if I got it right in terms of encapsulation and data hiding. Might want to refactor that.

[ibc]

I'm testing this PR right now, but I'll do in the flatbuffers branch to avoid conflicts once that is merged.

[ibc]

I see a potential problem:

	void DtlsTransport::Reset()
	{
		MS_TRACE();

		int ret;

		if (!IsRunning())
			return;

		MS_WARN_TAG(dtls, "resetting DTLS transport");

		// Stop the DTLS timer.
		this->timer->Stop();

		// We need to reset the SSL instance so we need to "shutdown" it, but we
		// don't want to send a Close Alert to the peer, so just don't call
		// SendPendingOutgoingDTLSData().
		SSL_shutdown(this->ssl);

		this->localRole.reset();
		this->state            = DtlsState::NEW;
		this->handshakeDone    = false;
		this->handshakeDoneNow = false;

		// Reset SSL status.
		// NOTE: For this to properly work, SSL_shutdown() must be called before.
		// NOTE: This may fail if not enough DTLS handshake data has been received,
		// but we don't care so just clear the error queue.
		ret = SSL_clear(this->ssl);

		if (ret == 0)
			ERR_clear_error();
	}

Note the:

// We need to reset the SSL instance so we need to "shutdown" it, but we// don't want to send a Close Alert to the peer, so just don't call
// SendPendingOutgoingDTLSData().
SSL_shutdown(this->ssl);

However in your PR we no longer call SendPendingOutgoingDTLSData() or any other method to send data. Instead it is done automatically by the sslBioOutCallback callback. The idea is that we reset mediasoup side DTLS without sending DTLS Close Alert to the endpoint and I'm afraid we are loosing this.

[ibc]

I see a potential problem:

	void DtlsTransport::Reset()
	{
		MS_TRACE();

		int ret;

		if (!IsRunning())
			return;

		MS_WARN_TAG(dtls, "resetting DTLS transport");

		// Stop the DTLS timer.
		this->timer->Stop();

		// We need to reset the SSL instance so we need to "shutdown" it, but we
		// don't want to send a Close Alert to the peer, so just don't call
		// SendPendingOutgoingDTLSData().
		SSL_shutdown(this->ssl);

		this->localRole.reset();
		this->state            = DtlsState::NEW;
		this->handshakeDone    = false;
		this->handshakeDoneNow = false;

		// Reset SSL status.
		// NOTE: For this to properly work, SSL_shutdown() must be called before.
		// NOTE: This may fail if not enough DTLS handshake data has been received,
		// but we don't care so just clear the error queue.
		ret = SSL_clear(this->ssl);

		if (ret == 0)
			ERR_clear_error();
	}

Note the:

// We need to reset the SSL instance so we need to "shutdown" it, but we// don't want to send a Close Alert to the peer, so just don't call
// SendPendingOutgoingDTLSData().
SSL_shutdown(this->ssl);

However in your PR we no longer call SendPendingOutgoingDTLSData() or any other method to send data. Instead it is done automatically by the sslBioOutCallback callback. The idea is that we reset mediasoup side DTLS without sending DTLS Close Alert to the endpoint and I'm afraid we are loosing this.

Let's ignore this use case since AFAIR it never worked fine. This is, if DtlsTransport::Reset() is called once the DTLS session was established, it never works again, and probably it's what the spec says. So whether we send a DTLS Close Alert or not is irrelevant because:

1. In most of the cases we do want to send it (when we really close/reset the DtlsTransport due to a fatal failure).
2. The use case of reseting the DTLS session once established is not a real/needed use case.

[ibc]

@pnts-se I'm gonna close this PR in favour of this one #1156 which is basically a clone of yours but it targets flatbuffers branch instead of v3 branch (to avoid complex git conflicts).

Thanks a lot!