Cannot connect to XRDP

[mdenty]

I have some Ubuntu Linux servers running XRDP (0.9.12 and 0.9.17)

The connection always fails with this error message:

Connection error: [TLS upgrade] custom error, caused by: received corrupt message of type InvalidContentType

It works fine with a Windows 11 ARM VM.

[CBenoit]

Thank you for the report.

I’m not sure how much of the issue is due to IronRDP as opposed to rustls (the TLS backend we are using).

I think it could help if you could provide an execution trace with the environment variable IRONRDP_LOG set to rustls=trace,tokio-rustls=trace and send us the ironrdp-client.log file.
Note that the name of the target server will appear in the logs, you should replace it with a placeholder if this information is sensitive.

[mdenty]

Sure, you'll find the log file attached.
rustls_ironrdp-client.log

After that log recording, I have tried to change in Cargo.toml file (client) the TLS feature:

[features]
# default = ["rustls"]
default = ["native-tls"]
rustls = ["ironrdp-tls/rustls"]
native-tls = ["ironrdp-tls/native-tls"]

and I have the exact same error message (it does not seem to activate native-tls).

Then, I also changed to native-tls in ironrdp-tls/Cargo.toml file, the error message is different :

Connection error: [TLS upgrade] custom error, caused by: connection closed via error

I've also changed the IRONRDP_LOG variable to rustls=trace,native-tls=trace,tokio-native-tls=trace,tokio-rustls=trace but the log file stays empty.

Here the call stack if useful.

[CBenoit]

Thank you for the detailed report! Much appreciated.

I have tried to change in Cargo.toml file (client) the TLS feature:

[features]
# default = ["rustls"]
default = ["native-tls"]
rustls = ["ironrdp-tls/rustls"]
native-tls = ["ironrdp-tls/native-tls"]

and I have the exact same error message (it does not seem to activate native-tls).

Good catch. This is addressed by #325

Then, I also changed to native-tls in ironrdp-tls/Cargo.toml file, the error message is different :

Connection error: [TLS upgrade] custom error, caused by: connection closed via error

The underlying cause is certainly the same.

I've also changed the IRONRDP_LOG variable to rustls=trace,native-tls=trace,tokio-native-tls=trace,tokio-rustls=trace but the log file stays empty.

This is because native-tls typically does not log anything in this case. (There were something like two or three log instructions in the whole crate last time I checked.)

Sure, you'll find the log file attached.

Unfortunately, nothing is looking wrong as far as I can see. However, the response to the ClientHello is not included. This is likely because rustls is not able to make any sense of it. Any chance you could record the traffic and send us a pcap? (I recommend wireshark.)

One last question: are you able to confirm that TLS is enabled on XRDP at all? Do you have any special configuration? I believe it is supposed to be enabled by default, but I’m not super familiar with it. The immediate failure after sending the ClientHello is suspicious (especially because of the "InvalidContentType" error). IronRDP only supports TLS for securing the RDP transport, and notably there is no desire to support the so-called "Standard RDP Security" (which relies on RC4) because it’s completely broken.

A good way to troubleshoot would be to record a pcap of a successful handshake made using mstsc or freerdp as client, and compare with the pcap of the failed handshake made using IronRDP. Maybe the problem is not directly related to TLS, but to how we negotiate the security layer.

[mdenty]

I captured the traces of both connections (IronRDP and MS Remote Desktop) and attached them in the zip file.

I can spot differences very early in the communication.
With IronRDP

With MS RDP

Again, the complete trace is inside the zip file mentioned at the beginning of this message.

J'espère que ça aidera à la résolution du problème.

FYI: the clients are running on macOS 14.1.2 (IronRDP: commit b87627b, MS RDP: 10.9.4 (2161)).
Server is Linux Ubuntu 22.04 with xrdp 0.9.17 / OpenSSL 3.0.2 installed with apt.

[CBenoit]

J'espère que ça aidera à la résolution du problème.

Merci ! 😛

The pcap provided me with everything I needed to know.

In mstsc, the requestedProtocols flags are set as follow:

This indicates three security protocols as supported client side, and the server may choose any of them. Standard RDP security is also always considered available by the server, so you could count four.

In comparison, IronRDP is sending this:

The "TLS security" flag is not set.

However, XRDP does not support NLA, and thus attempts to "downgrade" to the standard RDP security protocol:

But as mentioned previously, IronRDP purposefully does not support this security protocol, and fails.

Good to know: usage of the TLS transport itself is implied by all security protocols except standard RDP security, and it’s not a bug to not specify it again. By not specifying it, we are able to effectively enforce usage of NLA (CredSSP). Roughly, the TLS "security protocol" defined in the RDP protocol is both for wrapping the RDP transport into a TLS transport AND showing the Windows MS logon screen to the user in order to perform a graphical login (modulo the "autologon" option). That is: the authentication is not happening at the network level (using NLA), and the user enters the credentials directly on the remote machine, as if a physical machine was used. This is what differs from the other security protocols.

For context, see this "xweet" from @awakecoding (behind the scene, the options shown are enabling or disabling these flags as appropriate).

NLA being disabled is problematic for a few reasons. For one, it’s easy to perform man-in-the-middle attacks by exposing a fake RDP server and to collect credentials. Even with the certificate warning, many users are disregarding the warning popup and end up entering their credentials anyway. GoSecure conducted a study by exposing honeypot RDP servers, relying on this NLA → TLS security downgrade.

Today, it’s already possible to alter this behavior in the IronRDP client by providing the --security-protocol argument with the value Ssl.

The user interface for that, however, is less than ideal for many reasons, and I still consider this as a bug.
In addition to that, IronRDP should "fail fast" as soon as the server announced the usage of the standard RDP security instead of moving forward with TLS. I’ll address this.

[CBenoit]

The user interface has been fixed in #328

the clients are running on macOS 14.1.2 (IronRDP: commit b87627b, MS RDP: 10.9.4 (2161)).

FYI, there is a bug related to scancodes on macOS: #180
However,  I also happen to have a patch almost ready for this.

[mdenty]

Thank you very much Benoît for this detailed investigation.

However,  I also happen to have a patch almost ready for this.

Great news about the scancodes for macOS, I was planning to implement this feature myself.

EDIT : I tried the --security-protocol ssl option and this leads to an other error message: Connection error: [received bad MCS Channel Join Confirm] general error. I'm not sure that was expected.

[CBenoit]

EDIT : I tried the --security-protocol ssl option and this leads to an other error message: Connection error: [received bad MCS Channel Join Confirm] general error. I'm not sure that was expected.

Indeed, it’s not expected. Looks like this check fails:

IronRDP/crates/ironrdp-connector/src/channel_connection.rs

Lines 176 to 181 in bd8df6d

	if channel_join_confirm.initiator_id != user_channel_id
	|| channel_join_confirm.channel_id != channel_join_confirm.requested_channel_id
	|| channel_join_confirm.channel_id != channel_id
	{
	return Err(general_err!("received bad MCS Channel Join Confirm"));
	}

Which could mean that XRDP is advertising a specific ID for a given channel, and then joins it using yet a different channel ID. It’s not clear to me whether this is a bug on XRDP side or IronRDP. I opened a PR for improving the error reporting of this: #329
Do you think you could give it a try and report back?

[mdenty]

The new message is :
Connection error: [ChannelJoinConfirm] reason: unexpected requested_channel_id in MCS Channel Join Confirm: received 1004, got 1002

See attached log for xrdp 0.9.17.
ironrdp-client-xrdp-0.9.17.log
EDIT: For some reason the official MS RDP does connect anymore on this VM. I'll investigate... Still unsure about this, I've re-installed the package, activated TLS. Now the official RDP client connects, but I have a black screen.

I took an other log with both ssl and hybrid after reinstall:
ironrdp-client-reinstall.log

When I connect with IronRDP (with ssl), I have this log on server side:

[20231208-10:24:29] [INFO ] Socket 12: AF_INET6 connection received from ::ffff:192.168.41.1 port 61557
[20231208-10:24:29] [INFO ] Connected client computer name: macbookpro.loca
[20231208-10:24:29] [ERROR] SSL_read: Failure in SSL library (protocol error?)
[20231208-10:24:29] [ERROR] SSL: error:0A000126:SSL routines::unexpected eof while reading
[20231208-10:24:29] [ERROR] libxrdp_force_read: header read error
[20231208-10:24:29] [ERROR] Processing [ITU-T T.25] ChannelJoinRequest failed
[20231208-10:24:29] [ERROR] [MCS Connection Sequence] receive channel join request failed
[20231208-10:24:29] [ERROR] xrdp_sec_incoming: xrdp_mcs_incoming failed
[20231208-10:24:29] [ERROR] xrdp_rdp_incoming: xrdp_sec_incoming failed
[20231208-10:24:29] [ERROR] xrdp_process_main_loop: libxrdp_process_incoming failed
[20231208-10:24:29] [ERROR] xrdp_iso_send: trans_write_copy_s failed
[20231208-10:24:29] [ERROR] Sending [ITU T.125] DisconnectProviderUltimatum failed

If I use hybrid mode:

[20231208-10:23:58] [INFO ] Socket 12: AF_INET6 connection received from ::ffff:192.168.41.1 port 61546
[20231208-10:23:58] [ERROR] libxrdp_force_read: bad header length 3
[20231208-10:23:58] [ERROR] Processing [ITU-T T.125] Connect-Initial failed
[20231208-10:23:58] [ERROR] [MCS Connection Sequence] receive connection request failed
[20231208-10:23:58] [ERROR] xrdp_sec_incoming: xrdp_mcs_incoming failed
[20231208-10:23:58] [ERROR] xrdp_rdp_incoming: xrdp_sec_incoming failed
[20231208-10:23:58] [ERROR] xrdp_process_main_loop: libxrdp_process_incoming failed

I've also tested on a Raspberry Pi5 with xrdp-0.9.21 (stock install).
Connection error: [TLS upgrade] custom error, caused by: received corrupt message of type InvalidContentType
with ssl option
ironrdp-client-xrd-0.9.21-ssl.log
with hybrid option
ironrdp-client-xrdp-0.9.21-hybrid.log

[CBenoit]

The new message is : Connection error: [ChannelJoinConfirm] reason: unexpected requested_channel_id in MCS Channel Join Confirm: received 1004, got 1002

Interesting. 1002 is typically the ID used for the "user channel". It’s supposed to be joined using the "AttachUserConfirm" message.

I don’t have time for looking into the rest of the logs now. I’ll probably dig a bit more next week. :)

Keep me updated if you find anything else!

[CBenoit]

I took an other log with both ssl and hybrid after reinstall: ironrdp-client-reinstall.log

I can see in the logs that the standard RDP security was negotiated again.

2023-12-08T10:43:53.223062Z  INFO connect_begin: ironrdp_connector::connection: Server confirmed connection selected_protocol=SecurityProtocol(0x0) flags=ResponseFlags(EXTENDED_CLIENT_DATA_SUPPORTED)

"selected_protocol=SecurityProtocol(0x0)" means that the server selected the standard RDP security protocol.
(By the way, the quality of the logs was improved on master, it should now write "STANDARD_RDP_SECURITY" instead of "0x0".)

I've also tested on a Raspberry Pi5 with xrdp-0.9.21 (stock install). Connection error: [TLS upgrade] custom error, caused by: received corrupt message of type InvalidContentType with ssl option ironrdp-client-xrd-0.9.21-ssl.log with hybrid option ironrdp-client-xrdp-0.9.21-hybrid.log

Same problem for these:

2023-12-08T09:50:31.637034Z  INFO connect_begin: ironrdp_connector::connection: Server confirmed connection selected_protocol=SecurityProtocol(0x0) flags=ResponseFlags(EXTENDED_CLIENT_DATA_SUPPORTED)

and

2023-12-08T09:51:56.347600Z  INFO connect_begin: ironrdp_connector::connection: Server confirmed connection selected_protocol=SecurityProtocol(0x0) flags=ResponseFlags(EXTENDED_CLIENT_DATA_SUPPORTED)

Can you try again using a build from master? I think the latest patches will help with that.

Of course, if the standard RDP security is the only available security protocol, there is nothing IronRDP can do. If you forced the "SSL" / "TLS" security protocol using the CLI, and XRDP still selected the standard RDP security protocol, it may be that something is not right on XRDP side… if you are unable to make progress using a new build from master, maybe we could use a pcap again and see if something fishy is happening on either side?

[mdenty]

Hello, I got back to this issue.

I've checked out the commit 33c66ed which is HEAD of master at the time of writing.

2024-01-04T09:57:47.292741Z ERROR ThreadId(01) crates/ironrdp-client/src/gui.rs:222: error=Error { context: "ChannelJoinConfirm", kind: Reason("unexpected requested_channel_id in MCS Channel Join Confirm: received 1004, got 1002"), source: None }
Connection error: [ChannelJoinConfirm] reason: unexpected requested_channel_id in MCS Channel Join Confirm: received 1004, got 1002

For some reason, this version does not create the log file, I had to redirect the console output.
ironrdp-client.log

I'm still unsure about: how to force XRDP to use CredSSP.

I have also tested with the Raspberry Pi 5. The message is different, but still no connection.
ironrdp-client-pi.log

It works on a Windows client with a warning:

[CBenoit]

@mdenty Thank you for following up!

For some reason, this version does not create the log file, I had to redirect the console output.

Oh, this was changed recently in #330.
It’s still possible to log into a file by specifying the --log-file option.

I'm still unsure about: how to force XRDP to use CredSSP.

XRDP doesn’t support CredSSP (NLA). But this should not prevent IronRDP from connecting, because we also support not using NLA ("TLS alone" mode).

It works on a Windows client with a warning

Looks good. The warning is because XRDP is likely using a self-signed certificate that is (rightfully) not trusted by your Windows machine. One way to fix this is to install the certificate on your (client) Windows machine. (An organization should typically use a certificate issued by their own CA, or a well-known CA.)

So we are back with

[ChannelJoinConfirm] reason: unexpected requested_channel_id in MCS Channel Join Confirm: received 1004, got 1002

You confirmed that everything is working when using FreeRDP and the official Microsoft client, so IronRDP may be a bit too strict.

The check is here:

IronRDP/crates/ironrdp-connector/src/channel_connection.rs

Lines 183 to 190 in 33c66ed

	if channel_id != channel_join_confirm.requested_channel_id {
	return Err(reason_err!(
	"ChannelJoinConfirm",
	"unexpected requested_channel_id in MCS Channel Join Confirm: received {}, got {}",
	channel_id,
	channel_join_confirm.requested_channel_id,
	));
	}

Do you think you could try to remove it ocally and see what happens?
You can decrypt the TLS payload and record a pcap of the communication by following this guide: https://github.com/awakecoding/wireshark-rdp
However, it may not be safe to post the full pcap here the client is assuming the traffic is encrypted at this point and may send sensitive information to the server. (Though, to my knowledge, it doesn’t yet at this point.)

It’s safe to send the packets from the "Channel Connection" steps, described below:

Ideally I need to see what we have in the "attach user" and "join" packets.

Thank you!

[mdenty]

Do you think you could try to remove it ocally and see what happens?

It fails on the next test (line 192) with this message:

2024-01-05T11:07:59.115905Z ERROR ThreadId(01) crates/ironrdp-client/src/gui.rs:222: error=Error { context: "ChannelJoinConfirm", kind: Reason("unexpected channel_id in MCS Channel Join Confirm: received 1004, got 1002"), source: None }
Connection error: [ChannelJoinConfirm] reason: unexpected channel_id in MCS Channel Join Confirm: received 1004, got 1002

I've also removed this test and it now fails with this error message:
2024-01-05T11:12:04.601751Z ERROR ThreadId(01) crates/ironrdp-client/src/gui.rs:222: error=Error { context: "decode_send_data_indication", kind: Reason("received disconnect provider ultimatum: UserRequested"), source: None }

The log with both tests disabled:
ironrdp-client.log

I'll try to decode the stream, but I think I already tried and the macOS version of WireShark crashes in this case. I'll have to install it on the Linux side.

EDIT: Here is the pcap:
ironrdp-pcap.zip

[CBenoit]

EDIT: Here is the pcap: ironrdp-pcap.zip

Thank you! However, it appears that the keys file included in the archive is protected by password so I can’t use it.

[mdenty]

It's not. The zip was created under Ubuntu. I can open it with no issue on my Mac.
To avoid unnecessary ping-pong, here are the keys :

CLIENT_HANDSHAKE_TRAFFIC_SECRET d3b940e82f0831f7908e1791d5346bfb9d6c668b1ee4fb1750357041ddb1fada f1339a87aee8cbdb6557d51d79e38848d1f534711ceaabffb3e2e1f17547f29461796016caca2abb1e8aaa387bf30123
SERVER_HANDSHAKE_TRAFFIC_SECRET d3b940e82f0831f7908e1791d5346bfb9d6c668b1ee4fb1750357041ddb1fada f0a3a02c90a73ec95a127ce81660763606059fb7ea10b81111a185bcf74e3d7304a408a149c2ca8a8200eb4084fe6ff7
CLIENT_TRAFFIC_SECRET_0 d3b940e82f0831f7908e1791d5346bfb9d6c668b1ee4fb1750357041ddb1fada 7be75e974e6fb8c5debb8ee7f4804e39a986c1267808db8d39596ec5c4ec45df7135ee6620b88c5afbedd6b5ae7b9ae0
SERVER_TRAFFIC_SECRET_0 d3b940e82f0831f7908e1791d5346bfb9d6c668b1ee4fb1750357041ddb1fada 0806d1db8fadf04dac38e404ff58ab1ac33c01b45e2ec6f172c7c716c151dd946338e7ce1ccc5fd0102f00e3cd128ed4
EXPORTER_SECRET d3b940e82f0831f7908e1791d5346bfb9d6c668b1ee4fb1750357041ddb1fada db9f864d9d1b07b78286fde5e7f38cd34b08c47386a04186109181f70fe8c7606d9aa7ab52dfcba7aefcc4ba86e983c4

[CBenoit]

Oh, thank you! Yes, I think I went to the wrong tab in Wireshark. Apologies.

We can see in the trace that XRDP is sending the confirmation message without waiting for the request:

However, IronRDP expects the confirmation for a given ID only after sending a join request for it.
The official Microsoft server is properly sending the confirmation after receiving the request.
I think XRDP might be doing this to further minimize the overall connection sequence time by sending the confirmations ahead of time.

Incidentally, we had an open issue for sending the join requests in batch and then handling the confirmation in any order: #112
I implemented that in #347.
This should fix the problem with XRDP at the same time. Do you think you could give a try again?

PS: The user channel must also be joined in addition to the "attach user" messages. IronRDP was already doing that as expected. My bad, I misremembered that part.

[mdenty]

I tried to connect after checkout of branch batch-join-requests.

This leads to:

Connection error: [decode_send_data_indication] reason: received disconnect provider ultimatum: UserRequested

Here is the client side log:
ironrdp-client.log

And the server side log:

[20240109-10:12:15] [INFO ] Socket 12: AF_INET6 connection received from ::ffff:192.168.42.1 port 60910
[20240109-10:12:15] [INFO ] Connected client computer name: macbookpro.loca
[20240109-10:12:15] [INFO ] xrdp_load_keyboard_layout: Keyboard information sent by the RDP client, keyboard_type:[0x04], keyboard_subtype:[0x00], keylayout:[0x00000000]
[20240109-10:12:15] [INFO ] xrdp_load_keyboard_layout: model [] variant [] layout [us] options []
[20240109-10:12:15] [INFO ] TLS connection established from ::ffff:192.168.42.1 port 60910: TLSv1.3 with cipher TLS_AES_256_GCM_SHA384
[20240109-10:12:15] [ERROR] received wrong flags, likely decrypt not working
[20240109-10:12:15] [ERROR] xrdp_sec_recv: xrdp_sec_process_logon_info failed
[20240109-10:12:15] [ERROR] xrdp_rdp_recv: xrdp_sec_recv failed
[20240109-10:12:15] [ERROR] libxrdp_process_data: xrdp_rdp_recv failed
[20240109-10:12:15] [ERROR] xrdp_process_data_in: xrdp_process_loop failed

Off topic:
Is there a way to connect to a remote via a gateway?
I have a use case that use a RDP file to fix among other things the following entries:

gatewayusagemethod:i:2
gatewayprofileusagemethod:i:1
gatewaycredentialssource:i:0
full address:s:remote_bk.example.com
gatewayhostname:s:remote_gw.example.com
workspace id:s:remote_bk.example.com
use redirection server name:i:1
loadbalanceinfo:s:tsv://MS Terminal Services Plugin.1.EXAMPLE
use multimon:i:1
alternate full address:s:remote_bk.example.com

How can I do that with IronRDP?

[CBenoit]

Sounds like XRDP doesn’t like something about the Client Info PDU we are sending.

[20240109-10:12:15] [ERROR] received wrong flags, likely decrypt not working

IronRDP is sending these flags: DISABLE_CTRL_ALT_DEL | UNICODE | LOGON_NOTIFY | LOGON_ERRORS | NO_AUDIO_PLAYBACK | VIDEO_DISABLE.

VIDEO_DISABLE sounds suspicious. 🤔

Do you think you could investigate this further by playing a bit with the flags?

Is there a way to connect to a remote via a gateway?

Support for RD Gateway has not yet been evaluated, so it’s probably not possible.  However, it would be possible to use a Devolutions Gateway, though I’m not sure this is something you are interested about.
Side note: if you are interested for adding support for RD Gateway to IronRDP, I would welcome any PR for that.

[mdenty]

IronRDP is sending these flags: DISABLE_CTRL_ALT_DEL | UNICODE | LOGON_NOTIFY | LOGON_ERRORS | NO_AUDIO_PLAYBACK | VIDEO_DISABLE.

VIDEO_DISABLE sounds suspicious. 🤔

I tried to disable, one by one, all the flags in crates/ironrdp-connector/src/connection.rs line 736. Same result.

---

it would be possible to to use a Devolutions Gateway

My use case implies that I use a customer's (out of my hands) RDP server AND gateway.
I have not looked at your gateway software, but I doubt it will solve this use case, will it?

[CBenoit]

IronRDP is sending these flags: DISABLE_CTRL_ALT_DEL | UNICODE | LOGON_NOTIFY | LOGON_ERRORS | NO_AUDIO_PLAYBACK | VIDEO_DISABLE.

VIDEO_DISABLE sounds suspicious. 🤔

I tried to disable, one by one, all the flags in crates/ironrdp-connector/src/connection.rs line 736. Same result.

Maybe another flag is missing? The error shown in XRDP logs is not helping me much.
I guess I would need to setup an XRDP server on my side in order to test further. I may not have time for the foreseeable future though.
It would help to inspect an actual payload sent by mstsc, but it doesn’t support dumping of the TLS secrets as IronRDP, so it might be harder to put our hands on it.
One approach could be to connect mstsc to the IronRDP server example and dump the received packets using the Debug representation.

it would be possible to to use a Devolutions Gateway

My use case implies that I use a customer's (out of my hands) RDP server AND gateway. I have not looked at your gateway software, but I doubt it will solve this use case, will it?

I can’t tell for sure as I don’t know your use case and other constraints, but many of our customers, including MSPs, are using Devolutions Gateway as a replacement for RD Gateway, and some, in turns, install Gateways in their customers’ infrastructure.