Designing a new keyboard protocol

[denelon]

Based on discussion in:

• Add keyboard protocol support (the kitty one) #11509

Is there any interest in designing and implementing a new keyboard protocol for inclusion in Windows Terminal?

I'd like to start that discussion here to see if others are interested, and if there are, a new GitHub project can be created to collaborate, and once there is an implementation available, it can be included in Windows Terminal. At that point in time someone can create a new issue to have it added to Windows Terminal.

[o-sdn-o]

We need to add to the discussion description a list of keyboard events (text input events) that we would like to track in the system. Perhaps this will be just a list of all keyboard text input events/states available in the operating system. A list of capabilities of existing keyboard protocols will also help the discussion.

To be the protocol cross-platform, a list of operating systems (platforms) and their list of keyboard text input system caps is also required.

Due to the fact that mouse events (depending on keyboard modifiers), keyboard focus state, keyboard layout state, IME state (Hello TSF and Alt+Numpad), clipboard activity (e.g. paste/receive unmodified text block), drag&drop events, gesture events, closing/shutdown/logoff events, display power-saving state, viewport size/state events, etc. are closely coupled to text input, the keyboard protocol must be generalized to the Input Protocol or VT Input Protocol or something like that, and should support tracking of all related input events.

Unfortunately, any keyboard protocol without tracking related events will be defected in general, here is a minor example … (click to expand)

of the need to pass an IME composition preview to a console application.

 

• In the first video, the terminal, not knowing the layout of the text input fields in the application, piles up text on top of the entire viewport without respecting the application layout:

screen_record_Thu_08.29.2024_15-20-36.74.mp4

• In the second video, the application receives IME composition preview with synchronization of all necessary states and everything remains inside the application input fields:

screen_record_Thu_08.29.2024_15-21-42.08.mp4

Perhaps a full-fledged text input protocol should not depend on the type of application, be it CUI, TUI or GUI. It is intended to shape up events and states. VT transport is a secondary (insignificant) matter. It must be assumed that the terminal is a kind of vt-driven widget toolkit (List of widget toolkits).

[zadjii-msft]

Previous discussions on the topic:

• Raw keyboard input (terminal-wg/specifications#1)
• VT Input Mode Protocol (terminal-wg/specifications#35)
• libtickit's "fixterms" protocol
• the kitty protocol
• win32-input mode, (originally tracked in Scenario: Improved keyboard handling #4999)