[Feature]: Mouse tracking shouldn't block scrollback

[jace]

Feature description

The xterm control sequence for mouse tracking (code 1000 or 1006) also captures scroll events, reporting them as button codes 64 and 65. This is necessary for apps in alternate screen mode, but hurts usability in regular screen mode because it blocks viewport scrollback.

Termux's behaviour is consistent with other terminal emulators, but may need reconsidering. On a touchscreen, it's far easier to move the cursor in a readline prompt by tapping the screen than by using the soft arrow keys. Many REPL toolkits already have optional mouse support (for example, this zsh plugin), but turning on the feature blocks scrollback, and worse -- an accidental swipe of the screen may invoke unwanted functionality. Eg: the zsh plugin remaps scroll events to command history navigation, so a sloppy tap could accidentally erase user input.

Ideas for how to make this better:

1. Add a preference for mouse tracking: if the display is not in alternate screen mode, turning on mouse tracking will not capture finger dragging as scroll events. Mouse tracking translates dragging with finger down to scroll wheel events #1384 has a conflicting proposal for reporting finger dragging as mouse move events, but I feel these are distinct expectations based on alt screen mode, so both options could co-exist.

2. Implement the passive mouse tracking extension from Contour terminal. (I brought this up in more Mouse Click events #612 too.) This is entirely new functionality however. It's not a stable spec and will need implementations in every REPL, but it'll (hopefully) work across terminal apps as a new modern standard.

Maybe Termux can implement the user preference while waiting for the spec to mature? It seems like an easy fix for a huge usability gain. I'm not aware of any non-fullscreen app that benefits from blocking scrollback.

Additional information

Proposed VT extensions from Contour Terminal: https://github.com/contour-terminal/vt-extensions

[jace]

@trygveaa @fornwall: tagging you both since you've commented on the other issues -- hope that's okay!

[trygveaa]

Hm, I'm skeptical to adding a config option to deviate from the existing terminal emulators. Different terminals behaving differently often ends up with a mess for applications that has to have logic based on which terminal it's running in. And this isn't really something a user should decide on, so I don't think a user option is the best way.

The proposal from Contour looks interesting, but I see that it says to report the same coordinates regardless of scroll position. If so, I'm a bit unsure how you would use it to move the cursor in the shell? How would the shell know the difference between clicking on the last line vs. clicking on the second last line with one line scrolled up, given that they would return the same coordinate?

Note that several terminal emulators already implement the feature you're asking for in a different way, without any mouse reporting to the shell. The way they do it is to have the terminal send left/right arrow key sequences to the shell to move the cursor to where you click. This is supported in at least kitty, xterm, iTerm2, Konsole and DomTerm. I think they all work more or less the same way, though I haven't gone into detail on that.

[jace]

Note that several terminal emulators already implement the feature you're asking for in a different way, without any mouse reporting to the shell. The way they do it is to have the terminal send left/right arrow key sequences to the shell to move the cursor to where you click. This is supported in at least kitty, xterm, iTerm2, Konsole and DomTerm. I think they all work more or less the same way, though I haven't gone into detail on that.

This appears to be via control code 2001, defined as "Enable readline mouse button-1". This will be a great addition to Termux and I'm happy to take a stab at a PR -- I have neither Java nor Android dev experience, but this is a compelling enough reason.

From what I can tell, the 2001 mode is only supposed to work if the text cursor is on the same line as the mouse cursor, which is somewhat limiting on portrait-oriented screens. But better than nothing.

I haven't looked carefully at Contour's proposed extensions, but you're right -- it seems to have been imagined around alt-screen apps.

FWIW, I haven't found any justification for why mouse tracking in the main screen should also capture scroll events. Nobody seems to be using it (apart from remapping to up/down), and Termux already reserves long press events for itself in both main and alt screens. Scroll capture seems like one of those accidents of history that no longer deserves to be honoured.

[jace]

Turns out iTerm2 has an option to disable reporting mouse events, but it applies to both main and alt screen. Filed a ticket.

[trygveaa]

This appears to be via control code 2001, defined as "Enable readline mouse button-1". This will be a great addition to Termux and I'm happy to take a stab at a PR -- I have neither Java nor Android dev experience, but this is a compelling enough reason.

It seems to work a bit differently in xterm vs. the other four terminal emulators I mentioned. xterm requires you to print \e[?2001h to enable it, but the others don't.

The others instead require the shell to emit some sequences for the terminal to know where the prompt starts. See https://sw.kovidgoyal.net/kitty/shell-integration/#notes-for-shell-developers and https://iterm2.com/documentation-escape-codes.html for details about this. Some terminals have integrations with the shell to enable this automatically. They also typically have a preference in the terminal settings whether to enable this or not.

I'm not sure how xterm knows where the prompt starts, but maybe it doesn't need to since it only operates on a single line.

Note that I haven't tested iTerm2 and DomTerm, so not completely sure about the behavior there, but I think what I said is correct.

From what I can tell, the 2001 mode is only supposed to work if the text cursor is on the same line as the mouse cursor, which is somewhat limiting on portrait-oriented screens. But better than nothing.

This is only a limitation in xterm, the others work across a multi line prompt. Also note that a line here is until the line break, so even xterm works across a wrapped line, so it shouldn't matter that the screen is portrait-oriented.

FWIW, I haven't found any justification for why mouse tracking in the main screen should also capture scroll events. Nobody seems to be using it (apart from remapping to up/down), and Termux already reserves long press events for itself in both main and alt screens. Scroll capture seems like one of those accidents of history that no longer deserves to be honoured.

Yeah, I can't say that I have a use case for receiving wheel events in the normal screen buffer. It's more about not deviating from existing standards.

Implementing it with mouse events would give you the same problem as I mentioned with contours 2029 though. So moving the cursor wouldn't work (or would work incorrectly) when scrolled up. But I suppose it's an edge case, since the prompt is usually at the bottom, and you're scrolled to the bottom when editing it. I see that neither xterm or kitty allows you to move the cursor with the mouse when you're scrolled up. Konsole does though.

Not sure why all of these terminal emulators went with the approach of sending left/right key events. It seems a bit hacky compared to the shell moving the cursor itself.

[jace]

iTerm has implemented my request and closed the ticket. I went through the commit and I'm not sure I understand if this actually implements the request, so I'll have to wait for a release to test it.

Ghostty has made a public release and I found this in their documentation:

# Enables the ability to move the cursor at prompts by using `alt+click` on
# Linux and `option+click` on macOS.
#
# This feature requires shell integration (specifically prompt marking
# via `OSC 133`) and only works in primary screen mode. Alternate screen
# applications like vim usually have their own version of this feature but
# this configuration doesn't control that.
#
# It should be noted that this feature works by translating your desired
# position into a series of synthetic arrow key movements, so some weird
# behavior around edge cases are to be expected. This is unfortunately how
# this feature is implemented across terminals because there isn't any other
# way to implement it.
cursor-click-to-move = true

OSC 133 is not in ECMA-48 or XTerm Control Sequences, so I had missed it earlier. It's in the iTerm documentation you linked to. (For anyone else reading this, look for the heading "Shell Integration/FinalTerm").

Could this be an option for Termux? It'll have to forego the modifier key (alt/option) and translate a tap between the last FTCS_COMMAND_START and end-of-buffer into arrow key presses.

[trygveaa]

OSC 133 is not in ECMA-48 or XTerm Control Sequences, so I had missed it earlier. It's in the iTerm documentation you linked to. (For anyone else reading this, look for the heading "Shell Integration/FinalTerm").

Yes, this is the escape sequence the shell has to emit that I mentioned in my last comment and linked to the documentation for in kitty and iTerm2.

Could this be an option for Termux? It'll have to forego the modifier key (alt/option) and translate a tap between the last FTCS_COMMAND_START and end-of-buffer into arrow key presses.

As mentioned, this is the way all terminal emulators (that I've seen that has the feature of moving the cursor with the mouse) except xterm do it, so it's probably the way to go.