Touch control #77

XITRIX · 2021-02-03T23:02:55Z

I've made a basic touch-event system, hope it could be useful.

# Conflicts: # library/borealis.mk # library/include/borealis/platforms/switch/switch_platform.hpp

# Conflicts: # library/include/borealis/core/platform.hpp # library/include/borealis/platforms/glfw/glfw_platform.hpp # library/include/borealis/platforms/switch/switch_platform.hpp # library/include/borealis/views/view.hpp # library/lib/platforms/glfw/glfw_platform.cpp # library/lib/platforms/switch/switch_platform.cpp # library/lib/views/scrolling_frame.cpp # library/meson.build

natinusala · 2021-02-10T20:25:12Z

Hey, thanks for the PR! I'm sorry I didn't get to review it the first time, seeing that it's pretty large.

It's going to take me some time to process everything, can you maybe explain quickly how your implementation works / how is its architecture? Thanks!

natinusala · 2021-02-10T20:42:42Z

It does not look like your branch is building for Switch, I see glfw / OpenGL stuff being built 🙁

XITRIX · 2021-02-11T19:52:18Z

That's strange... Merged yoga branch right now and checked, make clean / make -j works fine.

natinusala · 2021-02-11T19:54:45Z

You added glfw back to the Switch Makefile it seems

XITRIX · 2021-02-11T20:05:10Z

Oh, I remember, in one of your old commits it was there, and I could not build nro, so I removed it by my self but not commited it, cause I thought you need it for any reason.
Is my last commit building? I checked, where is no glfw in makefile right now.

natinusala · 2021-02-11T20:07:26Z

I was able to try it by removing the glfw line in the Makefile.

It works quite well, good job! Don't forget to add your name to the copyrights if you didn't already 😄

I don't know how far you wanted to go with your code, but there are a few things I noticed:

when we use touch, the highlight doesn't disappear, while it does in HOS, it can be fixed by having a "touch" mode that's enabled when we use touch and disabled when we use the controller
some gestures are a bit sloppy, especially scrolling, it doesn't accelerate when we flick the finger for instance. I think RetroArch has some pretty good acceleration code that's GPL so we can use it!
pressing things only moves the highlight cursor, it doesn't actually "click" the thing. I think it should run the "A" (or "click") action if there's any on the clicked view. If you look on HOS, touching things with the finger actually clicks them.
if you try to touch something that's not fully in view (so if it's clipped by scrolling), you won't be able to touch it, whereas in what you did it becomes focused and scrolls to the center. That's tied to the "fancy" scrolling that's not yet implemented in borealis so we can take care of it later I think.
there's no audio but that's polishing for later 😛

How do you want to proceed? Do you want me to look at the code and review it before changing anything? Or do you want to tackle some of these issues first?

natinusala · 2021-02-11T20:11:47Z

In qlaunch when you touch on things, you need to touch them once to select them THEN touch them another time to click them. So actually borealis views should have two modes: a "touch to click" one and a "touch to highlight, touch another time to click" mode

XITRIX · 2021-02-11T20:20:09Z

I think if you'll review it, it will be better cause I'm new in c++ (I'm c#/swift developer).
I tried to implement something similar like iOS handles touches, and I had a problem with idea of how acceleration on scroll should work so I'll try to find how it works in RetroArch.
I know about current problems it have, cause I just used default giveFocus on items without any modifications. If you think it's ok, and you'd like me to continue touch development, I'll be happy to help with it :)

XITRIX · 2021-02-11T20:24:46Z

Also I have no sounds in borealis at all, do you have any idea of why that could be?

natinusala · 2021-02-11T21:16:24Z

Sound is only implemented on Switch right now, are you trying on Switch? It works for me, does it not work for you?

XITRIX · 2021-02-11T21:34:08Z

Yes, I'm trying on switch with atmosphere 17.1, hos 11.1

natinusala · 2021-02-11T21:36:32Z

Using touch or controller? Do you see anything relevant in the logs?

XITRIX · 2021-02-11T21:40:38Z

Controller, I know that I've not implemented sound with touch. That what inside logs, I'm not sure what qlaunch is, may be it's some kind related:

natinusala · 2021-02-11T21:59:10Z

Is your hbloader old by any chance?

XITRIX · 2021-02-11T22:07:01Z

It's v3.4.0

XITRIX · 2021-02-11T22:08:13Z

oh, it's hbmenu
I'll check right now, but I think it's latest one
upd. no way it's outdated, it was installed with latest atmosphere 18.0 (updated half an hour ago)

HookedBehemoth · 2021-02-11T22:46:37Z

This is a filesystem permission issue. The qlaunch romfs is accessible with hbloader since v2.3.3.

Don't use forwarders.

XITRIX · 2021-02-12T07:02:01Z

I got it, I don't know why, but I cannot run hbmenu by holding R while running any title (albums works fine, but it's an applet), so I created a forwarder for hbmenu. It worked fine when I was on SXOS, but after updating on Atmosphere it stoped working, do you know any reason of why that could happen?

natinusala · 2021-02-12T20:20:06Z

It looks like a botched Atmos install and/or config to me 👀

I'm going to read and review the changes when I can, it's a big PR so it might take me a while to take in lol

natinusala · 2021-03-23T07:41:08Z

Tell me how you would test scrolling without a super long text?

XITRIX · 2021-03-23T12:36:02Z

+1, I used it to test long scrolling view. True, it's on Russia, but I could replace it with lorem ipsum text. It was the first I found in google with "super long text" request.

Also I've found, that there is the problem with cyrillic character "Ц" in glfw version of font, it shows like " ," without "U".

EmmmaTech · 2021-03-23T16:43:49Z

It's understandable if the super long text is for the scrolling, but yeah the text should be changed to the lorem ipsum example.

You could also move the current example to another folder for Russian text.

EmmmaTech · 2021-03-23T16:47:06Z

Tell me how you would test scrolling without a super long text?

Well, I thought an alternative could be a bunch of elements that you scroll down on. For example, a bunch of pictures on the screen, maybe some big buttons that do nothing, rectangles? And yes, I'm aware my example is similar to the first tab in the demo.

XITRIX · 2021-03-24T23:07:15Z

I've changed sound processing a bit. Now I'm not calling play() inside recognizer's callback anymore, now I'm setting sound to the pointer, provided by recognitionLoop() and play it inside Application. If there are several recognizers with custom sound, only the closest to the firstFesponder will be taken.

EmmmaTech · 2021-03-25T00:23:21Z

I wanted to try this out, and I like it. It controls pretty smoothly, even on a Mac. Only one suggestion (might be hard to implement), I wish the scrolling on GLFW platforms was easier to do (what I mean is adding support for the scroll-wheel).

XITRIX · 2021-03-25T08:54:42Z

I thought a lot about scrolling by wheel but I cannot realise how could I do that without any problem for users of Borealis...
The main idea is to allow pan gesture recognizer work with both scenarious, like it works on iOS apps ported on Mac, it's gestures automatically translates to Mac's equivalents.
I think first of all I need to separate mouse from touches and implement separate recognition scenarious for it.

natinusala · 2021-03-26T17:44:54Z

Borealis is not about mouse, it's about controllers and touch. As mouse is yet another "input type" with different control schemes, I wouldn't focus on making mouse usable for now. Le jeu. 25 mars 2021 à 09:55, Vinogradov Daniil ***@***.***> a écrit :

…

I thought a lot about scrolling by wheel but I cannot realise how I could do that without any problem for users of Borealis... The main idea is to allow pan gesture recognizer work with both scenarious, like it works on iOS apps ported on Mac, it's gestures automatically translates to Mac's equivalents. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#77 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAI3CKCXSUXWQTTRSBDG77LTFL26HANCNFSM4XBXJ4ZA> .

XITRIX · 2021-03-28T18:23:19Z

I've added mouse scrolling, tried to use as less code as possible. Review it please.

natinusala · 2021-03-28T19:25:27Z

Is it implemented as part of touch or separately? Because scrolling using the wheel doesn't make sense in a touch context, since... touch screens don't have a mouse wheel. If it works it's fine but it's weird to have a hybrid touch emulation / scroll wheel on a computer.

I would rather implememt mouse as mouse with its own defined behavior and stuff

XITRIX · 2021-03-28T22:03:03Z

As you could guess, I'm using ios and macos as reference. Apple's UIKit has auto adaptation of its gesture recognizers for touch and mouse, so I can use the same recognizer on both platforms. I wanted to implement something like that, but right now I've implemented only scrolling wheel as separate part for mouse. May be I should fully separate their inputs and combine them in single struct i.e. "ScreenInput" and toss it through recognizers, but that will double the code of recognizers, cause then we'll need to handle both touch and mouse. I'm still not sure what will be the best solution, so decided to write something what will not have too many lines of code and will be easy to replace later.

EmmmaTech · 2021-03-28T22:25:32Z

Wow, it works really fluently! It feels like i'm using a web browser, which is good.

natinusala · 2021-03-29T20:45:49Z

@XITRIX And what purpose does it serve at the moment? Does mouse still act as a touch pointer on PC? Isn't it weird to have a half touch half mouse thing?

XITRIX · 2021-03-30T07:59:21Z

You could test it, it works perfectly fine now, like it should work on PC.
Tap works the same on touch and mouse click, Pan also works the same like drag and drop event, and I've made another recognizer - Scroll one, which with touch works the same way as pan, but on PC also handles scrolling wheel. Also I've added NO_TOUCH_SCROLLING define for platforms without touch input, so mouse will not be able to scroll by dragging that window.

natinusala

Here are some more tiny comments, I think I'm starting to get through the review!

Once I go through the whole code I'll try it once again on PC and on Switch to see how it looks and feels, and see if there are little things that need to be tweaked here and there.

library/include/borealis/core/audio.hpp

library/include/borealis/core/input.hpp

library/include/borealis/core/touch/scroll_gesture.hpp

natinusala · 2021-04-02T17:04:12Z

library/include/borealis/core/touch/scroll_gesture.hpp

+ *
+ * Child of Pan recognizer.
+ * The only difference is that Scroll recognizer will translate scrolling wheel to pan gestures.
+ * If NO_TOUCH_SCROLLING=true defined, will ignore scrolling by touch.


Shouldn't the "should touch scroll" boolean be defined in the platform instead of as a define?

I think while it's temporary solution there is no reason to create separate variable for that.
When mouse will be separated from touch, scroll will not trigger by mouse dragging the view.

library/include/borealis/core/touch/tap_gesture.hpp

library/include/borealis/platforms/switch/switch_input.hpp

natinusala · 2021-04-02T17:09:35Z

library/lib/core/application.cpp

@@ -149,6 +150,7 @@ void Application::createWindow(std::string windowTitle)
 bool Application::mainLoop()
 {
    static ControllerState oldControllerState = {};
+    static View* firstResponder;


touchFirstResponder ?

I think it will be used also for mouse , so may be leave it as is?

natinusala · 2021-04-02T17:10:24Z

library/lib/core/application.cpp

+    switch (touchState.phase)
+    {
+        case TouchPhase::START:
+            Logger::debug("Touched at X: " + std::to_string(touchState.position.x) + ", Y: " + std::to_string(touchState.position.y));


We have to remember to remove all debug prints once we're done

natinusala · 2021-04-02T17:11:18Z

library/lib/core/application.cpp

+    {
+        Sound sound = firstResponder->gestureRecognizerRequest(touchState, firstResponder);
+        float pitch = 1;
+        if (sound == SOUND_TOUCH)


IIRC in HOS there are more sounds that have their pitch randomized, did you poke around to see for yourself?

Could you hint me what else sounds have randomized pitch? It was the only sound I heard randomized.

library/lib/core/touch/pan_gesture.cpp

natinusala · 2021-04-02T17:16:00Z

TODO for me after I'm done reviewing the code:

Remember to remove all logs
See if everything behaves likes HOS
See if sounds behave like HOS
Think of how to easily integrate the "tap" gesture to the "click" action we already have, so that it just works (tm)

XITRIX added 8 commits February 1, 2021 01:09

Basic touch event system

831def7

Added Switch implementation

cdfe9dc

Implemented gesture recognizion system

72b07de

Added scroll with PanGestureRecognizer

3cd8f27

Merge branch 'yoga' into Touch-control

a43aa87

# Conflicts: # library/borealis.mk # library/include/borealis/platforms/switch/switch_platform.hpp

Naming fixes

c66fe06

Merge branch 'yoga' into Touch-control

5d1d99d

natinusala added the yoga label Feb 10, 2021

Merge remote-tracking branch 'Natinusala/yoga' into Touch-control

2b0b125

Improvements in recognition states and interruption

35277b2

PR review fixes

8bb51a9

XITRIX force-pushed the Touch-control branch from fb9db80 to 8bb51a9 Compare March 23, 2021 22:46

XITRIX added 5 commits March 24, 2021 01:57

Stop scrolling on tap

8d3ab5e

Merge remote-tracking branch 'origin/main' into Touch-control

2b3b7bf

Changed sound processing

a2f3899

Changed function to Event in tap recognizer

f63d92c

Gesture documentation update

d5035fe

Mouse scrolling

cdd150e

XITRIX added 2 commits March 28, 2021 21:48

Merge branch 'main' into Touch-control

859990c

Fix blink on mouse scroll

c8597c4

Fix scroll failed state chech

9eea766

natinusala requested changes Apr 2, 2021

View reviewed changes

Review fixes

cab3ac6

XITRIX force-pushed the Touch-control branch from c764fa2 to cab3ac6 Compare April 4, 2021 10:06

Merge remote-tracking branch 'origin/main' into Touch-control

1d39f59

Touch control #77

Are you sure you want to change the base?

Touch control #77

Conversation

XITRIX commented Feb 3, 2021

natinusala commented Feb 10, 2021

natinusala commented Feb 10, 2021

XITRIX commented Feb 11, 2021 • edited Loading

natinusala commented Feb 11, 2021

XITRIX commented Feb 11, 2021 • edited Loading

natinusala commented Feb 11, 2021

natinusala commented Feb 11, 2021

XITRIX commented Feb 11, 2021 • edited Loading

XITRIX commented Feb 11, 2021

natinusala commented Feb 11, 2021

XITRIX commented Feb 11, 2021

natinusala commented Feb 11, 2021

XITRIX commented Feb 11, 2021

natinusala commented Feb 11, 2021

XITRIX commented Feb 11, 2021

XITRIX commented Feb 11, 2021 • edited Loading

HookedBehemoth commented Feb 11, 2021

XITRIX commented Feb 12, 2021 • edited Loading

natinusala commented Feb 12, 2021

natinusala commented Mar 23, 2021

XITRIX commented Mar 23, 2021

EmmmaTech commented Mar 23, 2021

EmmmaTech commented Mar 23, 2021

XITRIX commented Mar 24, 2021

EmmmaTech commented Mar 25, 2021

XITRIX commented Mar 25, 2021 • edited Loading

natinusala commented Mar 26, 2021 via email

XITRIX commented Mar 28, 2021

natinusala commented Mar 28, 2021

XITRIX commented Mar 28, 2021

EmmmaTech commented Mar 28, 2021

natinusala commented Mar 29, 2021

XITRIX commented Mar 30, 2021 • edited Loading

natinusala left a comment

Choose a reason for hiding this comment

natinusala Apr 2, 2021

Choose a reason for hiding this comment

XITRIX Apr 4, 2021

Choose a reason for hiding this comment

natinusala Apr 2, 2021

Choose a reason for hiding this comment

XITRIX Apr 4, 2021

Choose a reason for hiding this comment

natinusala Apr 2, 2021

Choose a reason for hiding this comment

natinusala Apr 2, 2021

Choose a reason for hiding this comment

XITRIX Apr 4, 2021

Choose a reason for hiding this comment

natinusala commented Apr 2, 2021

XITRIX commented Feb 11, 2021 •

edited

Loading

XITRIX commented Feb 11, 2021 •

edited

Loading

XITRIX commented Feb 11, 2021 •

edited

Loading

XITRIX commented Feb 11, 2021 •

edited

Loading

XITRIX commented Feb 12, 2021 •

edited

Loading

XITRIX commented Mar 25, 2021 •

edited

Loading

XITRIX commented Mar 30, 2021 •

edited

Loading