UI input refactor

[MechWarrior99]

PR Details

This refactor focuses on the input handling of the Runtime UI system, making it easier to understand, and more consistant.

• Moved picking and input handling out of UIRenderFeature and in to UISystem, this means it now happens in the Update loop instead of the Draw loop.
• Changed naming of Touch and Click to Pointer.
• Added more properties to the PointerEventArgs (formerly TouchEventArgs), to give more control, and in the future allow for UIElements to be able to 'capture' a pointer's input (to allow for better drag actions).
• Added a UIDocument that acts as a container for a UI's state and info required for it. Along with a EntityProcessor for UIComponents to create these and pass them to the UISystem.

Additional Notes

I am not sure about the naming of UIDocument, doesn't quite feel the most descriptive. And the term 'document' feels like it could be used better else where later on perhapse. Maybe in the editor to refer to assets which can be opened, or maybe some other file format. Nothing else was coming to me in the moment though.

Types of changes

• Docs change / refactoring / dependency upgrade
• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist

• My change requires a change to the documentation.
• I have added tests to cover my changes.
• All new and existing tests passed.
• I have built and run the editor to try this change out.

[Doprez]

Are there any things to look out for with the consolidation of mouse and touch inputs? Im personally happy to have one generic pointer event handler as I dont think the actual events between mouse and touch change much.

[MechWarrior99]

Are there any things to look out for with the consolidation of mouse and touch inputs? Im personally happy to have one generic pointer event handler as I dont think the actual events between mouse and touch change much.

Actually it was already consolidated as pointer input, but was refered to as touch, and the touch routed event didn't contain info about the actual pointer. So really, it is just a name change and additional data.
The exception being inside of UpdatePointerOver in UISystem.PointerInput.cs, it uses the input.MousePosition still. In the future I want to change it to use the pointer position, but that felt a little larger scope than I wanted for this PR.

[Kryptos-FR]

Be aware that renaming makes it a breaking change, so this branch cannot be merged until we reach the next major update of Stride.

[Kryptos-FR]

Moved picking and input handling out of UIRenderFeature and in to UISystem, this means it now happens in the Update loop instead of the Draw loop.

What's the rationale? It is not explained in this PR. Is there an issue where this was discussed?

[MechWarrior99]

Moved picking and input handling out of UIRenderFeature and in to UISystem, this means it now happens in the Update loop instead of the Draw loop.

What's the rationale? It is not explained in this PR. Is there an issue where this was discussed?

Ahh, sorry I forgot to go over that. No issue for this, but I'll open(s) for future work. There are a couple of reasons.

1. A system made for handling rendering (the RenderFeature) was also handling user input, so was mixing responsibility.
2. UISystem was handling the keyboard input already, so that meant that UI input was being handled in two seperate locations. Making it harder to find the code you were looking for and felt rather messy (to me).
3. The pointer input was being handled in a Draw method instead of an Update method. I haven't dove deep enough in to the code to know if this truely matters. But having the pointer input be handled in the same location as the key input, and in an update loop instead of a draw loop felt like a better archetecture to me. This is a lesser of the 3 for me.

So this change means that all input is handled in the same class, i nthe same update method, along with any sort of picking related logic and utilities.

[MechWarrior99]

Be aware that renaming makes it a breaking change, so this branch cannot be merged until we reach the next major update of Stride.

Yup, thanks for the clarification. I made sure to mark the checkbox noting the breaking changes. And it is fine, there are other breaking changes to the UI system I want to make later on anyway, so best to do them all at once anyway. I will open issues for it/them a bit later to get a discussion on them going, once I have a more fleshed out plan.

[Kryptos-FR]

Ok so no particular reason to move the code just a feeling. Well there is good reason to do the picking in the rendering loop, that's because it involves the graphics api to do so.

Unless there is a good reason to change that behavior, I'd prefer we don't touch it. As the saying goes, if I ain't broke don't fix it.

Update loop and draw loop might run at different rate, so picking during the update loop could be incorrect and involve issues with multi threading.

@xen2 what's your take on this?

To me this PR is a good example of why it is better to discuss it with the team (either by opening an issue or on our Discord server) before starting to do any work on your own.

Final note: don't get me wrong, I'm all about refactoring if that makes the code cleaner. But we need to make sure it doesn't introduce any regression and is correct in term of architecture.

[Eideren]

@Kryptos-FR @MechWarrior99 did discuss this PR with us, see https://discord.com/channels/500285081265635328/500290856532574209/1283586888409550868
I don't think we should dissuade our users from contributing towards refactoring some of our systems if those make sense. I haven't had time yet to go through the changes this PR introduces.

I did mention over there that picking inside of Draw would be more correct though, but let's check the changes before dismissing this wholesale.

[MechWarrior99]

Hmm, I wouldn't say there isn't a good reason to move it @Kryptos-FR. Mind you I am not sure if the draw and update run at different rates (think they do?). But if the draw loop does run faster than the update loop (where the input events are populated) then it could raise the same event multiple times, which would be a issue for pointer pressed and released events. And if the draw does run slower, it could completely miss inputs, as the input events are cleared at the start of each update by the InputManager. This also means that all pointer events are raised on their UIElements in the draw loop, so any code that responds to those events will also run in the draw loop.

I probably should have said this more explcitly in the PR from the start!

As for graphics usage, while it does use the graphics API, in the end it actually is only the viewport (used to unproject a screen point to world space using a matrix, to get a ray). And the size of the dispaly (used for getting the world view projection matrix of a UIComonent's UI). To me these feel pretty minor compared to the amount of input related code, and for the most part do not require the current render state, so don't need to be in the draw loop.

(Full discloure, I am not completely sure that the way I am getting the viewport is the best in this PR, but from looking around the code base it seemed to be the most correct way to do.)

Looks like Eideren beat me to it. But yeah I did actually ask on discord before starting work, perhapse I should have linked it before, that's my bad. Maybe it is better to make a Issue for these things so they are a little easier to notice and not get hidden?

I do totally get not wanting to touch code that is already working fine, as it can introduce bugs and regressions as you say. But I think not refactoring old code to be easier to work with hampers the code quality and accessability of the project. If after review if its decided that this isn't the way to go about to, so be it.