[parley] new selection logic and an editor example #106

dfrg · 2024-08-15T01:32:32Z

Adds a new Selection type for dealing with cursor movement and selection.

Also includes a vello_editor example that serves as a testbed for checking the logic. This supports mouse selections, arrow keys (+ shift to extend selections), backspace, delete and basic text input.

Marked as draft because the code is a mess but making this public so people can play with it and report bugs.

Shows how to implement the next/previous line behaviors for cursors. Note that this is untested.

nicoburns

Just wanted to add that h_pos ought to be remembered such that the cursor will return back to it's original h pos when moving into a long enough line after moving through short (or empty) lines.

nicoburns · 2024-08-15T02:10:10Z

parley/src/layout/cursor.rs

+    Some(Cursor::from_point(
+        layout,
+        h_pos.unwrap_or(cursor.offset),
+        y,
+    ))


A bit silly to be working out the y pos from the line and then Cursor::from_point presumably turning that back into a line, but it'll certainly work.

Agreed. In theory, I'd rather see it use the data we already know will just be re-found to be the index we have available. But this is fine as-is.

I disagree. This stuff is really complex (the current code in masonry gets much of it wrong and will be worse with mixed direction text). Once you have fast and robust index/point primitives, you want to use them as much as possible.

Line movement is fundamentally a hit test operation anyway, given the expectation of maintaining the visual horizontal position.

I mean sure. It would be an optimisation to skip hit testing in the y direction because you already know which line you will hit (right?). If you think it's better not to do that to keep the code simpler then that seems fine. It's not something I feel particularly strongly about (and would also be something that would be easy to change later (perhaps when the codebase is more mature and better tested))

Point taken, we could factor the horizontal search out of from_position and reuse it for this.

DJMcNab · 2024-08-15T08:22:58Z

Isn't remembering the h_pos the job of a higher-level API?

DJMcNab

If you're not planning on doing testing for this, we can just land it.

DJMcNab · 2024-08-15T08:26:12Z

parley/src/layout/cursor.rs

+) -> Option<Cursor> {
+    let line = layout.get(line_index)?;
+    let metrics = line.metrics();
+    let y = metrics.baseline - metrics.line_height * 0.5;


Why is this subtraction needed? I can see that it could make things more robust when finding the lines?
That is, does the actual y within the range (metrics.baseline - metrics.line_height)..metrics.baseline have any significance?

Yeah, it’s just for robustness to ensure we choose a y position within the target line.

DJMcNab · 2024-08-15T08:28:32Z

parley/src/layout/cursor.rs

+    Some(Cursor::from_point(
+        layout,
+        h_pos.unwrap_or(cursor.offset),
+        y,
+    ))


Agreed. In theory, I'd rather see it use the data we already know will just be re-found to be the index we have available. But this is fine as-is.

dfrg · 2024-08-15T10:39:39Z

If you're not planning on doing testing for this, we can just land it.

I’m going to update this to include a selection primitive with the basic operations: next, prev, next_line, prev_line and selection rects which I’ve now tested to work properly. Stay tuned

commit 6b11f21 Author: Chad Brokaw <[email protected]> Date: Fri Aug 16 12:31:34 2024 -0400 selection and editing commit 951fc96 Author: Chad Brokaw <[email protected]> Date: Fri Aug 16 10:13:13 2024 -0400 checkpoint commit ecd232d Author: Chad Brokaw <[email protected]> Date: Thu Aug 15 21:30:27 2024 -0400 checkpoint

code is substantially changed

.vscode/launch.json

DJMcNab · 2024-08-16T16:42:11Z

examples/vello_editor/src/text.rs

+                        KeyCode::Backspace => {
+                            let start = if self.selection.is_collapsed() {
+                                let end = self.selection.focus().text_start;
+                                if let Some((start, _)) =


Suggested change

if let Some((start, _)) =

// TODO: Handle Emoji, etc.

if let Some((start, _)) =

Noted but I think this behavior should be an option. I believe some editors (mostly on desktop?) allow you to “decompose” emojis with backspace by removing modifiers. Mobile usually just deletes the whole thing for simplicity.

Parley tracks emoji clusters so adding this here should be significantly simpler than the current code in masonry that tries to parse them manually.

Hmm. Handy that we can use the existing data for this.

Decomposing also needs some care with the zero width joiners, of course.
I presumed that the decomposing behaviour I was seeing was a bug in vscode, but it does seem to be intentional.

Configuring would be good. Of course, it would also be possible to do even better here, but our job isn't to come up with new UX for emoji input/editing :)

Decomposing emoji is highly dependent on specific editor features that I've never seen implemented in a useful way in a real application. On Android this could conceivably be handled by input methods (through corrections), but on desktop there is no standard UI for it.

I think this is something to prototype in Masonry land, it'd be cool to have something like a ‘variants menu’, but also there's a question about keyboard and screen reader interpretation of that.

parley/src/layout/cursor.rs

DJMcNab · 2024-08-16T16:50:20Z

.vscode/launch.json

I'd probably remove this - consider then adding it to a local gitignore?

oops, yes, I missed that

I don't know why GitHub decided to hide the first time I made this comment from me...

DJMcNab

Another overall thought - we have written a lot of docs for very similar code before (e.g. the Selection struct), so I would really like to see some of that at least brought across (where licenses are compatible)

But this is looking really promising - using Cursors for the cursor position is compelling, and I'm annoyed I didn't realise I should have been doing that...

DJMcNab · 2024-08-19T12:03:37Z

Backlink to #52

DJMcNab

I'm really liking the behaviour I see running this. Not sure what the expectation normally is with arrow-keys in bidi text - this matches vscode (but not firefox)

DJMcNab · 2024-08-21T09:16:46Z

parley/src/layout/cursor.rs

+    }
+
+    pub fn next_logical<B: Brush>(&self, layout: &Layout<B>, extend: bool) -> Self {
+        self.maybe_extend(Cursor::from_byte_index(layout, self.focus.text_end), extend)


This doesn't quite work for emergency breaks. Pressing right at:

aaaaaaaaaaaaaaaaaaaaaa<|>a[emergency break] aaaaaa

should give:

aaaaaaaaaaaaaaaaaaaaaaa<|>[emergency break] aaaaaa

but actually gives:

aaaaaaaaaaaaaaaaaaaaaaa[emergency break] <|>aaaaaa

A similar thing happens with bidi text (and would also happen with differently styled rich text).

My understanding is that right now Parley doesn't keep track of cursor affinity. This is not unheard of (chrome doesn't have affinity either) but I also strongly prefer keeping affinity for soft breaks. It's also the behavior of most text editors.

I was under the impression that this logic already correctly has state for affinity, as discussed in #106 (comment)

In the example I have given, end and home work as expected for me, the issue is with the left and right keys.

I do have this working for soft line breaks but not emergency breaks (yet). This new behavior matches Firefox where moving right at the end of the line places the cursor at the beginning of the next (rather than skipping the first cluster). Note that this effectively requires two key presses to move between characters so affinity doesn’t handle this naturally.

PoignardAzur

Overall, this is good progress, and we could definitely use this in Masonry.

The biggest thing I see missing (beside first-class affinity) is word-by-word selection (eg what happens when you do Ctrl+Right). Everything else we can figure out later, but word selection is kinda essential for a first-class editor experience.

PoignardAzur · 2024-08-21T11:11:17Z

examples/vello_editor/src/main.rs

+            // Resize the surface when the window is resized
+            WindowEvent::Resized(size) => {
+                self.context
+                    .resize_surface(&mut render_state.surface, size.width, size.height);
+                render_state.window.request_redraw();
+                self.text.update_layout(size.width as _, 1.0);
+            }


This should also update the coordinates of the cursor/selection.

Right now the cursor position becomes stale when you resize the window.

I added a refresh method to both Cursor and Selection and inserted a call here.

PoignardAzur · 2024-08-21T11:15:23Z

parley/src/layout/cursor.rs

    /// Creates a new cursor for the specified layout and text position.
-    pub fn from_position<B: Brush>(
-        layout: &Layout<B>,
-        mut position: usize,
-        is_leading: bool,
-    ) -> Self {
+    pub fn from_byte_index<B: Brush>(layout: &Layout<B>, mut index: usize) -> Self {


My understanding from reading the code is that, if the byte index you give is in the middle of a grapheme, the cursor returned will basically store a range from the beginning to the end of that grapheme. This property is used by prev_logical where you request the grapheme that contains the byte before the beginning of the current grapheme.

I think that behavior should be described in the doc comment.

PoignardAzur · 2024-08-21T11:19:15Z

parley/src/layout/cursor.rs

+fn move_to_line<B: Brush>(
+    layout: &Layout<B>,
+    cursor: &Cursor,
+    h_pos: Option<f32>,
+    line_index: usize,
+) -> Option<Cursor> {


I'd like this method or one similar to be public. It would be helpful to implement PageUp and PageDown.

Agreed. I’ll just expose a single line motion method with a signed delta

PoignardAzur · 2024-08-21T11:28:45Z

parley/src/layout/cursor.rs

+        Cursor::from_point(layout, x, y).into()
+    }
+
+    pub fn from_byte_index<B: Brush>(layout: &Layout<B>, index: usize) -> Self {


This should take an affinity parameter or similar.

Done in as yet unpushed code

dfrg · 2024-08-21T11:56:59Z

I’ve been hacking away at this and have some rather large changes incoming.

Daniel: the new behavior matches either pango or firefox (cursor moves visually in bidi sections). The difference is how insertion is handled at boundaries. In pango apps, the visual cursor always represents the “strong” insertion point which is where text matching layout direction would appear. In firefox this depends on how you enter the boundary. (edit: this is controllable with an enum)

Olivier: word movement/selection with ctrl (+shift) is done and working including double click to select a word.

dfrg · 2024-08-21T11:58:58Z

I’ve got one tiny bug to fix and then I’ll push the new code for review and we can hopefully get it landed later today

dfrg · 2024-08-21T15:04:49Z

Updated with all of my changes now. There are a few debug prints that I'll remove before merging but I think this is in a pretty good state as far as functionality.

Basic rundown:

Visual cursor movement matches pango by default. I arrived at this decision after discussion with both Olivier and Behdad.
Allows both cursor and mouse navigation to the end of soft line breaks.
Word movement/selection is implemented. This works in logical order so will jump around for bidi text. I was unable to find any existing app that does visual movement here 🤷

There are a million little edge cases to handle and a lot of code changes here to address them. Happy to have suggestions for improvement during review but I'd like to defer substantial changes to both code and docs to future work.

edit: editor example also has copy/paste support

Please test and let me know how it works :)

PoignardAzur · 2024-08-21T15:23:41Z

Tried to run it, got this error:

  = note: /usr/bin/ld: cannot find -lxcb-shape: No such file or directory
          /usr/bin/ld: cannot find -lxcb-xfixes: No such file or directory

dfrg · 2024-08-21T15:31:55Z

Nice, this is probably the old clipboard crate. I'll try updating to clipboard_rs.

PoignardAzur · 2024-08-21T16:37:17Z

Amazing work!

Some semi-blocking problems:

Pressing up on the first line or down on the last line doesn't move the cursor to the beginning/end of the line.
Ctrl+Delete and Ctrl+Backspace aren't implemented.
Delete doesn't work correctly with non-collapsed selections (The text is correctly deleted, but the selection isn't collapsed.)

Some non-blocking nitpicks:

Double-click stops working if you drag the cursor around without releasing your click.
Double-click (select by word) is handled, but triple-click (select by line) isn't.
Ctrl+Home and Ctrl+End aren't implemented.

Despite all that, I think this is basically already usable in Masonry. I'll see if I can write an initial integration this evening.

xorgy · 2024-08-22T00:17:32Z

[target.'cfg(target_os = "android")'.dependencies]
winit = { version = "0.30.3", features = ["android-native-activity"] }

To fix the issue with the examples building for Android. You need to select an Activity type from android-activity to build winit for Android.

xorgy · 2024-08-22T01:02:12Z

You will also need to gate clipboard-rs because it doesn't build for Android.

waywardmonkeys · 2024-08-22T01:12:38Z

For the failed typos check, you can use a variation on this block from the Xilem typos config:

extend-ignore-re = [
    # Matches lorem ipsum text.
    # In general, regexes are only matched until the end of a line by typos,
    # and the repeated matcher at the end of both of these also ensures that
    # matching ends at quotes or symbols commonly used to terminate comments.
    "Lorem ipsum [a-zA-Z .,]*",
    "Phasellus in viverra dolor [a-zA-Z .,]*",
]

dfrg · 2024-08-22T01:12:47Z

You will also need to gate clipboard-rs because it doesn't build for Android.

ah, good catch. Is there any crate that supports this or is it something we'll have to implement?

dfrg · 2024-08-22T01:13:39Z

For the failed typos check, you can use a variation on this block from the Xilem typos config:

Thanks! Will fix this up too

DJMcNab · 2024-08-22T07:41:24Z

ah, good catch. Is there any crate that supports this or is it something we'll have to implement?

I don't think there's much advantage in trying to get it working, because most users on Android won't have a keyboard plugged in anyway.
Although I guess to use this demo, you need to have that plugged in anyway...

xorgy · 2024-08-22T16:03:13Z

Handling for Ime events is one thing that would be good before we go integrating this into Masonry (particularly making sure we can support preedit behavior). I think this can be done as an iteration once this lands though.

- support ctrl+home/end - fix delete with non-collapsed selections - move to start/end of line when maxing out up/down respectively - fix panic on empty text buffer - proper cursor placement when using left/right to collapse selection - fix weird cursor offset bug when line ends with RTL run containing only whitespace that is reordered - triple click to select line

dfrg · 2024-08-22T17:23:20Z

Amazing work!

❤️

Some semi-blocking problems:

Pressing up on the first line or down on the last line doesn't move the cursor to the beginning/end of the line.

Fixed in Selection::move_lines.

Ctrl+Delete and Ctrl+Backspace aren't implemented.

This behavior is sorta weird when the selection is not collapsed so I'm going to defer it.

Delete doesn't work correctly with non-collapsed selections (The text is correctly deleted, but the selection isn't collapsed.)

Good catch. This definitely used to work but it appears I broke it at some point. Fixed.

Some non-blocking nitpicks:

Double-click stops working if you drag the cursor around without releasing your click.

Will also defer this one. Selection::extend_to_point should probably take some sort of granularity.

Double-click (select by word) is handled, but triple-click (select by line) isn't.

Added this.

Ctrl+Home and Ctrl+End aren't implemented.

Added this as well.

Despite all that, I think this is basically already usable in Masonry. I'll see if I can write an initial integration this evening.

🎉

dfrg · 2024-08-22T17:24:20Z

Handling for Ime events is one thing that would be good before we go integrating this into Masonry (particularly making sure we can support preedit behavior). I think this can be done as an iteration once this lands though.

Adding this to the editor example would be valuable. I'll leave that to others :)

This line may have been mistakenly removed in linebender#106 (14070d5). Fix linebender#114.

This line may have been mistakenly removed in #106 (14070d5). Fix #114.

dfrg added 2 commits August 14, 2024 21:32

Add next_line/prev_line methods to Cursor

60b99d1

Shows how to implement the next/previous line behaviors for cursors. Note that this is untested.

swap ops

54d08a0

nicoburns reviewed Aug 15, 2024

View reviewed changes

DJMcNab previously approved these changes Aug 15, 2024

View reviewed changes

dfrg marked this pull request as draft August 16, 2024 16:33

dfrg changed the title ~~Add next_line/prev_line methods to Cursor~~ [parley] new selection logic and an editor example Aug 16, 2024

dfrg added 3 commits August 16, 2024 12:42

fix ci

e8ab0eb

copyright header

1d2402e

no_std

0938434

DJMcNab reviewed Aug 16, 2024

View reviewed changes

DJMcNab reviewed Aug 17, 2024

View reviewed changes

checkpoint bidi matches chrome impl

ee7f063

dfrg added 2 commits August 20, 2024 22:43

checkpoint visual movement

596a2af

remove experimental code; make clippy happy

f380d80

DJMcNab approved these changes Aug 21, 2024

View reviewed changes

PoignardAzur reviewed Aug 21, 2024

View reviewed changes

dfrg added 3 commits August 21, 2024 10:56

mostly final state

5aceaaa

remove .vscode directory

abc9da8

use system-ui font

d72da0d

dfrg marked this pull request as ready for review August 21, 2024 15:07

dfrg added 2 commits August 21, 2024 11:08

make no_std happy

8e4b199

use cmd on macos; add cut command

c352505

dfrg added 2 commits August 21, 2024 11:32

try clipboard_rs

0639357

ignore spurious cusor move on macos

66979c5

xorgy approved these changes Aug 22, 2024

View reviewed changes

really fix android builds this time

3beb203

dfrg added this pull request to the merge queue Aug 22, 2024

Merged via the queue into main with commit 14070d5 Aug 22, 2024
20 checks passed

dfrg deleted the move-line branch August 22, 2024 17:38

waywardmonkeys mentioned this pull request Sep 9, 2024

swash_render not rendering on main #114

Closed

waywardmonkeys added a commit to waywardmonkeys/parley that referenced this pull request Sep 12, 2024

parley: Fix swash example rendering.

74530c1

This line may have been mistakenly removed in linebender#106 (14070d5). Fix linebender#114.

waywardmonkeys mentioned this pull request Sep 12, 2024

parley: Fix swash example rendering. #115

Merged

github-merge-queue bot pushed a commit that referenced this pull request Sep 12, 2024

parley: Fix swash example rendering. (#115)

7a4cc48

This line may have been mistakenly removed in #106 (14070d5). Fix #114.

	if let Some((start, _)) =
	// TODO: Handle Emoji, etc.
	if let Some((start, _)) =

[parley] new selection logic and an editor example #106

[parley] new selection logic and an editor example #106

Conversation

dfrg commented Aug 15, 2024 • edited Loading

nicoburns left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DJMcNab commented Aug 15, 2024

DJMcNab left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dfrg commented Aug 15, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xorgy Aug 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DJMcNab left a comment

Choose a reason for hiding this comment

DJMcNab commented Aug 19, 2024 • edited Loading

DJMcNab left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PoignardAzur Aug 21, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PoignardAzur left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dfrg Aug 21, 2024 • edited Loading

Choose a reason for hiding this comment

dfrg commented Aug 21, 2024 • edited Loading

dfrg commented Aug 21, 2024

dfrg commented Aug 21, 2024 • edited Loading

PoignardAzur commented Aug 21, 2024

dfrg commented Aug 21, 2024

PoignardAzur commented Aug 21, 2024

xorgy commented Aug 22, 2024 • edited Loading

xorgy commented Aug 22, 2024 • edited Loading

waywardmonkeys commented Aug 22, 2024

dfrg commented Aug 22, 2024

dfrg commented Aug 22, 2024

DJMcNab commented Aug 22, 2024

xorgy commented Aug 22, 2024 • edited Loading

dfrg commented Aug 22, 2024

dfrg commented Aug 22, 2024

dfrg commented Aug 15, 2024 •

edited

Loading

dfrg commented Aug 15, 2024 •

edited

Loading

xorgy Aug 19, 2024 •

edited

Loading

DJMcNab commented Aug 19, 2024 •

edited

Loading

PoignardAzur Aug 21, 2024 •

edited

Loading

dfrg Aug 21, 2024 •

edited

Loading

dfrg commented Aug 21, 2024 •

edited

Loading

dfrg commented Aug 21, 2024 •

edited

Loading

xorgy commented Aug 22, 2024 •

edited

Loading

xorgy commented Aug 22, 2024 •

edited

Loading

xorgy commented Aug 22, 2024 •

edited

Loading