enhancement(LorieView.java): improve input handling

[twaik]

Making LorieView handle korean/chinese/turkish/other languages requiring composition and composed symbols.
@knyipab can you please test this? It replaces #620 solution.

@813ethan @ilhan-athn7 @robertkirkman @hansm629 can you please test this too?

[hansm629]

@twaik
Here is the test result video.

There is still a delay when typing in Korean,
and there is also a symptom where the text is entered as if it is automatically completed.

SHANA.korean.input.mp4

[twaik]

The solution is created for software keyboard, I did not try to fix hardware keyboard.
Also in the case if you use "prefer scancodes when possible" input will be handled by fcitx or similar mechanism in linux, not by my code.

[hansm629]

@twaik
Does that mean testing should be conducted using the Android software keyboard instead of a USB or Bluetooth hardware keyboard?

[robertkirkman]

In Android Studio if I click the Debug button using this PR, then reproduce the crash, the debugger does not seem to print any backtrace, but this appears in adb logcat, when I type any character in Chromium inside a Termux:X11 app built from this PR using Gboard, and I can confirm that with this PR I am able to type "Ç" but Backspace makes the app close, but without this PR I cannot type "Ç" but backspace does not make the app close.

logcat snippet

2024-12-11 08:24:05.312 17579-17579 DEVICES                 com.termux.x11                       D  found device "hall" sources 0x80000000
2024-12-11 08:24:05.312 17579-17579 DEVICES                 com.termux.x11                       D  requesting stylus false
2024-12-11 08:24:05.312 17579-17579 DEVICES                 com.termux.x11                       D  external keyboard connected false
2024-12-11 08:24:05.345 17579-17579 ViewRootIm...nActivity] com.termux.x11                       I  Relayout returned: old=(0,0,1440,2960) new=(0,0,1440,2960) req=(1440,2960)0 dur=16 res=0x1 s={true 538304110592} ch=false
2024-12-11 08:24:10.778 17579-17579 LorieNative             com.termux.x11                       D  Parsing text: æ
2024-12-11 08:24:10.778 17579-17579 LorieNative             com.termux.x11                       D  Sending unicode event: æ (U+E6)
2024-12-11 08:24:12.181 17579-17579 AndroidRuntime          com.termux.x11                       D  Shutting down VM
2024-12-11 08:24:12.203 17579-17579 AndroidRuntime          com.termux.x11                       E  FATAL EXCEPTION: main
                                                                                                    Process: com.termux.x11, PID: 17579
                                                                                                    java.lang.IndexOutOfBoundsException: replace (0 ... -1) has end before start
                                                                                                    	at android.text.SpannableStringBuilder.checkRange(SpannableStringBuilder.java:1328)
                                                                                                    	at android.text.SpannableStringBuilder.replace(SpannableStringBuilder.java:523)
                                                                                                    	at android.text.SpannableStringBuilder.delete(SpannableStringBuilder.java:231)
                                                                                                    	at android.text.SpannableStringBuilder.delete(SpannableStringBuilder.java:38)
                                                                                                    	at com.termux.x11.LorieView$1.delete(LorieView.java:82)
                                                                                                    	at android.view.inputmethod.BaseInputConnection.deleteSurroundingText(BaseInputConnection.java:252)
                                                                                                    	at com.android.internal.view.IInputConnectionWrapper.executeMessage(IInputConnectionWrapper.java:450)
                                                                                                    	at com.android.internal.view.IInputConnectionWrapper$MyHandler.handleMessage(IInputConnectionWrapper.java:89)
                                                                                                    	at android.os.Handler.dispatchMessage(Handler.java:107)
                                                                                                    	at android.os.Looper.loop(Looper.java:237)
                                                                                                    	at android.app.ActivityThread.main(ActivityThread.java:8167)
                                                                                                    	at java.lang.reflect.Method.invoke(Native Method)
                                                                                                    	at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:496)
                                                                                                    	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1100)
2024-12-11 08:24:12.302 17579-17579 Process                 com.termux.x11                       I  Sending signal. PID: 17579 SIG: 9
2024-12-11 08:24:12.403  1322-1841  InputDispatcher         pid-1322                             E  channel '8bb50eb com.termux.x11/com.termux.x11.MainActivity (server)' ~ Channel is unrecoverably broken and will be disposed!
2024-12-11 08:24:12.465  1322-3564  WindowManager           pid-1322                             E  win=Window{8bb50eb u0 com.termux.x11/com.termux.x11.MainActivity EXITING} destroySurfaces: appStopped=false win.mWindowRemovalAllowed=true win.mRemoveOnExit=true win.mViewVisibility=0 caller=com.android.server.wm.AppWindowToken.destroySurfaces:1249 com.android.server.wm.AppWindowToken.destroySurfaces:1230 com.android.server.wm.WindowState.onExitAnimationDone:5189 com.android.server.wm.-$$Lambda$01bPtngJg5AqEoOWfW3rWfV7MH4.accept:2 java.util.ArrayList.forEach:1262 com.android.server.wm.AppWindowToken.onAnimationFinished:3941 com.android.server.wm.AppWindowToken.commitVisibility:914 
2024-12-11 08:24:12.917   904-999   BufferQueueProducer     pid-904                              E  [SurfaceView - com.termux.x11/com.termux.x11.MainActivity@23442d7@9#0] dequeueBuffer: BufferQueue has been abandoned
�

termux-info of the testing device

Termux Variables:
TERMUX_APP_PACKAGE_MANAGER=apt
TERMUX_APP__APK_FILE=/data/app/com.termux-afP421Bfk35XjVh1_X0Sdw==/base.apk
TERMUX_APP__APK_RELEASE=F_DROID
TERMUX_APP__APP_VERSION_CODE=1020
TERMUX_APP__APP_VERSION_NAME=0.119.0-beta.1
TERMUX_APP__DATA_DIR=/data/user/0/com.termux
TERMUX_APP__IS_DEBUGGABLE_BUILD=false
TERMUX_APP__IS_INSTALLED_ON_EXTERNAL_STORAGE=false
TERMUX_APP__PACKAGE_NAME=com.termux
TERMUX_APP__PID=21866
TERMUX_APP__TARGET_SDK=28
TERMUX_VERSION=0.119.0-beta.1
TERMUX__SE_FILE_CONTEXT=u:object_r:app_data_file:s0:c33,c257,c512,c768
TERMUX__SE_INFO=untrusted:targetSdkVersion=28:complete
TERMUX__SE_PROCESS_CONTEXT=u:r:untrusted_app_27:s0:c33,c257,c512,c768
TERMUX__UID=10289
TERMUX__USER_ID=0
Packages CPU architecture:
aarch64
Subscribed repositories:
# sources.list
deb https://packages.termux.org/apt/termux-main stable main
# x11-repo (sources.list.d/x11.list)
deb https://packages.termux.org/apt/termux-x11 x11 main
# tur-repo (sources.list.d/tur.list)
deb https://tur.kcubeterm.com tur-packages tur tur-on-device tur-continuous
Updatable packages:
appstream/stable 1.0.4 aarch64 [upgradable from: 1.0.3]
clang/stable 19.1.5 aarch64 [upgradable from: 19.1.4]
cmake/stable 3.31.2 aarch64 [upgradable from: 3.31.1]
code-server/tur-packages 4.95.3 aarch64 [upgradable from: 4.89.1]
command-not-found/stable 2.4.0-54 aarch64 [upgradable from: 2.4.0-52]
firefox/x11 132.0.2-1 aarch64 [upgradable from: 132.0.2]
gst-plugins-bad/stable 1.24.10 aarch64 [upgradable from: 1.24.9-1]
gst-plugins-base/stable 1.24.10 aarch64 [upgradable from: 1.24.9]
gst-plugins-good/stable 1.24.10 aarch64 [upgradable from: 1.24.9-1]
gstreamer/stable 1.24.10 aarch64 [upgradable from: 1.24.9]
json-glib/stable 1.10.6 aarch64 [upgradable from: 1.10.0]
libcompiler-rt/stable 19.1.5 aarch64 [upgradable from: 19.1.4]
libdrm/stable 2.4.124 aarch64 [upgradable from: 2.4.123]
libllvm-static/stable 19.1.5 aarch64 [upgradable from: 19.1.4]
libllvm/stable 19.1.5 aarch64 [upgradable from: 19.1.4]
libopenmpt/stable 0.7.12 aarch64 [upgradable from: 0.7.11]
libpolly/stable 19.1.5 aarch64 [upgradable from: 19.1.4]
librav1e/stable 0.7.1-2 aarch64 [upgradable from: 0.7.1-1]
libsoup3/stable 3.6.1 aarch64 [upgradable from: 3.4.4]
lld/stable 19.1.5 aarch64 [upgradable from: 19.1.4]
lldb/stable 19.1.5 aarch64 [upgradable from: 19.1.4]
llvm-tools/stable 19.1.5 aarch64 [upgradable from: 19.1.4]
llvm/stable 19.1.5 aarch64 [upgradable from: 19.1.4]
llvmgold/stable 19.1.5 aarch64 [upgradable from: 19.1.4]
luanti/x11 1:5.10.0-4 aarch64 [upgradable from: 1:5.10.0-3]
mediainfo/stable 24.12 aarch64 [upgradable from: 24.11]
mesa-dev/stable 24.2.8 all [upgradable from: 24.2.7]
mesa/stable 24.2.8 aarch64 [upgradable from: 24.2.7]
mlir/stable 19.1.5 aarch64 [upgradable from: 19.1.4]
openal-soft/stable 1.24.1 aarch64 [upgradable from: 1.24.0]
openjpeg/stable 2.5.3 aarch64 [upgradable from: 2.5.2]
qt5-qtbase/x11 5.15.16 aarch64 [upgradable from: 5.15.14-2]
qt5-qtsvg/x11 5.15.16 aarch64 [upgradable from: 5.15.14-1]
qt5-qtx11extras/x11 5.15.16 aarch64 [upgradable from: 5.15.14-1]
ruby/stable 3.3.6 aarch64 [upgradable from: 3.3.5]
rust-std-aarch64-linux-android/stable 1.83.0-2 all [upgradable from: 1.82.0]
rust/stable 1.83.0-2 aarch64 [upgradable from: 1.82.0]
toxic/stable 0.16.0 aarch64 [upgradable from: 0.15.1]
tree-sitter/stable 0.24.5 aarch64 [upgradable from: 0.24.4]
unrar/stable 7.1.2 aarch64 [upgradable from: 7.1.1]
vim/stable 9.1.0900-1 aarch64 [upgradable from: 9.1.0850]
vulkan-headers/stable 1.4.303 all [upgradable from: 1.3.302]
vulkan-loader-generic/stable 1.4.303 aarch64 [upgradable from: 1.3.302]
vulkan-tools/stable 1.4.303 aarch64 [upgradable from: 1.3.302]
termux-tools version:
1.44.5
Android version:
10
Kernel build information:
Linux localhost 4.9.186-22990479 #1 SMP PREEMPT Thu Feb 24 18:21:21 KST 2022 aarch64 Android
Device manufacturer:
samsung
Device model:
SM-G960U
LD Variables:
LD_LIBRARY_PATH=
LD_PRELOAD=/data/data/com.termux/files/usr/lib/libtermux-exec.so
Installed termux plugins:
com.termux.api versionCode:51
com.termux.x11 versionCode:15

[twaik]

Does that mean testing should be conducted using the Android software keyboard instead of a USB or Bluetooth hardware keyboard?

Software keyboard only.

I can confirm that with this PR I am able to type "Ç" but Backspace makes the app close

Can you please try again?

[twaik]

@robertkirkman And what software keyboard layout did you use?

[twaik]

I mean I used gboard and it was fine on my device.

[twaik]

It lets us use cursor control, keyboard suggestions (with word replacings!) and some other stuff.

Screen_Recording_20241211_172139_TermuxX11.mp4

[twaik]

Korean input seems to be a bit more weird.

Screen_Recording_20241211_172818_TermuxX11.mp4

[hansm629]

@twaik
Should I also test this test with the option off?

Hardware keyboard scancodes workaround

Screen_Recording_20241212_005155_TermuxX11.mp4

[knyipab]

@twaik
Have you ever try to use backspace in Gboard English and other non-CJK langauges (with autocomplete and word suggestion)? The IME incorrectly think that it is working on some composed text that is yet to be commited, making backspace not functioning well.

This is the main reason why #620 needs to test the language used for Gboard and decide the input type dynamically. It would great if such behaviour can be suppressed.

[ilhan-athn7]

@813ethan @ilhan-athn7 @robertkirkman @hansm629 can you please test this too?

The cursor movement isn't usable on LatinIME
But this one works flawlessly on my end.

[twaik]

The cursor movement isn't usable on LatinIME

I did not find cursor control feature there. Probably it is not available there.

[robertkirkman]

@robertkirkman And what software keyboard layout did you use?

I used GBoard in Turkish mode, and now I have tested the current version, and the crash is not happening anymore in the same test, Backspace is now working. I believe the update you created fixed the problem.

Now I will describe a second problem that I noticed, which I believe I noticed happening at least once in both the original and the revised versions of this PR, and I am not sure how important this is because it seems like it only happens in Chromium and when I try to reproduce it in Firefox it never occurs, or at least I have not noticed it happening for approximately 20 minutes of testing firefox. Since Chromium is in tur-repo but Firefox is in x11-repo, I am not completely sure how high-priority this would be since it could be specifically just an issue with Chromium. What happens is, very occasionally, a symbol will appear that is different from the symbol that I touched, but only for one key stroke, and then the next touch of the same button produces the correct symbol.

Here is a video of that happening, at the timestamp in the video where it happened, 2:25, where for one key stroke, "é" became "í", and then "é" appeared on the next touch of the same button. https://youtu.be/jAVW-7wSjug?si=Fo8O9Hv0lQyf9yua&t=145

[ilhan-athn7]

The cursor movement isn't usable on LatinIME

I did not find cursor control feature there. Probably it is not available there.

screen-20241212-165829.mp4

[twaik]

It is not the same IME I have.

[813ethan]

seems fine with gboard chinese handwriting and cantonese input for now

logs like this appeared while typing tho, there seems to be one keysym log every character typed

(II) Added unknown keysym 0x10089ba to keycode 202
(II) Added unknown keysym 0x1005f97 to keycode 184
(II) Added unknown keysym 0x1005f88 to keycode 183

[813ethan]

the Deactivate special keys on additional key bar after each key press option is also not working on this pr

[twaik]

I think I need more data to reproduce "slow typing" and "autocomplete". I need output of termux-x11-preference, apk/link to google play/f-droid of IME you use, name of layout, other factors that probably will help me.

[twaik]

there seems to be one keysym log every character typed

Correct, it is the standard behaviour for Unicode input handling.

[twaik]

the Deactivate special keys on additional key bar after each key press option is also not working on this pr

Should be fixed now.

The cursor movement isn't usable on LatinIME

Should be fixed too but I am not sure about this.

[813ethan]

nice, seems fine for now with gboard

[ilhan-athn7]

The cursor movement isn't usable on LatinIME

Should be fixed too but I am not sure about this.

It doesn't, it also brings back the previuos bug, where i can't use Backspace and few letter keys

[twaik]

Probably I should check what happens with your IME. Can you send the apk of your keyboard?

[twaik]

@hansm629 I checked korean input with external keyboard and now I understand what you mean by saying "slow input". That is because of symbol composition, pending symbol was displayed at the bottom of screen. But it seems like I found some simple and pretty much dumb way to avoid using the composition displaying. Check the latest artifact.

[twaik]

@813ethan probably I need your confirmation too because I checked only pinyin 3x4 of samsung keyboard, I am not sure if it will work on other IMEs.

[813ethan]

seems working fine

gboard yueping/cantonese pinyin

screen-20250114-232850.mp4

[twaik]

Lol. Current solution simply erases old text (with backspace key) Android thinks we compose and replaces it with new text. Combined with input delay it shows interesting result.

Screen_Recording_20250114_172844_TermuxX11.mp4

[813ethan]

also autocomplete is not always working?

screen-20250115-123102.mp4

[twaik]

Yeah, seems so. I am already on it. Is it gboard?

[813ethan]

yes gboard

[twaik]

This build should fix this.

[twaik]

Have you ever try to use backspace in Gboard English and other non-CJK langauges (with autocomplete and word suggestion)? The IME incorrectly think that it is working on some composed text that is yet to be commited, making backspace not functioning well.

@knyipab Seems to be fixed now.

[813ethan]

bug seems to be still happening

[twaik]

Probably I'll disable predictive typing and word suggestion completely since it does not seem to be possible to make it work fine even on most common keyboards in our case (without direct access to ibus/fcitx/xim servers).

[ilhan-athn7]

Probably I'll disable predictive typing and word suggestion completely since it does not seem to be possible to make it work fine even on most common keyboards in our case (without direct access to ibus/fcitx/xim servers).

this scenario can be handled by adding a switch in preferences, not sure if it worth the effort

[twaik]

It is not worth the effort to make it work.

[knyipab]

It works great! Few issues from my test:

• when using gboard Chinese IME + English IME with physical keyboard (with Gboard option "Physical keyboard" -> "Show on-screen keyboard" off), pressing SHIFT key will stuck the SHIFT key in Termux:X11.
• when using gboard English IME with physical keyboard, Ctrl + Backspace deletes more words than expected which can be annoying. May be related to the current approach to compose text.

[twaik]

Ok, seems like there is no way to disable text prediction and suggestions except using InputType.TYPE_NULL or old, "dumb" keyboard option, which will block people from using CJK languages... Sad. Probably I should implement some fake & dumb cursor tracking to make it work fine.

[twaik]

• when using gboard English IME with physical keyboard, Ctrl + Backspace deletes more words than expected which can be annoying. May be related to the current approach to compose text.

It is so dumb... I found it out. When you do CTRL+Backspace Android sends both Press and Release events for CTRL, but only Press event for Backspace, forgetting to send backspace releasing event. I've found a workaround, starting delayed sending key release event in dispatchKeyEventPreIme (before IME consumes it) and cancelling it in dispatchKeyEvent in the case if actually did not consume it...

when using gboard Chinese IME + English IME with physical keyboard (with Gboard option "Physical keyboard" -> "Show on-screen keyboard" off), pressing SHIFT key will stuck the SHIFT key in Termux:X11.

I think same applies here.

[twaik]

Ok, I hope I did not break anything. Also I added a switch to disable both autosuggestions/predictive text and CJK languages (because I can not disable them separately). Should work fine but I am not so sure about this.