Input event not emitted when hovering / focused on wry webview child window

[tnthung]

Bevy version

Main branch (on Windows)

What you did

I'm trying to embed the wry webview inside the bevy app as a HUD system for HMR, but input issue kept me from moving forward. Following is the code I used to embed the webview:

use bevy::prelude::*;
use bevy::input::keyboard::KeyboardInput;
use bevy::input::mouse::{MouseButtonInput, MouseMotion};


fn main() {
  App::new()
    .add_plugins(DefaultPlugins
      .set(WindowPlugin {
        primary_window: Some(Window {
          clip_children: false,
          ..Default::default()
        }),
        ..Default::default()
      }))
    .add_systems(Startup, setup)
    .add_systems(Update, (
      print_key_events,
      print_mouse_movement,
      print_mouse_buttons,
    ))
    .run();
}

fn setup(world: &mut World) {
  use bevy::winit::*;
  use bevy::window::*;
  use bevy::ecs::system::SystemState;
  use wry::WebViewBuilder;
  use wry::raw_window_handle::HasWindowHandle;

  let (pw, ws) = SystemState::<(
    Single<Entity, With<PrimaryWindow>>,
    NonSend<WinitWindows>,
  )>::new(world).get(world);

  let Some(window) = ws.get_window(pw.into_inner()) else {
    error!("Cannot find the primary window.");
    return;
  };

  let Ok(handle) = window.window_handle() else {
    error!("Cannot get the window handle.");
    return;
  };

  let Ok(webview) = WebViewBuilder::new()
    .with_transparent(true)
    .with_focused(true)
    .build(&handle)
  else {
    error!("Unable to create webview");
    return;
  };

  world.insert_non_send_resource(webview);
  info!("Webview created");
}

fn print_key_events(mut key_events: EventReader<KeyboardInput>) {
  for event in key_events.read() {
    println!("{:?}", event);
  }
}

fn print_mouse_movement(mut mouse_events: EventReader<MouseMotion>) {
  for event in mouse_events.read() {
    println!("{:?}", event);
  }
}

fn print_mouse_buttons(mut mouse_events: EventReader<MouseButtonInput>) {
  for event in mouse_events.read() {
    println!("{:?}", event);
  }
}

What went wrong

1. When the bevy window is focused, no matter where the mouse is, the mouse motion will always report, except when the mouse is hovering on top of the webview region. When mouse is on top of webview, the mouse motion / mouse button events are not emitting whatsoever. But as long as the mouse is not within the webview rect, mouse related events will start emitting again.

2. If the webview is initially not focused (.with_focused(false)), the keyboard inputs are captured and the events are emitted with no problem. But once the webview got the focus, the keyboard events are stopped reporting, even if I've transfer the focus back to the bevy window, basically no way to get keyboard working again without restarting.

Additional information

The problem for mouse related events is, it can still emit even when the mouse is not within the bevy window, such as taskbar, another monitor, etc. Only when the webview is built, then that part of the window will reject to create any new events. I've dug through the bevy codebase to search the possible cause, what I found is mouse motion is actually listening for device_event of winit. If I'm not mistaken, the device event should be window independent? But from what I've observed, that seems not to be the case somehow.

Keyboard events are understandable since it's listening to window event. When the child window (webview) is focused, the windows won't send the message to bevy window. But the problem of not being able to re-receive the event when the focus transferred to bevy window (unfocus the child) is quite annoying.

These problem can be solve to some extents by using IPC to transfer the event to from webview to bevy but this method is far from ideal as it would introduce a lot of overhead and not solving 100% of the problem. I think it should be solvable via some low-level tinkering.

I have tried to using Win32 API to forward the events to from child window to bevy window, and it's not working. The subclass for the wry window does not reporting keyboard, mouse motion, and many more message, only the mouse button, window create, close that sort of stuffs are received.

I have some hypothesis about this related to the different event loop they are using under the hood, but I don't have time to refactor and verify that.