fix: File dialogs on Linux causing a crash on systems where GTK uses Wayland but GLFW uses X11

[btzy]

There has been reports of the file dialog crashing ImHex on Linux, including this and this. I think this occurs on systems where Wayland is the default compositor but the GLFW used in ImHex only supports X11.

The default Ubuntu 22.04 and Ubuntu 24.04 builds of ImHex use the GLFW from Ubuntu's default package repository, which supports only X11 but not Wayland. GLFW is only built with Wayland support from Ubuntu 24.10 onwards. This can be seen from the CI log https://github.com/WerWolv/ImHex/actions/runs/10167117281, which contain something like the lines below in the CMake configure step (they are added by #1833).

-- Performing Test GLFW_HAS_X11
-- Performing Test GLFW_HAS_X11 - Success
-- Performing Test GLFW_HAS_WAYLAND
-- Performing Test GLFW_HAS_WAYLAND - Failed

These tests detect if the GLFW package that ImHex is being built against supports X11 or Wayland (or both).

Using a build of ImHex like those above means that ImHex must use an X11 window, even when Wayland is available (e.g. with Sway). However, the system GTK may support Wayland, and even default to Wayland with depending on the system configuration. As the Ubuntu builds of ImHex use NFDe with the GTK backend and use the system GTK package, GTK may open a Wayland window. An X11 window cannot be made the parent of a Wayland window, thereby causing the crash.

NFDe has been updated to force GTK to use the same compositor as the parent window, and this PR fixes ImHex by updating NFDe.

[jumanji144]

Is this pr still a draft or can it be reviewed?

[btzy]

This PR can't be merged as is, because the commit a24692b has bumped the nativefiledialog version already, and the detection of X11 and Wayland is problematic in Linux due to people swapping out the glfw package for the glfw-wayland package after installation.

It would be ideal to reapply a modified commit that was reverted though, because it keeps the file dialog above the main window (on Windows, macOS, and Linux with GLFW 3.4+ only) - I can raise a PR if there is appetite for it.