3D Composition mechanism discussion

[jcyuan]

According to the Evergine integration process with MAUI, you can use the built-in View (ViewHandler) base class provided by MAUI. Override its CreatePlatformView method and return the SurfaceView required for different platforms. In this method, you can initialize your own 3D rendering logic by creating the hwnd or other platform-specific surface parameters needed for window creation. In this way, even the swapchain initialization is handled by developers themselfs, and there is no need for the UI framework to provide a base class and manage it.
Windows:
https://github.com/EvergineTeam/EverSneaks/blob/612b950cc3cad05709b28771b4938fa3b98bd5e0/EverSneaks.MAUI/Platforms/Windows/EvergineViewHandler.Windows.cs#L38-L48
https://github.com/EvergineTeam/EverSneaks/blob/612b950cc3cad05709b28771b4938fa3b98bd5e0/EverSneaks.MAUI/Platforms/Windows/EvergineViewHandler.Windows.cs#L103

Android:
https://github.com/EvergineTeam/EverSneaks/blob/612b950cc3cad05709b28771b4938fa3b98bd5e0/EverSneaks.MAUI/Platforms/Android/EvergineViewHandler.Android.cs#L76-L79
https://github.com/EvergineTeam/EverSneaks/blob/612b950cc3cad05709b28771b4938fa3b98bd5e0/EverSneaks.MAUI/Platforms/Android/EvergineViewHandler.Android.cs#L50

iOS:
https://github.com/EvergineTeam/EverSneaks/blob/612b950cc3cad05709b28771b4938fa3b98bd5e0/EverSneaks.MAUI/Platforms/iOS/EvergineViewHandler.iOS.cs#L30-L33
https://github.com/EvergineTeam/EverSneaks/blob/612b950cc3cad05709b28771b4938fa3b98bd5e0/EverSneaks.MAUI/Platforms/iOS/EvergineViewHandler.iOS.cs#L65

In the underlying layers of MAUI, on Windows, the created Surface is actually a WinUI SwapchainPanel
(https://learn.microsoft.com/en-us/uwp/api/windows.ui.xaml.controls.swapchainpanel?view=winrt-22621
and demos:
https://github.com/microsoftarchive/msdn-code-gallery-microsoft/tree/master/Official%20Windows%20Platform%20Sample/XAML%20SwapChainPanel%20DirectX%20interop%20sample).
On mobile platforms, CreatePlatformView itself can directly return the system-provided Surface, which is a surface supported by the system for composition, so no special handling is required. MAUI on Windows should also use Milcore (Windows) for composition, similar to how WPF uses it. I haven't delved into the specific details.

Therefore, if you want to initialize your own 3D rendering, you just need to add in XAML, and then initialize and return the SurfaceView for different platforms in your MyViewHandler.

I personally think this approach is elegant, and Avalonia doesn't need to provide mechanisms like swapchain, which greatly reduces the chances of introducing bugs.

According to this logic, it seems that ServerCompositor needs significant changes (or maybe the current architecture already has a similar mechanism that I haven't thoroughly studied). Additionally, Here is the rendering process for this mechanism in WPF:

Milcore (MIL-Media Integration Layer) is responsible for communication between WPF and DirectX. It consists of two main components: the Composition Engine and the Render Engine. The Composition Engine is responsible for creating the Composition Tree, while the Render Engine converts the Composition Tree into triangles that DirectX can recognize and notifies DirectX to perform rendering.

Actually, the Swapchain provided by Avalonia currently resembles such a mechanism. However, the integration process described above is more flexible and elegant (as seen in the Evergine's MAUI integration link). The previous Interop mechanism was extremely complex and almost impossible for developers who are not familiar with rendering to integrate. The MAUI integration approach offered by Evergine simplifies the entire process, making it more accessible for developers.

[jcyuan]

@maxkatz6 @kekekeks Because I saw that you were designing a new Interop interface before, I apologize if I am disturbing you by this mentioning.

[kekekeks]

The interop mechanism used by Avalonia is needed to avoid airspace problems.

As of WPF, it can only interop with externally provided DX9 texture which has various synchronization problems.

Avalonia interop API is mostly the same as WPF one, but provides synchronization primitives (dxgi mutex / vulkan semaphore) and interop mechanism negotiation since Avalonia can run on top of 4 different graphics APIs depending on the platform and configuration.

If you want to render into a native view like with maui, you can use NativeControlHost and provide your own native view, but you wont be able to have avalonia content on top of the interop area (the same problem as with WPF)

[jcyuan]

The interop mechanism used by Avalonia is needed to avoid airspace problems.

As of WPF, it can only interop with externally provided DX9 texture which has various synchronization problems.

Avalonia interop API is mostly the same as WPF one, but provides synchronization primitives (dxgi mutex / vulkan semaphore) and interop mechanism negotiation since Avalonia can run on top of 4 different graphics APIs depending on the platform and configuration.

If you want to render into a native view like with maui, you can use NativeControlHost and provide your own native view, but you wont be able to have avalonia content on top of the interop area (the same problem as with WPF)

but <ViewHandler> does not have airspace problem. and i don't need to handle those sync with it.

[kekekeks]

MAUI is using native views for every control.
Avalonia (like WPF) uses a single native view for presenting the entire window (if such thing as a "native view" even exists for the target platform, e. g. with wayland and fbdev there is no such thing).

[jcyuan]

I found that the SwapchainPanel is actually a ISwapChainPanelNative COM interface which created by IDXGIFactory2::CreateSwapChainForComposition For WinRT XAML, but i think Avalonia currently cannot use WinRT XAML, because Windows Composition Engine related.

btw, related topic: https://learn.microsoft.com/en-us/windows/win32/directcomp/directcomposition-portal

[kekekeks]

For some platforms (e. g. modern windows and macOS), we can potentially workaround airspace issues by using several target render surfaces, e. g. overlaying thos like this:

1. Content below the "native" visual (consumes full-window size texture 3 times for backbuffering)
2. "Native" visual
3. Content above the "native" (consumes full-window size texture 3 times for backbuffering).

Since WinRT visuals and macOS views support transparency and are properly composed by the system compositor, this will work (user input and hit-testing will still be a problem and would have to be routed through Avalonia anyway).

That will not work for X11 for various reasons. The only way to properly present content is to make it composited by Avalonia.

[jcyuan]

Thanks for replying @kekekeks , I just combine my replies here.

After I posted this, I thought about it and realized that yes, Avalonia indeed cannot solve the problem using this method. There are only two possible solutions:

1. Write a NativeServerCompositor, which encapsulates and interfaces with Windows.UI.Composition. Since XAML definition is merely for collecting drawing information, Avalonia has already serialized the drawing information to the server. Then, it can provide the drawing information to Windows.UI.Composition for composition. The obvious advantage of this is that it allows the underlying systems to handle the composition, thereby directly supporting complex functionalities like 3D blending, video blending, etc. However, the downside is also evident: it requires a significant amount of work and brings compatibility issues with the existing Avalonia as you mentioned above, yes.

2. Optimize with the current ServerCompositor mechanism:

   1. Encapsulate the texture of the current Graphic Backend into a <ViewHandler> and provide lock/unlock methods (developers need to consider the performance impact of the lock on UI rendering; Avalonia is not responsible). However, since this lock is merely a GPU-end lock for a texture used in the ServerCompositor and not a CPU-end lock (e.g., in Vulkan, it just sets the dstAccess to SHADER_READ_BIT for the texture. and a FencSync in OpenGL etc), it does not cause significant performance issues. Avalonia's logic can even consider creating a separate SharedContext or a separate Queue for this tag to provide maximum parallelism for Command List submission, making the lock/unlock performance issue almost negligible.

   2. Developers can use the same Graphic Backend as Avalonia to render content on this texture (since the Graphic Backend can be configured), which avoids the complexity of texture sharing between different backends.

   3. For the ServerCompositor, what is submitted to the server (the <ViewHandler> tag) is simply a texture of the current Backend, with no additional processing logic and it avoids the Interop action.

This approach should be the most developer-friendly and compatible with the existing Avalonia backends, while also solving the problem of simulating the Swapchain.

[kekekeks]

Write a NativeServerCompositor, which encapsulates and interfaces with Windows.UI.Composition

That will be be deeply tied to the underlying platform and quite significantly limited in what we can do. There is also no Windows.UI.Composition on non-windows platforms and similar things on mac and browser require UI changing the visual tree on the UI thread. NSView/UIView are also rather expensive in comparison to WinRT composition visuals. With X11/Wayland/Framebuffer we don't even have an underlying API to run on top.

So this approach isn't really feasible, especially if it's required for GPU interop alone.\

In general the app should consider the avalonia's renderer to be the same as the system compositor, because it plays the same role for in-app content and can be the only thing that drives the system display (e. g. with fbdev and DRM/KMS backend when we talk directly to DRI video drivers).

provide lock/unlock methods

Modern graphics APIs don't work on lock/unlock basis and instead use GPU-side fences/semaphores. We currently provide access to those. The API is rather complex because it tries to encapsulate DXGI, Vulkan and OpenGL on different OSes with future-proofing for Metal. Presentation time is also a concern.

If we just use lock/unlock for synchronizing access to GPU memory we'll have the same problem we currently have with our CPU-backed WriteableBitmap - if we lock the bitmap for update before the previous frame used the bitmap data, we can get out of sync with the rest of the visual tree, i. e.

1. app calls WriteableBitmap.Lock and updates the data for frame [1]
2. frame [1] is enqueued with layout and other visual state
3. frame[1] is started rendering on the render thread
4. app calls WriteableBitmap.Lock and updates the data data for frame [2]
5. frame[1] is consumed the bitmap data intended for frame[2] and presented it on-screen with the rest of the frame[1]
6. frame [2] is enqueued with layout and other visual state

This is an existing problem with our current implementation that we intend to fix at some point. If we use the same naive locking approach with GPU memory sharing, we'll have the same problem, so we'll internally need a copy to a separate texture anyway.

Developers can use the same Graphic Backend as Avalonia to render content on this texture

I believe that this is a misconception. Writing the same logic for 4 different graphics APIs is rather expensive and not feasible for legacy codebases that are e. g. using OpenGL. So we currently expose access to OS-provided GPU memory sharing mechanisms instead. It's also not always possible to use the same underlying graphics context or logical device because of threading issues and underlying API limitations (e. g. with Vulkan we aren't guaranteed to be able to create more than one logical queue per queue type and queue API access isn't thread safe).

Note that the provided API is low-level enough to implement more simplified and user-friendly APIs on top of it, but those implementations will still have to deal with underlying GPU synchronization concepts internally. The swapchain-like implementation currently available in GpuInterop sample and one in #14653 were experimental attempts to design such a user-friendly API. Both are implemented on top of low-level public APIs.

[jcyuan]

Modern graphics APIs don't work on lock/unlock basis and instead use GPU-side fences/semaphores.

I mean the lock/unlock I mentioned is exactly the GPU-side synchronization you talked about. My intention is to provide an abstract method in the ViewHandler (texture) class for developers to call, and its implementation on the Avalonia side is determined based on the current backend in use.

If we just use lock/unlock for synchronizing access to GPU memory we'll have the same problem we currently have with our CPU-backed WriteableBitmap - if we lock the bitmap for update before the previous frame used the bitmap data, we can get out of sync with the rest of the visual tree

CPU lock will have such problem but GPU 'lock' won't, I believe.

--

Essentially, after simplifying this issue, it boils down to the texture synchronization problem I mentioned. Here is a minimal demo: https://community.arm.com/cfs-file/__key/telligent-evolution-components-attachments/01-2066-00-00-00-00-42-84/Whitepaper_2D00_ThreadSync.pdf. This demo has a main renderer (equivalent to ServerCompositor), and it provides a texture in a new thread for the user to paint on to dynamically update the texture. Then, when the main renderer uses this texture, synchronization is required; otherwise, the user's painting will have issues.

Therefore, this entire process is identical to the problem we are currently discussing. It just needs to be abstracted. As for implementing the logic for all four Backends, it indeed requires some effort, but this is unavoidable.

[kekekeks]

https://github.com/AvaloniaUI/Avalonia/blob/82e3613939af5451597d3c19f3a48e0a8ba888f3/src/Avalonia.Base/Rendering/Composition/CompositionExternalMemory.cs

This set of interfaces allow you to:

1. import GPU memory you've allocated with your preferred drawing API regardless of what Avalonia is internally using for rendering as long as there is the way to share GPU memory
2. import a semaphore object
3. tell Avalonia that you are ready to present your render results and provide the semaphore handles to wait on and to set (lock/unlock primitive)

It's completely backend-agnostic, i. e. you can use those interfaces to present Vulkan, DirectX and OpenGL (via WGL_NV_DX_interop.txt) content when Avalonia is running on top of DirectX (ANGLE). You do not need to switch Avalonia to a different drawing API just to use that API in that app, all you need is to create your preferred GPU API context and use existing OS GPU memory sharing options.

For non-Windows platforms it's less fleshed out, e. g. for OpenGL we usually rely on shared OpenGL contexts, because it was the easiest way to implement interop.

So with Linux we the only external memory/semaphore type we currently support is Vulkan-created "opaque FDs". Technically you can also create your own EGL/GLX context in tandem with a Vulkan one and use OpenGL for drawing too. The other external memory type we are planning to support is DMA-BUF which can technically point to arbitrary memory supported by the GPU driver.

For macOS the plan is to support IOSurface and CVPixelBuffer objects that are functionally equivalent to DXGI handles with Windows.

So technically for all major platforms, the current API (which is currently properly implemented only for Windows/Linux) you can bring your own GPU context and use it for interop.

However the driver support (especially with Linux) might wary. Which is why we still have support for OpenGL context sharing there, so you need Avalonia to create a context for you.

There is also WebAssembly. WebAssembly is a very special platform where it's currently not possible to use shared OpenGL contexts and WebGPU is still experimental. Which is why we've added GPU context lease feature on top of Skia API lease. The feature allows you to run your code directly on Avalonia's render thread (threading support is coming to WASM with .NET 9, with <=NET8 wasm it's technically the UI thread) and have direct access to the GPU context Avalonia is using in the middle of compositing a scene.

While that API doesn't require sync primitive usage when using OpenGL (Vulkan/Metal would still need issuing sync commands), the downside is that you'll need to make your code to be thread safe and aware of composition batches and properly clean up your resources since you can't just kill the entire context and be on your merry way. Another downside when using OpenGL is that you need to properly reset the state.

[kekekeks]

What we currently lack is a higher level code on top of our existing low-level APIs that would provide a simple plug-and-pray method that would manage everything automatically. That OpenGL PR was an attempt to make one, however I don't really like the swapchain API that is a bit cumbersome to use and I also need to fix OpenGL fences in a separate PR before continuing work on it.

[jcyuan]

What we currently lack is a higher level code on top of our existing low-level APIs that would provide a simple plug-and-pray method that would manage everything automatically. That OpenGL PR was an attempt to make one, however I don't really like the swapchain API that is a bit cumbersome to use and I also need to fix OpenGL fences in a separate PR before continuing work on it.

That's exactly what I wanted to say! I completely agree with your statement.

Indeed, it can be quite challenging for developers to do their own interop, It's too low-level, and they would benefit from higher-level encapsulation. Hopefully, Avalonia will come up with a fantastic solution in the future, just like how Avalonia's XAML/Style syntax provides a user-friendly experience.

Additionally, I've always been curious about how HTML achieves this. The <canvas> and <video> elements in HTML make it incredibly easy for developers to use. I assume it's similar to Avalonia's current approach.