How reliable is Windows Graphics Desktop Capture in replicating the monitor frames ?

[LordBrkica]

Lets say that I have a person sitting in front of the monitor and at the same time I use the desktop capture api to store the frames in a video file.

Is it guaranteed that the screen that person sees correspond 1/1 to the video file ?

Or is it a case of screen being rendered by A and desktop capture renders the frames as a B and there may be differences (in framerate or quality) ?

If differences are possible - what may I do (with regards to the C++ desktop capture api) to eliminate any discrepancies, is it possible at all ?

[robmikh]

For the most part it should be the same but there are some corner cases where things are different.

The most obvious difference is in the case of protected content. Windows with DRM content or that have been marked with SetWindowDisplayAffinity will render as solid black instead.

Applications can also use SetWindowDisplayAffinity to exclude their window from captures.

And finally, the secure desktop isn't captured. So if your user gets a UAC prompt, it won't show up in your capture. Those scenarios may want to use DDA from the SYSTEM account instead.

[LordBrkica]

Ok, but my main concern are not those big/obvious border cases but rather possible minor graphical (regarding primarily framerate) deviations due to the way API itself works and/or how I use it.

What I am concerned about is the following:

• the line sender.TryGetNextFrame() fails or is delayed somehow. In this case even though my call failed (or took some time) the real screen may however process a frame -> so we have a discrepancy
• such an error maybe happens rarely and it is very difficult to see with a naked eye (unlike the cases where the entire window is missing or we have a black screen or such "obvious" problems)

[robmikh]

To be clear, the capture API will not stall DWM's rendering of the screen. So if your app isn't consuming frames quick enough you may miss frames. My recommendation would be to call TryGetNextFrame when the FrameArrived event fires and return the frame ASAP, and to use more frames in your frame pool if you're worried about missing something.

[LordBrkica]

Ok, if you could be so kind an take a look at my code:

    void WindowsGraphicsScreenCapturer::OnFrameArrived(winrt::Windows::Graphics::Capture::Direct3D11CaptureFramePool const& sender, winrt::Windows::Foundation::IInspectable const&)
    {
            auto frame = sender.TryGetNextFrame();
                
            if (frame != nullptr)
            {
                winrt::com_ptr<ID3D11Texture2D> surfaceTexture = GetDXGIInterfaceFromObject<ID3D11Texture2D>(frame.Surface());

                ID3D11Texture2D* backBuffer;
                D3D11_TEXTURE2D_DESC descIncomingFrame;
                surfaceTexture->GetDesc(&descIncomingFrame);
                    
                HRESULT createEmptyTextureResult = device->CreateTexture2D(&descIncomingFrame, nullptr, &backBuffer);

                winrt::check_hresult(createEmptyTextureResult);

                m_d3dContext->CopyResource(backBuffer, surfaceTexture.get());

                if (createEmptyTextureResult == S_OK)
                {
                    passFrame(backBuffer); 
                }
	    }
    }

I am not really sure how should I refactor it so that I am still able to do stuff on frame AND that the frame is returned ASAP.

When you say to use more frames in my framepool - how exactly would I implement that ?

[robmikh]

This looks fine, but I would clear out the MiscFlags in the texture description before using it to create a new texture. Textures from the capture API are shared textures and you probably don't need that. You may also want to consider reusing these textures instead of always creating new ones.

To configure the number of buffers in the frame pool, that is done at creation time (see Direct3D11CaptureFrame::Create and CreateFreeThreaded) or can be done later with a call to Recreate.

[tom-huntington]

Context

I'm getting some weird behaviour when using the Windows.Graphics.Capture apis in a WinUI3 app with a SwapChainPanel (which scrolls horizontally at a constant 60 fps, this is how I managed to debug it).

If you measure the duration (Direct3D11CaptureFrame.SystemRelativeTime ) between subsequent frames usually it's always exactly 60 fps (16666 ticks).

Direct3D11CaptureFrame.SystemRelativeTime: The QPC (Query Performance Counter) time at which the compositor rendered the frame.

However, what's happening in my app I'm being sent partial frames that never actually get displayed on the moniter. The duration between frames is frequently 0%, 3%, 10% (of the usual 166666 ticks).

In a frame obtained from Direct3D11CaptureFramePool.TryGetNextFrame, an animation transition of a UI component updates and then the swap chain presents in the subsequent frame, even though these updates both actually happened in the same frame on the monitor.

Redacted. This paragraph is incorrect

Out of order frames ---- The swap chain updates in one frame, and then in the subsequent frame the swap chain is back in the previous stale state (perhaps another composition is being updated). You always (eventually) get the frame that was actually displayed to the moniter (where the swapchain is in the correct position, i.e. its not skipped) although **you can't tell which frames are the stale ones**. (I'm suspicious that the non-decreasing `Direct3D11CaptureFrame.SystemRelativeTime` are lying, are being artificially made to be non-decreasing. I'm getting a lot of repeated timestamps and frames out of order).

Frames are Dropped and replaced by Stale ones

I believe (in normal apps where the frames are 166666 ticks apart) that the wrong frame is sometimes being chosen by Windows.Graphics.Capture. This results in stale frames (and skipping of frames that where actually displayed to the monitor) causing stutters. Potentially related

You can weakly reproduce this in https://github.com/microsoft/DirectX-Graphics-Samples/tree/master/Samples/Desktop/D3D12Fullscreen

Change this line for a constant frame rate:

ThrowIfFailed(m_swapChain->Present(1, 0/*presentFlags*/));

also change

const float translationSpeed = 0.05f;

recordedVideo_1.mp4

If you step through frame-by-frame (drag the seekbar thumb starting from halfway), you'll notice that the 7th frame is the same as the 6th frame. The fact that the 8th frame jumps ahead of the 7th frame by two renders/presents, shows that the 7th frame is stale. This bug is not very noticeable/common in the D3D12Fullscreen app, however, it does plague recordings of my SwapChainPanel app https://www.microsoft.com/en-us/p/speech-school/9nqsjr8bwr3g. But its probably a small bug occurring in all recordings with the Windows.Graphics.Capture namespace.

[robmikh]

Thanks for reporting this!

I have a few questions:
What build of Windows are you seeing this on?
What GPUs are on the repro system?
What are you using to record?
Can you reproduce this with other Windows.Graphics.Capture-based recorders? (e.g. CaptureVideoSample)

I'll see if I can reproduce this.

[tom-huntington]

What build of Windows are you seeing this on?

Version 21H2 Build 22000.2295

What GPUs are on the repro system?

3090. I haven't tested this on another gpu, so it might just be a driver bug. BE WARNED.

What are you using to record? Can you reproduce this with other Windows.Graphics.Capture-based recorders? (e.g. CaptureVideoSample)

All of them. The previous video was recorded using https://github.com/MicrosoftDocs/SimpleRecorder/

Redacted

The best way to reproduce it (smallest feedback loop) is side-by-side with your swapchain apps

recordedVideo5.mp4

In the video the jitter appears on both apps, but on the monitor it was only on the Win32CaptureSample. There's a lot of jitter in the above video, but below is another video (recorded with SimpleRecorder) with only a little jitter. This is typical when recording with OBS, the jitter varies a lot. By jitter I mean repeated frames 4,5,6,6,8,9 (where n is the Present index). These repeated frames are not noticeable in most situations, but are when in an app that scrolls at a constant framerate. The jitter becomes more noticeable when compressed/encoded because it requires a higher bitrate (becomes more lossy for a given bitrate).

recordedVideo_1.mp4

[tom-huntington]

The extra frames I was receiving were clustered into groups with a period of 166666 ticks. By taking the last frame in each cluster as the correct one, I was able to get the capture I wanted (I've got no idea which dependency was causing this bug, I cant reproduce it outside of my app).

The issue (repeated/skipped frames, that all Windows.Graphics.Capture apps have on my machine) is hard to notice when it is occurring, and sometimes its only very slight (its a synchronization thing). To verify that this issue is actually happening I recommend recording the D3D12Fullscreen app with the changes I mentioned, then using mpv (not vlc) to step through frame by frame (keyboard shortcut .) (or play at ~0.08 speed).