improve 10-bit hevc format handling

[airlied]

Attempting a 10-bit HEVC decode on radv results in some issues, I haven't solved then all yet.

The first one though is the format picking, I think there is one bug in radv, but there's a second bug.

With AMD hw the DPB and decode are distinct, for 10-bit the DPB needs to be in 10-bit, but the output can be 8-bit and the hardware has the ability to handle that.

So radv returns both an 8-bit and 10-bit format for the decode output, but you always pick the 8-bit, and I think you should probably be smarter and scan all returned formats and pick the optimal one?

[charlie-ht]

That's due to this

vk_video_samples/common/include/VkVideoCore/VulkanVideoCapabilities.h

Line 89 in f4de028

referencePicturesFormat = supportedDpbFormats[0];

I changed this for my WIP CTS tests for 10-bit here: KhronosGroup/VK-GL-CTS@94425ce#diff-d73087a9725d59f7a76bdaf9f683fa4c773a10122bbce3b419b81fceda71ebb2R698

We need something similar (and probably smarter) for the samples app.

[zlatinski]

Aha, okay. Thank you, charlie-ht and airlied. Indeed, we'll need a fix for that in the sample app. I'll look into fixing this soon.