shader_recompiler: Maintain loss of precision when folding half-float unpack.

[squidbus]

When folding a pair of half unpack and pack instructions, maintain the loss of precision that should have happened from converting to 16-bit float using a quantize op.

Fixes some missing lighting in CUSA05637. The game stores a set of control bits in the alpha channel during rendering in order to control lighting properties in later shaders. This involves dividing to convert it to a normalized float when writing, and multiplying to convert it back to an 8-bit integer when reading. When writing they compensate for the loss of precision when packing to 16-bit float on PS4 by adding a constant 0.5 / 255, but without the loss in precision this ends up changing the bits when transformed back into an integer in the lighting shader.

The lighting components that are now visible in this game seem to have some color problems but those appear to be a separate issue from this.

[Randomuser8219]

This breaks textures on both Sonic and the huge worm in Stage 1.

[squidbus]

This breaks textures on both Sonic and the huge worm in Stage 1.

No, it does not. What you are seeing are the incorrect lighting colors mentioned in the PR description.

[Randomuser8219]

Oh, that explains it.

[squidbus]

Status of this: It seems that precision can still be an issue due to different rounding behavior with conversion to 16-bit float, so this will probably need a different solution.