Upscaling certain textures creates black spots/splotches on originally bright spots.

[ProtoBelisarius]

Hi, I have problems with my Image getting black spots when using a texture upscaled with Ultrasharp in img2img.
(using 6800XT 16GB with Linux and dont have these problems with other images, like t2i created people/landscapes)

I changed checkpoints, tried different vae and no vae, tried 2-15 cfg, 10-80 steps with euler,eulera,dpm++ 2m/SDE (Karras), clip skip 1/2/3/4, different seeds, I have 61 Tokens in the positive and 22 in the negative prompt, nothing removes these spots or changes their position. (I also converted the images to jpg from png but no change)
Settings for Tiled Diffusion are at default, and I tried either 4x upscale with the integrated function from 1k or
no upscale with 4k input.
Tiled Vae settings are 1024 - 128 and I tested up until 1536 - 256 but around then I get oom and it also doesnt change anything.

Only when I use a photo editor and a heal/clone tool to copy over darker parts on the brightest parts of the image, where later spots appear, then no black spots appear. But as I want to upscale hundreds of textures that way, I cant spend so much time running them through, identifying bright spots and painting over them, then running them again, especially as I cant guarantee that I wont miss any.
Different upscalers(SwinIR/UltraMix/Remacri/etc) dont create black spots but white spots there (but have significantly worse quality for my specific image here, so Im not using those).
Different textures also get these black spots, but at different persistent locations(their respective bright spots).

Is this a problem with multidiffusion or something else? Its a 4k texture, so I cant test it with normal img2img, but Ultimate SDUpscale doesnt produce these black spots. And manually darkening the brighter parts before img2img removes this issue.
Any suggestions welcome, I have no idea what to do anymore.

Left is zoomed in part of the original, middle is a preview during generation and right is the end result.

[Kahsolt]

Could you please make a further test, any bright patches will also turn to be black? (e.g. edit the picture is PS, copy the brightest parts of the image to other corners of the canvas, even draw some pure white areas, and run through your workflow again)

If so, this might be caused by value explosion in VAE decoding. Please upload that ref image (or just the bright parts cropping) to help us reproducing and debugging :)

[ProtoBelisarius]

Sorry for the late answer.
Copying brighter parts all over actually removed the black spots.
Also added links to image hoster in case git does something to the images.

Original Image (256x256 from .dds to .png)(https://ibb.co/S3JHxm1)

upscaled with Ultrasharp (4096x4096 .jpg) (in .png: https://ibb.co/YcnsyCh)

tiled diffusion result (4096x4096 .jpg) (in .png: https://ibb.co/w4Rwd5y)

tiled diffusion with coppied bright spots all over (same prompt and settings as result above)