[BUG]: lama anime output

[wolfkingal2000]

Model
Which model are you using? lama anime

Describe the bug
so in other software give this output like iopaint
VoxelCubes/PanelCleaner#121
this issues
Screenshots
this my output

System Info
Software version used

• Platform: Windows-10-10.0.19045-SP0
• Python version: 3.10.9
• torch: 2.3.1+cu121
• torchvision: 0.18.1+cu121
• Pillow: 10.4.0
• diffusers: 0.27.2
• transformers: 4.42.3
• opencv-python: 4.10.0.84
• accelerate: 1.0.1
• iopaint: 1.5.2
• rembg: 2.0.59

[wolfkingal2000]

@Sanster
I have an idea to address this issue. It seems adding a blur mask—similar to the one used in Stable Diffusion—could solve the problem. This mask would handle the surrounding area, ensuring it matches seamlessly. Additionally, incorporating a histogram-matching process to synchronize the mask with the rest of the image could further enhance the result.

[minh-nguyenhoang]

imgs: torch.Tensor =...
masks: torch.Tensor = ...
inpainted_images: torch.Tensor = ...
mask_clone = masks.clone()
mask_clone[:,:,0,0] = 0     # prevent all black mask
img_means = (imgs * (1-mask_clone)).mean(dim=(2, 3), keepdim=True) / (1-mask_clone).mean(dim=(2, 3), keepdim=True)
img_stds = (((imgs - img_means).pow(2) * (1-mask_clone)).mean(dim=(2, 3), keepdim=True) / (1-mask_clone).mean(dim=(2, 3), keepdim=True)).sqrt()
inpainted_means = (inpainted_images * (1-mask_clone)).mean(dim=(2, 3), keepdim=True) / (1-mask_clone).mean(dim=(2, 3), keepdim=True)
inpainted_stds = (((inpainted_images - inpainted_means).pow(2) * (1-mask_clone)).mean(dim=(2, 3), keepdim=True) / (1-mask_clone).mean(dim=(2, 3), keepdim=True)).sqrt()
inpainted_images = (inpainted_images - inpainted_means) / inpainted_stds * img_stds + img_means
inpainted_images = inpainted_images * masks + imgs * (1-masks)

This is a simple statistical equalization process that you can try to apply to minimize the color discrepancy. You can further blur the mask to gain smoother transition in the last step.

[wolfkingal2000]

imgs: torch.Tensor =...
masks: torch.Tensor = ...
inpainted_images: torch.Tensor = ...
mask_clone = masks.clone()
mask_clone[:,:,0,0] = 0     # prevent all black mask
img_means = (imgs * (1-mask_clone)).mean(dim=(2, 3), keepdim=True) / (1-mask_clone).mean(dim=(2, 3), keepdim=True)
img_stds = (((imgs - img_means).pow(2) * (1-mask_clone)).mean(dim=(2, 3), keepdim=True) / (1-mask_clone).mean(dim=(2, 3), keepdim=True)).sqrt()
inpainted_means = (inpainted_images * (1-mask_clone)).mean(dim=(2, 3), keepdim=True) / (1-mask_clone).mean(dim=(2, 3), keepdim=True)
inpainted_stds = (((inpainted_images - inpainted_means).pow(2) * (1-mask_clone)).mean(dim=(2, 3), keepdim=True) / (1-mask_clone).mean(dim=(2, 3), keepdim=True)).sqrt()
inpainted_images = (inpainted_images - inpainted_means) / inpainted_stds * img_stds + img_means
inpainted_images = inpainted_images * masks + imgs * (1-masks)

This is a simple statistical equalization process that you can try to apply to minimize the color discrepancy. You can further blur the mask to gain smoother transition in the last step.

How can add this?

[minh-nguyenhoang]

@wolfkingal2000 For me, I modify the code in the models file directly. Below is an example from iopaint.model.zits, while you can change it in iopaint.model_manager.ModelManager.__call__, but you will need to modify the code a little bit (use numpy, mask normalization,...) to apply it to all model at once.

• Old forward:

        ...
        mask = mask[:, :, 0]
        items = load_image(image, mask, device=self.device)

        self.wireframe_edge_and_line(items, config.zits_wireframe)

        inpainted_image = self.inpaint(
            items["images"],
            items["masks"],
            items["edge"],
            items["line"],
            items["rel_pos"],
            items["direct"],
        )

        inpainted_image = inpainted_image * 255.0
        ...

• New forward:

        ...
        mask = mask[:, :, 0]
        items = load_image(image, mask, device=self.device)

        self.wireframe_edge_and_line(items, config.zits_wireframe)

        inpainted_image = self.inpaint(
            items["images"],
            items["masks"],
            items["edge"],
            items["line"],
            items["rel_pos"],
            items["direct"],
        )
        imgs: torch.Tensor = items["images"]
        masks: torch.Tensor = items["masks"]
        inpainted_images: torch.Tensor = inpainted_image
        mask_clone = masks.clone()
        mask_clone[:,:,0,0] = 0     # prevent all white mask
        img_means = (imgs * (1-mask_clone)).mean(dim=(2, 3), keepdim=True) / (1-mask_clone).mean(dim=(2, 3), keepdim=True)
        img_stds = (((imgs - img_means).pow(2) * (1-mask_clone)).mean(dim=(2, 3), keepdim=True) / (1-mask_clone).mean(dim=(2, 3), keepdim=True)).sqrt()
        inpainted_means = (inpainted_images * (1-mask_clone)).mean(dim=(2, 3), keepdim=True) / (1-mask_clone).mean(dim=(2, 3), keepdim=True)
        inpainted_stds = (((inpainted_images - inpainted_means).pow(2) * (1-mask_clone)).mean(dim=(2, 3), keepdim=True) / (1-mask_clone).mean(dim=(2, 3), keepdim=True)).sqrt()
        inpainted_images = (inpainted_images - inpainted_means) / inpainted_stds * img_stds + img_means
        inpainted_images = inpainted_images * masks + imgs * (1-masks)

        inpainted_image = inpainted_images * 255.0
       ...