problem with StableDiffusionXLControlNetInpaintPipeline

[D222097]

the example of StableDiffusionControlNetInpaintPipeline set the mask area to -1 when make control_image, but XL version set 0 in the example. Would that be more reasonable? Why change?

[D222097]

I don't know if anyone has the same problem: when I use the controlnet-inpainting model via diffusers StableDiffusionControlNetInpaintPipeline, the result didn't follow the prompt, but the same model works well on the webui.This is only the case with XL version(no matter the mask area is set to 0 or -1)
prompt: masterpiece, best quality, ultra-detailed,8k, top quality, extremely detailed,high resolution,4k, ultra high resolution,masterpiece,best quality, a blue suv is driving down the road
raw img:

mask:

diffusers output:

webui output:

[willjejones]

Currently experiencing the exact same issue. SDXL control net inpainting works as expected in the webui, but not via diffusers with this pipeline. Or at least, it does not work consistently at all.

No errors - an image is returned, but often with a barely perceivable change to the masked area. Feels as though the prompt is ignored, (although the mask seems to be used).

[D222097]

I ran a more detailed test, the result was strange.: https://discuss.huggingface.co/t/problem-with-xl-controlnet-inpaint/63257

[patrickvonplaten]

cc @yiyixuxu can you check this?

[yiyixuxu]

not same issue but all related to implementation difference between diffusers vs auto1111

#5808
#5498
#5600

[yiyixuxu]

@D222097 thanks for reporting this issue!
sorry! I'm just able to find time to look into this just now.

for this StableDiffusionControlNetInpaintPipeline example, Could you please share the exact script you used for diffusers and your auto1111 settings? I would need to know all the parameters you used, such as guidance_scale and controlnet_conditioning_scale...

Thanks again!

[D222097]

@yiyixuxu Hi! Sorry for the late reply.
auto1111-webui setting:
masterpiece, best quality, the white suv is driving down the road
Steps: 30, Sampler: DPM++ 2M SDE Karras, CFG scale: 7, Seed: 529028879, Size: 1024x1024, Model hash: 0f1b80cfe8, Model: dreamshaperXL10_alpha2Xl10, Denoising strength: 0.75, Mask blur: 4, ControlNet 0: "Module: inpaint_global_harmonious, Model: car_1113_dreamshaper-xl-1-0_pretrained-cn-e80 [0a6e7ba8], Weight: 1, Resize Mode: Crop and Resize, Low Vram: False, Guidance Start: 0, Guidance End: 1, Pixel Perfect: False, Control Mode: Balanced, Save Detected Map: True", Version: v1.6.1


raw img:

mask:

diffusers-pipeline-result:

webui-result:

[D222097]

diffuser pipeline script：

import os
import cv2
import numpy as np
import os
import torch
from diffusers.utils import load_image
from diffusers import (StableDiffusionXLControlNetInpaintPipeline, 
                       ControlNetModel, 
                       DDIMScheduler, 
                       DPMSolverMultistepScheduler, 
                       DDPMScheduler,
                       StableDiffusionXLInpaintPipeline,
                       StableDiffusionXLControlNetPipeline,
                       StableDiffusionXLControlNetImg2ImgPipeline)
from PIL import Image


def gen_image(data_pth, pipe, name, caption, seed):

    img_pth = f'{data_pth}/img'
    img_files = os.listdir(img_pth)
    generator = torch.Generator(device="cuda").manual_seed(seed)

    for i, img_file in enumerate(img_files):
        img_name = img_file.split('.')[0]
        img_file_pth = f'{data_pth}/img/{img_file}'
        msk_file_pth = f'{data_pth}/msk/{img_name}.png'
       
        prompt = caption

        init_image = load_image(img_file_pth)
        init_image = init_image.resize((1024, 1024))
        mask_image = load_image(msk_file_pth)
        mask_image = mask_image.resize((1024, 1024))
        control_image = make_inpaint_condition(init_image, mask_image)

        images = pipe(
            prompt=prompt,
            # negative_prompt=neg_prompt,
            num_inference_steps=30,
            num_images_per_prompt=1,
            generator=generator,
            guidance_scale=7.0,  #
            eta=1.0,
            controlnet_conditioning_scale=0.5, # 
            image=init_image,
            mask_image=mask_image,
            control_image=control_image,
            )['images']
        
        output_path = f'{data_pth}/res'
        if os.path.exists( output_path) == False:
            os.makedirs(output_path) 
        for j, image in enumerate(images) :
            image.save(os.path.join(output_path, name+"_"+str(i)+"_"+str(j)+".png") )
      
def make_inpaint_condition(image, image_mask):
    image = np.array(image.convert("RGB")).astype(np.float32) / 255.0
    image_mask = np.array(image_mask.convert("L")).astype(np.float32) / 255.0

    assert image.shape[0:1] == image_mask.shape[0:1], "image and image_mask must have the same image size"
    image[image_mask < 0.5] = -1  # set as masked pixel  ##
    image = np.expand_dims(image, 0).transpose(0, 3, 1, 2)
    image = torch.from_numpy(image)
    return image

if __name__=='__main__':
    data_pth = ""
    name = "diffusers-pipe-res"
    caption = "masterpiece, best quality, the white suv is driving down the road"
    seed = 529028879
   
    controlnet = ControlNetModel.from_pretrained(
       "", 
       torch_dtype=torch.float32
     ).to('cuda')
    pipe = StableDiffusionXLControlNetInpaintPipeline.from_pretrained(
       "/mnt/share_disk/Lykon/dreamshaper-xl-1-0", 
       controlnet=controlnet, 
       torch_dtype=torch.float32
     )
    # pipe.scheduler = DDPMScheduler.from_config(pipe.scheduler.config)
    pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)
    pipe.to('cuda')
    gen_image(data_pth, pipe, name, caption, seed)

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.