Detection Detailer

An object detection and auto-mask extension for Stable Diffusion web UI. See Installation.

Segmentation

Default models enable person and face instance segmentation.

Detailing

With full-resolution inpainting, the extension is handy for improving faces without the hassle of manual masking.

Installation

1. Use git clone https://github.com/dustysys/ddetailer.git from your SD web UI /extensions folder. Alternatively, install from the extensions tab with url https://github.com/dustysys/ddetailer
2. Start or reload SD web UI.

The models and dependencies should download automatically. To install them manually, follow the official instructions for installing mmdet. The models can be downloaded here and should be placed in /models/mmdet/bbox for bounding box (anime-face_yolov3) or /models/mmdet/segm for instance segmentation models (dd-person_mask2former). See the MMDetection docs for guidance on training your own models. For using official MMDetection pretrained models see here, these are trained for photorealism. See Troubleshooting if you encounter issues during installation.

Usage

Select Detection Detailer as the script in SD web UI to use the extension. Click 'Generate' to run the script. Here are some tips:

• anime-face_yolov3 can detect the bounding box of faces as the primary model while dd-person_mask2former isolates the head's silhouette as the secondary model by using the bitwise AND option. Refer to this example.
• The dilation factor expands the mask, while the x & y offsets move the mask around.
• The script is available in txt2img mode as well and can improve the quality of your 10 pulls with moderate settings (low denoise).

Troubleshooting

If you get the message ERROR: 'Failed building wheel for pycocotools' follow these steps.

Any other issues installing, open an issue.

Credits

hysts/anime-face-detector - Creator of anime-face_yolov3, which has impressive performance on a variety of art styles.

skytnt/anime-segmentation - Synthetic dataset used to train dd-person_mask2former.

jerryli27/AniSeg - Annotated dataset used to train dd-person_mask2former.

open-mmlab/mmdetection - Object detection toolset. dd-person_mask2former was trained via transfer learning using their R-50 Mask2Former instance segmentation model as a base.

AUTOMATIC1111/stable-diffusion-webui - Web UI for Stable Diffusion, base application for this extension.