Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to use this in Automatic1111 Stable Diffusion Web UI (for SD 1.5, XL etc) #8

Open
SMUsamaShah opened this issue Apr 25, 2024 · 13 comments

Comments

@SMUsamaShah
Copy link

SMUsamaShah commented Apr 25, 2024

How to use this in Automatic1111 Stable Diffusion Web UI? Can you add steps to readme if its as simple as changing a few lines?

I want to use it with SD 1.5 mainly because it seems to give SD 1.5 a new life (to gen high res images)

@Luke2642
Copy link

Luke2642 commented Apr 26, 2024

You can't do the one line thing in Auto1111.

HiDiffusion is built on the diffusers framework. The "one line" thing only works on UIs that use the diffusers framework.

A quick google didn't turn up many options, just https://github.com/invoke-ai/InvokeAI which is still under active development, and an old one, https://github.com/abhishekkrthakur/diffuzers

There's a couple of possible places it could go in invokeAI:

InvokeAI\invokeai\backend\model_manager\load\model_loaders\stable_diffusion.py
InvokeAI\invokeai\backend\model_manager\load\model_loaders\generic_diffusers.py

But diffuzers looks super simple:

diffuzers\diffuzers\text2img.py

Best of luck!

@CCpt5
Copy link

CCpt5 commented Apr 26, 2024

ComfyUI developer Kijai added a simple SD1.5 wrapper implementation of it in his ELLA node: https://github.com/kijai/ComfyUI-ELLA-wrapper. See commit from April 24: https://github.com/kijai/ComfyUI-ELLA-wrapper/commits/main/

However, I've had (and confirmed w/ him that he has had) a reported issue w/ generating non-square resolutions. The fix pushed yesterday which closed the thread here on it did not resolve it unfortunately: (Closed case here):#6

Also per that link the SD.Next developer Vlad is working on incorporating it on his (formerly fork) of A1111. I think it's only in his dev branch for now but that might be the UI to have full implementation first.

Even if the non-square problem is worked out,, Kijai's comfy wrapper wouldn't support SDXL since ELLA doesn't. Aparently there are a lot of things to consider when making a node for this specifically. I hope the developers here can make nodes/extensions happen!!

@ShenZhang-Shin
Copy link
Collaborator

Is the non-square resolution problem still happens? Man, let me know what model and resolution you are using.

@CCpt5
Copy link

CCpt5 commented Apr 26, 2024

Is the non-square resolution problem still happens? Man, let me know what model and resolution you are using.

I ran the error I kept getting last night after trying the latest commit you pushed through Claude Opus. This was their reply on it:
123123444
sdffvsv

A lot of us are very excited about your work here, so I hope we can get it running in Comfy and/or A111 (automatic).

@ShenZhang-Shin
Copy link
Collaborator

you run hidiffusion on ComfyUI?
I see the problem is the size mismatch between hidden_states and res_hidden_states. But I still don’t know the image resolution.
If I know it, I can reproduce the problem on my device and solve it easily.

@CCpt5
Copy link

CCpt5 commented Apr 27, 2024

you run hidiffusion on ComfyUI? I see the problem is the size mismatch between hidden_states and res_hidden_states. But I still don’t know the image resolution. If I know it, I can reproduce the problem on my device and solve it easily.

Developer Kijai added a very limited wrapper implementation of it in a node he created for ELLA. His wrapper only supports the SD1.5 but implemented here: kijai/ComfyUI-ELLA-wrapper@457c9b8

Can find a bunch of the comfy developers in the Banodoco server on Discord (AnimateDiff etc): https://discord.gg/r22ZNk3J

I had issues w/ 1080x720 for example. Don't know exactly what I had been trying when I got that error above. I asked the developer of the node (Kijai) and he replied that he had issues even after the latest commit (fix) also. Here are the relevant convos:

45151514444

fsdffv

sdsxcx

@florestefano1975
Copy link

florestefano1975 commented Apr 27, 2024

I created this nodes to start testing this technology on ComfyUI: https://github.com/florestefano1975/ComfyUI-HiDiffusion

@CCpt5
Copy link

CCpt5 commented Apr 27, 2024

I created this nodes to start testing this technology on ComfyUI: https://github.com/florestefano1975/ComfyUI-HiDiffusion

Thanks for a way to trial. I assume you can only use the base model it pulls w/ your wrapper here?

@Remember2015
Copy link

I created this nodes to start testing this technology on ComfyUI: https://github.com/florestefano1975/ComfyUI-HiDiffusion

So, does this plugin work properly now?

@xiao2mo
Copy link

xiao2mo commented May 11, 2024

you run hidiffusion on ComfyUI? I see the problem is the size mismatch between hidden_states and res_hidden_states. But I still don’t know the image resolution. If I know it, I can reproduce the problem on my device and solve it easily.

Developer Kijai added a very limited wrapper implementation of it in a node he created for ELLA. His wrapper only supports the SD1.5 but implemented here: kijai/ComfyUI-ELLA-wrapper@457c9b8

Can find a bunch of the comfy developers in the Banodoco server on Discord (AnimateDiff etc): https://discord.gg/r22ZNk3J

I had issues w/ 1080x720 for example. Don't know exactly what I had been trying when I got that error above. I asked the developer of the node (Kijai) and he replied that he had issues even after the latest commit (fix) also. Here are the relevant convos:

45151514444

fsdffv

sdsxcx

Hi,Exactly the same question here. I just use the code for SDXL inference not the ComfyUI. Have u solved the problem yet?

@xiao2mo
Copy link

xiao2mo commented May 11, 2024

-> 1710 hidden_states = torch.cat([hidden_states, res_hidden_states], dim=1)
1712 if self.training and self.gradient_checkpointing:
1714 def create_custom_forward(module, return_dict=None):

RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 90 but got size 45 for tensor number 1 in the list.

I just don't know why the feature map is half

@greasebig
Copy link

is there any originally implemented extension that can load A1111 webui model?

@ShenZhang-Shin
Copy link
Collaborator

-> 1710 hidden_states = torch.cat([hidden_states, res_hidden_states], dim=1) 1712 if self.training and self.gradient_checkpointing: 1714 def create_custom_forward(module, return_dict=None):

RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 90 but got size 45 for tensor number 1 in the list.

I just don't know why the feature map is half

You can try the latest hidiffusion version and the problem would be fixed

pip3 install hidiffusion -U

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

8 participants