Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DC-AE, SANA] fix SanaMultiscaleLinearAttention apply_quadratic_attention bf16 #10595

Merged
merged 14 commits into from
Jan 16, 2025
Merged

[DC-AE, SANA] fix SanaMultiscaleLinearAttention apply_quadratic_attention bf16 #10595

Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
2643c74
autoencoder_dc tiling
chenjy2003 Jan 9, 2025
03fba80
Merge remote-tracking branch 'upstream/main' into DC-AE-tiling
chenjy2003 Jan 9, 2025
c0b1ca5
add tiling and slicing support in SANA pipelines
chenjy2003 Jan 10, 2025
d80dea5
create variables for padding length because the line becomes too long
chenjy2003 Jan 10, 2025
0a581bc
Merge remote-tracking branch 'upstream/main' into DC-AE-tiling
chenjy2003 Jan 10, 2025
c3b9a8e
add tiling and slicing support in pag SANA pipelines
chenjy2003 Jan 10, 2025
baa9e6f
Merge remote-tracking branch 'upstream/main' into DC-AE-tiling
chenjy2003 Jan 10, 2025
a8f277e
Merge branch 'main' into DC-AE-tiling
chenjy2003 Jan 10, 2025
2fad762
revert changes to tile size
a-r-r-o-w Jan 11, 2025
d2f2d2b
make style
a-r-r-o-w Jan 11, 2025
84854b4
Merge branch 'main' into DC-AE-tiling
a-r-r-o-w Jan 11, 2025
477937e
add vae tiling test
a-r-r-o-w Jan 11, 2025
bba83a4
fix SanaMultiscaleLinearAttention apply_quadratic_attention bf16
chenjy2003 Jan 16, 2025
7a30b76
Merge remote-tracking branch 'upstream/main' into DC-AE-tiling
chenjy2003 Jan 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/diffusers/models/attention_processor.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -899,7 +899,7 @@ def apply_quadratic_attention(self, query: torch.Tensor, key: torch.Tensor, valu
scores = torch.matmul(key.transpose(-1, -2), query)
scores = scores.to(dtype=torch.float32)
scores = scores / (torch.sum(scores, dim=2, keepdim=True) + self.eps)
hidden_states = torch.matmul(value, scores)
hidden_states = torch.matmul(value, scores.to(value.dtype))
return hidden_states

def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
Expand Down