Massive Speedup with Tiny AutoEncoder (Edit: No. It was a mistake)

[muerrilla]

Just a heads up:
If using diffusers, you can get a massive speedup by using Tiny AutoEncoder instead of the original VAE (40% faster on my 4060ti) for a minor trade-off in quality.

EDIT: This was a mistake on my part. See my comment below.

with VAE:

with TAE:

[lawrence-cj]

Nice work! The main reference time is spent on model forward, so may I have a detailed time speed up here?

[muerrilla]

Sure thing. Although I did a system reboot and now I'm getting better times on both options, than I was getting yesterday, without changing anything. So the speed increase is more like 33% today! 😆

On a 4060ti 16GB, Python 2.1.0+cu11.8 with no CPU offloading and no betterTransformers, Using DPM-Solver for 20 steps at 1024x1024:
Text encoding + sampling takes 10.0 seconds, and decoding takes 2.1 seconds with TAE and 8.2 seconds with VAE. So that's 12.1s vs 18.2s (yesterday I was getting 13.1s vs. 22.2s).

[lawrence-cj]

That's a promising result. We test the difference decoder on V100 spec., the result are shown below. I think the inference speed on different GPU spec. really has a big gap.

[muerrilla]

Damn! What are the sampling times like on that beast of a GPU?

[lawrence-cj]

The whole process could be done within 8 seconds. 👍 : )

[muerrilla]

Damn! I just went through this again and realized the drastic difference was caused by me not having disabled CUDA - Sysmem Fallback, so the VAE was pushing the VRAM usage towards the limit and the whole process was being offloaded to RAM (which was not happening when using TAE).

So I fixed it, and now the difference between the two is like 1s. What an embarrassment! 😆🤦‍♂️