fix offload gpu tests etc (#10366) #4300

Job	Run time
Setup Torch Pipelines CUDA Slow Tests Matrix	1m 51s
ONNX CUDA Tests	4m 16s
Flax TPU Tests	2m 56s
PyTorch Compile CUDA tests	2m 44s
Torch CUDA Tests (models)	6m 25s
Torch CUDA Tests (schedulers)	4m 48s
Torch CUDA Tests (lora)	5m 49s
Torch CUDA Tests (others)	40s
Torch CUDA Tests (single_file)	3m 44s
Examples PyTorch CUDA tests on Ubuntu	31m 5s
PyTorch xformers CUDA tests	4m 57s
Torch Pipelines CUDA Tests (cogvideo)	3m 29s
Torch Pipelines CUDA Tests (controlnet)	12m 19s
Torch Pipelines CUDA Tests (ddpm)	39s
Torch Pipelines CUDA Tests (flux)	3m 29s
Torch Pipelines CUDA Tests (ip_adapters)	2m 13s
Torch Pipelines CUDA Tests (kandinsky2_2)	5m 22s
Torch Pipelines CUDA Tests (kandinsky3)	3m 33s
Torch Pipelines CUDA Tests (latent_consistency_models)	3m 0s
Torch Pipelines CUDA Tests (ltx)	2m 47s
Torch Pipelines CUDA Tests (pixart_alpha)	2m 33s
Torch Pipelines CUDA Tests (stable_diffusion)	3m 44s
Torch Pipelines CUDA Tests (stable_diffusion_2)	6m 42s
Torch Pipelines CUDA Tests (stable_diffusion_3)	1m 18s
Torch Pipelines CUDA Tests (stable_diffusion_adapter)	1m 52s
Torch Pipelines CUDA Tests (stable_diffusion_xl)	6m 21s
Torch Pipelines CUDA Tests (stable_video_diffusion)	51s
	2h 9m 27s

Provide feedback