Is training possible?

[ArchAngelAries]

I've tried training models, textual inversions, etc and it just fails with errors. I don't think I'm doing anything wrong, but I'm just wondering if it's possible or not to train anything using this version? I'm on an AMD 7900 XT 20GB.

[lshqqytiger]

Because my graphic card has just 8GB of vram, I can't test training.
If you attach error message and traceback, I can try to fix that error.

[allancorrigall]

I'm also attempting to train hypernetwork and its crashing for me around at around. 700-800 steps. this is the error im getting w/ 6800 XT

File "D:\SD\auto1\stable-diffusion-webui-directml\modules\hypernetworks\hypernetwork.py", line 660, in train_hypernetwork
scaler.scale(loss).backward()
File "D:\SD\auto1\stable-diffusion-webui-directml\venv\lib\site-packages\torch_tensor.py", line 488, in backward
torch.autograd.backward(
File "D:\SD\auto1\stable-diffusion-webui-directml\venv\lib\site-packages\torch\autograd_init_.py", line 197, in backward
Variable.execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
File "D:\SD\auto1\stable-diffusion-webui-directml\venv\lib\site-packages\torch\autograd\function.py", line 267, in apply
return user_fn(self, *args)
File "D:\SD\auto1\stable-diffusion-webui-directml\venv\lib\site-packages\torch\utils\checkpoint.py", line 157, in backward
torch.autograd.backward(outputs_with_grad, args_with_grad)
File "D:\SD\auto1\stable-diffusion-webui-directml\venv\lib\site-packages\torch\autograd_init.py", line 197, in backward
Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
RuntimeError: The application made a call that is invalid. Either the parameters of the call or the state of some object was incorrect.
Enable the D3D debug layer in order to see details via debug messages.

[lshqqytiger]

Known issue. microsoft/DirectML#407

[ArchAngelAries]

Known issue. microsoft/DirectML#407

So does that mean training isn't possible? I'm so sick of paying for google colab just to train anything. I can't even train something small and simple like a textual inversion. error I'm getting:

Traceback (most recent call last):
File "D:\AI\stable-diffusion-webui-directml_updated\modules\textual_inversion\textual_inversion.py", line 503, in train_embedding
scaler.scale(loss).backward()
File "D:\AI\stable-diffusion-webui-directml_updated\venv\lib\site-packages\torch_tensor.py", line 488, in backward
torch.autograd.backward(
File "D:\AI\stable-diffusion-webui-directml_updated\venv\lib\site-packages\torch\autograd_init_.py", line 197, in backward
Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
RuntimeError: The GPU device instance has been suspended. Use GetDeviceRemovedReason to determine the appropriate action.

Error completing request
Arguments: ('task(4ahsy4h4s004lcp)', 'EllaBalinska', '0.01:100, 0.007:1000, 0.005:2500, 0.0001:150000\n', 1, 1, 'C:\Users\archa\Downloads\EllaBalinska\Processed\New folder', 'textual_inversion', 512, 512, False, 150000, 'disabled', '0.1', True, 0.1, 'deterministic', False, 250, 250, 'subject_filewords_custom.txt', True, False, '', '', 20, 0, 7, -1.0, 512, 512) {}
Traceback (most recent call last):
File "D:\AI\stable-diffusion-webui-directml_updated\modules\call_queue.py", line 56, in f
res = list(func(*args, **kwargs))
File "D:\AI\stable-diffusion-webui-directml_updated\modules\call_queue.py", line 37, in f
res = func(*args, **kwargs)
File "D:\AI\stable-diffusion-webui-directml_updated\modules\textual_inversion\ui.py", line 33, in train_embedding
embedding, filename = modules.textual_inversion.textual_inversion.train_embedding(*args)
File "D:\AI\stable-diffusion-webui-directml_updated\modules\textual_inversion\textual_inversion.py", line 632, in train_embedding
shared.sd_model.first_stage_model.to(devices.device)
File "D:\AI\stable-diffusion-webui-directml_updated\venv\lib\site-packages\pytorch_lightning\core\mixins\device_dtype_mixin.py", line 113, in to
return super().to(*args, **kwargs)
File "D:\AI\stable-diffusion-webui-directml_updated\venv\lib\site-packages\torch\nn\modules\module.py", line 989, in to
return self._apply(convert)
File "D:\AI\stable-diffusion-webui-directml_updated\venv\lib\site-packages\torch\nn\modules\module.py", line 641, in _apply
module._apply(fn)
File "D:\AI\stable-diffusion-webui-directml_updated\venv\lib\site-packages\torch\nn\modules\module.py", line 641, in _apply
module._apply(fn)
File "D:\AI\stable-diffusion-webui-directml_updated\venv\lib\site-packages\torch\nn\modules\module.py", line 664, in _apply
param_applied = fn(param)
File "D:\AI\stable-diffusion-webui-directml_updated\venv\lib\site-packages\torch\nn\modules\module.py", line 987, in convert
return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
RuntimeError: The GPU device instance has been suspended. Use GetDeviceRemovedReason to determine the appropriate action.

[lshqqytiger]

zhangxiang1993, a member of DirectML team said that they will fix it in their future release.

[tr7zw]

Because my graphic card has just 8GB of vram, I can't test training.

At least on my 2070SUPER with 8GB training works just fine. Currently looking into buying a gpu like the RX 6800 XT for the 16GB of vram to be able to train on bigger datasets.

DirectML team said that they will fix it in their future release.

The linked issue got closed a month ago, and there has been a torch-directml release 4 days ago. Any update on how viable training with an AMD gpu now is?