MusicGen training failure

[jhurlbut]

I got this exception in the logs. I am assuming this is to do with a lack of enough GPU memory? I have trained with similar settings previously though

nendo-server | Exception: [' return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,', ' File "/opt/conda/lib/python3.8/subprocess.py", line 516, in run', ' raise CalledProcessError(retcode, process.args,', "subprocess.CalledProcessError: Command '['python', '-m', 'dora', '-P', 'audiocraft', 'run', 'solver=musicgen/musicgen_base_32khz', 'model/lm/model_scale=small', 'conditioner=text2music', 'device=cuda', 'continue_from=//pretrained/facebook/musicgen-small', 'datasource.max_sample_rate=44100', 'datasource.train=/home/nendo/.cache/nendo/models/musicgen/085df796-cb6b-4251-9d17-758c720114e5/StrictlyTraining/Strictly-prompt:asdfwert-epoch256-batch2-medium-rate.0001', 'dataset.valid.num_samples=1', 'dataset.generate.num_samples=1', 'dataset.train.num_samples=152', 'optim.epochs=256', 'optim.lr=0.0001', 'schedule.lr_scheduler=cosine', 'schedule.cosine.warmup=8', 'schedule.polynomial_decay.warmup=8', 'schedule.inverse_sqrt.warmup=8', 'schedule.linear_warmup.warmup=8', 'classifier_free_guidance.training_dropout=0.3', 'logging.log_updates=10', 'dataset.batch_size=2', 'optim.optimizer=adamw', 'dataset.train.permutation_on_files=True', 'optim.updates_per_epoch=100']' returned non-zero exit status 1.", '']