DDP has slower training speed

[jyuntins]

Hi, I am trying to Finetune HMR2.0 on 3 RTX3090. It works fine when I only train it on 1 gpu.
When I set trainer.devices=3, I received the error:
ValueError: ctypes objects containing pointers cannot be pickled

I found a workaround to solve this is to use ddp strategy just as you did in the ddp trainer config file.
However, when I set the trainer to ddp, the speed drops from 2.5 it/s to 7s/it.

I wonder which config file is used when you trained hmr2.0? Is there anyway to accelerate the training speed in this configuration?

[wtx9527]

Hi, I have the same problem, have you solved it?

[jyuntins]

Hi, @wtx9527
No, I didn't solve it.

[wangsen1312]

I met the same problem when I use the multi GPU training