Add support for data parallel QLoRA training via DeepSpeed Zero stages 0, 1 and 2. #11337
Job | Run time |
---|---|
25m 53s | |
30m 36s | |
23m 59s | |
12m 19s | |
27m 44s | |
29m 31s | |
17m 24s | |
12m 43s | |
22m 46s | |
2s | |
29m 2s | |
36m 27s | |
8m 58s | |
36m 31s | |
10m 48s | |
5h 24m 43s |
Job | Run time |
---|---|
25m 53s | |
30m 36s | |
23m 59s | |
12m 19s | |
27m 44s | |
29m 31s | |
17m 24s | |
12m 43s | |
22m 46s | |
2s | |
29m 2s | |
36m 27s | |
8m 58s | |
36m 31s | |
10m 48s | |
5h 24m 43s |