How to restart one DPA-2 training task in DP-GEN?

[chenggoj]

Hi folks,

I am just a new user. I am running an unfinished DPA-2 model by DP-GEN. I am setting "stop_batch": 500000,. So I have already finished 001 and 002 models. 003 was still running but was stopped and 000 has not started yet.

The command I am using in machine.json is "command": "CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --nnodes=1 --nproc_per_node=auto --no-python dp --pt". If I would like to restart the DP-GEN job, what should I do in this case?

Is this CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --nnodes=1 --nproc_per_node=auto --no-python dp --restart --pt enough? Do I need to specify the model.ckpt.pt? How to specify the model.ckpt.pt precisely?

I know for DeePMD-kit, I should use "dp train --restart model.ckpt". I am just not sure what should I do in DP-GEN. I am REALLY confused.
Could anyone please give me one example to show how to set a restart task in DP-GEN?

One more question, if the DP-GEN task was stopped at the stage of LAMMPS or VASP. What should I do to restart them then?
Thanks for your time.

The tree structure of DeePMD-kit is here.

└── 24c4d5b438050112eb93d40c5a20dd236857aaa1
├── 000
│   └── input.json -> /expanse/projects/qstore/mia344/cjiang1/NNPs-traing-DeepMD-kit/DP-GEN/run_test_DPA2/iter.000000/00.train/000/input.json
├── 001
│   ├── checkpoint
│   ├── dpa2.hdf5
│   ├── input.json -> /expanse/projects/qstore/mia344/cjiang1/NNPs-traing-DeepMD-kit/DP-GEN/run_test_DPA2/iter.000000/00.train/001/input.json
│   ├── input_v2_compat.json
│   ├── lcurve.out
│   ├── model.ckpt-492000.pt
│   ├── model.ckpt-494000.pt
│   ├── model.ckpt-496000.pt
│   ├── model.ckpt-498000.pt
│   ├── model.ckpt-500000.pt
│   ├── model.ckpt.pt -> model.ckpt-500000.pt
│   ├── out.json
│   └── train.log
├── 002
│   ├── checkpoint
│   ├── dpa2.hdf5
│   ├── input.json -> /expanse/projects/qstore/mia344/cjiang1/NNPs-traing-DeepMD-kit/DP-GEN/run_test_DPA2/iter.000000/00.train/002/input.json
│   ├── input_v2_compat.json
│   ├── lcurve.out
│   ├── model.ckpt-492000.pt
│   ├── model.ckpt-494000.pt
│   ├── model.ckpt-496000.pt
│   ├── model.ckpt-498000.pt
│   ├── model.ckpt-500000.pt
│   ├── model.ckpt.pt -> model.ckpt-500000.pt
│   ├── out.json
│   ├── test2.py
│   ├── test.py
│   └── train.log
├── 003
│   ├── checkpoint
│   ├── dpa2.hdf5
│   ├── input.json -> /expanse/projects/qstore/mia344/cjiang1/NNPs-traing-DeepMD-kit/DP-GEN/run_test_DPA2/iter.000000/00.train/003/input.json
│   ├── input_v2_compat.json
│   ├── lcurve.out
│   ├── model.ckpt-220000.pt
│   ├── model.ckpt-222000.pt
│   ├── model.ckpt-224000.pt
│   ├── model.ckpt-226000.pt
│   ├── model.ckpt-228000.pt
│   ├── model.ckpt-230000.pt
│   ├── model.ckpt.pt -> model.ckpt-230000.pt
│   ├── out.json
│   └── train.log
├── 24c4d5b438050112eb93d40c5a20dd236857aaa1.json
├── 7eb153230c83736e655201e8443dbdf4ad716db0_flag_if_job_task_fail
├── 7eb153230c83736e655201e8443dbdf4ad716db0_job_id
├── 7eb153230c83736e655201e8443dbdf4ad716db0_last_err_file
├── 7eb153230c83736e655201e8443dbdf4ad716db0.out
├── 7eb153230c83736e655201e8443dbdf4ad716db0.sub
├── 7eb153230c83736e655201e8443dbdf4ad716db0.sub.run
└── data.init
└── init

7 directories, 50 files

[Chandler-Lu]

https://github.com/deepmodeling/dpgen/blob/master/doc/run/overview-of-the-run-process.md