Skip to content

Commit

Permalink
Add auto save ckpt at the end of training (#260)
Browse files Browse the repository at this point in the history
Currently, checkpoints will not be saved until the training steps
satisfy the saving frequency. This PR adds an auto-save ckpt function at
the end of training.
  • Loading branch information
YSLIU627 authored Feb 15, 2025
1 parent 1703c34 commit 8003e87
Showing 1 changed file with 4 additions and 0 deletions.
4 changes: 4 additions & 0 deletions verl/trainer/ppo/ray_trainer.py
Original file line number Diff line number Diff line change
Expand Up @@ -964,4 +964,8 @@ def fit(self):
val_metrics = self._validate()
pprint(f'Final validation metrics: {val_metrics}')
logger.log(data=val_metrics, step=self.global_steps)
if self.config.trainer.save_freq > 0 and \
(self.global_steps - 1) % self.config.trainer.save_freq != 0:
with _timer('save_checkpoint', timing_raw):
self._save_checkpoint()
return

0 comments on commit 8003e87

Please sign in to comment.