Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rework of train_model function #65

Closed
wants to merge 8 commits into from
Closed

Rework of train_model function #65

wants to merge 8 commits into from

Conversation

Old-Shatterhand
Copy link
Member

I reworked the metric-handling in the train_model function by changing the following:

  • Adding metrics. The metric calculations are still fully based on the scikit-learn. It now tracks the following:
    • ACC, MCC, and AUROC (new) for single- and multi-class classification
    • ACC (new), MCC (new), LRAP, and NDCG for multilabel classification
    • MSE, MAE (new), and R2 (new) for regression
  • Weighting of the metrics when summarizing based on individual batch sizes (the last batch might be smaller than the others). It only has minor effects, but it is still more precise.
  • The best performances are now taken from the best model (not the best ever-seen value which might come from a different model).
  • The best model is determined based on the lowest loss value (applies to all classification and regression settings).
  • The function has a new boolean parameter return_metrics, which defaults to False to preserve the old behaviour. If set to True, the function returns the best model and all training and validation metrics as a tuple of type Tuple[torch.nn.Module, dict["train": dict[...], "val": dict[...]]].
  • Added comment to the optimizer argument that it has to be a SAM optimizer for classification settings. The standard optimizers like Adam don't have first_step and second_step functionality.
  • Telling names. Before, best_acc could have been an actual accuracy, an MSE, or some LRAP. This functionality has been replaced by a variable called best_lead_metric that is inferred from the validation ACC, MSE, or LRAP.

The new code is tested for a LectinOracle training (works) and should not tamper with the old behaviour, i. e., everything possible before, is still possible. Only the selection of the best model has changed to be loss-based and not lead-metric-based (best_acc before).

@Old-Shatterhand Old-Shatterhand added the enhancement New feature or request label Nov 14, 2024
@Bribak
Copy link
Contributor

Bribak commented Nov 14, 2024

Thank you & great job!
But could you merge into dev instead? Since the documentation is generated from master, any development work could/would derail, or at least alter, the documentation of the currently stable version. We'll then roll it into master with v1.5

@Bribak Bribak deleted the dev_dl branch November 14, 2024 15:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants