Change how eval function works #65
Conversation
|
I'm not sure I understand why this is needed? In case we want to eval only energies and not forces? |
|
@frostedoyster do you have an idea why the regression test is failing? |
examples/basic_usage/usage.sh
Outdated
|
|
||
| metatensor-models eval model.pt qm9_reduced_100.xyz | ||
| metatensor-models eval options.yaml model.pt --eval_on="training" |
There was a problem hiding this comment.
Not sure about this... should we just allow the user to input a single dataset (or multiple) without specific names? In particular, having a training set makes little sense when you want to evaluate a model. This will be very confusing
There was a problem hiding this comment.
I did another comment. We can change this but this means people have to write another yaml file.
| from .formatter import CustomHelpFormatter | ||
|
|
||
|
|
||
| logger = logging.getLogger(__name__) | ||
|
|
||
| CHOICES_EVAL_ON = ["training", "validation", "test"] |
There was a problem hiding this comment.
Not a big fan. Here IMO the datasets should have custom names given by the user. I might want to evaluate the "universal" PET model on my toy "silicon" dataset and on a "water" dataset, which will be read from files, but training, validation, test don't make much sense
There was a problem hiding this comment.
I see your point. But then people have to write a new .yaml file? Is this what we want?
| @@ -192,70 +178,68 @@ def _train_model_hydra(options: DictConfig) -> None: | |||
| ) | |||
| train_targets = read_targets(train_options["targets"]) | |||
| train_dataset = Dataset(train_structures, train_targets) | |||
| train_size = 1.0 | |||
There was a problem hiding this comment.
Is this hardcoded now?
There was a problem hiding this comment.
No, if you read further below thsi is adjusted.
Co-authored-by: Filippo Bigi <[email protected]>
docs/src/getting-started/usage.rst
Outdated
| @@ -66,25 +66,36 @@ The sub-command to evaluate an already trained model is | |||
|
|
|||
| metatensor-models eval | |||
|
|
|||
| Besides the trained `model` you will also have to provide a file containing the | |||
There was a problem hiding this comment.
| Besides the trained `model` you will also have to provide a file containing the | |
| Besides the trained `model`, you will also have to provide a file containing the |
the eval function will now take an
options.yamlas the train function to evaluate the RMSE if such data is available. The user can also just removes thetargetsif he doesn't have target data. Then we only predict and write a file without prompting the RMSE values.the internal functions of eval will now also be used to report the test, train and validation rmse at the end of the training.
TODO
trainandeval.