Download from HuggingFace with private token

[frostedoyster]

As in the title

Contributor (creator of pull-request) checklist

• Tests updated (for new features and bugfixes)?
• Documentation updated (for new features)?
• Issue referenced (for PRs that solve an issue)?

---

📚 Documentation preview 📚: https://metatrain--390.org.readthedocs.build/en/390/

[abmazitov]

I managed to adapt the PET.load_checkpoint() method, so now it works with the HuggingFace checkpoint that we have. However the mtt export API is still not well-adapted. I keep getting the following error:

Traceback (most recent call last):
  File "/home/mazitov/apps/anaconda3/envs/metatrain/lib/python3.11/site-packages/metatrain/__main__.py", line 105, in main
    export_model(**args.__dict__)
TypeError: export_model() got an unexpected keyword argument 'huggingface_api_token'

which is related to the fact that in the load_model function, we pop the 'huggingface_api_token' key from kwargs, which is the copy of the args.__dict__, so the original dict still has the unexpected key.

---

Even though loading the model explicitly worked for me, it only worked while loading from the abmazitov/pet-mad and not from the metatensor/pet-mad. Maybe I did something wrong...

[Luthaf]

Suggested change

	(Downloading private models is also supported, only through HuggingFace and using the
	``--huggingface_api_token`` flag.)
	Downloading private HuggingFace models is also supported, by specifying the corresponding API token with the ``--huggingface_api_token`` flag.

[Luthaf]

should we/can we had a test for this on CI? Maybe creating a cosmo account on HuggingFace & uploading a dummy model+checkpoint there.

[Luthaf]

I dont understand why this is required?

[PicoCentauri]

And if you want the file to be directly downloaded to the current directory you can do

path, _ = urlretrieve(
    url=str(path),
    filename=str(Path.cwd() / str(path).split("/")[-1]))

See https://docs.python.org/3/library/urllib.request.html#urllib.request.urlretrieve.

[Luthaf]

I like this! Down the line, we could maybe add a way to read the API token from env variables or a file; and use such mechanism for all external integrations (W&B seems like a very logical one to add next).

[PicoCentauri]

If you have a private model on Hugging Face, there are effective third-party tools like wget, curl, or the Hugging Face web interface to download it. Adding a private API interface introduces overhead and complexity to the codebase, and for a private Hugging Face repo, this doesn’t provide significant benefits, as there are accessible alternatives. The presented workaround also introduces an additional dependency, and I’m concerned it will require maintenance soon. We’re moving around URL schemas, and, to my knowledge, Hugging Face doesn’t have a stable schema, right?

As for W&B, it’s a different story, and I fully support adding a functionality, because users can’t access the benefits of W&B without us integrating it directly into the code.