Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding default parameter for max_new_tokens in TransformersModel #604

Merged
merged 3 commits into from
Feb 13, 2025

Conversation

matfrei
Copy link
Contributor

@matfrei matfrei commented Feb 11, 2025

Related to issue #414.
This PR adds a default value of 5000 tokens for max_new_tokens amd max_tokens to TransformersModel.__call__ to avoid using the corresponding transformers default value (= 20 tokens) which is too low for agentic tasks.

Usage example:
Modify the last few lines of the Text2SQL example to be

from smolagents import CodeAgent, TransformersModel

agent = CodeAgent(
    tools=[sql_engine],
    model=TransformersModel("meta-llama/Meta-Llama-3.1-8B-Instruct"),
    max_steps=5
)

agent.run("Can you give me the name of the client who got the most expensive receipt?")

@aymeric-roucher
Copy link
Collaborator

I'd say it's better to set this ddefault max_new_tokens value directly in the init! The way parameters in models work is that any parameter passed in call will override the init value anyway.

@matfrei
Copy link
Contributor Author

matfrei commented Feb 12, 2025

Yeah, I was kinda divided on that. I chose this solution as found this less likely to break unexpectedly should the interface ever change.
But sure, I'll move it to init!

@aymeric-roucher aymeric-roucher merged commit f3ee605 into huggingface:main Feb 13, 2025
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants