Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

local-apps: update node-llama-cpp snippet #1169

Merged
merged 2 commits into from
Feb 26, 2025

Conversation

giladgd
Copy link
Contributor

@giladgd giladgd commented Feb 4, 2025

This PR updates the code snippet of node-llama-cpp to use a tag-based model URI, like in llama.cpp and Ollama.
The implementation for this uses the Ollama support on Hugging Face, and is based on this PR: ggml-org/llama.cpp#11195

Examples:

# chat with the model
npx -y node-llama-cpp chat hf:mradermacher/Llama-3.2-3B-Instruct-GGUF:Q4_K_M

# estimate the model compatibility with the current machine hardware
npx -y node-llama-cpp inspect estimate hf:mradermacher/Llama-3.2-3B-Instruct-GGUF:Q4_K_M

Copy link
Member

@Vaibhavs10 Vaibhavs10 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Amazing! Let's wait for @ngxson to give an LGTM too and merge!

` --model "hf:${model.id}/${filepath ?? "{{GGUF_FILE}}"}" \\`,
` --prompt 'Hi there!'`,
].join("\n"),
content: `npx -y node-llama-cpp chat hf:${model.id}${tagName}`,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice, this looks quite clean 👍

@giladgd
Copy link
Contributor Author

giladgd commented Feb 12, 2025

@ngxson Let me know if you need me to make changes to the PR to merge it

@giladgd
Copy link
Contributor Author

giladgd commented Feb 25, 2025

Do you need any input from me to merge this PR? It'd be a bummer to see this PR get stale...

@julien-c
Copy link
Member

@ngxson i think we can merge if you approve

@ngxson
Copy link
Member

ngxson commented Feb 26, 2025

The failed CI doesn't relate to this PR, so I assume it's ok to merge?

@julien-c
Copy link
Member

The failed CI doesn't relate to this PR, so I assume it's ok to merge?

yes imo

@ngxson ngxson merged commit f0361f3 into huggingface:main Feb 26, 2025
2 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants