Does it support private deployment

[su-zelong]

May I ask if this plugin supports local models packaged in OpenAI format using VLLM

[WissamAntoun]

Hey, I created a version of this app last week that addresses this https://github.com/WissamAntoun/GalacTex

[su-zelong]

Hey, I created a version of this app last week that addresses this https://github.com/WissamAntoun/GalacTex

Thank you very much for your reply. I have carefully read this app. Is it currently only supported for Galactica models? Can I respond if I use URLs from other local models? For example: chatglm+vllm

[WissamAntoun]

it was intended for galactica, but it should support any model supported by vllm

[achiyae]

Interesting. I was not aware of vLLM/Galactica.

Yesterday, I released version 1.4.0, which should allow connecting any vLLM model.
This version changes the API to the chat completion API and adds the ability to configure the plugin via a JSON editor that was added to the configuration page. Since vLLM supports chat completion API - the plugin will work with vLLM as well. See this image:

With this editor, you can change the URL and configure the hyperparameters for the model. You can do that for each command and for all of them. You will still need to set the API key to enable the plugin, but it doesn't have to be a real API key.

[su-zelong]

Interesting. I was not aware of vLLM/Galactica.

Yesterday, I released version 1.4.0, which should allow connecting any vLLM model. This version changes the API to the chat completion API and adds the ability to configure the plugin via a JSON editor that was added to the configuration page. Since vLLM supports chat completion API - the plugin will work with vLLM as well. See this image:

With this editor, you can change the URL and configure the hyperparameters for the model. You can do that for each command and for all of them. You will still need to set the API key to enable the plugin, but it doesn't have to be a real API key.

What a great job! This extension will definitely become popular.

[achiyae]

Thanks!

Let me know how if it works indeed.
If it does, I would appreciate it if you could add an explanation on how to do that to README.md.

Help us spreading the word by staring the project and adding a review to the extension homepage.