Differences between model prompts #89
Replies: 2 comments
-
Hi @jacob-mink! Indeed, the behaviour depends on the LLM model. Some models, like |
Beta Was this translation helpful? Give feedback.
-
Hi @jacob-mink ! We have done several experiments to improve the prompts for the various LLMs that we have explored. Of course, this is a continuous process and the prompts can still be improved - in the last month we have put more energy in improving the prompts and assessing the performance of open source models. Some results are available here and the nemoguardrails/eval package contains various tools to test topical rails, but also execution rails (each tool has its own detailed README on how to use). It would be great for people from the community to improve the current existing prompts, especially as now we have provided these initial set of tools and datasets to assess the performance of the models with different prompts. What we have observed is that LLMs that are run in chat vs completion mode act differently:
Hope this helps. |
Beta Was this translation helpful? Give feedback.
-
tl;dr: The bot spits out intents as responses instead of generating a response. I'm curious if it has to do with the generate_user_intent prompt.
Hi NeMo-Guardrails team. As I've been playing with the tools you've provided and been perusing the repository, I've noticed a behavior that led me to the llm/prompts/ folder. I haven't filed an issue yet because I'm not really sure that it IS an issue, maybe just a normal LLM limitation.
Let me describe what I've observed first. The behavior is a failure to generate & parse an appropriate intent - things along the lines of
Then a prompt like "What's up?" might have the end output of the bot be something like "user ask status". Obviously, this isn't even one of the described utterances... but where did it come from?
Now, the research. I found a couple key differences in the prompts for determining user intent. Taking a look at https://github.com/NVIDIA/NeMo-Guardrails/blob/main/nemoguardrails/llm/prompts/openai-chatgpt.yml#L7 and comparing it to https://github.com/NVIDIA/NeMo-Guardrails/blob/main/nemoguardrails/llm/prompts/cohere.yml#L24, the key difference is that the cohere intent prompt SPECIFICALLY asks the LLM to generate an intent, while the OpenAI ChatGPT one does not.
So is this a conscious choice to not have that specific instruction for ChatGPT, or is it an issue? Has anyone else noticed the bot spitting out some weird intent messages instead of an actual response?
Beta Was this translation helpful? Give feedback.
All reactions