What happens if embeddings_only is set to False?

[jackkwok]

The FAQ talked about embeddings_only boolean flag. If it is set to True, user messages are matched to the canonical examples in rails.co. But what happens if embeddings_only is set to False? The doc doesn't seem to explain it, at least I don't see it.

The issue I have with embeddings_only set to True is that to many users messages are mismatched to the wrong sample utterances in rails.co. Due to the large variety of allowed user questions, it's impracticable for me to give a comprehensive list of example utterances.

Please help me understand embeddings_only.

[drazvan]

Hi @jackkwok! The default value is False. That means that by default the LLM is used to determine the canonical form for the user message. In 0.10.0 we'll add support to specify a threshold for the confidence level. This should limit significantly the number of LLM calls, but still use the LLM for more difficult cases.

[jackkwok]

Thanks @drazvan : What does it take to add this thresholding feature? Which files do I need to modify? I am not sure I can wait til v0.10.0.

[drazvan]

Hi @jackkwok !

The critical bit is here:
https://github.com/NVIDIA/NeMo-Guardrails/blob/develop/nemoguardrails/embeddings/basic.py#L273

It would need to also specify include_distances as per the Annoy API (https://github.com/spotify/annoy?tab=readme-ov-file#full-python-api - see the a.get_nns_by_vector sginature). After the results are retrieved, you also need to filter the ones above the threshold.

In terms of propagating a config, e.g. search_threshold, make sure you update this bit as well: https://github.com/NVIDIA/NeMo-Guardrails/blob/develop/nemoguardrails/rails/llm/llmrails.py#L396.

We plan to implement this in the next couple of weeks. So, it should land in the develop branch by the end of this month (and released at some point mid-June). Let me know if you want to start contributing this and create a PR.