Purporse of orientation parameter in rag_tuning.

[Nithyashree-coder]

Hello team,
I was trying out use case example rag_tuning. I have a doubt on the parameter you are optimising.

1. You are taking orientation as kind and item. What is the significance of this parameter for RAG?
2. Please provide an example input and output of orientation parameter. If it is "item", how the input and output will be? If it is "kind", how the input and output will be? Please provide with an example context, questions and answers.

[t-schn]

Hi,

if the orientation is 'kind', then inputs and outputs are grouped together. It is is item, then all fields for items are grouped together.

E.g, in JSON {"inputs": [item1, item2..], "outputs": [item1, item2..]} vs {item1: {"input": <>, "output": <>}.

This makes sense trying out to see if locality impacts performance.

[Nithyashree-coder]

Thank you for your answer.
As per documentation, SAMMO predominantly works with optimization of prompts. I actually want to know the information on how the orientation parameter "kind" and "item" is optimizing the prompts. Hence I downloaded the model parameters in beam_search.json
to check the optimised prompt.
Q1. In the model iteration, if "item" is given in iteration how do i see the optimized prompt for that iteration?

I tried beam search with the following parameter values.

However, while running the code, I couldn't find orient= "kind" being iterated. I could only see orient = "item being iterated."

I tried increasing the parameters as below:

However still item = "kind" is not being iterated across the beam search.

The documentation was only having contents relating to enumerative search with RAG. However i want to also perform instruction tuning along with optimization of orientation parameter.

Q2. How to ensure both orient = "kind" and "item" are being iterated in beam search along with Instruction Optimisation?

PS: Below is the metaprompt design for your reference

[t-schn]

Good question! Three ways:

1. Use GridSearch instead of BeamSearch.
2. Increase the number of starting points that BeamSearch samples. The one_of operator will only be used to sample starting points. So, if the starting set does not contain both variants, they might not be generated.
3. Make sure you have a Mutator that does the parameter replacement. You can refer to the compression example to see how this can be done.

[Nithyashree-coder]

@t-schn ,
I tried with Enumerative Search (Grid Search) . But it doesn't have allow me to pass mutation operators. I have mutate from parameter though but it is expecting Output type data. We are still sending metaprompt output to it. Isnt it possible to perform induced instruction operation in enumerative search?

[Nithyashree-coder]

@t-schn , thank you for the reply I will check regarding this. However, I also wanted to get information on the optimised prompt from the iteration (refer Q1 of my previous comment). I couldn't see it. I checked both cache file and also the model.json file (I have attached the file above) but i couldn't find it. How to know that when "item" or "kind" is given as orientation parameter, then the particular input is optimised? I want to know the optimised prompt for the iteration.

[t-schn]

You can check the optimizer._state property for a list. Also, you should see whether the item property will be chosen in the chosen action column when you print the report.

[Nithyashree-coder]

Thank you for your reply. I checked with the code "optimizer._state["fit"][1]" but I am still unable to see the optimized prompt for orientation = "item". The "item" property is chosen as per the below screenshot.

[Nithyashree-coder]

But when I use test data with dtest_sammo_beam_search.outputs[0].plot_call_trace(), I am able to see optimized prompt in the metaprompt section.

Can this also be viewed for the training data for how the prompt got optimised for "item"/"kind"?