docs: add ADR for RAG Evaluations Framework #842
Conversation
jalling97
commented
Jul 26, 2024
•
jalling97
commented
- Adds the ADR for the overall RAG Evaluations Framework
- This will be added to for each component of the RAG Evals MVP Epic and closed when all pieces are done and all decisions have been documented in the ADR.
✅ Deploy Preview for leapfrogai-docs canceled.
|
jalling97
added this to the Current - RAG UX Enhancements | Model Directory | API Odds and Ends milestone
jalling97
changed the title
jalling97
changed the title
jalling97
changed the title
jalling97
changed the title
adr/0007-rag-eval-framework.md
Outdated
|
|
||
| #### Rationale | ||
|
|
||
| These datasets were created because it filled a gap in the openly available datasets that could have been used. For example, in QA datasets, there did not exist any dataset that had all **4** components listed above. Many had the questions, answers, and context, but none also included the source documents in a readily accessible manner. Therefore, the fastest and most effective course of action was to generate a QA dataset from source documentation using the [DeepEval Synthesizer](https://docs.confident-ai.com/docs/evaluation-datasets-synthetic-data). |
There was a problem hiding this comment.
Explain why we picked the reference documents we used as sources for this data set. It was a mix of technical documentation & DoD policy docs to capture the type of data we generally expect our mission users will be working with.
adr/0007-rag-eval-framework.md
Outdated
|
|
||
| ## Context | ||
|
|
||
| LeapfrogAI uses RAG to provide context-aware responses to users who have specific data they need to reference. In order to make sure RAG is operating at the levels we need it to, we need to get measurable feedback from our RAG pipeline to make it better. We also need a standard to show to mission heroes that we are in fact operating at that level. We do this with RAG-focused evaluations. This ADR documents all of the decisions and lessons learned for enabling a full-scale RAG evaluations pipeline MVP. |
There was a problem hiding this comment.
In addition to evaluating and improving our "default" RAG implementation in LeapfrogAI, developing a standard approach to evals will help make delivery folks more efficient at evaluating custom RAG implementations they may build on and around the core LeapfrogAI.
|
|
||
| #### Decision | ||
|
|
||
| The three models that will initially be evaluated are going to be: |
There was a problem hiding this comment.
It may be good to provide a small sections about why we chose the quantization and sizes of the models here. For example, reasons like consistency with # of active params or vRAM limitations, so that it doesn't seem like we selected 4-bit quantization for these model sizes arbitrarily.
There was a problem hiding this comment.
Made some updates, let me know what you think!
adr/0007-rag-eval-framework.md
Outdated
|
|
||
| In order to reach an MVP product, a single LLM judge will be utilized for the evaluations that require it. This will be the first stage so that the evaluation framework can begin receiving results. As progress is made, additional LLM-based judges will be incorporated to develop an LLM-jury styled approach. For context, please see the following [paper](https://arxiv.org/pdf/2404.18796). | ||
|
|
||
| Claude 3.5 Sonnet was chosen to be used as the first judge due to it's high levels of [performance](https://artificialanalysis.ai/models/claude-35-sonnet), which is crucial when utilizing an LLM judge. Additionally, it exists outside the family of models that will be evaluated against, which has been shown to be effective in comparison to using models of the same family due to [self-enhancement bias](https://arxiv.org/pdf/2306.05685). |
There was a problem hiding this comment.
Can you provide a little more context on what performance here? Maybe talk about its reasoning scores, like MMLU or GPQA? Might also be good to mention its large context window for digesting and analyzing the in/output for the evaluated models.
There was a problem hiding this comment.
Made some updates, let me know what you think!
Co-authored-by: Justin Law <[email protected]>
Co-authored-by: Justin Law <[email protected]>
