Evidence - FeedbackEvaluator Storage & Basic UI #12758
Open
+577
−16
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dandrabik
changed the title
dandrabik
requested review from
brendanshean,
emilia-friedberg and
anathomical
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
WHAT
Run the FeedbackEvaluator after trials complete, store in a DB table and roll-up, show in UI.
WHY
We want to show this info in the UI of the research tool to make prompt engineering on the feedback prompt more efficient.
HOW
Create a new table
evidence_research_gen_ai_feedback_evaluationsand run evaluations when GEvalScores are calculated. Show in UI.Screenshots
https://www.notion.so/quill/Feedback-Evaluation-Metrics-Storage-185d42e6f941807191e3ddc2da872cd7
What have you done to QA this feature?
I've run some tests locally in the console while connected to the staging DB. I also ran the metrics on staging and seemed to work properly. Viewed the UI locally to ensure the metrics show properly.