Evidence - FeedbackEvaluator Storage & Basic UI #12758
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
emilia-friedberg
approved these changes
Feb 3, 2025
There was a problem hiding this comment.
This looks good to me -- nice work cleanly following the existing paradigm on the frontend. It looks like you might be having the same issue with snapshot datetimes that @brendanshean did last week, maybe he can speak to how he resolved that.
brendanshean
approved these changes
Feb 3, 2025
services/QuillLMS/engines/evidence/app/models/evidence/research/gen_ai/trial.rb
Outdated
Show resolved
Hide resolved
...S/engines/evidence/spec/workers/evidence/research/gen_ai/run_trial_evaluation_worker_spec.rb
Outdated
Show resolved
Hide resolved
anathomical
approved these changes
Feb 3, 2025
dandrabik
commented
Feb 5, 2025
services/QuillLMS/engines/evidence/app/models/evidence/research/gen_ai/trial.rb
Show resolved
Hide resolved
brendanshean
approved these changes
Feb 5, 2025
services/QuillLMS/engines/evidence/app/models/evidence/research/gen_ai/trial.rb
Show resolved
Hide resolved
dandrabik
added a commit
that referenced
this pull request
Feb 5, 2025
Evidence - FeedbackEvaluator Storage & Basic UI (#12758)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
WHAT
Run the FeedbackEvaluator after trials complete, store in a DB table and roll-up, show in UI.
WHY
We want to show this info in the UI of the research tool to make prompt engineering on the feedback prompt more efficient.
HOW
Create a new table
evidence_research_gen_ai_feedback_evaluationsand run evaluations when GEvalScores are calculated. Show in UI.Screenshots
https://www.notion.so/quill/Feedback-Evaluation-Metrics-Storage-185d42e6f941807191e3ddc2da872cd7
What have you done to QA this feature?
I've run some tests locally in the console while connected to the staging DB. I also ran the metrics on staging and seemed to work properly. Viewed the UI locally to ensure the metrics show properly.