Code

Eval config eval#226

Code

Closed

scosman wants to merge 81 commits intomainKiln-AI/Kiln:mainfrom eval_config_evalKiln-AI/Kiln:eval_config_evalCopy head branch name to clipboard

+11,053-1,046

Commits

Commits on Feb 12, 2025

• New datamodel for evals w tests.

  scosman

  committed
  c17f9d9

  Copy full SHA for c17f9d9

Commits on Feb 13, 2025

• title to json key function w tests

  scosman

  committed
  f5596e2

  Copy full SHA for f5596e2

• checkpoint of g_eval work, has working json_schema output, and initial framework. Not up and running yet

  scosman

  committed
  3aa608e

  Copy full SHA for 3aa608e

Commits on Feb 14, 2025

• Refactor our prompt ID system to 1 uniform ID, with a pydantic type and validator

  scosman

  committed
  92f7bcc

  Copy full SHA for 92f7bcc

Commits on Feb 15, 2025

• Add a prompt serialization in the eval config model. I might move this to the EvalRun but working with tests for now.

  scosman

  committed
  0055af9

  Copy full SHA for 0055af9

• Add in progress eval adaptor, and g_eval implementation

  scosman

  committed
  56f7e08

  Copy full SHA for 56f7e08

• G-evals are working with tests!!

  Show description for 8c015f3
  scosman

  committed
  8c015f3

  Copy full SHA for 8c015f3

Commits on Feb 16, 2025

• Add LLM as Judge evaluator

  scosman

  committed
  41c0e45

  Copy full SHA for 41c0e45

• Add comment

  scosman

  committed
  107f598

  Copy full SHA for 107f598

• Fix python 3.10 issue, and update cursor rules with 3.10+

  scosman

  committed
  e5bd880

  Copy full SHA for e5bd880

• Remove TODOs

  scosman

  committed
  d925443

  Copy full SHA for d925443

• CR feedback

  scosman

  committed
  76ee204

  Copy full SHA for 76ee204

• Remove unused import

  scosman

  committed
  d4fad9e

  Copy full SHA for d4fad9e

• Merge pull request #200 from Kiln-AI/g_eval

  Show description for dd32153
  scosman

  authored
  dd32153

  Copy full SHA for dd32153

• Big change:

  Show description for 4232eb1
  scosman

  committed
  4232eb1

  Copy full SHA for 4232eb1

• Remove console.log

  scosman

  committed
  446fafe

  Copy full SHA for 446fafe

• Better API typing: check for valid PromptId using pydantic types

  scosman

  committed
  6e72cf5

  Copy full SHA for 6e72cf5

• Update prompt_builder_from_id to take a typed string for extra typechecking

  scosman

  committed
  9d73952

  Copy full SHA for 9d73952

Commits on Feb 17, 2025

• Remove duplicate data storage inside the adapter

  scosman

  committed
  b261e4e

  Copy full SHA for b261e4e

• Fix ID parsing

  scosman

  committed
  e0ab86c

  Copy full SHA for e0ab86c

• Merge pull request #204 from Kiln-AI/evals_run_config

  Show description for 886d550
  scosman

  authored
  886d550

  Copy full SHA for 886d550

• Fix prompt name UI

  scosman

  committed
  781c66f

  Copy full SHA for 781c66f

Commits on Feb 18, 2025

• Refactoring:

  Show description for 24c8348
  scosman

  committed
  24c8348

  Copy full SHA for 24c8348

• Add tests, and refactor the run_config files

  scosman

  committed
  33e60ba

  Copy full SHA for 33e60ba

• Merge pull request #205 from Kiln-AI/evals_run_config_tests

  Show description for 04964b6
  scosman

  authored
  04964b6

  Copy full SHA for 04964b6

• New eval_run data structure

  scosman

  committed
  cc8daa8

  Copy full SHA for cc8daa8

• Better pydantic typing for dataset filters, similar to promptIDs.

  Show description for 6542391
  scosman

  committed
  6542391

  Copy full SHA for 6542391

• Merge pull request #209 from Kiln-AI/evals_runner

  Show description for d593572
  scosman

  authored
  d593572

  Copy full SHA for d593572

• Add datasets to evals:

  Show description for de6dff7
  scosman

  committed
  de6dff7

  Copy full SHA for de6dff7

Commits on Feb 19, 2025

• Add a fancy async evaluation runner. Not complete but checkpoint with tests

  scosman

  committed
  cfa6955

  Copy full SHA for cfa6955

• Add real task collection for eval runner.

  Show description for cad3644
  scosman

  committed
  cad3644

  Copy full SHA for cad3644

• Finalize the eval running, with a run_job method

  scosman

  committed
  240bc8c

  Copy full SHA for 240bc8c

• Add test

  scosman

  committed
  45f7d8f

  Copy full SHA for 45f7d8f

• Merge pull request #210 from Kiln-AI/evals_runner

  Show description for 72e222d
  scosman

  authored
  72e222d

  Copy full SHA for 72e222d

Commits on Feb 20, 2025

• Evals now define the scores it should produce!

  Show description for 9de4611
  scosman

  committed
  9de4611

  Copy full SHA for 9de4611

• Merge pull request #212 from Kiln-AI/eval_ui

  Show description for 04615dd
  scosman

  authored
  04615dd

  Copy full SHA for 04615dd

• Checkpoint of UI. CI won't pass, but saving considerable progress.

  scosman

  committed
  d8da2ca

  Copy full SHA for d8da2ca

Commits on Feb 21, 2025

• Huge update:

  Show description for 749d86c
  scosman

  committed
  749d86c

  Copy full SHA for 749d86c

• Fix up main evals screen. Design far from final but it's functional.

  scosman

  committed
  2473489

  Copy full SHA for 2473489

• WIP evaluator view

  scosman

  committed
  a9a6bb5

  Copy full SHA for a9a6bb5

• Make the "add config" functional, and better labels/code sharing

  scosman

  committed
  1ffa3e8

  Copy full SHA for 1ffa3e8

Commits on Feb 22, 2025

• Checkpoint: create task_runs, async run API (not done), list task_runs, improved UI

  scosman

  committed
  170cb34

  Copy full SHA for 170cb34

• Functional eval runs!

  Show description for f493980
  scosman

  committed
  f493980

  Copy full SHA for f493980

• Nice eval progress UI, and fix a bug where the eval runner didn't work with structured tasks

  scosman

  committed
  5ae11a2

  Copy full SHA for 5ae11a2

• Fix all linter warnings/errors

  scosman

  committed
  f419006

  Copy full SHA for f419006

• reaname evals_api to eval_api for consistency

  scosman

  committed
  897a086

  Copy full SHA for 897a086

• CR feedback

  scosman

  committed
  e0510bb

  Copy full SHA for e0510bb

• CR feedback: better links, better errors

  scosman

  committed
  34868d0

  Copy full SHA for 34868d0

• CR feedback

  scosman

  committed
  91e775b

  Copy full SHA for 91e775b

• Merge pull request #216 from Kiln-AI/eval_screen

  Show description for 627083e
  scosman

  authored
  627083e

  Copy full SHA for 627083e

• Eval results!!

  scosman

  committed
  2a2d1a9

  Copy full SHA for 2a2d1a9

• Check completeness, for out UI

  Show description for 02792e6
  scosman

  committed
  02792e6

  Copy full SHA for 02792e6

Commits on Feb 23, 2025

• Nice UI for eval incomplete warnings

  scosman

  committed
  c73b0e6

  Copy full SHA for c73b0e6

• Show eval set size in UI

  scosman

  committed
  dcf3a00

  Copy full SHA for dcf3a00

• CR feedback: better names and strings

  scosman

  committed
  13755c7

  Copy full SHA for 13755c7

• Merge pull request #219 from Kiln-AI/eval_results

  Show description for fcb75be
  scosman

  authored
  fcb75be

  Copy full SHA for fcb75be

• Remove TODO, not needed

  scosman

  committed
  55532b1

  Copy full SHA for 55532b1

• Merge pull request #218 from Kiln-AI/eval_ui

  Show description for 2bd73c1
  scosman

  authored
  2bd73c1

  Copy full SHA for 2bd73c1

• Nicer evals UI for tablewq

  scosman

  committed
  24463a9

  Copy full SHA for 24463a9

• Merge pull request #220 from Kiln-AI/eval_ui

  Show description for 461a74c
  scosman

  authored
  461a74c

  Copy full SHA for 461a74c

• Much better prompt system for evals

  Show description for e0a5532
  scosman

  committed
  e0a5532

  Copy full SHA for e0a5532

• CR feedback

  scosman

  committed
  e3a6a27

  Copy full SHA for e3a6a27

• improve comment

  scosman

  committed
  a46b942

  Copy full SHA for a46b942

• Merge pull request #221 from Kiln-AI/better_prompts

  Show description for 0759fb2
  scosman

  authored
  0759fb2

  Copy full SHA for 0759fb2

• Okay: fix the root of my 2 prompt issue

  Show description for 3f21c36
  scosman

  committed
  3f21c36

  Copy full SHA for 3f21c36

• Merge pull request #222 from Kiln-AI/better_prompts

  Show description for bbb8e27
  scosman

  authored
  bbb8e27

  Copy full SHA for bbb8e27

• UI improvements

  scosman

  committed
  0af1cdf

  Copy full SHA for 0af1cdf

Commits on Feb 24, 2025

• Add R1 and Llama 3.1 70B to g_eval support.

  Show description for 7d3cccb
  scosman

  committed
  7d3cccb

  Copy full SHA for 7d3cccb

• New UI: detailed results screen

  scosman

  committed
  f0d4144

  Copy full SHA for f0d4144

Commits on Feb 25, 2025

• Add eval config comparison summary API

  scosman

  committed
  7e51c3e

  Copy full SHA for 7e51c3e

• WIP UI for evaluating eval configs

  scosman

  committed
  113475c

  Copy full SHA for 113475c

Commits on Feb 26, 2025

• Eval runner updated to be more powerful. Run a eval_config analysis, as well as x product of eval_configs and task runs.

  scosman

  committed
  a8bb4db

  Copy full SHA for a8bb4db

• Fix bug in how we collected runs

  scosman

  committed
  f6dec21

  Copy full SHA for f6dec21

• Fix 2 issues:

  Show description for ee1318e
  scosman

  committed
  ee1318e

  Copy full SHA for ee1318e

• Fully functionaly UI for finding the eval-config which works best for your score. Includes the ability to run the eval-config-eval.

  scosman

  committed
  1133e1a

  Copy full SHA for 1133e1a

• - All setting current eval config for an eval through UI

  Show description for 50811b1
  scosman

  committed
  50811b1

  Copy full SHA for 50811b1

• Add 2 new scores: normalized MSE and MAE

  scosman

  committed
  a493ccd

  Copy full SHA for a493ccd

• Improved UI for config eval comparisons

  scosman

  committed
  43eb784

  Copy full SHA for 43eb784

• More improve copy/UI.

  scosman

  committed
  36c064d

  Copy full SHA for 36c064d

• Fix issue where the run_eval progress disappeared. We triggerd loading, which tool the whole svelte componenet out of dom

  scosman

  committed
  935092f

  Copy full SHA for 935092f

• String changes, final CR feedback

  scosman

  committed
  ee30223

  Copy full SHA for ee30223