Skip to content

Commit

Permalink
Add a section for test expectations (#95)
Browse files Browse the repository at this point in the history
  • Loading branch information
maximusunc committed Sep 25, 2024
1 parent 3dfbef0 commit 44b1b81
Showing 1 changed file with 13 additions and 0 deletions.
13 changes: 13 additions & 0 deletions docs/architecture/sri/testing/sri_testing_infrastructure.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,19 @@ The SRI Testing Infrastructure includes all components dealing with the automate
- [Benchmarks Test Runner](https://github.com/TranslatorSRI/Benchmarks) - Quality Assurance Testing
- (Coming Soon) [Performance Test Runner]() - Quality Assurance Testing

## Test Expectations
- Pass/Fail Tests
- TopAnswer: The expected curie result is in the top 30 results.
- Acceptable: The expected curie result is in the top 50% of results.
- BadButForgivable: The expected curie result is either in the bottom 50% of results or not present.
- NeverShow: The expected curie result is not present in the results.
- TRAPI Validation
- The entire response message passes the current version of TRAPI.
- Benchmarks
- Given a set of "ground-truth" results for a given set of input queries, calculate the ratio of overlap between ground-truth and a response message results within a top-k range.
- Performance
- Run successively heavier load tests against the system to see how many concurrent users can send queries while returning consistent results.

## Automated Pipeline
- A CronJob spins up a Docker Image of the Test Harness
- Test Harness downloads the Tests Repo and grabs the specified "Test Suite" based on the CronJob command
Expand Down

0 comments on commit 44b1b81

Please sign in to comment.