Skip to content
This repository has been archived by the owner on Feb 14, 2023. It is now read-only.

Metric Definitions: Added metric definitions endpoints + reponse bodies #33

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

RobGeada
Copy link
Contributor

@RobGeada RobGeada commented Feb 13, 2023

  1. The OpenAPI specification now has short metric descriptions in the tags:
openapi: 3.0.3
info:
  title: trustyai-service API
  version: 1.0.0-SNAPSHOT
tags:
- name: Disparate Impact Ratio Endpoint
  description: Disparate Impact Ratio (DIR) measures imbalances in classifications
    by calculating the ratio between the proportion of the majority and protected
    classes getting a particular outcome.
- name: Statistical Parity Difference Endpoint
  description: Statistical Parity Difference (SPD) measures imbalances in classifications
    by calculating the difference between the proportion of the majority and protected
    classes getting a particular outcome.
  1. General metric definitions can be queried by:
    curl -X GET http://localhost:8080/metrics/spd/definition
    returns
    Statistical Parity Difference (SPD) measures imbalances in classifications by calculating the difference between the proportion of the majority and protected classes getting a particular outcome. Typically, -0.1 < SPD < 0.1 indicates a fair model, while a value outside those bounds indicates an unfair model for the groups and outcomes in question.

  2. Specific metric descriptions are returned in metric query bodies:

curl -X POST --location "http://localhost:8080/metrics/dir/" \
    -H "Content-Type: application/json" \
    -d "{
          \"protectedAttribute\": \"gender\",
          \"favorableOutcome\": 1,
          \"outcomeName\": \"income\",
          \"privilegedAttribute\": 1,
          \"unprivilegedAttribute\": 0
        }"

returns
{"type":"metric","name":"DIR","value":0.17292587250695893,"specificDefinition":"The DIR of 0.172926 indicates that the likelihood of Group:gender=1 receiving Outcome:income=1 is 0.172926 times that of Group:gender=0.","id":"2b3e009a-ddf7-496e-843d-812893c3a478","timestamp":"2023-02-13T13:42:03.448+00:00","thresholds":{"lowerBound":0.8,"upperBound":1.2,"outsideBounds":true}}

@danielezonca
Copy link
Member

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants