Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Exploration: counterfactual accuracy #251

Open
2 tasks
dzeber opened this issue Feb 22, 2021 · 0 comments
Open
2 tasks

Exploration: counterfactual accuracy #251

dzeber opened this issue Feb 22, 2021 · 0 comments

Comments

@dzeber
Copy link
Contributor

dzeber commented Feb 22, 2021

Once implemented (#197), the counterfactual accuracy evaluation needs to be tested out against a variety of datasets/models in order to surface any issues or potential improvements. This can be run as a data analysis project using the datasets in the repo and added to the examples dir. This can be considered done with at least 2-3 dataset/model pairs, but more is fine.

Based on these results, we would also like determine a protocol for making an automated pass/fail decision based on this evaluation, ie. a rule of thumb for what a "good" result looks like. Otherwise, if the evaluation is found not to be a good fit for such a protocol, this should be documented.

  • Road-testing against multiple modelling problems
  • Protocol for automated decision-making, if appropriate.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant