New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Include Cohen's kappa in confusion matrices #26

Open

cimentadaj opened this issue Feb 23, 2021 · 0 comments

Owner

cimentadaj commented Feb 23, 2021

Whe talking about confusion matrices you should add cohen's kappa. You can refer to this paper: https://www.nature.com/articles/s41467-020-20206-z.pdf using it as a measure to check that two coders correctly classified search queries the same way. The difference between Cohen's Kappa and simply accuracy is is that Cohen’s Kappa is a quantitative measure of reliability for two raters that are rating the same thing, corrected for how often that the raters may agree by chance. See how they use it here: https://github.com/stefanjwojcik/ms_flu/blob/master/Survey%20and%20MRP/Main_Replication_Code.R#L98 and an explanation here: https://towardsdatascience.com/cohens-kappa-9786ceceab58

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment