Custom probability thresholds for binary classification ?

[yangxg]

Dear Authors:

I am aware that there is a plan to add the feature of custom probability threshold for classification in workflow package. But it seems not published yet. I also notice that the probably package is very helpful to find out the approriate threshold value (the link), but I don't know how to integrate it into the workflow.

As this feature is important to one of my projects, so I am wondering is there a alternative approach to archieve it in the whole tidymodels workflow?

Anyway, I am also look forward to the one-line solution in the future! Thanks again for the great work!

Xiaoguang

[Teett]

Hi!!

I was wondering if there is any update in this issue? This is kind of an important feature when dealing with classification problems that require a higher specificity (e.g. for mortality prediction and health risks) and at the moment I feel like the only way is to do it in tidymodels is something of this sort:

collect_predictions(final_model) %>%
  mutate(corrected_class = as_factor(case_when(.pred_alive > 0.75 ~ "alive",
                                               TRUE ~ "dead"))) %>% # Manual threshhold
  conf_mat(truth = vital_state, estimate = corrected_class) 

I think this workaround is accurate for a confusion matrix? But we still need to be able to calculate the new metrics and I can't find a way to do it in a simple way using yardstick.

Thank you!!

[StevenWallaert]

Hi,
Would there be any update on this?

I am aware of {probably} and how to look for an optimal thershold after training.
However, I would like to tune the threshold within a workflow and treat the threshold as any other hyperparameter.

Many thanks!