Skip to content

Commit

Permalink
Merged origin/main into survival-workflows-tests
Browse files Browse the repository at this point in the history
  • Loading branch information
hfrick committed Nov 13, 2023
2 parents 34052e0 + 04ee2d1 commit db738d3
Show file tree
Hide file tree
Showing 24 changed files with 371 additions and 89 deletions.
1 change: 1 addition & 0 deletions DESCRIPTION
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@ Suggests:
tibble,
tidyr,
tune,
withr,
workflows,
workflowsets,
xgboost,
Expand Down
30 changes: 30 additions & 0 deletions README.Rmd
Original file line number Diff line number Diff line change
Expand Up @@ -30,3 +30,33 @@ knitr::opts_chunk$set(
* Whose run-time is not practical for individual packages.

These tests are run on a cron job and are run for both CRAN versions and the current GitHub development versions.


## PR-pairs with package repos

PRs on extratests typically are part of a PR pair since they test changes in package repositories. The following workflow ensures that the CI run triggered by the PR on extratests runs with the changes in the corresponding PR on the package repository.

Normal development

* [pkg repo] Make changes
* [extratests repo] Write tests

Set version for the change

* [pkg repo] Give it new dev version number in DESCRIPTION, e.g. `1.1.0.9001` instead of `1.1.0.9000`
* [extratests repo] Add `skip_if_not_installed()` to the tests with that dev version number as `minimum_version`.

Open PRs and point GHA to the changes

* [pkg repo] Make a PR
* [extratests repo] in `GH-R-CMD-check.yaml`, point GHA to the pkg PR by appending `#<PR number>`, e.g. `try(pak::pkg_install("tidymodels/parsnip#991"))`
* Without pointing GHA to that branch, the tests will be skipped based on the version number.
* If the branch information is added to the DESCRIPTION via `Remotes:` instead, the "CRAN workflow" will also run the dev version.
* [extratests repo] Make a PR, link it to the pkg-PR in the PR description
* [extratests repo] Make a review comment to change remote back to main before merging

Clean-up and merge (after PR review and approval)

* [pkg repo] Merge PR
* [extratests repo] Point remote back to main
* [extratests repo] Merge PR
46 changes: 43 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,51 @@ Spark](https://github.com/tidymodels/extratests/workflows/spark-R-CMD-check/badg

`extratests` is an internal package used for tests that

- Depend on multiple tidymodels packages
- Depend on multiple tidymodels packages

- Involve special/extra packages.
- Involve special/extra packages.

- Whose run-time is not practical for individual packages.
- Whose run-time is not practical for individual packages.

These tests are run on a cron job and are run for both CRAN versions and
the current GitHub development versions.

## PR-pairs with package repos

PRs on extratests typically are part of a PR pair since they test
changes in package repositories. The following workflow ensures that the
CI run triggered by the PR on extratests runs with the changes in the
corresponding PR on the package repository.

Normal development

- \[pkg repo\] Make changes
- \[extratests repo\] Write tests

Set version for the change

- \[pkg repo\] Give it new dev version number in DESCRIPTION,
e.g. `1.1.0.9001` instead of `1.1.0.9000`
- \[extratests repo\] Add `skip_if_not_installed()` to the tests with
that dev version number as `minimum_version`.

Open PRs and point GHA to the changes

- \[pkg repo\] Make a PR
- \[extratests repo\] in `GH-R-CMD-check.yaml`, point GHA to the pkg PR
by appending `#<PR number>`,
e.g. `try(pak::pkg_install("tidymodels/parsnip#991"))`
- Without pointing GHA to that branch, the tests will be skipped based
on the version number.
- If the branch information is added to the DESCRIPTION via `Remotes:`
instead, the “CRAN workflow” will also run the dev version.
- \[extratests repo\] Make a PR, link it to the pkg-PR in the PR
description
- \[extratests repo\] Make a review comment to change remote back to
main before merging

Clean-up and merge (after PR review and approval)

- \[pkg repo\] Merge PR
- \[extratests repo\] Point remote back to main
- \[extratests repo\] Merge PR
15 changes: 9 additions & 6 deletions tests/testthat/_snaps/glmnet-linear.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,15 +3,17 @@
Code
linear_reg(penalty = 0.01) %>% set_engine("glmnet") %>% fit(mpg ~ ., data = mtcars[
-(1:4), ]) %>% predict(mtcars[-(1:4), ], penalty = 0:1)
Error <rlang_error>
`penalty` should be a single numeric value. `multi_predict()` can be used to get multiple predictions per row of data.
Condition
Error in `.check_glmnet_penalty_predict()`:
! `penalty` should be a single numeric value. `multi_predict()` can be used to get multiple predictions per row of data.

---

Code
linear_reg() %>% set_engine("glmnet") %>% fit(mpg ~ ., data = mtcars[-(1:4), ])
Error <rlang_error>
For the glmnet engine, `penalty` must be a single number (or a value of `tune()`).
Condition
Error in `.check_glmnet_penalty_fit()`:
! For the glmnet engine, `penalty` must be a single number (or a value of `tune()`).
* There are 0 values for `penalty`.
* To try multiple values for total regularization, use the tune package.
* To predict multiple penalties, use `multi_predict()`
Expand All @@ -21,6 +23,7 @@
Code
linear_reg(penalty = 0.01) %>% set_engine("glmnet") %>% fit(mpg ~ ., data = mtcars) %>%
multi_predict(mtcars, type = "class")
Error <rlang_error>
For class predictions, the object should be a classification model.
Condition
Error in `check_pred_type()`:
! For class predictions, the object should be a classification model.

20 changes: 12 additions & 8 deletions tests/testthat/_snaps/glmnet-logistic.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,25 +2,28 @@

Code
multi_predict(class_fit, newdata = wa_churn[1:4, vars], type = "prob")
Error <rlang_error>
Please use `new_data` instead of `newdata`.
Condition
Error in `multi_predict()`:
! Please use `new_data` instead of `newdata`.

# error traps

Code
logistic_reg(penalty = 0.01) %>% set_engine("glmnet") %>% fit(Class ~ log(
funded_amnt) + int_rate + term, data = lending_club) %>% predict(lending_club,
penalty = 0:1)
Error <rlang_error>
`penalty` should be a single numeric value. `multi_predict()` can be used to get multiple predictions per row of data.
Condition
Error in `.check_glmnet_penalty_predict()`:
! `penalty` should be a single numeric value. `multi_predict()` can be used to get multiple predictions per row of data.

---

Code
logistic_reg() %>% set_engine("glmnet") %>% fit(Class ~ log(funded_amnt) +
int_rate + term, data = lending_club)
Error <rlang_error>
For the glmnet engine, `penalty` must be a single number (or a value of `tune()`).
Condition
Error in `.check_glmnet_penalty_fit()`:
! For the glmnet engine, `penalty` must be a single number (or a value of `tune()`).
* There are 0 values for `penalty`.
* To try multiple values for total regularization, use the tune package.
* To predict multiple penalties, use `multi_predict()`
Expand All @@ -31,6 +34,7 @@
logistic_reg(penalty = 0.01) %>% set_engine("glmnet") %>% fit(Class ~ log(
funded_amnt) + int_rate + term, data = lending_club) %>% multi_predict(
lending_club, type = "time")
Error <rlang_error>
For event time predictions, the object should be a censored regression.
Condition
Error in `check_pred_type()`:
! For event time predictions, the object should be a censored regression.

20 changes: 12 additions & 8 deletions tests/testthat/_snaps/glmnet-multinom.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,23 +2,26 @@

Code
multi_predict(xy_fit, newdata = hpc[rows, 1:4], penalty = c(0.1, 0.5))
Error <rlang_error>
Please use `new_data` instead of `newdata`.
Condition
Error in `multi_predict()`:
! Please use `new_data` instead of `newdata`.

# error traps

Code
multinom_reg(penalty = 0.01) %>% set_engine("glmnet") %>% fit(class ~ ., data = hpc_data) %>%
predict(hpc_data, penalty = 0:1)
Error <rlang_error>
`penalty` should be a single numeric value. `multi_predict()` can be used to get multiple predictions per row of data.
Condition
Error in `.check_glmnet_penalty_predict()`:
! `penalty` should be a single numeric value. `multi_predict()` can be used to get multiple predictions per row of data.

---

Code
multinom_reg() %>% set_engine("glmnet") %>% fit(class ~ ., data = hpc_data)
Error <rlang_error>
For the glmnet engine, `penalty` must be a single number (or a value of `tune()`).
Condition
Error in `.check_glmnet_penalty_fit()`:
! For the glmnet engine, `penalty` must be a single number (or a value of `tune()`).
* There are 0 values for `penalty`.
* To try multiple values for total regularization, use the tune package.
* To predict multiple penalties, use `multi_predict()`
Expand All @@ -28,6 +31,7 @@
Code
multinom_reg(penalty = 0.01) %>% set_engine("glmnet") %>% fit(class ~ ., data = hpc_data) %>%
multi_predict(hpc_data, type = "numeric")
Error <rlang_error>
For numeric predictions, the object should be a regression model.
Condition
Error in `check_pred_type()`:
! For numeric predictions, the object should be a regression model.

24 changes: 13 additions & 11 deletions tests/testthat/_snaps/glmnet-poisson.md
Original file line number Diff line number Diff line change
@@ -1,18 +1,20 @@
# error traps

Code
poisson_reg(penalty = 0.1) %>% set_engine("glmnet") %>% fit(mpg ~ ., data = mtcars[
-(1:4), ]) %>% predict(mtcars[-(1:4), ], penalty = 0:1)
Error <rlang_error>
`penalty` should be a single numeric value. `multi_predict()` can be used to get multiple predictions per row of data.

---
# model errors on missing penalty value

Code
poisson_reg() %>% set_engine("glmnet") %>% fit(mpg ~ ., data = mtcars[-(1:4), ])
Error <rlang_error>
For the glmnet engine, `penalty` must be a single number (or a value of `tune()`).
Condition
Error in `.check_glmnet_penalty_fit()`:
! For the glmnet engine, `penalty` must be a single number (or a value of `tune()`).
* There are 0 values for `penalty`.
* To try multiple values for total regularization, use the tune package.
* To predict multiple penalties, use `multi_predict()`

# predict() errors with multiple penalty values

Code
poisson_reg(penalty = 0.1) %>% set_engine("glmnet") %>% fit(mpg ~ ., data = mtcars[
-(1:4), ]) %>% predict(mtcars[-(1:4), ], penalty = 0:1)
Condition
Error in `.check_glmnet_penalty_predict()`:
! `penalty` should be a single numeric value. `multi_predict()` can be used to get multiple predictions per row of data.

16 changes: 8 additions & 8 deletions tests/testthat/_snaps/parsnip-extension-messaging.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

Code
bag_tree() %>% set_engine("rpart") %>% set_mode("regression")
Message <rlang_message>
Message
! parsnip could not locate an implementation for `bag_tree` regression model specifications using the `rpart` engine.
i The parsnip extension package baguette implements support for this specification.
i Please install (if needed) and load to continue.
Expand All @@ -20,7 +20,7 @@

Code
bag_tree() %>% set_mode("censored regression")
Message <rlang_message>
Message
! parsnip could not locate an implementation for `bag_tree` censored regression model specifications.
i The parsnip extension package censored implements support for this specification.
i Please install (if needed) and load to continue.
Expand All @@ -38,7 +38,7 @@

Code
bag_tree()
Message <rlang_message>
Message
! parsnip could not locate an implementation for `bag_tree` model specifications.
i The parsnip extension packages censored and baguette implement support for this specification.
i Please install (if needed) and load to continue.
Expand All @@ -56,7 +56,7 @@

Code
bag_tree() %>% set_engine("rpart")
Message <rlang_message>
Message
! parsnip could not locate an implementation for `bag_tree` model specifications using the `rpart` engine.
i The parsnip extension packages censored and baguette implement support for this specification.
i Please install (if needed) and load to continue.
Expand Down Expand Up @@ -102,7 +102,7 @@

Code
bag_tree() %>% set_mode("regression") %>% set_engine("rpart")
Message <rlang_message>
Message
! parsnip could not locate an implementation for `bag_tree` regression model specifications using the `rpart` engine.
i The parsnip extension package baguette implements support for this specification.
i Please install (if needed) and load to continue.
Expand All @@ -120,7 +120,7 @@

Code
bag_tree() %>% set_mode("classification") %>% set_engine("C5.0")
Message <rlang_message>
Message
! parsnip could not locate an implementation for `bag_tree` classification model specifications using the `C5.0` engine.
i The parsnip extension package baguette implements support for this specification.
i Please install (if needed) and load to continue.
Expand Down Expand Up @@ -182,7 +182,7 @@

Code
decision_tree() %>% set_engine("partykit") %>% set_mode("regression")
Message <rlang_message>
Message
! parsnip could not locate an implementation for `decision_tree` regression model specifications using the `partykit` engine.
i The parsnip extension package bonsai implements support for this specification.
i Please install (if needed) and load to continue.
Expand Down Expand Up @@ -216,7 +216,7 @@

Code
decision_tree() %>% set_mode("regression") %>% set_engine("partykit")
Message <rlang_message>
Message
! parsnip could not locate an implementation for `decision_tree` regression model specifications using the `partykit` engine.
i The parsnip extension package bonsai implements support for this specification.
i Please install (if needed) and load to continue.
Expand Down
31 changes: 31 additions & 0 deletions tests/testthat/_snaps/parsnip-model-formula.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# error without model formula (workflow, no tune)

Code
gam_fit <- gam_wflow %>% fit(mtcars)
Condition
Error:
! When working with generalized additive models, please supply the model specification to `workflows::add_model()` along with a `formula` argument.
i See `?parsnip::model_formula()` to learn more.

# error without model formula (workflow, with tune)

Code
show_notes(gam_res)
Output
unique notes:
--------------------------------------------------------------------------------
Error:
! When working with generalized additive models, please supply the model specification to `workflows::add_model()` along with a `formula` argument.
i See `?parsnip::model_formula()` to learn more.

# error without model formula (no workflow, with tune)

Code
show_notes(gam_res)
Output
unique notes:
--------------------------------------------------------------------------------
Error:
! When working with generalized additive models, please supply the model specification to `workflows::add_model()` along with a `formula` argument.
i See `?parsnip::model_formula()` to learn more.

5 changes: 3 additions & 2 deletions tests/testthat/_snaps/parsnip-survival-censoring-model.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@

Code
predict(alt_obj, time = 100)
Error <rlang_error>
Don't know how to predict with a censoring model of type: reverse_km
Condition
Error in `predict()`:
! Don't know how to predict with a censoring model of type: reverse_km

Loading

0 comments on commit db738d3

Please sign in to comment.