BDD it() function not reporting test coverage with test fixture

[mjfrigaard]

I noticed the new it() functions don't seem to load helper functions after load_all().

R/ folder:

R
├── bddtestcoverage-package.R
└── tidy_ggp2movies.R

tests/ folder:

tests
├── testthat
│   ├── fixtures
│   │   ├── make_test_data.R
│   │   └── test_data.rds
│   ├── helper.R
│   └── test-tidy_ggp2movies.R
└── testthat.R

Helper function

In tests/testthat/helper.R, I have the following helper:

test_logger <- function(msg, start = NULL, end = NULL) {
  if (is.null(start) & is.null(end)) {
    cat("\n")
    logger::log_info("{msg}")
  } else if (!is.null(start) & is.null(end)) {
    cat("\n")
    logger::log_info("\n[ START {start} = {msg}]")
  } else if (is.null(start) & !is.null(end)) {
    cat("\n")
    logger::log_info("\n[ END {end} = {msg}]")
  } else {
    cat("\n")
    logger::log_info("\n[ START {start} = {msg}]")
    cat("\n")
    logger::log_info("\n[ END {end} = {msg}]")
  }
}

In test-tidy_ggp2movies.R:

When the test_logger() helper is inside it() (with a nested describe()):

testthat::describe(
  description = "
    Feature: tidy_ggp2movies Function
      In order to tidy up the movie data
      As a data scientist
      I want to convert ggplot2moves::movies into a 'tidy' data.frame", code = {
    testthat::describe(
      description = "
      Background:
      Given a dataframe 'movies_data' with columns:
        'title', 'year', 'length', 'budget', 'rating',
        'votes', 'mpaa', 'Action', 'Animation', 'Comedy',
        'Drama', 'Documentary', 'Romance', and 'Short'", code = {
        testthat::it(
          description = "
      Scenario: Correct formatting of mpaa as factor with specified levels and labels
          When the function tidy_ggp2movies is applied to the movies_data
          Then the 'mpaa' column should be of type factor
          And have levels 'G', 'PG', 'PG-13', 'R', 'NC-17'",
          code = {
            test_logger(start = "fixture", msg = "test_data.rds")
            # load fixture
            test_data <- readRDS(file = test_path("fixtures", "test_data.rds"))
            tidy_ggp2 <- tidy_ggp2movies(test_data)
            testthat::expect_true(is.factor(tidy_ggp2$mpaa))
            testthat::expect_equal(
              object = levels(tidy_ggp2$mpaa),
              expected = c("G", "PG", "PG-13", "R", "NC-17")
            )
            test_logger(end = "fixture", msg = "test_data.rds")
          })
      })
  })

devtools::load_all(".")
ℹ Loading bddtestcoverage

The test fails (can't find the helper).

devtools:::test_active_file()
[ FAIL 1 | WARN 0 | SKIP 0 | PASS 0 ]

── Error (test-tidy_ggp2movies.R:22:13): 
      Scenario: Correct formatting of mpaa as factor with specified levels and labels
          When the function tidy_ggp2movies is applied to the movies_data
          Then the 'mpaa' column should be of type factor
          And have levels 'G', 'PG', 'PG-13', 'R', 'NC-17' ──
Error in `test_logger(start = "fixture", msg = "test_data.rds")`: could not find function "test_logger"
[ FAIL 1 | WARN 0 | SKIP 0 | PASS 0 ]

And if I switch it to a single describe() function:

testthat::describe(
  description = "
    Feature: tidy_ggp2movies Function
      In order to tidy up the movie data
      As a data scientist
      I want to convert ggplot2moves::movies into a 'tidy' data.frame", code = {
    testthat::it(
      description = "
      Scenario: Correct formatting of mpaa as factor with specified levels and labels
        Given a dataframe 'movies_data' with columns:
          'title', 'year', 'length', 'budget', 'rating',
          'votes', 'mpaa', 'Action', 'Animation', 'Comedy',
          'Drama', 'Documentary', 'Romance', and 'Short'
        When the function tidy_ggp2movies is applied to the movies_data
        Then the 'mpaa' column should be of type factor
        And have levels 'G', 'PG', 'PG-13', 'R', 'NC-17'",
      code = {
        test_logger(start = "fixture", msg = "test_data.rds")
        # load fixture
        test_data <-
          readRDS(file = test_path("fixtures", "test_data.rds"))
        tidy_ggp2 <- tidy_ggp2movies(test_data)
        testthat::expect_true(is.factor(tidy_ggp2$mpaa))
        testthat::expect_equal(
          object = levels(tidy_ggp2$mpaa),
          expected = c("G", "PG", "PG-13", "R", "NC-17")
        )
        test_logger(end = "fixture", msg = "test_data.rds")
      })
  })

devtools::load_all(".")
ℹ Loading bddtestcoverage

The test still fails:

[ FAIL 1 | WARN 0 | SKIP 0 | PASS 0 ]

── Error (test-tidy_ggp2movies.R:54:9): 
      Scenario: Correct formatting of mpaa as factor with specified levels and labels
        Given a dataframe 'movies_data' with columns:
          'title', 'year', 'length', 'budget', 'rating',
          'votes', 'mpaa', 'Action', 'Animation', 'Comedy',
          'Drama', 'Documentary', 'Romance', and 'Short'
        When the function tidy_ggp2movies is applied to the movies_data
        Then the 'mpaa' column should be of type factor
        And have levels 'G', 'PG', 'PG-13', 'R', 'NC-17' ──
Error in `test_logger(start = "fixture", msg = "test_data.rds")`: could not find function "test_logger"
[ FAIL 1 | WARN 0 | SKIP 0 | PASS 0 ]

But if the helper is inside test_that() (with nested describe()):

testthat::describe(
  description = "
    Feature: tidy_ggp2movies Function
      In order to tidy up the movie data
      As a data scientist
      I want to convert ggplot2moves::movies into a 'tidy' data.frame", code = {
    testthat::describe(
      description = "
      Background:
      Given a dataframe 'movies_data' with columns:
        'title', 'year', 'length', 'budget', 'rating',
        'votes', 'mpaa', 'Action', 'Animation', 'Comedy',
        'Drama', 'Documentary', 'Romance', and 'Short'", code = {
        testthat::test_that("
      Scenario: Correct formatting of mpaa as factor with specified levels and labels
          When the function tidy_ggp2movies is applied to the movies_data
          Then the 'mpaa' column should be of type factor
          And have levels 'G', 'PG', 'PG-13', 'R', 'NC-17'",
          code = {
            test_logger(start = "fixture", msg = "test_data.rds")
            # load fixture
            test_data <- readRDS(file = test_path("fixtures", "test_data.rds"))
            tidy_ggp2 <- tidy_ggp2movies(test_data)
            testthat::expect_true(is.factor(tidy_ggp2$mpaa))
            testthat::expect_equal(
              object = levels(tidy_ggp2$mpaa),
              expected = c("G", "PG", "PG-13", "R", "NC-17")
            )
            test_logger(end = "fixture", msg = "test_data.rds")
          })
      })
  })

devtools::load_all(".")
ℹ Loading bddtestcoverage

The test passes.

devtools:::test_active_file()
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 0 ]
INFO [2023-10-28 12:11:58] [ START fixture = test_data.rds]
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 2 ]
INFO [2023-10-28 12:11:59] [ END fixture = test_data.rds]

And if the helper is inside the nested describe(), but outside it():

testthat::describe(
  description = "
    Feature: tidy_ggp2movies Function
      In order to tidy up the movie data
      As a data scientist
      I want to convert ggplot2moves::movies into a 'tidy' data.frame", code = {
    testthat::describe(
      description = "
      Background:
      Given a dataframe 'movies_data' with columns:
        'title', 'year', 'length', 'budget', 'rating',
        'votes', 'mpaa', 'Action', 'Animation', 'Comedy',
        'Drama', 'Documentary', 'Romance', and 'Short'", code = {
        test_logger(start = "fixture", msg = "test_data.rds")
        testthat::it("
      Scenario: Correct formatting of mpaa as factor with specified levels and labels
          When the function tidy_ggp2movies is applied to the movies_data
          Then the 'mpaa' column should be of type factor
          And have levels 'G', 'PG', 'PG-13', 'R', 'NC-17'",
          code = {
            # load fixture
            test_data <- readRDS(file = test_path("fixtures", "test_data.rds"))
            tidy_ggp2 <- tidy_ggp2movies(test_data)
            testthat::expect_true(is.factor(tidy_ggp2$mpaa))
            testthat::expect_equal(
              object = levels(tidy_ggp2$mpaa),
              expected = c("G", "PG", "PG-13", "R", "NC-17")
            )
          })
        test_logger(end = "fixture", msg = "test_data.rds")
      })
  })

devtools::load_all(".")
ℹ Loading bddtestcoverage

[ FAIL 0 | WARN 0 | SKIP 0 | PASS 0 ]
INFO [2023-10-28 12:13:37] [ START fixture = test_data.rds]
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 2 ]
INFO [2023-10-28 12:13:37] [ END fixture = test_data.rds]

This also passes.

I'm not sure if test_that() and it() are supposed to have different functionality (or if it changes with nested describe() calls)?

These describe() functions are great for Gherkin Features and Scenarios (and nesting is nice for combining Scenarios under Features.) :)

testthat in DESCRIPTION:

Suggests: 
    testthat (>= 3.0.0)
Config/testthat/edition: 4

This package is here in case I'm missing something I'm not thinking of.

Thank you in advance for your time (and all your continued hard work on this package :)!)

[hadley]

Could you please create a simple reprex package? Your code here is a good start but you could make the examples even simpler, and you could assume that testthat is loaded so you don't have to preface every call with testthat::.

[mjfrigaard]

Ok--I re-installed testtthat and now the tests are passing :) However, I noticed that when I run the test coverage for a test using a fixture the code coverage report from devtools:::test_coverage_active_file() is 0.00%

tests/ folder

tests
 └── tests
      ├── testthat
      │   ├── fixtures
      │   │   ├── make_test_data.R <- takes sample from ggplot2movies::movies 
      │   │   └── test_data.rds    <- sample from ggplot2movies::movies 
      │   ├── helper.R             <- contains test_logger() helper function 
      │   └── test-collapse_movies_genres.R
      └── testthat.R

Here are the tests:

# it() with single describe() 
describe("Feature: (As a ..., In order to ..., I want to ...)", {
    it("Scenario: (Given ... , When ..., Then ...)", {
        # load fixture
        test_data <- readRDS(file = test_path("fixtures", "test_data.rds"))
        movie_genres <- collapse_movies_genres(test_data)
        expect_true(is.factor(movie_genres$mpaa))
      })
  })
# it() with nested describe() 
describe("Feature: (As a ..., In order to ..., I want to ...)", {
  describe("Background (Given ...)", {
          it("Scenario: (Given ... , When ..., Then ...)", {
          # load fixture
          test_data <- readRDS(file = test_path("fixtures", "test_data.rds"))
          movie_genres <- collapse_movies_genres(test_data)
          expect_true(is.factor(movie_genres$mpaa))
        })
    })
})

These pass:

devtools:::test_active_file()
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 2 ]

devtools:::test_coverage_active_file()

Would you like me to close this issue and open a new one for the code coverage?

Thank you so much for your prompt response, too! I really appreciate it :)

[mjfrigaard]

I also put all this information in the package README

[mjfrigaard]

Hi again 😄 --I created another small example package that reproduces the issue. I kept the functions simple and only used base R to remove additional dependencies.

I'm still getting 0.00% test coverage on tests using fixtures when I use test_coverage_active_file()

[hadley]

Actually, you might want to file this in the covr package, since that's what actually does the coverage computation.

[mjfrigaard]

Thank you! I will move it over 😃

[mjfrigaard]

Issue in covr (for reference)