Allow httptest2 to be used with roxygen2/pkgdown documentation

[DavZim]

Not sure how easy this is to implement, but something like the following would be great.

Allow httptest2 to be used with roxygen2 + pkgdown documentation. That is, whenever the website is build, all code examples are run to be displayed on the website.
If possible, I want to be able to store and replay the API calls as my build server might not have internet, I might not want to use it (cost, time, sensitive information) or I simply don't want to share credentials with my build server.

The background of this request is that I write an R package that wraps API that require a secret and cost money when using it. Think something along the lines of OpenAI.
I want to test my package and show example responses back to the user without using the actual API N-times whenever I rebuild the Readme (see also #33) or the code documentation using `pkgdown::build_site(). Also, I dont want to share secrets with build servers...
The request is authorized but the response elements are non-sensitive, so I am fine with storing them alongside the repository.

##################################################
# in foo.R

#' Example function
#' 
#' @parms ... something
#' @export
#' @return the value of the API call
#' @examples 
#' # this code will be called whenever pkgdown builds the site. Capture and replay this with httptest2
#' foo()
foo <- function(...) {
  req <- httr2::request("https://example.com")
  # ...
  req |> httr2::req_perform()
}

##################################################
# in console run by developer

# either capture all calls once, or start capturing and run some example calls to update the return
httptest2::start_capturing() # or whatever command should be run here
pkgdown::build_site()
httptest2::stop_capturing()

[nealrichardson]

I agree that would be nice. At one point, I looked into making an Rd macro that would do this, but for some reason couldn't get it to work right. The same idea that works for mocking in vignettes should work for man examples though.

Maybe it's more tractable if you limit the problem to just pkgdown, but you still need the examples not to error when CRAN runs them in R CMD check, so that's why I think there needs to be something in the Rd itself that handles the mocking, and preferably wrapped in \dontshow{} so that the setup and teardown don't pollute the rendered examples and confuse the reader.

[DavZim]

In our internal package we have something like this. The downside is that it means the developer has to replace a call from httr2::req_peform() |> httr2::resp_body_json() with req_perform_body_with_mock().

But start_mock_recording() and stop_mock_recording() work with Vignettes, Readme, Pkgdown builds etc as all mocks are stored centrally.

# Internal functions that wrap httr2::req_perform() |> httr2::resp_body_json()
# so that we can record and load mock responses for testing and documentation (eg pkgdown)
# without requiring an internet connection or valid API credentials.
#
# This function replaces req |> httr2::req_perform() |> httr2::resp_body_json()
# if its in a test or pkgdown build it will load pre-recorded respones from disk
# we can record these mock API calls using start_mock_recording() and stop_mock_recording()
#
# If you want to see if a request was sent ot the API or loaded from disk, set MOCK_SILENT="false"
#
# For the developer of the package only!
req_perform_body_with_mock <- function(req) {
  # base case: make the request and return the body of the response
  if (!is_mock_recording() && !is_testing() && !is_pkgdown()) {
    if (identical(Sys.getenv("MOCK_SILENT"), "false"))
      cat(paste0("Making API call to '", req$url, "'- no mocking\n"))
    return(httr2::req_perform(req) |> httr2::resp_body_json())
  }

  # if either we are recording or testing/pkgdown, we need to handle the request differently
  # either we call the API and store the responses to the mock-dir or we load the response
  # from tests/testthat/mock_responses/{mock_id}.json

  mock_dir <- get_path_to_mock_response()
  mock_id <- get_mock_id(req)
  if (!dir.exists(mock_dir)) dir.create(mock_dir, recursive = TRUE, showWarnings = FALSE)

  file <- file.path(mock_dir, paste0(mock_id, ".json"))

  if (is_mock_recording()) {
    # perform the request and save the response
    resp <- httr2::req_perform(req)
    body <- resp |> httr2::resp_body_json()
    jsonlite::write_json(body, file, pretty = TRUE, auto_unbox = TRUE, null = "null")
    cat(sprintf("Mock response saved id '%s' with %s bytes to '%s'\n",
                mock_id, length(resp$body), req$url))
    return(body)
  } else if (is_testing() || is_pkgdown()) {

    if (identical(Sys.getenv("MOCK_SILENT"), "false"))
      cat(paste0("Loading mock response from '", file, "'\n"))

    # check if the file exists
    if (!file.exists(file))
      stop(sprintf(
        "Mock response file '%s' does not exist. Run `start_mock_recording()` to create it.",
        file
      ))

    # if exists, load
    body <- jsonlite::read_json(file)
    return(body)
  }
}

# gets a unique ID for a request
get_mock_id <- function(req) {
  args <- list(
    method = req$method,
    url = req$url,
    body = req$body,
    query = req$query
  )
  url_short <- gsub("https?://", "", req$url)
  # extract the first 5 parts of the URL based on /
  url_short <- paste(strsplit(url_short, "/")[[1]][1:5], collapse = "_")
  paste(
    req$method,
    url_short,
    rlang::hash(args),
    sep = "-"
  )
}


# helper functions to be able to perform a request with mock responses in tests/pkgdown builds
is_testing <- function() {
  identical(Sys.getenv("TESTTHAT"), "true") ||
    endsWith(getwd(), "tests/testthat")
}

is_pkgdown <- function() {
  identical(Sys.getenv("PKGDOWN"), "true") ||
    endsWith(getwd(), "docs/reference")
}

get_path_to_mock_response <- function() {
  # get the path to the mock_responses dir
  # this can be
  # /docs/reference => in pkgdown
  # /tests/testthat => in tests
  # /vignettes => in vignettes
  paste0(
    gsub("(/$|/docs/reference/?|/tests/testthat/?|/vignettes/?)", "", getwd()),
    "/tests/testthat/mock_responses"
  )
}

start_mock_recording <- function() {
  Sys.setenv(RECORD_MOCK_API_CALLS = "true")
}
stop_mock_recording <- function() {
  Sys.unsetenv("RECORD_MOCK_API_CALLS")
}
is_mock_recording <- function() {
  identical(Sys.getenv("RECORD_MOCK_API_CALLS"), "true")
}