doctest

Introduction to doctest

Documentation examples and tests are similar in some ways:

  • They are self-contained pieces of code.

  • They should cover the software’s most important functions and typical uses.

  • They should be simple and clear: complex examples are hard for users to understand, and complex test code can introduce testing bugs.

This similarity makes it attractive to use “doctests”, which combine tests and documentation. Indeed, several languages, including Python and Rust, have doctests built in.1 R also checks for errors in examples when running R CMD check.

The doctest package extends this idea. It lets you write testthat tests, by adding tags to your roxygen documentation. This helps you check that your examples do what they are supposed to do.

Example

Here’s some roxygen documentation for a function:


#' Fibonacci function 
#' 
#' @param n Integer
#' @return The nth Fibonacci number
#' 
#' @doctest
#'
#' @expect type("integer")
#' fib(2)
#'
#' n <- 6 
#' @expect equal(8)
#' fib(n)
#' 
#' @expect warning("not numeric")
#' fib("a")
#'
#' @expect warning("NA")
#' fib(NA)
fib <- function (n) {
  if (! is.numeric(n)) warning("n is not numeric")
  ...
}

Instead of an @examples section, we have a @doctest section.

This will create tests like:

# Generated by doctest: do not edit by hand
# Please edit file in R/<text>

test_that("Doctest: fib", {
  # Created from @doctest for `fib`
  # Source file: <text>
  # Source line: 7
  expect_type(fib(2), "integer")
  n <- 6
  expect_equal(fib(n), 8)
  expect_warning(fib("a"), "not numeric")
  expect_warning(fib(NA), "NA")
})

The .Rd file will be created as normal, with an example section like:

\examples{
fib(2)

n <- 6 
fib(n)
fib("a")
fib(NA)
}

Usage

Install doctest from r-universe:

install.packages("doctest", repos = c("https://hughjonesd.r-universe.dev", 
                                      "https://cloud.r-project.org"))

Or from CRAN:

install.packages("doctest")

Or get the development version:

devtools::install("hughjonesd/doctest")

To use doctest in your package, alter its DESCRIPTION file to add the dt_roclet roclet and "doctest" package to roxygen:

Roxygen: list(roclets = c("collate", "rd", "namespace", 
              "doctest::dt_roclet"), packages = "doctest")

Then use roxygen2::roxygenize() or devtools::document() to build your package documentation.

Adding doctests to your package

Here’s a simple workflow to start using doctest:

  1. Alter your package DESCRIPTION as above.

  2. In your roxygen documentation, replace @examples by @doctest.

  3. In the package directory run roxygen2::roxygenize() or devtools::document() to create documentation. You should see Rd files created as normal in the man/ directory, including \examples sections.

  4. Add @expect tags to your @doctest sections.

  5. Run roxygenize() again. You will now see new files created in the tests/testthat directory, with the name test-doctest-<topic name>.R.

  6. Run devtools::test() and check that your tests pass.

At present, you can’t use doctest from the RStudio keyboard shortcut Ctrl + Shift + D, because this always uses the standard roxygen2 roclets. However, you can bind the RStudio addin “Devtools: document a package” to a keyboard shortcut. This will use the roclets from your package DESCRIPTION file.

You don’t need to add doctest as a dependency to your package. Just like roxygen2 itself, you can use it to create help files and tests without it being installed for users. However, you may wish to add it in Suggests:, to help other developers working on the package:

usethis::use_package("doctest", type = "Suggests")

Tags

The doctest package adds these tags to roxygen:

@doctest

Use @doctest instead of @examples:

#' @doctest
#' 
#' # ... examples for your function

The content of @doctest will be used in the .Rd “examples” section, and in a testthat test.

You can have more than one @doctest section. Each section creates one test like test_that("Test name", {...}). You can name the doctest, or leave it blank for a default name. All the sections will be merged into a single .Rd example.

#' @doctest Positive numbers
#' x <- 1
#' @expect equal(x)
#' abs(x)
#'
#' @doctest Negative numbers
#' x <- -1
#' @expect equal(-x)
#' abs(x)

@expect

@expect writes a testthat expectation.

#' @expect equal(4)
#' 2 + 2

You can use any expect_* function from testthat. Omit the expect_ at the start of the call.

The expression on the next line will be substituted as the first argument into the expect call:

expect_equal(2 + 2, 4)

Use a dot . to substitute in different places:

#' @expect equal(., rev(.))
#' c("T", "E", "N", "E", "T")

This becomes:

expect_equal(c("T", "E", "N", "E", "T"), rev(c("T", "E", "N", "E", "T")))

@expectRaw

@expectRaw writes an expectation, without substituting the next expression:

#' x <- 2 + 2
#' @expectRaw equal(x, 4)

@snap

@snap is shorthand for @expect snapshot(). This creates a snapshot test, which is useful for checking that complex examples haven’t changed.

@testRaw

@testRaw adds an arbitrary line of code to your test:

#' @testRaw skip_on_cran("Takes too long to run")
#' 
#' @expect equal(6765)
#' fib(20)

Tests are only written if they contain at least one @expect or @expectRaw tag, so use those tags to create expectations, not @testRaw.

@omit and @resume

While @testRaw includes a line of code in the test but not the example, @omit does the opposite: it includes all following code in the example but not the test. You can use @resume to restart including lines without creating a new expectation.

#' myfunc(1)
#' 
#' @omit
#' # No need to test plotting
#' plot(1:10, my_func(1:10))
#' 
#' @resume
#' x <- NA
#' @expect warning()
#' myfunc(x)

If you are using @testRaw and @omit a lot, it is probably a good idea to separate out the test and the example. You can do this by renaming the test-doctest- file, and removing the “Generated by doctest” line within it. Then change your @doctest tag back to @examples.

@doctestExample

@doctestExample filename.R includes the R code in filename.R as an example. It is a drop-in replacement for roxygen2’s @example. The R code isn’t checked for doctest tags and isn’t included in any tests.

Caveats

  • Don’t use @doctest and @examples in the same topic. That won’t work.

  • Doctest currently ignores \dontrun and \donttest macros. Potentially, that could lead to dangerous code being included in tests. To avoid this, use the @omit tag.

  • Each @doctest section should include a complete self-contained example, that would work inside a test_that expression. Don’t rely on variables from a previous @doctest.

  • You can include expectations within e.g. if blocks or for loops. Don’t forget that each roxygen tag must be indented with a single space:

#' # Right:
#' if (TRUE) {
#' @expect equals(4)
#'   2+2
#' }
#' # Wrong:
#' if (TRUE) {
#'   @expect equals(4)
#'   2+2
#' }

Writing good doctests

Tests and documentation are similar, but not identical. Tests need to cover difficult corner cases. Examples need to convey the basics to the user. I like the following advice:

… write the best possible documentation, and [R] makes sure the code samples in your documentation actually compile and run [and do what they are supposed to do]

Programming Rust, Blandy, Orendorff and Tindall, 2021

In particular, use doctest as an addition to manually created tests, not a substitute for them. Use doctest to make sure your examples do what they expect, and for simple tests of basic functionality. If it’s hard to specify what to test for, consider using @snap to capture output:

#' @snap
summary(model)

For more complex test cases, write a test file manually.

To see an example of using the doctest package in “production”, check out vignette("conversion").

  1. https://docs.python.org/3/library/doctest.html, https://doc.rust-lang.org/rustdoc/write-documentation/documentation-tests.html↩︎