Integration testing with Shiny

This section describes features that are still in development and not yet available in the version of Shiny that is on CRAN. If you want to try these features, you’ll need to install Shiny from GitHub using: remotes::install_github("rstudio/shiny")

There are many different types of automated testing, but a comprehensive testing strategy for a complex Shiny app might cover:

  1. unit testing to test the individual functions in your app (testthat or a similar package)
  2. integration testing to test the interactions between different functions in your app
  3. functional testing to test the overall function of the app from the user’s perspective in a browser (shinytest, shinyloadtest)

The testModule() and testServer() functions in Shiny address the “integration testing” point above for modules and Shiny server functions, respectively. We’ll take a look at both below, but the mechanics of writing the tests are identical regardless of whether you’re testing a module or an application.

(If you haven’t seen modules before, we recommend that you go take a look at the article on modules first, as this article will presume familiarity.)

library(shiny)
module <- function(input, output, session) {
  myreactive <- reactive({
    input$x * 2
  })
  output$txt <- renderText({
    paste0("I am ", myreactive())
  })
}

This module

  • depends on one input (x),
  • creates an intermediate reactive (myreactive), and
  • updates an output (txt) reactively.

It would be nice to write tests that confirm that the module behaves the way we expect. We can do so using the testModule function.

testModule(module, {
  cat("Initially, input$x is NULL, right?", is.null(input$x), "\n")

  # Give input$x a value.
  session$setInputs(x = 1)

  cat("Now that x is set to 1, myreactive is: ", myreactive(), "\n")
  cat("\tand output$txt is: ", output$txt, "\n")
  
  # Now update input$x to a new value
  session$setInputs(x = 2)

  cat("After updating x to 2, myreactive is: ", myreactive(), "\n")
  cat("\tand output$txt is: ", output$txt, "\n")
})
## Initially, input$x is NULL, right? TRUE 
## Now that x is set to 1, myreactive is:  2 
##  and output$txt is:  I am 2 
## After updating x to 2, myreactive is:  4 
##  and output$txt is:  I am 4

There are a few things to notice in this example.

First, the test expression provided here assumes the existence of some variables – specifically, input, output, and myreactive. This is safe because the test code provided to testModule is run in a child environment of the module. This means that any parameters passed in to your module (such as input, output, and session) are readily available, as are any intermediate objects or reactives that you define in the module. However, because it’s a child environment, your test code is less likely to accidentally modify anything in the module itself.

Second, you’ll need to give values to any inputs that you want to be defined; by default, they’re all NULL. We do that using the session$setInputs() method. The real session object that you’d see when running a Shiny app does not have this method; however, the session object used in testModule differs from the real session object Shiny uses. This allows us to tailor it to be more suitable for testing purposes by modifying or creating new methods such as setInputs().

Last, you’re likely used to assigning to output, but here we’re reading from output$txt in order to check its value. When running inside testModule, you can simply reference an output and it will give the value produced by the render function.

Automated Tests

Realistically, we don’t want to just print the values for manual inspection; we’ll want to leverage them in automated tests. That way, we’ll be able to build up a collection of tests that we can run against our module in the future to confirm that it always behaves correctly. You can use whatever testing framework you’d like (or none at all!), but we’ll use the expect_* functions from the testthat package in this example.

# Bring in testthat just for its expectations
library(testthat)
testModule(module, {
  session$setInputs(x = 1)
  expect_equal(myreactive(), 2)
  expect_equal(output$txt, "I am 2")
  session$setInputs(x = 2)
  expect_equal(myreactive(), 4)
  expect_equal(output$txt, "I am 4")
})

If there’s no error, then we know our tests ran successfully. If there were a bug, we’d see an error printed. For example:

tryCatch({
  testModule(module, {
    session$setInputs(x = 1)

    # This expectation will fail
    expect_equal(myreactive(), 99)
  })
}, error = function(e){
  print("There was an error!")
  print(e)
})
## [1] "There was an error!"
## myreactive() not equal to 99.
## 1/1 mismatches
## [1] 2 - 99 == -97

Testing Shiny Apps

In addition to testing Shiny modules, you can also test Shiny applications. The testServer function will automatically extract the server portion given an application’s directory and you can test it just like you do any other module.

# app.R -- written into tempdir()
ui <- fluidPage(
 textInput("name", "Name: "),
 textOutput("greeting")
)

server <- function(input, output) {
 output$greeting <- renderText({
 paste0("Hello, ", input$name, "!")
 })
}

shinyApp(ui = ui, server = server)
testServer({
  session$setInputs(name = "Shiny User")
  expect_equal(output$greeting, "Hello, Shiny User!")
  
  session$setInputs(name = "New Name")
  expect_equal(output$greeting, "Hello, New Name!")
}, appDir = tempdir())

Promises

testModule can handle promises inside of render functions.

library(promises)
library(future)
plan(multisession)
module <- function(input, output, session){
  output$async <- renderText({
    # Stash the value since you can't do reactivity inside of a promise
    # See https://rstudio.github.io/promises/articles/shiny.html#shiny-specific-caveats-and-limitations
    t <- input$times
    
    # A promise chain that repeats the letter A and then collapses it into a string
    future({ rep("A", times = t) }) %...>%
      paste(collapse = "")
  })
}
testModule(module, {
  session$setInputs(times = 3)
  expect_equal(output$async, "AAA")

  session$setInputs(times = 5)
  expect_equal(output$async, "AAAAA")
})

As you can see, no special precautions were required for a render function that uses promises. Behind-the-scenes, the code in testModule will stall when trying to read from an output that returned a promise. This allows you to interact with asynchronous outputs in your tests as if they were synchronous.

Modules with additional inputs

testModule can also handle modules that accept additional arguments such as this one.

module <- function(input, output, session, arg1, arg2){
  output$txt1 <- renderText({ arg1 })

  output$txt2 <- renderText({ arg2 })
}

Additional arguments should be passed after the test expression as named parameters.

testModule(module, {
  expect_equal(output$txt1, "val1")
  expect_equal(output$txt2, "val2")
}, arg1 = "val1", arg2 = "val2")

Accessing a module’s returned value

Some modules return reactive data as an output. For such modules, it can be helpful to test the returned value as well. The returned value from the module is made available as a property on the mock session object as demonstrated in this example.

module <- function(input, output, session){
  reactive({
    return(input$a + input$b)
  })
}
testModule(module, {
  session$setInputs(a = 1, b = 2)
  expect_equal(session$returned(), 3)
  # And retains reactivity
  session$setInputs(a = 2)
  expect_equal(session$returned(), 4)
})

Timer and Polling

Testing behavior that relies on timing is notoriously difficult. Modules will behave differently on different machines and under different conditions. In order to make testing with time more deterministic, testModule uses simulated time that you control, rather than the actual computer time. Let’s look at what happens when you try to use “real” time in your testing.

module <- function(input, output, session){
  rv <- reactiveValues(x=0)

  observe({
    # Cause the observer to invalidate every 0.1 seconds
    invalidateLater(100)
    isolate(rv$x <- rv$x + 1)
  })
}
testModule(module, {
  expect_equal(rv$x, 1) # The observer runs once at initialization

  Sys.sleep(1) # Sleep for a second

  expect_equal(rv$x, 1) # The value hasn't changed
})

This behavior may be surprising. It seems like rv$x should have been incremented 10 times (or perhaps 9, due to computational overhead). But in truth, it hasn’t changed at all. This is because testModule doesn’t consider the actual time on your computer, but instead it uses simulated time.

In order to cause testModule to progress through time, instead of Sys.sleep, we’ll use session$elapse – another method that exists only on our mocked session object. Using the same module object as above:

testModule(module, {
  expect_equal(rv$x, 1) # The observer runs once at initialization

  session$elapse(100) # Simulate the passing of 100ms

  expect_equal(rv$x, 2) # The observer was invalidated and the value updated!

  # You can even simulate multiple events in a single elapse
  session$elapse(300)
  expect_equal(rv$x, 5)
})

As you can see, using session$elapse caused testModule to recognize that (simulted) time had passed which triggered the reactivity as we’d expect. This approach allows you to deterministically control time in your tests while avoiding expensive pauses that would slow down your tests. Using this approach, this test can complete in only a fraction of the 100ms that it simulates.

You should note that only certain time-based functions are aware of this mocked time that can be managed via elapse. Shiny functions like reactivePoll, invalidateLater, and reactiveTimer will all abide by this simulated time, but time-based functions in other packages (such as later::later are not aware of mocked time and will not behave according to these rules.)

Complex Outputs (plots, htmlwidgets)

Work in progress – We intend to add more helpers to make it easier to inspect and validate the raw HTML/JSON content. Our integration testing tools do not yet provide functionality for validating such output.

Thus far, we’ve seen how to validate simple outputs like numeric or text values. Real Shiny modules and applications often use more complex outputs such as plots or htmlwidgets. Validating the correctness of these is not as simple, but is doable.

You can access the data for even complex outputs in testModule, but the structure of the output may initially be foreign to you.

module <- function(input, output, session){
  output$plot <- renderPlot({
    df <- data.frame(length = iris$Petal.Length, width = iris$Petal.Width)
    plot(df)
  })
}
testModule(module, {
  print(str(output$plot))
})
## List of 4
##  $ src     : chr "data:image/png;base64,iVBORw0KGgoAAAANSUhEU"| __truncated__
##  $ width   : num 600
##  $ height  : num 400
##  $ coordmap:List of 2
##   ..$ panels:List of 1
##   .. ..$ :List of 4
##   .. .. ..$ domain :List of 4
##   .. .. .. ..$ left  : num 0.764
##   .. .. .. ..$ right : num 7.14
##   .. .. .. ..$ bottom: num 0.004
##   .. .. .. ..$ top   : num 2.6
##   .. .. ..$ range  :List of 4
##   .. .. .. ..$ left  : num 59
##   .. .. .. ..$ right : num 570
##   .. .. .. ..$ bottom: num 326
##   .. .. .. ..$ top   : num 58
##   .. .. ..$ log    :List of 2
##   .. .. .. ..$ x: NULL
##   .. .. .. ..$ y: NULL
##   .. .. ..$ mapping: Named list()
##   ..$ dims  :List of 2
##   .. ..$ width : num 600
##   .. ..$ height: num 400
## NULL

As you can see, there are a lot of internal details that go into a plot. Behind-the-scenes, these are all the details that Shiny will use to correctly display a plot in a user’s browser. You don’t need to learn about all of these properties – and they’re all subject to change.

In terms of your testing strategy, you shouldn’t bother yourself with “is Shiny generating the correct structure so that the plot will render in the browser?” That’s a question that the Shiny package itself needs to answer (and one for which we have our own tests). The goal for your tests should be to ask “is the code that I wrote producing the plot I want?” There are two components to that question:

  1. Does the plot generate without producing an error?
  2. Is the plot visually correct?

testModule is great for assessing the first component here. By merely referencing output$plot in your test, you’ll confirm that the plot was generated without an error. The second component is better suited for a shinytest test which actually loads the Shiny app in a headless browser and confirms that the content visually appears the same as it did previously. Doing this kind of test in testModule would be complex and may not be reliable as graphics devices differ slightly from platform to platform; i.e. the exact bits in the src field of your plot will not necessarily be reproducible between different versions of R or different operating systems.

For htmlwidgets, you can adopt a similar strategy. The goal is not to confirm that the htmlwidget’s render function is behaving properly – but rather that the data that you intend to render is indeed getting passed through.

We could modify the above example to better represent this approach.

module <- function(input, output, session){
  # Move any complex logic into a separate reactive which can be tested comprehensively
  plotData <- reactive({
    data.frame(length = iris$Petal.Length, width = iris$Petal.Width)
  })

  # And leave the `render` function to be as simple as possible to lessen the need for
  # integration tests.
  output$plot <- renderPlot({
    plot(plotData())
  })
}
testModule(module, {
  # Confirm that the data reactive is behaving as expected
  expect_equal(nrow(plotData()), 150)
  expect_equal(ncol(plotData()), 2)
  expect_equal(colnames(plotData()), c("length", "width"))

  # And now the plot function is so simple that there's not much need for
  # automated testing. If we did wish to evaluate the plot visually, we could
  # do so using the shinytest package.
  output$plot # Just confirming that the plot can be accessed without an error
})

You could adopt a similar strategy with other plots or htmlwidgets: move the complexity into reactives that can be tested, and leave the render functions as simple as possible.

Flushing Reactives

Reactivity differs from imperative programming in that the processing required to update reactives can be deferred and batched together. While this is a boon for the computational speed of a reactive system, it does create some ambiguity about when the reactives should be processed or “flushed”.

testModule will do its best to automatically “flush” the reactives at the right time. There are two triggers that will cause a reactive flush:

  1. Calling session$setInputs() - After setting the updated inputs, the reactives will be flushed.
  2. Calling session$elapse() - After the scheduled callbacks are executed, reactives will be flushed.

However, there may be other times that a Shiny module author might want to trigger a reactive flush. For instance, you might want to flush the reactives after updating an element in a reactiveValues in your module like this one.

module <- function(input, output, session){
  rv <- reactiveValues(a=1)
  output$txt <- renderText({
    rv$a
  })
}

testModule(module, {
  expect_equal(output$txt, "1")
  
  rv$a <- 2
  
  # testModule has no innate knowledge of our `rv` variable; 
  # therefore, it hasn't been updated
  expect_equal(output$txt, "1")
  
  # We'll need to manually force a flush of the reactives
  session$flushReact()
  
  expect_equal(output$txt, "2")
})

As you can see, we can use session$flushReact() to trigger a reactive flush at any point we’d like. In this example, testModule knows nothing about our rv variable. Therefore if we want to observe reactive changes that occur after manually updating this variable, we’d need to explicitly flush the reactives.



If you have questions about this article or would like to discuss ideas presented here, please post on RStudio Community. Our developers monitor these forums and answer questions periodically. See help for more help with all things Shiny.


Start
Build
Improve