End-to-end testing with shinytest2 (2023)

This is the final part of a series of three blog posts about using the{shinytest2} package to develop automated tests for shiny applications.In the posts we cover

• the purpose of browser-driven end-to-end tests for a shinydeveloper, and tools (like {shinytest2}) that help implementthem;

• how to write and run a simple test using{shinytest2};

• how best to design your test code so that it supports your futurework (this post).

By this point in the blog series, we have created a simple shinyapplication as an R package, added the {shinytest2} testinginfrastructure, and have written, ran, broken and fixed a {shinytest2}test case. Here, we will add a new feature to the application. As inreal (programming) life, we will add a new test for this feature, andensure that our old test still passes.

UI-driven end-to-end tests require a bit more code than unit tests. Forexample, starting the app and navigating around to set up some initialstate will require a few lines of code. But these are things you’lllikely need to do in several tests. As you add more and more test casesand these commonalities reveal themselves, it pays to extract out somehelper functions and / or classes. By doing so, your tests will looksimpler, the behaviour that you are testing will be more explicit, andyou’ll have less code to maintain. We’ll show some software designs thatmay simplify your {shinytest2} code.

This post builds upon the previous posts in the series, but is quite abit more technical than either of them. In addition to shinydevelopment, you’ll need to know how to define functions in R and forthe last section you’ll need to know about object-oriented programmingin R (specifically using R6). The ideas in that section may be ofinterest even if you aren’t fluent with R6 classes yet.

Let’s get started.

The initial application

Our initial shiny application had a text field where the user couldenter their name and a “Greet” button. The source code can be obtainedfromgithub.On clicking the button, a greeting (“Hello !”) is displayed inthe app. The source code for the user interface and server function isshown below.

# In ./R/ui.Rui = function(req) { fluidPage( textInput("name", "What is your name?"), actionButton("greet", "Greet"), textOutput("greeting") )}# In ./R/server.Rserver = function(input, output, session) { output$greeting = renderText({ req(input$greet) paste0("Hello ", isolate(input$name), "!") })}

For this app we have a single test that checks that the greeting isdisplayed once the user has entered their name and clicked the “Greet”button.

# ./tests/testthat/test-e2e-greeter_accepts_username.Rtest_that("the greeter app updates user's name on clicking the button", { # GIVEN: the app is open shiny_app = shinyGreeter::run() app = shinytest2::AppDriver$new(shiny_app, name = "greeter") app$set_window_size(width = 1619, height = 970) # WHEN: the user enters their name and clicks the "Greet" button app$set_inputs(name = "Jumping Rivers") app$click("greet") # THEN: a greeting is printed to the screen values = app$expect_values(output = "greeting", screenshot_args = FALSE)})

Do you require help building a Shiny app? Would you like someone to take over the maintenance burden?If so, check outour Shiny and Dash services.

Writing your second test

We’ll add a second bit of functionality to the app first. A simplechange, might be to greet the user in Spanish:

# In the UItextOutput("spanish_greeting")# In the serveroutput$spanish_greeting = renderText({ req(input$greet) paste0("Hola ", isolate(input$name), "!")})

The first thing to note is that with the change to the app, the firsttest still passes. It would have failed had we not restricted our testto just look at the greeting variable (for example, if we had usedapp$expect_values() to make a snapshot of all the variables that arein-play and to take an image of the app).

We want to add a new test to check the spanish_greeting as well as thegreeting variable.

To add a new test to the app, we could use the {shinytest2} recorder (asin the previous post), or we could just copy and paste the first test,and modify the bits we need to. We’ll do the latter.

# ./tests/testthat/test-e2e-greeter_accepts_username.R# ... snip ...test_that("the greeter app prints a Spanish greeting to the user", { # GIVEN: The app is open shiny_app = shinyGreeter::run() app = shinytest2::AppDriver$new(shiny_app, name = "spanish_greeter") app$set_window_size(width = 1619, height = 970) # WHEN: the user enters their name and clicks the "Greet" button app$set_inputs(name = "Jumping Rivers") app$click("greet") # THEN: a Spanish greeting is printed to the screen values = app$expect_values(output = "spanish_greeting", screenshot_args = FALSE)})

Note that we have changed the name argument to AppDriver$new(), thisallows us to have multiple test cases in the same script – were theAppDrivers for the English- and the Spanish-test both givenname="greeter", the snapshots would both be written to the same file.

Use functions to simplify and clarify your test code

The new test is almost identical to the previous one we wrote. That kindof duplication should set off alarm bells – more duplication means moremaintenance.

In R, the simplest way to reduce code duplication is by writing afunction.

Simplify the set-up code

Let’s add a function to get the app into the pre-test state:

initialise_test_app = function(name) { shiny_app = shinyGreeter::run() app = shinytest2::AppDriver$new(shiny_app, name = name) app$set_window_size(width = 1619, height = 970) app}

With that we can start the test-version of the app usingapp = initialise_test_app("greeter") in the first test andapp = initialise_test_app("spanish_greeter") in the second. Thisremoves a few lines of code, and would make it easier to write newtests, but the main purpose of doing this is to make the test code moreprominent.

The Spanish test now looks like:

test_that("the greeter app prints a Spanish greeting to the user", { # GIVEN: The app is open app = initialise_test_app("spanish_greeter") # WHEN: the user enters their name and clicks the "Greet" button app$set_inputs(name = "Jumping Rivers") app$click("greet") # THEN: a Spanish greeting is printed to the screen values = app$expect_values(output = "spanish_greeting", screenshot_args = FALSE)})

Make the user steps more descriptive

What’s actually happening when the following code runs?

app$set_inputs(name = "Jumping Rivers")app$click("greet")

First we set the value for the input$name variable to be “JumpingRivers” and then we click on a button that has the HTML identifier“greet”. These are quite ‘internal’ concerns. What’s really happening isthat the user is entering their username into the app (clicking thebutton is part of that process).

This is a really simple app, so it shouldn’t take long to work out whatthe above code does here. But in more complicated apps, and when testingmore complicated workflows, the series of steps that define the useractions can be quite extensive.

Having well-defined functions that are responsible for the differentsteps in a test workflow is really valuable. With these, your non-codingcolleagues will find it easier to follow the connection between what thecode is testing and how the test is defined.

Even in this simple setting, it might be beneficial to introduce afunction:

enter_username = function(app, username) { app$set_inputs(name = username) app$click("greet") # return the app object, so that you can pipe together the actions invisible(app)}

Then you can rewrite the test steps:

test_that("the greeter app prints a Spanish greeting to the user", { # GIVEN: The app is open app = initialise_test_app("spanish_greeter") # WHEN: the user enters their name and clicks the "Greet" button enter_username(app, "Jumping Rivers") # THEN: a Spanish greeting is printed to the screen values = app$expect_values(output = "spanish_greeting", screenshot_args = FALSE)})

Another benefit of introducing functions for commonly repeated parts ofyour test actions, relates to refactoring. Suppose the input$namevariable was renamed in the app. With the initial two tests, toaccommodate the change in this variable name we would have had to touchtwo different places in the code – one in the English test and one inthe Spanish test. Now we only have to modify a single line inenter_username(). A similar issue happens when decomposing apps intoshiny modules (because the HTML identifiers for different elements willchange with the refactoring).

Make your expectations descriptive too …

The snapshot tests used by {shinytest2} are wonderful if you need tocompare many values at once, or you need to do visual comparison of thecontents of your app. But they can make your test cases a bit opaque. Inthe above, on entering their username, two welcome messages were printedto the screen. While each test was running, {shinytest2} compared theobserved value for a given welcome message to a previously storedvalue – but that previously stored value is stored a distance from theplace where the test is defined. Hiding the expectations away like thismay make it hard for a new developer to see why the actions performed inthe “WHEN” steps of a test should culminate in the values observed inthe “THEN” step.

{shinytest2} provides some additional methods that help extract specificvalues. With these, you can use the expectation functions from{testthat} much as you would when unit-testing functions in R.

For example, we might rewrite the first test like so:

test_that("the greeter app updates user's name on clicking the button", { # GIVEN: The app is open app = initialise_test_app("greeter") # WHEN: the user enters their name and clicks the "Greet" button enter_username(app, "Jumping Rivers") # THEN: a greeting is printed to the screen message = app$get_value(output = "greeting") expect_equal(message, "Hello Jumping Rivers!")})

The source code for this version of the application can be obtained fromgithub.

The Page Object Model

The functions that were introduced above hid the details of the app awayfrom us. By using these functions, we don’t need to know which HTMLelement or shiny variable we need to interact with or modify whensetting the username.

A pattern called the “Page Object Model”(POM) takes this ideaof hiding an app’s internal details away from the test author evenfurther. The POM is common in UI-based end-to-end testing in otherlanguages. Here, a class is defined that contains methods forinteracting with the app (but does not contain any code to perform testexpectations). The test code calls methods provided by the POM, so thatthe test code is more concise and descriptive. A neat way to achievethis design in R, is by using R6 classes. Here, we might have a classthat has a method for opening the app, and a method enter_username.

The AppDriver class provided by {shinytest2} is an R6 class. Itprovides a lot of methods that we used above (expect_values,get_value) for interacting with the app. So by now, you have someexperience of using an R6 object. We can inherit from the AppDriverclass to create a POM that is specific for our app as follows:

GreeterApp = R6::R6Class( "GreeterApp", # Alternatively you could pass an AppDriver in at initiation inherit = shinytest2::AppDriver, public = list( width = 1619, height = 970, initialize = function(name) { shiny_app = shinyGreeter::run() super$initialize(shiny_app, name = name) self$set_window_size(width = self$width, height = self$height) }, enter_username = function(username) { self$set_inputs(name = username) self$click("greet") invisible(self) } ))

With that class in place, we can rewrite our original test as follows:

test_that("the greeter app updates user's name on clicking the button", { # GIVEN: The app is open app = GreeterApp$new("greeter") # WHEN: the user enters their name and clicks the "Greet" button app$enter_username("Jumping Rivers") # THEN: a greeting is printed to the screen message = app$get_value(output = "greeting") expect_equal(message, "Hello Jumping Rivers!")})

Adding all this design work into setting up your tests might seem like alot of unnecessary work. But, it does make it easier to add new tests,it makes it simpler to keep your tests passing as you refactor your appand it makes your tests easier to follow.

If your tests hinder your ability to add new features to your app, orprevent you from restructuring your app it may be worth restructuringyour test code.

The source code for the application in its current form can be obtainedfromgithub.

Conclusion

This blog series was a brief introduction to UI-based end-to-end testsfor web applications and to the new package {shinytest2}. These kinds oftests are very powerful and with {shinytest2}’s test recorder, they arerelatively easy to construct. But, because the whole app is within theirscope, these tests can be quite frail and difficult to follow. So if youfind that small changes to your app may lead seemingly unconnected teststo fail, or that keeping your tests passing requires you to make verysimilar changes in multiple places, you may benefit from some of theideas in this post:

• Can you introduce some functions (or POM methods) to clarify what ishappening in each step of your test?
• Can you ensure that the assertion in your test is only comparingdata that is directly relevant to that test?

The shinytest2 vignettes (Robusttesting,Testing indepth)discuss some of the ideas in this post in more depth, and with aslightly different perspective.

For updates and revisions to this article, see the original post

Related