# install.packages('pak')
::pak('mjfrigaard/shinypak') pak
14 System tests
This chapter covers using shinytest2
and testthat
to perform system tests on the features and scenarios in your app-package.
“Failure to allow enough time for system test, in particular, is peculiarly disastrous. Since the delay comes at the end of the schedule, no one is aware of schedule trouble until almost the delivery date. Bad news, late and without warning, is unsettling to customers and to managers.” - ‘The Mythical Man-Month’, Frederick P. Brooks Jr.
System (or end-to-end) tests simulate real user interactions in a ‘pre-production’ environment to verify the whole application (or system) works.1 Approaches to system testing vary, but in general, we’ll want to run a system test for each feature in our application before a release.
If we’ve been documenting our unit and integration tests with BDD feature and scenario descriptions, the system tests can be used to confirm the functional requirements for the primary execution path (or user experience) before release.
14.1 Current tests
Launch app with the shinypak
package:
launch('13_tests-modules')
The current files in our tests folder are below:
tests
├── testthat
│ ├── fixtures
│ │ ├── make-tidy_ggp2_movies.R
│ │ └── tidy_ggp2_movies.rds
│ ├── helper.R
│ ├── test-mod_scatter_display.R
│ ├── test-mod_var_input.R
│ └── test-scatter_plot.R
└── testthat.R
3 directories, 7 files
The output from devtools::test()
is below:
::test() devtools
ℹ Testing sap
✔ | F W S OK | Context
⠏ | 0 | mod_scatter_display
Loading required package: shiny
INFO [2024-05-20 04:19:55] [ START display = selected_vars initial values]
⠋ | 1 | mod_scatter_display
INFO [2024-05-20 04:19:55] [ END display = selected_vars initial values]
INFO [2024-05-20 04:19:55] [ START display = scatterplot[['alt']] = 'Plot object']
⠙ | 2 | mod_scatter_display
INFO [2024-05-20 04:19:56] [ END display = scatterplot[['alt']] = 'Plot object']
INFO [2024-05-20 04:19:56] [ START display = inputs() creates ggplot2 object]
INFO [2024-05-20 04:19:56] [ END display = inputs() creates ggplot2 object]
✔ | 3 | mod_scatter_display
⠏ | 0 | mod_var_input
INFO [2024-05-20 04:19:56] [ START var_inputs = initial returned()]
INFO [2024-05-20 04:19:56] [ END var_inputs = initial returned()]
INFO [2024-05-20 04:19:56] [ START var_inputs = updated returned()]
INFO [2024-05-20 04:19:56] [ END var_inputs = updated returned()]
✔ | 2 | mod_var_input
⠏ | 0 | scatter_plot
INFO [2024-05-20 04:19:56] [ START fixture = tidy_ggp2_movies.rds]
INFO [2024-05-20 04:19:56] [ START fixture = tidy_ggp2_movies.rds]
INFO [2024-05-20 04:19:56] [ START data = sap::movies]
⠙ | 2 | scatter_plot
INFO [2024-05-20 04:19:56] [ END data = sap::movies]
✔ | 2 | scatter_plot
══ Results ═══════════════════════════════════════════════════════════════════════
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 7 ]
Nice code.
14.2 Setting up shinytest2
shinytest2
requires a few steps to get up and running (most notably the chromote
package), but you’ll find excellent documentation on the package website.2
The shinytest2::use_shinytest2()
performs the following setup steps:
✔ Adding
shinytest2::load_app_env()
totests/testthat/setup-shinytest2.R
✔ Adding
*_.new.png
to.gitignore
✔ Adding
_\\.new\\.png$
to.Rbuildignore
✔ Adding
shinytest2
toSuggests
field inDESCRIPTION
We also get some advice on using shinytest2
functions in our code:
• In your package code, use `rlang::is_installed("shinytest2")` or
`rlang::check_installed("shinytest2")` to test if shinytest2 is installed • Then directly refer to functions with `shinytest2::fun()`
After setting up shinytest2
, be sure you can create a new chromote session like the one below:
library(chromote)
<- ChromoteSession$new()
b $view() b
14.3 Record a test
If we launch the test recorder with shinytest2::record_test()
, change the inputs in our application, click on Expect Shiny values and Save test and exit, a test is recorded to a new test file: tests/testthat/test-shinytest2.R
The test runs and saves the PNG snapshot and test values to the tests/testthat/_snaps/
folder:
[ FAIL 0 | WARN 2 | SKIP 0 | PASS 1 ]
Warning (test-shinytest2.R:23:3): {shinytest2} recording: sap-feature-01
Adding new file snapshot: 'tests/testthat/_snaps/sap-feature-01-001_.png'
Warning (test-shinytest2.R:23:3): {shinytest2} recording: sap-feature-01
Adding new file snapshot: 'tests/testthat/_snaps/sap-feature-01-001.json'
[ FAIL 0 | WARN 2 | SKIP 0 | PASS 1 ]
If we view the contents test-shinytest2.R
we find each action we performed in the test recorder has a corresponding call in the test:
library(shinytest2)
test_that("{shinytest2} recording: sap-feature-01", {
<- AppDriver$new(name = "sap-feature-01", height = 887, width = 1241)
app $set_inputs(`vars-y` = "imdb_num_votes")
app$set_inputs(`vars-x` = "critics_score")
app$set_inputs(`vars-z` = "genre")
app$set_inputs(`vars-alpha` = 0.7)
app$set_inputs(`vars-size` = 3)
app$set_inputs(`vars-plot_title` = "New plot title")
app$expect_values()
app })
- 1
-
Initialize the
AppDriver$new()
with the name of the test and the dimensions of the Chromium browser.
- 2
-
Change the y axis (
vars-y
) to ‘IMBD number of votes’ (imdb_num_votes
)
- 3
-
Change the x axis (
vars-x
) to ‘Critics Score’ (critics_score
)
- 4
-
Change the color by (
vars-z
) to ‘Genre’ (genre
)
- 5
-
Change the point opacity (
vars-alpha
) to ‘0.7’
- 6
-
Change the point size (
vars-alpha
) to ‘3’
- 7
-
Change the plot title to (
vars-plot_title
) to ‘New plot title’ - 8
- Export the test values
We’ll use this initial test as a template for writing the steps in our test Scenario
s.
14.4 shinytest2
and BDD
There are multiple ways to approach your test layout with testthat
’s describe()
, it()
and/or test_that()
functions. Below is an example with dedicated Feature
and Scenario
descriptions, a reference to the feature number in the it()
(or test_that()
) call:
describe("Feature 1: Scatter plot data visualization dropdowns
As a film data analyst
I want to explore variables in the movie review data
So that I can analyze relationships between movie reivew sources", {
describe("Scenario: Change dropdown values for plotting
Given the movie review application is loaded
When I choose the variable [ ] for the x-axis
And I choose the variable [ ] for the y-axis
And I choose the variable [ ] for the color
And I choose the size of the points to be [ ]
And I choose the opacity of the points to be [ ]
And I enter '[ ]' for the plot title
Then the scatter plot should show [ ] on the x-axis
And the scatter plot should show [ ] on the y-axis
And the points on the scatter plot should be colored by [ ]
And the size of the points on the scatter plot should be [ ]
And the opacity of the points on the scatter plot should be [ ]
And the title of the plot should be '[ ]'", {
it("Feature 01", {
<- AppDriver$new(name = "feature-01",
app height = 800, width = 1173)
$set_inputs(`vars-y` = "imdb_num_votes")
app$set_inputs(`vars-x` = "critics_score")
app$set_inputs(`vars-z` = "genre")
app$set_inputs(`vars-alpha` = 0.7)
app$set_inputs(`vars-size` = 3)
app$set_inputs(`vars-plot_title` = "New plot title")
app$expect_values()
app
})
}) })
With this approach you can create the test file as soon as you have a Feature
description (and come back later to fill in the Scenario
s and tests).
An alternative approach is to use nested describe()
functions and include each of the Scenario
’s Then
steps in the it()
or test_that()
call (these are what will actually be tested):
library(shinytest2)
describe("Feature 1: Scatter plot data visualization dropdowns
As a film data analyst
I want to explore variables in the movie review data
So that I can analyze relationships between movie reivew sources", {
describe("Scenario A: Change dropdown values for plotting
Given the movie review application is loaded
When I choose the variable [ ] for the x-axis
And I choose the variable [ ] for the y-axis
And I choose the variable [ ] for the color", {
it("Then the scatter plot should show [ ] on the x-axis
And the scatter plot should show [ ] on the y-axis
And the points on the scatter plot should be colored by [ ]", {
<- AppDriver$new(name = "feature-01-senario-a",
app height = 800, width = 1173)
$set_inputs(`vars-y` = "imdb_num_votes")
app$set_inputs(`vars-x` = "critics_score")
app$set_inputs(`vars-z` = "genre")
app$expect_values()
app
})
})
describe("Scenario B: Change dropdown values for plotting
Given the movie review application is loaded
When I choose the size of the points to be [ ]
And I choose the opacity of the points to be [ ]
And I enter '[ ]' for the plot title", {
it("Then the size of the points on the scatter plot should be [ ]
And the opacity of the points on the scatter plot should be [ ]
And the title of the plot should be '[ ]'", {
<- AppDriver$new(name = "feature-01-senario-b",
app height = 800, width = 1173)
$set_inputs(`vars-alpha` = 0.7)
app$set_inputs(`vars-size` = 3)
app$set_inputs(`vars-plot_title` = "New plot title")
app$expect_values()
app
})
}) })
- 1
-
Test for scenario A
- 2
- Test for scenario B
An important note with this approach is the different names for each AppDriver$new()
(otherwise we’d be overwriting the previous snapshot/values).
14.4.1 Testing apps in inst/
If we want to test a feature for one of the alternative applications in sap
, we can pass their location to the app_dir
argument of AppDriver$new()
. In the test below, the scenario describes changing inputs for x, y, and color, and removing the missing values from the graph:
library(shinytest2)
describe(
"Feature 1: Scatter plot data visualization dropdowns
As a film data analyst
I want to explore movie review variables from IMDB (ggplot2movies::movies data)
So that I can analyze relationships between movie attributes and ratings", {
describe(
"Scenario: Change dropdown values for plotting
Given the movie review application is loaded
When I choose the variable ['Length'] for the x-axis
And I choose the variable ['Rating'] for the y-axis
And I choose the variable ['Genre'] for the color
And I click the ['Remove missing'] checkbox", code = {
it("Then the scatter plot should show ['Length'] on the x-axis
And the scatter plot should show ['Rating'] on the y-axis
And the points on the scatter plot should be colored by ['Genre']
And the missing values should be removed from the plot", {
test_logger(start = 'ggp2movies-feat-01', msg = "update x, y, z, missing")
<- AppDriver$new(app_dir = system.file("dev", package = "sap"),
app name = "ggp2movies_app-feature-01",
wait = FALSE, timeout = 30000,
height = 800, width = 1173)
$set_inputs(`vars-y` = "length")
app$set_inputs(`vars-x` = "rating")
app$set_inputs(`vars-z` = "genre")
app$set_inputs(`plot-missing` = TRUE)
app$expect_values()
app
test_logger(end = 'ggp2movies-feat-01', msg = "update x, y, z, missing")
})
}) })
- 1
-
Use
system.file()
to access application ininst/dev
- 2
-
Adjust the
wait
andtimeout
so the test will run
Not that I’ve changed the wait
and timeout
arguments in AppDriver$new()
because this tests takes over 10 seconds to complete (which I can see with my test_logger()
output):
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 0 ]
INFO [2024-05-21 12:39:03] [ START ggp2movies-feat-01 = update x, y, z, missing]
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 1 ]
INFO [2024-05-21 12:40:11] [ END ggp2movies-feat-01 = update x, y, z, missing]
When I confirm this in the output png
file, I can see the x, y, and color values have been changed (and the missing values have been removed).
14.5 The test.mode
option
If you recall, we’ve included an argument in both of our standalone app functions to allow for options
to be passed to shinyApp()
.
launch_app(options = list())
If we’re testing our application, we can include the test.mode = TRUE
option, which will return any values passed to exportTestValues()
:
launch_app(options = list(test.mode = TRUE), run = 'p')
We can also include this in our .Rprofile
as:3
options(shiny.testmode = TRUE)
To export values, place the name of exported reactive values in curly brackets ({}
). Below is an example using the inputs()
reactive object in the mod_scatter_display_server()
:
exportTestValues(
x = { inputs()$x },
y = { inputs()$y },
z = { inputs()$z },
alpha = { inputs()$alpha },
size = { inputs()$size },
title = { inputs()$plot_title }
)
In our test, we can create the AppDriver$new()
object, extract the values with get_values()
, then write tests against any of the exported values:
<- AppDriver$new(name = "test-values",
app height = 800, width = 1173,
wait = FALSE, timeout = 300000)
<- app$get_values()
test_values
'export']] test_values[[
$`plot-alpha`
[1] 0.5
$`plot-size`
[1] 2
$`plot-title`
[1] ""
$`plot-x`
[1] "imdb_rating"
$`plot-y`
[1] "audience_score"
$`plot-z`
[1] "mpaa_rating"
14.6 Test _snaps/
After writing our system tests and running devtools::test()
, the tests/testthat/_snaps/
folder contains the follow folders and files:
tests/testthat/_snaps/
├── app-feature-01
│ ├── feature-01-senario-a-001.json
│ ├── feature-01-senario-a-001_.png
│ ├── feature-01-senario-b-001.json
│ └── feature-01-senario-b-001_.png
├── ggp2_app-feature-01
│ ├── ggp2launch_app-feature-01-001.json
│ └── ggp2launch_app-feature-01-001_.png
└── shinytest2
├── sap-feature-01-001.json
└── sap-feature-01-001_.png
4 directories, 8 files
These outputs correspond to the three system test files in the tests/testthat/
:
tests/testthat
├── fixtures/
│ ├── make-tidy_ggp2_movies.R
│ └── tidy_ggp2_movies.rds
├── helper.R
├── setup-shinytest2.R
├── test-app-feature-01.R
├── test-ggp2_app-feature-01.R
├── test-mod_scatter_display.R
├── test-mod_var_input.R
├── test-scatter_plot.R
└── test-shinytest2.R
6 directories, 18 files
Launch app with the shinypak
package:
launch('14_tests-system')
Recap
Behavior-driven development (or behavior-driven testing) fills the gulf between non-technical stakeholders and developers by encouraging natural, descriptive language (often complete sentences) to define and communicate the application requirements.
Capturing the application’s desired behaviors in Feature
s (As a , I want, So that) and Scenario
s (Given, When, Then) provides a testing script that’s clear and easy to follow. Using the BDD format also makes system tests easier to update if the features and scenarios change.
System tests should strive to replicate the production conditions, even when/if it’s not possible to perfectly replicate the environment.↩︎
A great place to start is the Getting Started vignette.↩︎
We covered the
.Rprofile
in Section 4.6.3.↩︎