This quarto project covers the YouTube series on shinytest2 (Part 1, Part 2, and Part 3). The quoted text comes from the transcripts, which I’ve edited to capture the ‘gist’ of what is covered in each video.
Setup
Check out my GitHub repository for getting setup with Chromium and chromote (and the app test files).
The inital application for the tutorial is below:
view greet app
library(shiny)library(magrittr)ui <-fluidPage(textInput("name", "what is your name"),actionButton("greet", "Greet"),textOutput("greeting"),textOutput("first_letter"))server <-function(input, output, session) { output$greeting <-renderText({req(input$name)paste0("Hello ", input$name, "!") }) %>%bindEvent({input$greet}) first_letter <-reactive({req(input$name)tolower(stringr::str_extract(input$name, "^.")) }) %>%bindEvent({input$greet}) output$first_letter <-renderText({paste0("The first letter in your name is ", first_letter(), "!") })}shinyApp(ui, server)
shinytest2 is a wonderful package to help do regression testing for your Shiny applications. And we’re going to perform this regression testing inside testthat. And it’s not regression testing like linear model or something like that. This is a consistent behavior over time, and so we do not want it to regress, or we do not want it to change. That’s what I mean for regression testing.
Before we get into shinytest2 and how it works, I just want to do this demo app. This demo app is short, a little toy example.
App ui
We have a fluidPag() that has a textInput(). It’s like what’s your name. We can fill it out. And then we also have an actionButton() of "greet", the text "Greet". And then we have two textOutput()s. One is a greeting, like "Hello, Barrett!" And then the other one is what is the first letter in your name. And it’s neat, like it’s short little app.
view ui
ui <-fluidPage(textInput("name", "what is your name"),actionButton("greet", "Greet"),textOutput("greeting"),textOutput("first_letter"))
So if I have it on here, I have "Barrett"
I click Greet. We get, "Hello Barrett!"
And if I’m here, we could do like "Shinytest2"
"What is your name?"Shinytest2."
Greet, "Hello Shinytest2."The first letter is S.
So it works. It’s a nice little app.
App server
view server
server <-function(input, output, session) { output$greeting <-renderText({req(input$name)paste0("Hello ", input$name, "!") }) %>%bindEvent({input$greet}) first_letter <-reactive({req(input$name)tolower(stringr::str_extract(input$name, "^.")) }) %>%bindEvent({input$greet}) output$first_letter <-renderText({paste0("The first letter in your name is ", first_letter(), "!") })}
The app for the server– what it does is we have a renderText() that requires the name and given a Greet click, we will say, "Hello" name. And then for the first letter calculation, we require a name, and we extract the first character of the name. And we send it tolower(). And we will bind all of that calculations to when the Greet button is clicked. Given this first letter, whenever it updates, we will then call renderText() and say the first letter in your name is this first letter. Awesome.
So let’s imagine that I have—well I have now created this app, or I am working on a team, and I want to make sure that behaviors are consistent. And tests are worth so much, because you don’t necessarily need to communicate that because the test is there. And if the test breaks, then we can figure out what’s happening.
And this also allows you to do other things like sweeping code changes. Like what if I want to change the guts of how first letter is calculated. I know this is a small function, but it could be much more complicated. And having tests allow you to do these sweeping changes with confidence, otherwise you just look out and go, no, that’s close enough. I don’t know. I hope I covered everything. At least with tests, we can say all of the tests pass. And if you’re missing something, we can add them in and then make your test more robust.
Recording tests
When you run record_test(), it opens up Running record_test() your application in the Chrome browser or Chromium based browser. Mine by default is set to Chrome.
It’ll have your app in this IFrame. And on the sidebar is actually the recorder. And we can see there’s some buttons to call Expect Shiny values or Expect screenshot. And then there’s also an area of all the code that will be executed later that is able to replay your recording. So let’s make a recording:
Input, output, expect shiny values
This is
“What is your name?”*
My name is "Barrett"
So let’s Greet Barrett. And it’ll say,
“Hello Barret! The first letter in your name is b!”
And we can see that I set inputs, name is "Barrett". And then I clicked the Greet button, and then an output value is updated. Awesome.
…I can save this as “hello-barrett” because that is the name that I used and just for my reference…I need to say, Expect Shiny values. And this will keep track– the expect values will keep track of all your input, output, and exported values. And exported values, we’ll get to in another video. So once we have at least one expectation, we can then click Save test and exit.
This will save the test and then immediately run the test– play back the test again. And there are some warnings, because we have some new snapshots. But that’s OK. And it looks like it all passed, because there’s also nothing to say that the snapshots would fail. So let’s take a look at what was created.
view record_test() output
• Saving test runner: tests/testthat.R• Saving test file: tests/testthat/test-shinytest2.R✔ Adding 'shinytest2::load_app_env()' to 'tests/testthat/setup-shinytest2.R'• Modify '/Users/mjfrigaard/projects/dev/greet/tests/testthat/test-shinytest2.R'• Running recorded test: tests/testthat/test-shinytest2.R✔ | F W S OK | Context✔ | 2 1 | shinytest2 [2.5s] ────────────────────────────────────────────────────────────────────────────────Warning (test-shinytest2.R:7:3): {shinytest2} recording: hello-barretAdding new file snapshot: 'tests/testthat/_snaps/hello-barret-001_.png'Warning (test-shinytest2.R:7:3): {shinytest2} recording: hello-barretAdding new file snapshot: 'tests/testthat/_snaps/hello-barret-001.json'────────────────────────────────────────────────────────────────────────────────══ Results ═════════════════════════════════════════════════════════════════════Duration: 2.9 s[ FAIL 0 | WARN 2 | SKIP 0 | PASS 1 ]
tests/ folder
…and then the tests folder, it will be a sibling to that. We’ll go inside that in a second.
And in there, we will have a testthat.R which internally, it just has shinytest2::test_app()
view testthat.R contents
shinytest2::test_app()
And then next to the testthat.R is a testthat folder. There there’s three parts here. The first part is the _snaps folder.
setup-shinytest2.R
The second one is the setup.R. setup.R is there for … more complicated apps … where you have an R folder or you have global.R in your Shiny application. We load all of this information inside setup.R, so that your tests have that information available.
view setup-shinytest2.R contents
# Load application support files into testing environmentshinytest2::load_app_env()
We then say app$set_inputs…I typed in "Barrett" for the name
app$set_inputs(name ="Barret")
I then clicked the Greet button
app$click("greet")
…and then we called app$expect_values.*
app$expect_values()})
Expect values is actually a wrapper around teststhat, expect_snapshot_file(). This file will contain a JSON representation of your input, output, and export values. And that is stored in the _snaps folder.
So we will open up _snaps. We’ll open up shinytest2, because that is the name of the file, shinytest2. So that name. And then there was the first JSON file that was created.
And this one if we click it, has my input, output, and export.
{"input":{"greet":1,"name":"Barret"},"output":{"first_letter":"The first letter in your is b!","greeting":"Hello Barret!"},"export":{}}
So my inputs is at the time of calling expect_values(), which was after setting an input and after clicking Greet button. We have input of "greet": 1 for one time to click the button. And name was "Barrett"
The output was "first_letter" was, "The first letter your name is b!" And the greeting is, "Hello Barrett!" We didn’t export any test values yet. But we’ll get into that in a later video.
"output":{"first_letter":"The first letter in your is b!","greeting":"Hello Barret!"}
So this is our test. This is awesome. We can, if we want, change the height and width. We can add more tests. This is our test file to work with.
Image files
A note on .png files created during testing:
As a side note, you might notice that there’s a PNG file here created next to the JSON file. This is there, because images typically are fairly brittle. And we don’t like to use them if we can help it, because there’s lots of things outside of our control that we can’t account for, such as if the R version updates, or let’s say you change your version of DT, or let’s say you update your system font. There’s many different things that are outside of Shiny or R that are not in your control. And screenshots will fail if this happens. And no one likes to see that, but that’ll happen. So if you can, try to do everything as much as possible using your JSON or your values, and try to just test the things that you own. Unless you’re a package developer, then by all means, we need to start taking pictures and making sure things render properly. But otherwise, we can give that responsibility back to the package developers, not us as app developers. But the image is there to take us a snapshot to let us know what the app look like when we called expect values.
So let’s do, hello-barrett-001.
And this will open up my application. And we can see that Chromote believes the app looked like this where it says,
"What is your name?""Barret."
Greet
"Hello Barret!" "Your first letter name is b"
Awesome. That worked out great.
Debugging
So one of the things that we can do interactively is we can take our app, and we can run it in the console
As long as we’re sitting in our working directory of our Shiny application. This uses the AppDriver object. And this object is something that drives shinytest2 And it runs your Shiny application in the background. And it also does your communication with Chromote. And so it handles all of that. And we can set our inputs. We can also retrieve inputs, and outputs, or different values, and click many different things
app$view()
So if we do this, one of the things that I really enjoy is that we can call app$view, and view will open up a web page Using app$view() to open a visual representation of our headless Chrome browser. And I think this is really neat because now it’s not a black box as to what is happening.
run in console
app$view()
app$set_inputs()
So now I have my app. And this is what Chromote is representing as– what is happening within the app. So if we actually just step through this code when in time, we can actually see the app update to the left. So we can call set_inputs(name="Barret").
run in console
app$set_inputs(name ="Barret")
Click Greet.
run in console
app$click("greet")
app$get_values()
And then if I wanted to, I could call app$expect_values(), or I could even say app$get_values. And this will return all the values.
run in console
app$get_values()
$input
$input$greet
[1] 2
attr(,"class")
[1] "shinyActionButtonValue" "integer"
$input$name
[1] "Barret"
$output
$output$first_letter
[1] "The first letter in your name is b!"
$output$greeting
[1] "Hello Barret!"
$export
named list()
app$get_value()
Or if I wanted to do something like just a particular one, I could say app$get_values(output = "greeting"). And it’s just that one particular value. Or since if it’s just your one value, you can say get value and it’ll turn back the regular result.
run in console
app$get_values(output ="greeting")
[1] "Hello Barret!"
Run tests
So it’s very exciting. You can debug your headless tiny application, and headless Chrome instance, and your Shiny application all at the same time. Save the code that you want. And then we can replay them afterwards. Or just going in the top right and saying, Run tests. You will load up. We’ll get there and eventually pass one for that expect_values, because that’s the only expectation in this whole test file.
==> Testing R file using 'testthat'
Loading required package: shiny
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 1 ]
Test complete
So sometimes just knowing your inputs and your outputs is not enough because you knowthey can hide kind of the middle interactions of your reactives. I know this is a smaller app but it tries to get across the point of we can export these inner reactive values it’s not just your inputs and outputs you can export these inner reactives
I know the phrasing of the first_letter in your name is b or whatever the first part of your name is but I want to actually just have that letter as a value that I can test against so we have our recording and I’m going to adjust this copy and adjust this recording and I’m going to do some alterations for it so we can export those test values.
We’ll say "export values"…the app is not going to be doing anything specific with screenshots…the app is not dependent on the width so I’m going to remove the height and width restriction and just use the defaults…
AppDriver name
test_that(desc ="export test values", code = {# use default height/width app <- AppDriver$new(name ="export values") })
I’m going to set the inputs of name = Barrett" and I’m going to click greet
set_inputs & actionButton
test_that(desc ="export test values", code = {# use default height/width app <- AppDriver$new(name ="export values") app$set_inputs(name ="Barret") app$click("greet")})
One thing we could do here is app$get_value()…and we want to say output = first_letter…
get_value(output = )
test_that(desc ="export test values", code = {# use default height/width app <- AppDriver$new(name ="export values") app$set_inputs(name ="Barret") app$click("greet")# extract value from output value: first_letter_phrase <- app$get_value(output ="first_letter")})
if I’m running the application this should be this phrase so I’m gonna say not just the first_letter but the first letter phrase…and we’ll say expect_equal() to first_letter_phrase and "The first letter in my name is b!"
expect_equal()
test_that(desc ="export test values", code = {# use default height/width app <- AppDriver$new(name ="export values") app$set_inputs(name ="Barret") app$click("greet")# extract value from first_letter_phrase <- app$get_value(output ="first_letter")expect_equal(object = first_letter_phrase, expected ="The first letter in your name is b!")})
For now I’m just going to comment that earlier test we’re going to just run all the tests right here
complete test
library(testthat)library(shinytest2)# test_that("{shinytest2} recording: hello-barret", {# app <- AppDriver$new(name = "hello-barret", height = 483, width = 862)# app$set_inputs(name = "Barret")# app$click("greet")# app$expect_values()# })test_that(desc ="export test values", code = {# use default height/width app <- AppDriver$new(name ="export values") app$set_inputs(name ="Barret") app$click("greet")# extract value from output$first_letter first_letter_phrase <- app$get_value(output ="first_letter")expect_equal(object = first_letter_phrase, expected ="The first letter in your name is b!")})
Run in console
…just because it’s fun to see this interactively…I’m taking a guess that first letter phrase is this first letter phrase…it does equal "The first letter your name is b!"
in console
app <- AppDriver$new(name ="export values")app$set_inputs(name ="Barret")app$click("greet")first_letter_phrase <- app$get_value(output ="first_letter")first_letter_phrase#>> [1] "The first letter in your name is b!"
so this expect_equal() passed and we a little warning of deleting unused screenshots or snapshots…because I commented this test…so that is like those tests don’t exist anymore…so we shall delete them and that’s okay…if I undo this and rerun it…those snapshots will come right back
exportTestValues()
So this is how we can get the whole phrase…but the thing i was interested in was the letter b…this is where…[we use]…exportTestValues()…and that we can export those values…so there’s a method under shiny called export test values
exportTestValues
shiny::exportTestValues()
exportTestValues() is a key to reactive expression set of arguments
I could say first_letter (it doesn’t really matter it’s my own custom key but just gonna for consistency I’m gonna just call it first_letter)…this will equal the reactive equal to the output of the reactive expression
In app.R:
exportTestValues from app
server <-function(input, output, session) { output$greeting <-renderText({req(input$name)paste0("Hello ", input$name, "!") }) %>%bindEvent({input$greet}) first_letter <-reactive({req(input$name)tolower(stringr::str_extract(input$name, "^.")) }) %>%bindEvent({input$greet}) output$first_letter <-renderText({paste0("The first letter in your name is ", first_letter(), "!") })shiny::exportTestValues(first_letter =first_letter() )}
Reset AppDriver
I will kill the old app with app$stop, and then let’s actually get us back into the state that we are at…so let’s make a new AppDriver
Let’s call app$set_inputs() let’s app$click()…let’s look at the values that are currently in the app with app$get_values()
In console:
so app$get_values() will return all the input, output, and exported values
set_input/actionButton
app$set_inputs(name ="Barret")app$click("greet")app$get_values()#>> $input#>> $input$greet#>> [1] 1#>> attr(,"class")#>> [1] "shinyActionButtonValue" "integer" #>>#>> $input$name#>> [1] "Barret"#>>#>>#>> $output#>> $output$first_letter#>> [1] "The first letter in your name is b!"#>>#>> $output$greeting#>> [1] "Hello Barret!"#>>#>>#>> $export#>> $export$first_letter#>> [1] "b"
so we have our inputs: we have input$greet and input$name…we have our outputs of output$first_letter and output$greeting…and then we have export$first_letter, and now we have access to this reactive value ["b"]
Test exported values
It doesn’t need to be an output, we can just use an exported value. So for this one we can also say app$get_value() of export = first_letter
get_value/export
app$get_value(export ="first_letter")
I’m more interested in export$first_letter, so i will say export = first_letter and we’ll do expect_equal() to "b"
expect_equal() get_value/export b
test_that(desc ="export test values", code = {# use default height/width app <- AppDriver$new(name ="export values") app$set_inputs(name ="Barret") app$click("greet")# extract value from output$first_letter first_letter_phrase <- app$get_value(output ="first_letter")expect_equal(object = first_letter_phrase, expected ="The first letter in your name is b!")expect_equal(object = app$get_value(export ="first_letter"),expected ="b" )})
and if I run this it passes
==> Testing R file using 'testthat'Loading required package: shiny[ FAIL 0 |WARN0|SKIP 0 |PASS 2 ]Test complete
just for proof if I change it to a "j"
expect_equal() get_value/export j
test_that(desc ="export test values", code = {# use default height/width app <- AppDriver$new(name ="export values") app$set_inputs(name ="Barret") app$click("greet")# extract value from output$first_letter first_letter_phrase <- app$get_value(output ="first_letter")expect_equal(object = first_letter_phrase, expected ="The first letter in your name is b!")expect_equal(object = app$get_value(export ="first_letter"),expected ="b" )})
and I run it it’ll say
==> Testing R file using 'testthat'Loading required package: shiny[ FAIL 1 |WARN0|SKIP 0 |PASS 1 ]── Failure (test-shinytest2.R:23:3): export test values ────────────────────────app$get_value(export="first_letter")not equal to "j".1/1 mismatchesx[1]:"b"y[1]:"j"[ FAIL 1 |WARN0|SKIP 0 |PASS 1 ]Test complete
error found actual is "b" but you expected "j"
so you do want "b", so I’ll change it back and run it just to confirm…then I will run all the tests because…we should be able to run these all the time quickly…
final exported values unit test
test_that(desc ="export test values", code = {# use default height/width app <- AppDriver$new(name ="export values") app$set_inputs(name ="Barret") app$click("greet")# extract value from output$first_letter first_letter_phrase <- app$get_value(output ="first_letter")expect_equal(object = first_letter_phrase, expected ="The first letter in your name is b!")expect_equal(object = app$get_value(export ="first_letter"),expected ="b" )})
So this example has some serious effects for production sized apps, where you have authentication or things that testing should not be confirming.
Testing production apps
One of the difficulties when doing production size apps and … unit tests is interacting with outside sources. This can be authentication … This can be database requests… And one of the things that we recommend you do is you listen to a flag or a global option of shiny.testmode.
And if shiny.testmode is TRUE, I recommend having a different UI, or server, or reactive behavior given this flag. And you can have things like the authentication isn’t even addressed.
If I do a restricted database call, maybe instead I load a CSV of information. We’re not testing the database–we’re actually just testing the Shiny app. So we need to try to isolate our test to what we can control. And using this flag, we can turn that behavior on or turn it off. And it’s really great…as how you can override your apps’ default behavior.
Authenticated database request
So let’s look at the app that we’ve been doing in the past two videos where we ask our name, and it’ll tell us the first part, or first letter of our name. And then let’s just add in a third field of penguins. And so to lighten the mood a little bit, we’re going to say “how many penguins are going to be coming for dinner?”.
textOutput penguins
ui <-fluidPage(textInput("name", "what is your name"),actionButton("greet", "Greet"),textOutput("greeting"),textOutput("first_letter"),# "how many penguins are going to be coming for dinner?"textOutput("penguins"))
And so let’s just add this into our app. I’m just going to run it for demonstration purposes.
And I click Greet. It said, "Hello Barret."
"The first letter in your name is b!"
And there are [322] penguins coming for dinner.
…if I keep hitting Greet, if we look as to where this penguin_count() is coming from, it’s coming from this authenticated_database_request(), or something that I’m going to have returned random data.
Below is a simulated authenticated_database_request() reactive:
output$penguins <-renderText({paste0("There are ", penguin_count(), " penguins coming for dinner") })
And so if I click Greet again, now it’s [336], [328]…
This is not deterministic at all. And this is, therefore, not useful for testing. Because if you keep running this test over and over, it will fail. There will be no way for it to be correct, because the data changes underneath. So instead, we can listen to our flag.
Test database request
So I’m going to change, just wrap this one part and say if we’ll get an option of shiny.testmode, we’ll default this value to FALSE. But if it’s TRUE, then I want to return a test_database_request().
Below is a simulated test_database_request() reactive:
This is added to the control flow for penguin_count(), along with "shiny.testmode"
So this is something that returns static data, and will only retrieve what I want it to have. Otherwise, if I’m not in test mode, then let’s just do the authenticated_database_request(). This will be stored into penguin_data, and then the penguin_data will count the number of rows, and then hopefully we have a consistent count. Now, if I just run my app and reload app, this is not in testing mode. So we should still see our original behavior.
So I see Barret. We see Greet. It’s something like [331], or different amount of each time when I click Greet. But let’s run this just looking at our application…
In the console
Back in test-shinytest2.R
So if I run this with the app, and I call app$view(), so [the app] opens up into chromium based browser…I have "What's your name."
We call app Greet. We can see that updated. And here it says, "Hello Barret. The first letter your name is b."
run app$click() in console
app$click("greet")
And there are 68 penguins coming for dinner. It’s a funny phrase, but it’s 68. It’s not 300. And so if I click Greet again, and again, and again, and again, 68 is now constant. And it’s still going to stay there.
If I look at the values to see what we have, app$get_values(), I have output$penguins, there are 68 penguins coming for dinner.
run app$get_values() in console
app$get_values()#>> $input#>> $input$greet#>> [1] 2#>> attr(,"class")#>> [1] "shinyActionButtonValue" "integer" #>> #>> $input$name#>> [1] "Barret"#>> #>> #>> $output#>> $output$first_letter#>> [1] "The first letter in your name is b!"#>> #>> $output$greeting#>> [1] "Hello Barret!"#>> #>> $output$penguins#>> [1] "There are 68 penguins coming for dinner"#>> #>> #>> $export#>> $export$first_letter#>> [1] "b"
Export test value
I could actually update that, so that we have another exported test value of the penguin_count(). So if I update with the penguin_count(), say penguin_count = a reactive expression, where I export that penguin_count()…save it…
And then I’m just going to restart those interactive tests, where I run app, and I click Greet. And let’s view it. Now, I don’t need to necessarily open up the viewer at the beginning. It’s usually useful when debugging errors.
And app$get_value() of export = penguin_count is 68.
run app$get_value() in console
app$get_value(export ="penguin_count")#>> [1] 68
Perfect. So now we can make assertions on this value, and it’ll be consistent every single time. This is just so useful for getting around those database calls that take 5, 10 minutes to execute, or you’re asking a different API about something that is outside the control of Shiny. We want these unit tests to be consistent and fast. That way you want to do them, and you want to do them often.
