15 Specifications

Published

2024-09-14

TLDR Test Specifications

Specifications, Requirements and Features

User Specifications: Describe what the end-users expect the application to accomplish
Features: User-focused descriptions of the high-level capabilities or characteristics of an application, and often represent a bundle of smaller functionalities
Functional Requirements: The precise, measurable, and testable ‘functions’ an application should perform
BDD Features and Scenarios: Descriptions of the expected behavior of the application features using a set of special keywords (Feature, Scenario, Given, When, Then, And, But).
- describe(): ‘specifies a larger component or function and contains a set of specifications’. Include feature descriptions and any relevant background information the describe() blocks
- it(): contains the test code or expectations. These are use to test each functional requirement (or Then statements from scenarios).
Traceability matrix: A table that ‘traces’ the user specifications to features and functional requirements (and the tests they give rise to) to verify that the application has been developed correctly.

This chapter focuses on what to test–or specifically, how to figure out what to test. This process usually involves converting a list of user needs into testable requirements. The following chapters in this section have examples of tests, but don’t go into depth on how to write tests. Excellent resources exist for writing unit tests in R packages ¹ and testing shiny apps. ² ³ The goal of this chapter is to illustrate the connections between the user’s needs, the code below R/, and developing tests.

Accessing the code examples

I’ve created the shinypak R package In an effort to make each section accessible and easy to follow:

Install shinypak using pak (or remotes):

# install.packages('pak')
pak::pak('mjfrigaard/shinypak')

Review the chapters in each section:

library(shinypak)
list_apps(regex = '^11')
## # A tibble: 2 × 2
##   branch         last_updated       
##   <chr>          <dttm>             
## 1 11_debug-print 2024-11-20 22:09:54
## 2 11_tests-specs 2024-09-03 22:33:22

Launch an app:

launch(app = "11_tests-specs")

install.packages("testthat")
library(testthat)

(If you’re using devtools, you won’t have to worry about installing testthat and covr)

15.1 Application requirements

Information about the various tasks and activities an application is expected to perform in typically stored in some kind of software requirements specification (SRS) document.⁴ The SRS can include things like distinct design characteristics, budgetary, technological, or timeline restrictions, etc. This document breaks down an application’s intended purpose (i.e., the problem it’s designed to solve) into three general areas: user specifications, features, and functional requirements.

15.1.1 User Specifications

Launch app with the shinypak package:

launch('11_tests-specs')

The user specifications are the goals and objectives stakeholders want to achieve with the application. They use terms like ‘deliver value’ and ‘provide insight’ and provide the basis for deriving the application’s features. ⁵

15.1.2 Features

Features translate the user expectations into an application capabilities. Generally speaking, features capture the tasks and activities user should “be able to” accomplish with the application (i.e., explore data with a graph).

15.1.3 Functional Requirements

Functional requirements are written for developers and provide the technical details on each feature. A single feature often gives rise to multiple functional requirements (these are where users needs come into direct contact with code)⁶

In summary

The areas above help direct the development process, albeit from slightly different perspectives.

The user specifications capture the needs and expectations of the end-user.
The features describe the high-level capabilities of the application.
Functional requirements are the testable, specific actions, inputs, and outputs.

15.2 Application developemnt

The Shiny application development process follows something like the figure below:

The figure above is an oversimplification, but it highlights a common separation (or ‘hand-off’) between users/stakeholders and developers. In the sections below, we’ll look at two common development processes: test-driven and behavior-driven development.

Test-driven development

If sap was built using test-driven development (TDD), the process might look something like this:

Gather user needs and translate into application features:
1. Document the application’s capabilities for exploring movie review variables from IMDB and Rotten Tomatoes.
2. Include feature descriptions for displaying continuous variables (i.e., ‘critics score’ and ‘audience score’) categorical variables (i.e., ‘MPAA’ ), graph visual attributes (size, color, opacity), and an optional plot title.
Write Tests:
1. Write tests to ensure the graph displays relationships between a set of continuous and categorical variables when the app launches.
Run Tests:
1. Before writing any code, these tests will fail.
Develop Features:
1. Write UI, server, module, and utility functions for user inputs and graph outputs.
Rerun Tests:
1. If the graph has been correctly implemented in the application, the tests should pass.
Write more Tests:
1. Add more tests for additional functionalities (e.g., an option to remove missing values from graph).

Starting with tests and writing just enough code to get them to pass often results in developing less (but better) code. The drawback to this approach is a strict focus on the function being tested and not the overall objective of the application.

Behavior-driven development

In behavior-driven development (BDD) (or behavior-driven testing), users and developers work together to understand, define and express application behaviors in non-technical language, ⁷

“Using conversation and examples to specify how you expect a system to behave is a core part of BDD” - BDD in Action, 2ed

Placing an emphasis on writing human-readable expectations for the application’s behaviors makes it easier to develop tests that can focus on verifying each user need exists (and is functioning properly). In BDD, the application’s expected capabilities are captured in Features and illustrated with concrete examples, or Scenarios.

`Feature`

In BDD, a Feature describes an implemented behavior or capability in the application, from a user’s perspective. Typically, these are written in the Gherkin format using specific keywords:⁸

As a ...
I want ...
So that ...

Below is an example Gherkin Feature for the graph in launch_app():

Feature: Visualization
    As a user
    I want to see the changes in the plot
    So that I can visualize the impact of my customizations

As you can see, the feature uses plain language and the wording is user-centric, so it remains accessible to both developers and users (or other non-technical stakeholders).

`Scenario`

A Gherkin Scenario provides a concrete example of how the Feature works and has the following general format:

Given ...
When ...
Then ...

An example Scenario for launch_app() might be:

  Scenario: Viewing the Data Visualization
    Given I have launched the application
    And it contains movie review data from IMDB and Rotten Tomatoes
    And the data contains variables like 'Critics Score' and 'MPAA'
    When I interact with the controls in the sidebar panel
    Then the graph should update with the selected options

`Background`

Instead of repeating any pre-conditions in each Scenario (i.e., the steps contained in the “Given” and first “And” statement), we can establish the context with a Background:

  Background: Launching the application
    Given I have launched the application
    And it loads with movie review data from IMDB and Rotten Tomatoes
    
  Scenario: Viewing the Data Visualization
    Given the data contains variables like 'Critics Score' and 'MPAA'
    When I interact with the controls in the sidebar panel
    Then the graph should update with the selected options

Adopting the Gherkin format (or something similar) provides a common language to express an application’s behavior:

As developers, we can work with users and shareholders to write specifications that describe the expected behavior of each Feature
When developing tests, we can group the tests by their Feature and Scenarios
Each test can execute a step (i.e., the Then statements).

In the next section we’ll cover how to map test code for each Scenario step with testthat.

15.3 BDD and `testthat`

testthat’s BDD functions (describe() and it()) allow us add Gherkin-style features and scenarios to our test files, ensuring the application remains user-centric while meeting the technical specifications.⁹

15.3.1 Describe the feature

We can use the language from our Feature, Background, and Scenario to in the description in the argument of describe():

testthat::describe(
  description = "Feature: Visualization
                   As a user
                   I want to see the changes in the plot
                   So that I can visualize the impact of my customizations",
  code = {
  
})

We can also nest describe() calls, which means we can include the Background (or other relevant information):

describe(
  "Feature: Visualization
      As a user
      I want to see the changes in the graph
      So that I can visualize the impact of my customizations.", 
  code = {
    
  describe(
    "Background: Launching the application
        Given I have launched the application
        And it loads with movie review data from IMDB and Rotten Tomatoes", 
    code = {   
               
      })
    
  })

1: BDD Feature (title and description)
2: Background (preexisting conditions before each scenario)

15.3.2 Confirm it with a test

Inside describe(), we can include multiple it() blocks which “functions as a test and is evaluated in its own environment.”

In the example below, we’ll use an it() block to test the example scenario from above:¹⁰

testthat::describe(
  "Feature: Visualization
      As a user
      I want to see the changes in the graph
      So that I can visualize the impact of my customizations.", 
  code = {
  
  testthat::describe(
    "Background: Launching the application
        Given I have launched the application
        And it loads with movie review data from IMDB and Rotten Tomatoes",
      code = {
      
    testthat::it(
      "Scenario: Viewing the Data Visualization
         Given the data contains variables like 'Critics Score' and 'MPAA'
         When I interact with the controls in the sidebar panel
         Then the graph should update with the selected options",
        code = {
          # test code
        })
      
    })
  
})

1: BDD Feature (title and description)
2: Background (preexisting conditions before each scenario)
3: Scenario (a concrete examples that illustrates a feature)
4: Test code

In the scenario above, Then contains the information required for the testthat expectation. This could be expect_snapshot_file() or vdiffr::expect_doppelganger()–whichever makes sense from the user’s perspective.

These are generic examples, but hopefully the tests in the upcoming chapters convey how helpful and expressive BDD functions can be (or they inspire you to properly implement what I’m attempting to do in your own app-packages).¹¹

15.4 Traceability Matrix

After translating the user needs into functional requirements, we can identify what needs to be tested by building a look-up table (i.e., a matrix).

I like to store early drafts of the requirements and traceability matrix in a vignette:¹²

usethis::use_vignette("specs")

Adding our first vignette to the vignettes/ folder does the following:

Adds the knitr and rmarkdown packages to the Suggests field in DESCRIPTION¹³

Suggests: 
    knitr,
    rmarkdown

Adds knitr to the VignetteBuilder field¹⁴

VignetteBuilder: knitr

Adds inst/doc to .gitignore and *.html, *.R to vignettes/.gitignore

The first column in the traceability matrix contains the user specifications, which we can ‘trace’ over to the functional requirements and their relevant tests.¹⁵

Traceability Matrix
Specifications	Features	Requirements	Test
S1: The application should source movie review data from platforms like IMDB or Rotten Tomatoes	F1.1: Feature: Movie Review Dataset Variables *As a* user *I want to* have a dataset with variables from IMDB and Rotten Tomatoes *In order to* provide comprehensive movie reviews Background: *Given* the application has access to IMDB and Rotten Tomatoes APIs	Scenario: Movie Review Continuous and Categorical Variables *When* the application loads from IMDB and Rotten Tomatoes movie review data *Then* the dataset should include a continuous critic ratings variable *And* the dataset should include a continuous audience ratings variable *And* the dataset should include a categorical mpaa ratings variable *And* the dataset should include a categorical genres variable

Building a traceability matrix ensures:

All user specifications have accompanying application features.
Each feature has been broken down into precise, measurable, and testable functional requirements.
Tests have been written for each functional requirement.

Recap

Launch app with the shinypak package:

launch('11_tests-specs')

Understanding the relationship between user specifications, features, and functional requirements gives us the information we need to build applications that satisfy the technical standards while addressing user needs. Documenting requirements in Gherkin-style features and scenarios allows us to capture the application’s behavior without giving details on how the functionality is implemented.

In the next chapter, we’re going to cover various tools to improve the tests in your app-package. The overarching goal of these tools is to reduce code executed outside of your tests (i.e., placed above the call to test_that() or it()).

Recap: Test Specifications

Specifications

Scoping tests: user specifications outline software goals and needs, and the functional requirements provide the technical details to achieve them.
- User specifications: descriptions of what a user expects the application to do (i.e., the user ‘wish list’ of features they want in the application).
- Features: detailed list of the main capabilities and functions the application needs to offer to users.
- Functional requierments: testable, specific step-by-step instructions for ensuring the application does what it’s supposed to do.
- Traceability matrix: tracking tool for connecting the users ‘wish list’ (i.e, specifications) to what’s being tested.

See a typo, error, or something missing?

Please open an issue on GitHub

Unit tests are covered extensively in R Packages, 2ed and the testthat documentation ↩︎
Mastering shiny dedicates an entire Chapter to Testing.) shinytest2 also has excellent documentation (and videos), and I highly recommend reading through those resources.↩︎
I will cover a few tips and tricks I’ve learned for testing module server functions with testServer() because they’re not in the documentation.↩︎
Read more about what goes in the Software Requirements Specification ↩︎
User Specifications are sometimes referred to as “user stories,” “use cases,” or “general requirements”↩︎
‘Features’ and ‘functional requirements’ are sometimes used interchangeably, but they refer to different aspects of the application. Features are high-level capabilities an application should have, and often contain a collection of smaller functionalities (broken down into the specific functional requirements).↩︎
Read more about behavior-driven development ↩︎
Gherkin is the domain-specific language format used for expressing software behaviors. Tools like Cucumber or SpecFlow maps and executes the Gherkin descriptions against the code to generate a pass/fail report status for each requirement.↩︎
Read more about describe() and it() in the testthat documentation. and in the appendix.↩︎
Each it() block contains the expectations (or what you would traditionally include in test_that()).↩︎
For an excellent description on the relationships between behavior-driven development, test-driven development, and domain-driven design, I highly recommend BDD in Action, 2ed by John Ferguson Smart and Jan Molack.↩︎
Storing the traceability matrix in a vignette is great for developers, but using an issue-tracking system with version control is also a good idea, like GitHub Projects or Azure DevOps.↩︎
We briefly covered the Suggests field in Dependencies, but in this case it specifically applies to “packages that are not necessarily needed. This includes packages used only in examples, tests or vignettes…” - Writing R Extensions, Package Dependencies ↩︎
The documentation on VignetteBuilder gives a great example of why knitr and rmarkdown belong in Suggests and not Imports.↩︎
When building tables in vignettes, I highly recommend using the Visual Markdown mode.↩︎