The goal of dfdiffs is to answer the following questions:
- What rows are here now that weren’t here before?
- What rows were here before that aren’t here now?
- What values have been changed?
The dfdiffs Shiny app-package is built with bslib. Its comparison engine is pure base R, but it owes a debt to the diffdf package: the tolerance-based numeric comparison in compare_values() is a direct port of diffdf’s own formula.
You can access a development version of the application here.
Installation
You can install the development version of dfdiffs from GitHub with:
# install.packages("devtools")
devtools::install_github("mjfrigaard/dfdiffs")Packages
dfdiffs depends on:
library(bslib)
library(data.table)
library(ggplot2)
library(haven)
library(janitor)
library(openxlsx)
library(purrr)
library(reactable)
library(readxl)
library(shiny)
library(tibble)dfdiffs’s own comparison engine (create_new_data(), create_deleted_data(), create_changed_data(), create_modified_data(), and the helpers behind them) is built on base R alone, so it doesn’t require dplyr, tidyr, tidyselect, or diffdf. Those packages (plus arsenal) remain in Suggests only because a few vignettes demonstrate them directly, side-by-side, as points of comparison.
The Shiny app previews and displays data with reactable (an interactive JS widget). GitHub renders this README as plain markdown and strips the JS that a reactable table needs, so the tables below use knitr::kable(), which GitHub can render natively.
Package functions
dfdiffs has a function for each of the questions posed above, and each function comes with a pair of example datasets that demonstrate how it works (which we’ll cover below).
What rows are here now that weren’t here before?
To check for new data, we’ll use T1Data and T2Data.
Timepoint 1 data (original)
T1Data represents data collected at the first timepoint (T1).
T1Data |> knitr::kable()| subject | record | start_date | mid_date | end_date | text_var | factor_var |
|---|---|---|---|---|---|---|
| A | 1 | 2022-01-28 | 2022-03-20 | 2022-03-30 | Patient reports mild headache after morning dose. | headache |
| A | 2 | 2022-01-25 | 2022-03-15 | 2022-03-29 | Patient reports occasional nausea following meals. | nausea |
| B | 3 | 2022-01-26 | 2022-03-19 | 2022-03-25 | Patient reports persistent fatigue throughout the day. | fatigue |
| C | 4 | 2022-01-29 | 2022-03-18 | 2022-03-27 | Patient reports brief dizziness upon standing. | dizziness |
| D | 5 | 2022-01-30 | 2022-03-16 | 2022-03-26 | Patient reports mild rash on the left forearm. | rash |
| D | 6 | 2022-01-27 | 2022-03-17 | 2022-03-31 | Patient reports low-grade fever in the evening. | fever |
Timepoint 2 data (new)
T2Data is the ‘new’ dataset, representing data collected at the second timepoint (T2).
T2Data |> knitr::kable()| subject | record | start_date | mid_date | end_date | text_var | factor_var |
|---|---|---|---|---|---|---|
| D | 5 | 2022-01-30 | 2022-03-16 | 2022-03-26 | Patient reports mild rash on the left forearm. | rash |
| D | 6 | 2022-01-27 | 2022-03-17 | 2022-03-31 | Patient reports low-grade fever in the evening. | fever |
| D | 5 | 2022-04-04 | 2022-04-13 | 2022-04-22 | Patient reports lower back pain after activity. | back pain |
| C | 4 | 2022-01-29 | 2022-03-18 | 2022-03-27 | Patient reports brief dizziness upon standing. | dizziness |
| B | 3 | 2022-01-26 | 2022-03-19 | 2022-03-25 | Patient reports persistent fatigue throughout the day. | fatigue |
| B | 4 | 2022-04-02 | 2022-04-14 | 2022-04-20 | Patient reports difficulty sleeping through the night. | insomnia |
| A | 1 | 2022-01-28 | 2022-03-20 | 2022-03-30 | Patient reports mild headache after morning dose. | headache |
| A | 2 | 2022-01-25 | 2022-03-15 | 2022-03-29 | Patient reports occasional nausea following meals. | nausea |
| A | 2 | 2022-04-04 | 2022-04-15 | 2022-04-21 | Patient reports dry cough lasting several days. | cough |
create_new_data()
The create_new_data() function returns the ‘new data’ (i.e., the rows that are here now but weren’t here before).
create_new_data(
compare = T2Data,
base = T1Data) |>
knitr::kable()| subject | record | start_date | mid_date | end_date | text_var | factor_var | |
|---|---|---|---|---|---|---|---|
| 3 | D | 5 | 2022-04-04 | 2022-04-13 | 2022-04-22 | Patient reports lower back pain after activity. | back pain |
| 6 | B | 4 | 2022-04-02 | 2022-04-14 | 2022-04-20 | Patient reports difficulty sleeping through the night. | insomnia |
| 9 | A | 2 | 2022-04-04 | 2022-04-15 | 2022-04-21 | Patient reports dry cough lasting several days. | cough |
We can check this against the NewData dataset, which should match the output from create_new_data().
NewData |> knitr::kable()| subject | record | start_date | mid_date | end_date | text_var | factor_var |
|---|---|---|---|---|---|---|
| D | 5 | 2022-04-04 | 2022-04-13 | 2022-04-22 | Patient reports lower back pain after activity. | back pain |
| B | 4 | 2022-04-02 | 2022-04-14 | 2022-04-20 | Patient reports difficulty sleeping through the night. | insomnia |
| A | 2 | 2022-04-04 | 2022-04-15 | 2022-04-21 | Patient reports dry cough lasting several days. | cough |
What rows were here before that aren’t here now?
To test for deleted data, we’ll compare CompleteData and IncompleteData, then check the result against DeletedData.
CompleteData <- dfdiffs::CompleteData
IncompleteData <- dfdiffs::IncompleteData
DeletedData <- dfdiffs::DeletedDataA complete dataset
CompleteData represents a ‘complete’ set of data.
CompleteData |> knitr::kable()| subject | record | start_date | mid_date | end_date | text_var | factor_var |
|---|---|---|---|---|---|---|
| A | 1 | 2021-12-28 | 2022-01-27 | 2022-02-26 | Vital signs recorded at screening visit. | vitals |
| A | 2 | 2021-12-28 | 2022-01-27 | 2022-02-26 | Concomitant medication reported at baseline. | conmed |
| B | 1 | 2021-12-26 | 2022-01-25 | 2022-02-24 | Laboratory sample collected for hematology panel. | labs |
| B | 2 | 2021-12-26 | 2022-01-25 | 2022-02-24 | ECG performed during screening assessment. | ecg |
| C | 1 | 2021-12-30 | 2022-01-29 | 2022-02-28 | Physical exam completed with no abnormalities noted. | exam |
| D | 1 | 2021-12-27 | 2022-01-26 | 2022-02-25 | Medical history reviewed and confirmed complete. | history |
| A | 3 | 2021-12-28 | 2022-01-27 | 2022-02-26 | Vital signs recorded at follow-up visit. | vitals |
| B | 3 | 2021-12-26 | 2022-01-25 | 2022-02-24 | Concomitant medication updated at visit two. | conmed |
| D | 2 | 2021-12-27 | 2022-01-26 | 2022-02-25 | Laboratory sample collected for chemistry panel. | labs |
An incomplete dataset
This is a dataset with rows removed from CompleteData.
IncompleteData |> knitr::kable()| subject | record | start_date | mid_date | end_date | text_var | factor_var |
|---|---|---|---|---|---|---|
| A | 1 | 2021-12-28 | 2022-01-27 | 2022-02-26 | Vital signs recorded at screening visit. | vitals |
| B | 1 | 2021-12-26 | 2022-01-25 | 2022-02-24 | Laboratory sample collected for hematology panel. | labs |
| B | 2 | 2021-12-26 | 2022-01-25 | 2022-02-24 | ECG performed during screening assessment. | ecg |
| A | 3 | 2021-12-28 | 2022-01-27 | 2022-02-26 | Vital signs recorded at follow-up visit. | vitals |
| D | 2 | 2021-12-27 | 2022-01-26 | 2022-02-25 | Laboratory sample collected for chemistry panel. | labs |
create_deleted_data()
Running create_deleted_data() checks for rows that were deleted between CompleteData and IncompleteData.
create_deleted_data(
compare = IncompleteData,
base = CompleteData) |>
knitr::kable()| subject | record | start_date | mid_date | end_date | text_var | factor_var | |
|---|---|---|---|---|---|---|---|
| 2 | A | 2 | 2021-12-28 | 2022-01-27 | 2022-02-26 | Concomitant medication reported at baseline. | conmed |
| 5 | C | 1 | 2021-12-30 | 2022-01-29 | 2022-02-28 | Physical exam completed with no abnormalities noted. | exam |
| 6 | D | 1 | 2021-12-27 | 2022-01-26 | 2022-02-25 | Medical history reviewed and confirmed complete. | history |
| 8 | B | 3 | 2021-12-26 | 2022-01-25 | 2022-02-24 | Concomitant medication updated at visit two. | conmed |
The deleted data
The output above is identical to the data stored in DeletedData.
DeletedData |> knitr::kable()| subject | record | start_date | mid_date | end_date | text_var | factor_var |
|---|---|---|---|---|---|---|
| A | 2 | 2021-12-28 | 2022-01-27 | 2022-02-26 | Concomitant medication reported at baseline. | conmed |
| B | 3 | 2021-12-26 | 2022-01-25 | 2022-02-24 | Concomitant medication updated at visit two. | conmed |
| C | 1 | 2021-12-30 | 2022-01-29 | 2022-02-28 | Physical exam completed with no abnormalities noted. | exam |
| D | 1 | 2021-12-27 | 2022-01-26 | 2022-02-25 | Medical history reviewed and confirmed complete. | history |
What values have been changed?
To answer this question, we have two options: create_changed_data() and create_modified_data(). Both build on the same diff_values_by_key()/compare_values() engine and differ only in the shape of their output: create_changed_data() returns $num_diffs/$var_diffs (used by compare_data()), and create_modified_data() returns $diffs_byvar/$diffs (used by mod_compare and create_comparison_report()).
To check for changes between two datasets, we’ll use InitialData and ChangedData.
InitialData <- dfdiffs::InitialData
ChangedData <- dfdiffs::ChangedDataInitial data
InitialData |> knitr::kable()| subject_id | record | text_value_a | text_value_b | created_date | updated_date | entered_date |
|---|---|---|---|---|---|---|
| A | 1 | Issue unresolved | Fatigue | 2021-07-29 | 2021-09-29 | 2021-09-29 |
| A | 2 | Issue unresolved | Fatigue | 2021-07-29 | 2021-10-03 | 2021-10-29 |
| B | 3 | Issue resolved | Fever | 2021-07-16 | 2021-09-02 | 2021-08-18 |
| C | 4 | Issue resolved | Joint pain | 2021-08-24 | 2021-10-03 | 2021-10-03 |
| C | 5 | Issue resolved | Joint pain | 2021-08-24 | 2021-09-20 | 2021-10-20 |
Changed data
ChangedData |> knitr::kable()| subject_id | record | text_value_a | text_value_b | created_date | updated_date | entered_date |
|---|---|---|---|---|---|---|
| A | 1 | Issue resolved | Fatigue | 2021-07-29 | 2021-10-03 | 2021-11-30 |
| A | 2 | Issue resolved | Fatigue | 2021-07-29 | 2021-11-27 | 2021-11-30 |
| B | 3 | Issue resolved | Fever | 2021-07-16 | 2021-10-20 | 2021-11-21 |
| C | 4 | Issue resolved | Joint pain, stiffness and swelling | 2021-08-24 | 2021-10-13 | 2021-11-11 |
| C | 5 | Issue resolved | Joint pain | 2021-08-24 | 2021-10-14 | 2021-11-16 |
create_changed_data()
create_changed_data() creates a list of tables.
changed <- create_changed_data(
compare = ChangedData,
base = InitialData)
names(changed)
#> [1] "num_diffs" "var_diffs"Counts of changes (num_diffs)
The counts of changes by variable are stored in num_diffs.
changed$num_diffs |> knitr::kable()| Variable name | Modified Values |
|---|---|
| subject_id | 0 |
| record | 0 |
| text_value_a | 2 |
| text_value_b | 1 |
| created_date | 0 |
| updated_date | 5 |
| entered_date | 5 |
Changes by row (var_diffs)
The changes by row are stored in var_diffs.
changed$var_diffs |> knitr::kable()| Variable name | rownumber | Current Value | Previous Value |
|---|---|---|---|
| text_value_a | 1 | Issue resolved | Issue unresolved |
| text_value_a | 2 | Issue resolved | Issue unresolved |
| text_value_b | 4 | Joint pain, stiffness and swelling | Joint pain |
| updated_date | 1 | 2021-10-03 | 2021-09-29 |
| updated_date | 2 | 2021-11-27 | 2021-10-03 |
| updated_date | 3 | 2021-10-20 | 2021-09-02 |
| updated_date | 4 | 2021-10-13 | 2021-10-03 |
| updated_date | 5 | 2021-10-14 | 2021-09-20 |
| entered_date | 1 | 2021-11-30 | 2021-09-29 |
| entered_date | 2 | 2021-11-30 | 2021-10-29 |
| entered_date | 3 | 2021-11-21 | 2021-08-18 |
| entered_date | 4 | 2021-11-11 | 2021-10-03 |
| entered_date | 5 | 2021-11-16 | 2021-10-20 |
create_modified_data()
The create_modified_data() function also creates a list of tables.
modified <- create_modified_data(
compare = ChangedData,
base = InitialData)
names(modified)
#> [1] "diffs" "diffs_byvar"Counts of changes (diffs_byvar)
The counts of changes by variable are stored in diffs_byvar.
modified$diffs_byvar |> knitr::kable()| Variable name | Modified Values |
|---|---|
| subject_id | 0 |
| record | 0 |
| text_value_a | 2 |
| text_value_b | 1 |
| created_date | 0 |
| updated_date | 5 |
| entered_date | 5 |
Changes by row
The changes by row are stored in diffs.
modified$diffs |> knitr::kable()| Variable name | rownumber | Current Value | Previous Value |
|---|---|---|---|
| text_value_a | 1 | Issue resolved | Issue unresolved |
| text_value_a | 2 | Issue resolved | Issue unresolved |
| text_value_b | 4 | Joint pain, stiffness and swelling | Joint pain |
| updated_date | 1 | 2021-10-03 | 2021-09-29 |
| updated_date | 2 | 2021-11-27 | 2021-10-03 |
| updated_date | 3 | 2021-10-20 | 2021-09-02 |
| updated_date | 4 | 2021-10-13 | 2021-10-03 |
| updated_date | 5 | 2021-10-14 | 2021-09-20 |
| entered_date | 1 | 2021-11-30 | 2021-09-29 |
| entered_date | 2 | 2021-11-30 | 2021-10-29 |
| entered_date | 3 | 2021-11-21 | 2021-08-18 |
| entered_date | 4 | 2021-11-11 | 2021-10-03 |
| entered_date | 5 | 2021-11-16 | 2021-10-20 |