Skip to content

Validation and QC

In a regulated study, a number is not finished when one program computes it. It is finished when a second, independent program computes the same number, and a compare tool finds no difference. In this chapter you write the second program, compare the two, and compare your ADaM data sets with the reference. Then you check the data rules and every warning, and you write down what you found for a reviewer.

Part II built the ADaM data sets in programs/data/. Part IV proves them. Run the programs of chapters 9 to 12 first, in file order.

Double programming means two programmers build the same value from the same specification, each alone. The first program is the production program. The second is the QC program (QC means quality control). When two programs agree, both are much more likely to be right.

The production program here is the ADSL of chapter 9. The specification for four of its variables:

Variable Rule
TRTSDT The date of the first dose: the earliest EX start date
TRTEDT The date of the last dose: the latest EX end date. A subject with an open dose row gets RFENDTC
AGEGR1 <65, 65-80 or >80
SAFFL "Y" for a randomised subject who took the study drug

The QC program reads the same specification, but it does not read the production code. It also uses other code on purpose. If both programs use the same function in the same way, they can share the same mistake.

programs/15_1_qc_adsl.R
# Chapter 15: the QC program. Same specification as chapter 9, other code
library(dplyr)
dm <- pharmaversesdtm::dm |>
filter(ARMCD != "Scrnfail")
ex <- pharmaversesdtm::ex
# TRTSDT: sort the EX rows, keep the first row of each subject
first_dose <- ex |>
arrange(USUBJID, EXSTDTC, EXSEQ) |>
slice_head(n = 1, by = USUBJID) |>
transmute(USUBJID, TRTSDT = as.Date(EXSTDTC))
# TRTEDT: a subject with an open dose row ends at RFENDTC, else at the
# latest EX end date. Text dates in ISO 8601 sort in date order
last_dose <- ex |>
summarise(
open = any(is.na(EXENDTC)),
last_end = max(EXENDTC, na.rm = TRUE),
.by = USUBJID
)
#> Warning:
#> There were 2 warnings in `summarise()`.
#> The first warning was:
#> i In argument: `last_end = max(EXENDTC, na.rm = TRUE)`.
#> i In group 86: `USUBJID = "01-705-1018"`.
#> Caused by warning in `max()`:
#> ! no non-missing arguments, returning NA
#> i Run `dplyr::last_dplyr_warnings()` to see the 1 remaining warning.
qc <- dm |>
select(USUBJID, AGE, RFENDTC) |>
left_join(first_dose, by = join_by(USUBJID)) |>
left_join(last_dose, by = join_by(USUBJID)) |>
mutate(
TRTEDT = as.Date(if_else(open, RFENDTC, last_end)),
# whole years: (-Inf, 64] is "<65", (64, 80] is "65-80"
AGEGR1 = as.character(cut(
AGE,
breaks = c(-Inf, 64, 80, Inf), labels = c("<65", "65-80", ">80")
)),
SAFFL = if_else(USUBJID %in% ex$USUBJID, "Y", "N")
) |>
select(USUBJID, TRTSDT, TRTEDT, AGEGR1, SAFFL)
qc |>
filter(USUBJID == "01-701-1015")
#> # A tibble: 1 x 5
#> USUBJID TRTSDT TRTEDT AGEGR1 SAFFL
#> <chr> <date> <date> <chr> <chr>
#> 1 01-701-1015 2014-01-02 2014-07-02 <65 Y
count(qc, AGEGR1, SAFFL)
#> # A tibble: 3 x 3
#> AGEGR1 SAFFL n
#> <chr> <chr> <int>
#> 1 65-80 Y 144
#> 2 <65 Y 33
#> 3 >80 Y 77
saveRDS(qc, "data/15_qc_adsl.rds")

The two programs differ at each step:

Variable Production (chapter 9) QC (this program)
TRTSDT min() of the dates arrange(), then the first row with slice_head()
TRTEDT max() gives NA for an open row, then coalesce() any(is.na()) decides, if_else() chooses
AGEGR1 case_when() cut() with break points
SAFFL from SUPPDM from EX: the subject has a dose row

cut() puts each number into an interval and gives the interval a label. By default an interval is closed on the right: (64, 80] holds 65 to 80. Section 15.3 shows why this detail matters.

For 01-701-1015 the QC program gives TRTSDT 2014-01-02, TRTEDT 2014-07-02 and AGEGR1 = "<65", as chapter 9 did. All 254 subjects have SAFFL = "Y".

diffdf() of the package diffdf compares two data frames. The first one is the base (the data you trust, here production). The second one is the compare data (the data you check, here QC). keys names the columns that identify a row: diffdf() pairs the rows by key, not by position.

programs/15_2_compare.R
# Chapter 15: compare production (chapter 9) with the QC program
library(dplyr)
library(diffdf)
vars <- c("USUBJID", "TRTSDT", "TRTEDT", "AGEGR1", "SAFFL")
prod <- readRDS("data/adsl.rds") |>
select(all_of(vars)) |>
haven::zap_label()
qc <- readRDS("data/15_qc_adsl.rds")
diffdf(prod, qc, keys = "USUBJID")
#> Warning message:
#> In diffdf(prod, qc, keys = "USUBJID") :
#> There are columns in BASE and COMPARE with differing attributes !!
#> Differences found between the objects!
#>
#> Summary of BASE and COMPARE
#> ==================================================================
#> PROPERTY BASE COMP
#> ------------------------------------------------------------------
#> Name prod qc
#> Class "tbl_df, tbl, data.frame" "tbl_df, tbl, data.frame"
#> Rows(#) 254 254
#> Columns(#) 5 5
#> ------------------------------------------------------------------
#>
#>
#> There are columns in BASE and COMPARE with differing attributes !!
#> ===============================================================
#> VARIABLE ATTR_NAME VALUES.BASE VALUES.COMP
#> ---------------------------------------------------------------
#> USUBJID label NULL "Unique Subject Identifier"
#> ---------------------------------------------------------------
#>
# The values agree; only a label differs. Compare the values only
diffdf(prod, haven::zap_label(qc), keys = "USUBJID")
#> No issues were found!

The first compare finds one difference, and it is not a value. The column USUBJID of the QC data has a label, Unique Subject Identifier, which it took from DM. The program removed the labels of production with haven::zap_label(). diffdf() compares the attributes of each column too, and a label is an attribute.

The second compare removes the labels of the QC data as well: “No issues were found!”. The four variables agree for all 254 subjects.

A real QC program is not always right the first time. Here is a QC program with one mistake: the break points of cut() are 65 and 80, not 64 and 80.

programs/15_3_boundary.R
# Chapter 15: a QC program with one mistake. cut() closes each group on the right
library(dplyr)
library(diffdf)
prod <- readRDS("data/adsl.rds") |>
select(USUBJID, AGE, AGEGR1) |>
haven::zap_label()
qc <- prod |>
mutate(AGEGR1 = as.character(cut(
AGE,
breaks = c(-Inf, 65, 80, Inf), labels = c("<65", "65-80", ">80")
)))
diffdf(prod, qc, keys = "USUBJID")
#> Warning message:
#> In diffdf(prod, qc, keys = "USUBJID") :
#> Not all Values Compared Equal
#> Differences found between the objects!
#>
#> Summary of BASE and COMPARE
#> ==================================================================
#> PROPERTY BASE COMP
#> ------------------------------------------------------------------
#> Name prod qc
#> Class "tbl_df, tbl, data.frame" "tbl_df, tbl, data.frame"
#> Rows(#) 254 254
#> Columns(#) 3 3
#> ------------------------------------------------------------------
#>
#>
#> Not all Values Compared Equal
#> =============================
#> Variable No of Differences
#> -----------------------------
#> AGEGR1 4
#> -----------------------------
#>
#>
#> =======================================
#> VARIABLE USUBJID BASE COMPARE
#> ---------------------------------------
#> AGEGR1 01-707-1206 65-80 <65
#> AGEGR1 01-709-1309 65-80 <65
#> AGEGR1 01-715-1107 65-80 <65
#> AGEGR1 01-715-1319 65-80 <65
#> ---------------------------------------
#>

diffdf() writes a warning, Not all Values Compared Equal, and lists 4 subjects: 01-707-1206, 01-709-1309, 01-715-1107 and 01-715-1319. All four are exactly 65 years old. The interval (-Inf, 65] includes 65, so the QC program puts them in <65.

Four of 254 subjects is a small number. In a frequency table of AGEGR1, nobody sees it. A full compare finds it at once, and names the subjects. That is why QC compares every value, not only the totals.

A diffdf report has one section for each kind of problem. This program makes three subjects with four planned changes, so that every section appears:

programs/15_4_diffdf_parts.R
# Chapter 15: the parts of a diffdf() report, on three subjects with planned changes
library(dplyr)
library(diffdf)
base <- readRDS("data/adsl.rds") |>
filter(USUBJID %in% c("01-701-1015", "01-701-1023", "01-701-1028")) |>
select(USUBJID, AGE, SEX, TRTSDT) |>
haven::zap_label()
compare <- base |>
filter(USUBJID != "01-701-1028") |> # a subject only in BASE
mutate(
SEX = if_else(USUBJID == "01-701-1023", "F", SEX), # a value differs
AGE = as.integer(AGE), # another type
COUNTRY = "USA" # a column only in COMPARE
)
res <- diffdf(base, compare, keys = "USUBJID")
#> Warning message:
#> In diffdf(base, compare, keys = "USUBJID") :
#> There are columns in BASE and COMPARE with different classes !!
#> There are rows in BASE that are not in COMPARE !!
#> There are columns in COMPARE that are not in BASE !!
#> Not all Values Compared Equal
res
#> Differences found between the objects!
#>
#> Summary of BASE and COMPARE
#> ==================================================================
#> PROPERTY BASE COMP
#> ------------------------------------------------------------------
#> Name base compare
#> Class "tbl_df, tbl, data.frame" "tbl_df, tbl, data.frame"
#> Rows(#) 3 2
#> Columns(#) 4 5
#> ------------------------------------------------------------------
#>
#>
#> There are columns in BASE and COMPARE with different classes !!
#> ==================================
#> VARIABLE CLASS.BASE CLASS.COMP
#> ----------------------------------
#> AGE numeric integer
#> ----------------------------------
#>
#>
#> There are rows in BASE that are not in COMPARE !!
#> =============
#> USUBJID
#> -------------
#> 01-701-1028
#> -------------
#>
#>
#> There are columns in COMPARE that are not in BASE !!
#> =========
#> COLUMNS
#> ---------
#> COUNTRY
#> ---------
#>
#>
#> Not all Values Compared Equal
#> =============================
#> Variable No of Differences
#> -----------------------------
#> SEX 1
#> -----------------------------
#>
#>
#> ======================================
#> VARIABLE USUBJID BASE COMPARE
#> --------------------------------------
#> SEX 01-701-1023 M F
#> --------------------------------------
#>
# The result is a list. A program can test it and use its parts
diffdf_has_issues(res)
#> [1] TRUE
names(res)
#> [1] "DataSummary" "VarClassDiffs" "ExtRowsBase" "ExtColsComp"
#> [5] "NumDiff" "VarDiff_SEX"
diffdf_issuerows(base, res)
#> USUBJID AGE SEX TRTSDT
#> 1 01-701-1023 64 M 2012-08-05
Section What it means
Summary of BASE and COMPARE the names, classes, rows and columns of the two data frames
columns with different classes the same column has another type: AGE is a decimal number in one and a whole number in the other
rows in BASE that are not in COMPARE a key in one data frame only: a lost subject
columns in COMPARE that are not in BASE a column in one data frame only
Not all Values Compared Equal the number of differences per column, then each difference with its key

The result of diffdf() is a list, so a program can use it:

  • diffdf_has_issues(res) is TRUE or FALSE. A batch QC run can test it.
  • names(res) shows the sections. res$NumDiff is the table of differences per column, and res$VarDiff_SEX lists the differences of SEX.
  • diffdf_issuerows(base, res) gives the rows of base with a value difference. Here it is 01-701-1023. It does not list the subject that is missing from compare.

Two right formulas can give numbers that differ in the last digits. A computer stores decimal numbers in binary, so CHG / BASE * 100 and (AVAL / BASE - 1) * 100 are not always exactly equal.

programs/15_5_tolerance.R
# Chapter 15: two right formulas for PCHG differ in the last digits. tolerance
library(dplyr)
library(diffdf)
advs <- readRDS("data/advs.rds") |>
filter(PARAMCD == "WEIGHT", !is.na(CHG))
prod <- advs |>
transmute(USUBJID, VSSEQ, PCHG = CHG / BASE * 100)
qc <- advs |>
transmute(USUBJID, VSSEQ, PCHG = (AVAL / BASE - 1) * 100)
# How many values are not exactly equal, and by how much at most?
nrow(advs)
#> [1] 1542
sum(prod$PCHG != qc$PCHG)
#> [1] 1236
max(abs(prod$PCHG - qc$PCHG))
#> [1] 1.421085e-14
# tolerance = 0: every tiny difference counts
res <- diffdf(prod, qc,
keys = c("USUBJID", "VSSEQ"), tolerance = 0,
suppress_warnings = TRUE
)
res$NumDiff
#> # A tibble: 1 x 2
#> Variable `No of Differences`
#> * <chr> <int>
#> 1 PCHG 1236
# The default tolerance, sqrt(.Machine$double.eps), is about 1.5e-08
diffdf(prod, qc, keys = c("USUBJID", "VSSEQ"))
#> No issues were found!

1236 of the 1542 weight changes are not exactly equal. The largest difference is 1.421085e-14, a number with 13 zeros after the decimal point. No table shows it.

tolerance is the largest difference that diffdf() still counts as equal. With tolerance = 0, diffdf() reports all 1236. The default is sqrt(.Machine$double.eps), about 1.5e-08. With it, “No issues were found!”. Keep the default. Change it only when the specification says how close two values must be.

suppress_warnings = TRUE stops the warning of diffdf(). Use it when a program reads the result itself, as program 15_7 does.

Double programming checks the code against the specification. It cannot check the specification. Two programs that follow the same wrong rule agree. A second look comes from outside: the reference ADaM data sets of pharmaverseadam, built by other people for the same study.

programs/15_6_reference.R
# Chapter 15: our ADSL against the reference ADSL of pharmaverseadam
library(dplyr)
library(diffdf)
vars <- c("USUBJID", "TRTSDT", "TRTEDT", "AGEGR1", "SAFFL")
ours <- readRDS("data/adsl.rds") |>
select(all_of(vars)) |>
haven::zap_label()
ref <- pharmaverseadam::adsl |>
filter(ARMCD != "Scrnfail") |>
select(all_of(vars)) |>
haven::zap_label()
res <- diffdf(ours, ref, keys = "USUBJID")
#> Warning message:
#> In diffdf(ours, ref, keys = "USUBJID") :
#> Not all Values Compared Equal
res$NumDiff
#> # A tibble: 2 x 2
#> Variable `No of Differences`
#> * <chr> <int>
#> 1 TRTEDT 6
#> 2 AGEGR1 254
# 254 of 254 differ: look for a rule, not for 254 bugs
inner_join(ours, ref, by = join_by(USUBJID), suffix = c("", "_REF")) |>
count(AGEGR1, AGEGR1_REF)
#> # A tibble: 3 x 3
#> AGEGR1 AGEGR1_REF n
#> <chr> <chr> <int>
#> 1 65-80 >64 144
#> 2 <65 18-64 33
#> 3 >80 >64 77

Two variables differ:

  • TRTEDT differs for 6 subjects. These are the subjects with an open dose row of section 9.3.
  • AGEGR1 differs for all 254 subjects. When every row differs, look for a rule, not for 254 bugs. The count shows the rule: the reference uses two groups, 18-64 and >64. Every <65 of yours is 18-64 there (33 subjects). Every 65-80 (144) and >80 (77) is >64. The two specifications define other age groups.

The same check for the whole study is one function, called once for each data set:

programs/15_7_study_qc.R
# Chapter 15: one QC run over three data sets: the number of differences per variable
library(dplyr)
library(purrr)
library(diffdf)
# The differences of one comparison as a tibble: one row per variable
qc_diff <- function(ours, ref, keys, vars) {
ours <- haven::zap_label(select(ours, all_of(c(keys, vars))))
ref <- haven::zap_label(select(ref, all_of(c(keys, vars))))
res <- diffdf(ours, ref, keys = keys, suppress_warnings = TRUE)
tibble(
variable = vars,
n_ours = nrow(ours),
n_diff = map_int(vars, \(v) {
nrow(res[[paste0("VarDiff_", v)]] %||% tibble())
})
)
}
ref_adlb <- pharmaverseadam::adlb |>
filter(LBCAT == "CHEMISTRY", is.na(DTYPE))
bind_rows(
ADSL = qc_diff(
readRDS("data/adsl.rds"),
filter(pharmaverseadam::adsl, ARMCD != "Scrnfail"),
keys = "USUBJID", vars = c("TRTSDT", "TRTEDT", "AGEGR1", "SAFFL")
),
ADAE = qc_diff(
readRDS("data/adae.rds"), pharmaverseadam::adae,
keys = c("USUBJID", "AESEQ"), vars = c("ASTDT", "AENDT", "TRTEMFL")
),
ADLB = qc_diff(
readRDS("data/adlb.rds"), ref_adlb,
keys = c("USUBJID", "LBSEQ"), vars = c("AVAL", "BASE", "CHG")
),
.id = "dataset"
)
#> # A tibble: 10 x 4
#> dataset variable n_ours n_diff
#> <chr> <chr> <int> <int>
#> 1 ADSL TRTSDT 254 0
#> 2 ADSL TRTEDT 254 6
#> 3 ADSL AGEGR1 254 254
#> 4 ADSL SAFFL 254 0
#> 5 ADAE ASTDT 1191 0
#> 6 ADAE AENDT 1191 0
#> 7 ADAE TRTEMFL 1191 4
#> 8 ADLB AVAL 32740 0
#> 9 ADLB BASE 32740 0
#> 10 ADLB CHG 32740 777
# Where are the 777 CHG differences? At visits that are not a scheduled week
inner_join(
readRDS("data/adlb.rds") |> select(USUBJID, LBSEQ, VISIT, CHG),
ref_adlb |> select(USUBJID, LBSEQ, CHG_REF = CHG),
by = join_by(USUBJID, LBSEQ)
) |>
filter(is.na(CHG) != is.na(CHG_REF)) |>
count(
visit = stringr::word(VISIT, 1), ours = !is.na(CHG), ref = !is.na(CHG_REF)
)
#> # A tibble: 3 x 4
#> visit ours ref n
#> <chr> <lgl> <lgl> <int>
#> 1 AMBUL FALSE TRUE 18
#> 2 RETRIEVAL FALSE TRUE 18
#> 3 UNSCHEDULED FALSE TRUE 741
  • qc_diff() keeps the key and the variables, removes the labels, and runs diffdf() without a warning. It then counts the rows of each VarDiff_ section. %||% gives the right side when the left side is NULL: a variable without differences has no section.
  • bind_rows() with names and .id = "dataset" stacks the three results and adds the name of each data set as a column.

The table is the QC summary of the study:

  • ADSL: 6 TRTEDT and 254 AGEGR1, as above.
  • ADAE: TRTEMFL differs on 4 events. They are the 4 events of 01-705-1303 of section 11.4. The reference ends the treatment of this subject at its own TRTEDT, 2013-12-30, and counts an event only up to 30 days after it.
  • ADLB: CHG differs on 777 rows. They are all at visits that are not a scheduled week: 741 unscheduled, 18 retrieval and 18 ambulatory ECG rows. The reference has a CHG there, yours has none (section 10.6). Chapter 10 counted more rows, because its join also matched the derived rows of the reference.

Not one of these differences is a bug in your code. Each one is a rule that differs between two specifications.

You do not decide which rule is right. You write the finding down and ask the statistician. A QC finding is a short note: what differs, how many rows, the cause, and the question. The reviewer must understand it without running a program.

QC finding 15-01: ADSL.TRTEDT, 6 of 254 subjects differ from the reference
Compared: data/adsl.rds against pharmaverseadam::adsl, key USUBJID
Found: 6 differences, all subjects with an EX row without an end date
(01-704-1233, 01-705-1018, 01-705-1031, 01-705-1303,
01-705-1377, 01-705-1382). For 01-705-1018 and 01-705-1382
the reference has no TRTEDT.
Cause: our rule uses RFENDTC for an open dose row (section 9.3).
The reference uses the latest end date that exists.
Impact: ADAE.TRTEMFL differs on 4 events of 01-705-1303.
Question: Which rule does the SAP want for an open dose row?
Status: open. Programs 15_6 and 15_7 show the numbers.

Write one note per rule, not per subject. The AGEGR1 finding is a second note: the specification of this book has three age groups, the reference has two. Keep the notes with the QC programs, and add the answer of the statistician when it comes. The next programmer then knows why the numbers differ.

Some mistakes need no second program. A data rule is a fact that must be true for every row: a key is unique, ADSL has one row per subject, a required value is not missing. A rule check is a short program that counts the rows that break each rule.

programs/15_8_rules.R
# Chapter 15: a QC program for data rules. Each rule is one row: PASS or FAIL
library(dplyr)
adsl <- readRDS("data/adsl.rds")
adae <- readRDS("data/adae.rds")
advs <- readRDS("data/advs.rds")
# A rule: its name, its data set, and the number of rows that break it
rule <- function(name, dataset, n_bad) {
result <- if_else(n_bad == 0, "PASS", "FAIL")
tibble(rule = name, dataset, n_bad, result)
}
n_dup <- function(data, ...) sum(duplicated(select(data, ...)))
agegr1 <- c("<65", "65-80", ">80")
lost <- anti_join(adae, adsl, by = join_by(USUBJID))
checks <- bind_rows(
rule("one row per subject", "ADSL", n_dup(adsl, USUBJID)),
rule("TRTSDT is not missing", "ADSL", sum(is.na(adsl$TRTSDT))),
rule("TRTEDT >= TRTSDT", "ADSL", sum(adsl$TRTEDT < adsl$TRTSDT)),
rule("AGEGR1 in its codelist", "ADSL", sum(!adsl$AGEGR1 %in% agegr1)),
rule("USUBJID, AESEQ is a key", "ADAE", n_dup(adae, USUBJID, AESEQ)),
rule("every subject is in ADSL", "ADAE", nrow(lost)),
rule("TRTEMFL is Y or empty", "ADAE", sum(!adae$TRTEMFL %in% c("Y", NA))),
rule("USUBJID, VSSEQ is a key", "ADVS", n_dup(advs, USUBJID, VSSEQ)),
rule("AVAL is not missing", "ADVS", sum(is.na(advs$AVAL)))
)
checks
#> # A tibble: 9 x 4
#> rule dataset n_bad result
#> <chr> <chr> <int> <chr>
#> 1 one row per subject ADSL 0 PASS
#> 2 TRTSDT is not missing ADSL 0 PASS
#> 3 TRTEDT >= TRTSDT ADSL 0 PASS
#> 4 AGEGR1 in its codelist ADSL 0 PASS
#> 5 USUBJID, AESEQ is a key ADAE 0 PASS
#> 6 every subject is in ADSL ADAE 0 PASS
#> 7 TRTEMFL is Y or empty ADAE 0 PASS
#> 8 USUBJID, VSSEQ is a key ADVS 0 PASS
#> 9 AVAL is not missing ADVS 8 FAIL
# A FAIL is a question for the data, not yet a bug: look at the rows in VS
advs |>
filter(is.na(AVAL)) |>
left_join(
pharmaversesdtm::vs |> select(USUBJID, VSSEQ, VSSTAT),
by = join_by(USUBJID, VSSEQ)
) |>
count(PARAMCD, VSSTAT)
#> # A tibble: 3 x 3
#> PARAMCD VSSTAT n
#> <chr> <chr> <int>
#> 1 DIABP NOT DONE 2
#> 2 PULSE NOT DONE 3
#> 3 SYSBP NOT DONE 3
  • rule() makes one row of the result: the rule, the data set, the number of bad rows, and PASS or FAIL.
  • duplicated() is TRUE for each row that has the same values as an earlier row. On the key columns, it finds a key that is not unique.
  • anti_join() keeps the ADAE rows without a subject in ADSL (chapter 5).

Eight rules pass. One fails: 8 ADVS rows have no AVAL. A FAIL is a question for the data, not yet a bug. The rows in VS have VSSTAT = "NOT DONE": the test was not done at that visit. So the rule is too strict. Exercise 15-5 writes it the way the data needs it.

Run the rule check after every change to Part II. It takes a second, and it finds a lost key or a copied row before a table shows a wrong N.

A program can give the right numbers and still have problems: a value that became NA, a join that copied rows. R says so in a warning or a message in the console. A reviewer reads them all, so you read them first.

This table program has three problems:

programs/15_logcheck/ae_counts.R
# A small table program with three problems that a reviewer must see
library(dplyr)
adsl <- pharmaverseadam::adsl |> filter(SAFFL == "Y")
adae <- pharmaverseadam::adae |> filter(TRTEMFL == "Y")
# Problem 1: a lab-style value "<5" cannot become a number
dose <- as.numeric(c("54", "81", "<5"))
# Problem 2: summarise() by two groups keeps one of them
by_sev <- adae |>
group_by(TRT01A, AESEV) |>
summarise(n_subj = n_distinct(USUBJID))
# Problem 3: the join copies rows: many AEs per subject, many rows per subject
both <- adae |>
select(USUBJID, AEDECOD) |>
left_join(adae |> select(USUBJID, AESEV), by = join_by(USUBJID))

The log check runs the program and collects everything that R says:

programs/15_9_log_check.R
# Chapter 15: a log check. Run a program and collect every warning and message
library(dplyr)
check_log <- function(file) {
found <- tibble(kind = character(), text = character())
keep <- function(kind, cnd) {
text <- gsub("\\s+", " ", conditionMessage(cnd))
found <<- add_row(found, kind = kind, text = substr(text, 1, 60))
}
withCallingHandlers(
source(file, local = new.env()),
warning = function(w) {
keep("WARNING", w)
invokeRestart("muffleWarning")
},
message = function(m) {
keep("MESSAGE", m)
invokeRestart("muffleMessage")
}
)
found
}
log <- check_log("15_logcheck/ae_counts.R")
log
#> # A tibble: 3 x 2
#> kind text
#> <chr> <chr>
#> 1 WARNING NAs introduced by coercion
#> 2 MESSAGE `summarise()` has regrouped the output. i Summaries were com
#> 3 WARNING Detected an unexpected many-to-many relationship between `x`
count(log, kind)
#> # A tibble: 2 x 2
#> kind n
#> <chr> <int>
#> 1 MESSAGE 1
#> 2 WARNING 2
# In a test, the same check is one line: this step must give no warning
testthat::expect_no_warning(
pharmaverseadam::adae |> count(TRT01A, AESEV)
)
testthat::expect_no_warning(as.numeric("<5"))
#> Error : Expected `as.numeric("<5")` not to throw any warnings.
#> Actually got a <simpleWarning> with message:
#> NAs introduced by coercion
  • withCallingHandlers() runs code and calls a function of yours for each condition: a warning, a message or an error. The program goes on after each one.
  • keep() adds a row to found. <<- assigns to found in the function around it, not to a new local copy.
  • invokeRestart("muffleWarning") stops the warning from reaching the console. It is now a row in the table instead.
  • source(file, local = new.env()) runs the program in a new, empty environment, so its objects do not mix with yours.

The check finds the three problems: 2 warnings and 1 message. NAs introduced by coercion is the text "<5" that became NA. The message says that summarise() kept a group. The many-to-many warning is the join that copied rows.

In a test, one line does the same for one step. testthat::expect_no_warning() is silent when the code gives no warning. When it does, it stops with an error that quotes the warning. Chapter 19 uses such tests for a function library.

  • Double programming: a second program, from the same specification, with other code. Then a compare must find no differences.
  • diffdf(base, compare, keys = ...) compares two data frames by key. It reports rows and columns in one data frame only, other types, other attributes and other values.
  • The result of diffdf() is a list. diffdf_has_issues(), $NumDiff and diffdf_issuerows() let a program use it.
  • tolerance ignores rounding noise. Keep the default unless the specification says otherwise.
  • A compare with the reference checks the specification, not the code. When all rows differ, look for a rule.
  • Write each finding as a short note: what, how many, the cause, the question.
  • Check the data rules (keys, one row per subject, required values) and every warning and message.

Exercise 15-1. Compute TRTDURD = TRTEDT - TRTSDT + 1 from your ADSL and compare it with TRTDURD of the reference ADSL. Which subjects differ, and why?

Exercise 15-2. In program 15_3, add right = FALSE to cut(). How many subjects differ now, and what do they have in common?

Exercise 15-3. List the EX rows of the 6 subjects behind the TRTEDT differences. What do all six have in common?

Exercise 15-4. check_log() stops at the first error, and the error is not in its table. Change it so that an error becomes a row of kind ERROR. Test it on a program with a warning and an error.

Exercise 15-5. Write the rule “AVAL is not missing” again: AVAL may be missing only when VSSTAT is "NOT DONE", and then it must be missing. Does ADVS pass?

The answers are in Answers, Parts IV and V.

Found an error or have a question? Write to datastep.studio@gmail.com· Every program runs in DataStep Studio for R, the R extension for VS Code and Positron.

Data: the CDISC pilot study (cdiscpilot01), from the R packagespharmaversesdtm andpharmaverseadam (Apache License 2.0).

R is a project of the R Foundation. SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. This book is not affiliated with or endorsed by SAS Institute Inc. or the R Foundation. CDISC, SDTM and ADaM are trademarks of the Clinical Data Interchange Standards Consortium.

© 2026 DataStep Labs. All rights reserved.