Skip to content

Debug a pipeline

A pipeline is a chain of steps: ae |> inner_join() |> mutate() |> filter(). You see what goes in and what comes out. You do not see what each step did in between. When a count is wrong, the question is: which step lost the records, and why? The pipe debugger answers it, step by step and record by record.

The pipe debugger and trace are in the Professional edition, free during Early Access until 17 February 2027. The keys and commands are on the cheat sheet.

This program makes the treatment-emergent adverse events from SDTM: an event that starts on or after the first dose day.

programs/23_1_lost.R
# Chapter 23: treatment-emergent adverse events from SDTM, and some records get lost on the way
library(dplyr)
adsl <- pharmaverseadam::adsl |>
filter(SAFFL == "Y") |>
select(USUBJID, TRT01A, TRTSDT)
ae <- pharmaversesdtm::ae
teae <- ae |>
inner_join(adsl, by = "USUBJID") |>
mutate(ASTDT = as.Date(AESTDTC), ASTDY = as.numeric(ASTDT - TRTSDT) + 1) |>
filter(ASTDY >= 1) |>
select(USUBJID, AESEQ, AEDECOD, AESTDTC, ASTDY, TRT01A)
teae |>
summarise(records = n(), subjects = n_distinct(USUBJID))
#> # A tibble: 1 x 2
#> records subjects
#> <int> <int>
#> 1 1120 218

The log of the run is clean:

$ dsr run programs/23_1_lost.R
4 NOTE: adsl has 254 rows and 3 columns.
7 NOTE: ae has 1191 rows and 35 columns.
7 NOTE: Empty columns in ae (missing on every row): AELLTCD, AEPTCD, AEHLTCD, AEHLGTCD, AEBDSYCD, AESOCCD, AEACN. A bug until you know why.
9 NOTE: teae has 1120 rows and 6 columns.
# A tibble: 1 × 2
records subjects
<int> <int>
1 1120 218
Workspace: new: adsl (tbl_df 254 x 3), ae (tbl_df 1191 x 35), teae (tbl_df 1120 x 6)
RESULT: CLEAN (0.02 s)

1191 records went in, 1120 came out. Some events started before the first dose, so a drop is expected. But how many, and are they the right ones? The log sees only the result of the pipeline at line 9, not its steps.

Put the cursor in the pipeline and choose Debug the Pipeline at the Cursor, or click the code lens Debug pipeline above it. The Pipeline view opens. In a terminal, give the file and the line of the pipeline:

$ dsr pipe programs/23_1_lost.R:9
Pipeline at programs/23_1_lost.R:9 (teae <- ...): 4 steps
0 ae 1191 rows
1 inner_join(adsl, by = "USUBJID") 1191 -> 1191 +TRT01A +TRTSDT
inner join with adsl (254 rows) by USUBJID
29 rows of adsl with no match: not used, e.g. 01-701-1033; 01-701-1234; 01-701-1345 @pipe[1]:unmatched:y
2 mutate(ASTDT = as.Date(AESTDTC), ASTDY = as.numeric(ASTDT - 1191 -> 1191 +ASTDT +ASTDY
3 filter(ASTDY >= 1) 1191 -> 1120
dropped 71 rows, e.g. 01-701-1111 / 1; 01-701-1111 / 4
45 rows: ASTDY >= 1 is FALSE, e.g. 01-701-1111 / 1 (ASTDY = -4); 01-701-1111 / 4 (ASTDY = -4); 01-701-1111 / 3 (ASTDY = -60) @pipe[3]:dropped:false
26 rows: ASTDY >= 1 is NA (filter() drops NA without a word), e.g. 01-701-1118 / 1 (ASTDY = NA); 01-701-1148 / 8 (ASTDY = NA); 01-701-1180 / 4 (ASTDY = NA) @pipe[3]:dropped:na
(rows matched by USUBJID+AESEQ)
4 select(USUBJID, AESEQ, AEDECOD, AESTDTC, ASTDY, TRT01A) 1120 -> 1120 -STUDYID -DOMAIN -AESPID -AETERM ...
RESULT: CLEAN (data after step k: @pipe[k]; rows step k dropped: @pipe[k]:dropped)

(The list of the columns that step 4 removes is cut here.) Each step has the rows before and after it, and the columns it added (+) or removed (-). A record is named by its key, USUBJID / AESEQ.

  • Step 1, the join, kept all 1191 records. 29 subjects of the safety population have no adverse event; the join does not use them.
  • Step 3, the filter, dropped 71 records, for two reasons:
    • 45 rows: ASTDY >= 1 is FALSE. These events started before the first dose. Dropping them is right.
    • 26 rows: ASTDY >= 1 is NA. filter() drops a row whose condition is NA, and it says nothing.

Each line of the answer ends with a name such as @pipe[3]:dropped:na: the set of those records. The data viewer and every data tool take it. In the Pipeline view, click show them. In a terminal:

$ dsr query '@pipe[3]:dropped:na' --vars 'USUBJID AESEQ AEDECOD AESTDTC TRTSDT' --n 30
@pipe[3]:dropped:na: 26 of 26 rows, rows 1-26
USUBJID AESEQ AEDECOD AESTDTC TRTSDT
01-701-1118 1 COUGH 2003 2014-03-12
01-701-1148 8 DYSPEPSIA 2012-02 2013-08-23
01-701-1180 4 HEADACHE 2002 2013-02-12
01-701-1192 4 COUGH 2010-06 2012-07-22
01-701-1192 9 COUGH 2010-06 2012-07-22
01-701-1239 9 FATIGUE 2014-03 2014-01-11
01-701-1239 10 HORDEOLUM 2014-04 2014-01-11
01-701-1363 2 HEADACHE 1986 2013-05-30
...
01-716-1418 5 HEADACHE 2013-07 2013-05-05
01-716-1418 7 HEADACHE 2013-07 2013-05-05
01-716-1418 6 VISION BLURRED 2013-07 2013-05-05
01-716-1418 8 VISION BLURRED 2013-07 2013-05-05
01-717-1004 1 ENURESIS 2013-05 2014-01-14
01-717-1357 1 DIZZINESS 1994-04 2013-05-01
01-718-1355 3 DERMATITIS ATOPIC 1982 2013-02-28

(Rows 9 to 19 are cut here.) All 26 start dates are partial: only a year, or a year and a month. as.Date() cannot make a date from 2014-03, so ASTDY is NA. Many of these events are years before the study. But look at subject 01-701-1239: FATIGUE in March 2014 and HORDEOLUM in April 2014, after the first dose on 11 January 2014. These two are treatment-emergent, and the program lost them.

The fix is not a change of the filter alone. A partial date needs a rule: an imputation (a completed date, with a flag), as the statistical analysis plan says. ADaM keeps the imputed date in ASTDT and the flag in ASTDTF (chapter 11).

Where does each record of this subject leave the pipeline, and why? Type the subject in the Follow field of the Pipeline view, or click a key such as 01-701-1239 / 9. In a terminal, --follow prints the steps (as above), then:

$ dsr pipe programs/23_1_lost.R:9 --follow 01-701-1239
...
Follow USUBJID == "01-701-1239":
0 ae 10 rows: 01-701-1239 / 1, 01-701-1239 / 10, 01-701-1239 / 2, 01-701-1239 / 3, 01-701-1239 / 4, 01-701-1239 / 5, ...
1 inner_join(adsl, by = "USUBJID") 10 -> 10
+TRT01A = "Xanomeline High Dose"
+TRTSDT = "2014-01-11"
2 mutate(ASTDT = as.Date(AESTDTC), ASTDY = as. 10 -> 10
+ASTDT = "2014-01-12" (01-701-1239 / 1), NA (01-701-1239 / 10), "2014-01-12" (01-701-1239 / 2), "2014-01-13" (01-701-1239 / 3), "2014-01-20" (01-701-1239 / 4), "2014-01-20" (01-701-1239 / 5), "2014-01-12" (01-701-1239 / 6), "2014-02-20" (01-701-1239 / 7)
+ASTDY = 2 (01-701-1239 / 1), NA (01-701-1239 / 10), 2 (01-701-1239 / 2), 3 (01-701-1239 / 3), 10 (01-701-1239 / 4), 10 (01-701-1239 / 5), 2 (01-701-1239 / 6), 41 (01-701-1239 / 7)
3 filter(ASTDY >= 1) 10 -> 8
left: 01-701-1239 / 10 (ASTDY >= 1 is NA: ASTDY = NA)
left: 01-701-1239 / 9 (ASTDY >= 1 is NA: ASTDY = NA)
4 select(USUBJID, AESEQ, AEDECOD, AESTDTC, AST 8 -> 8
RESULT: CLEAN | FOLLOW: 10 rows of USUBJID == "01-701-1239" at the start, 8 at the end (left at step 3: 2) (data after step k: @pipe[k]; rows step k dropped: @pipe[k]:dropped)

The subject has 10 events. The join adds the arm and the first dose date, the mutate() adds the study day, and the filter removes the two events with partial dates. --follow also takes a full key ('01-701-1239 / 9') or a condition ('USUBJID == "01-701-1239" & AESEQ == 9').

23.5 Try a fix without a change of the file

Section titled “23.5 Try a fix without a change of the file”

What would each later step give with other code in step 3? --try runs the pipeline again from that step, on the data that step 3 had:

$ dsr pipe programs/23_1_lost.R:9 --try 3 'filter(ASTDY >= 1 | is.na(ASTDY))'
...
Try step 3: filter(ASTDY >= 1 | is.na(ASTDY)) instead of filter(ASTDY >= 1)
3 filter(ASTDY >= 1 | is.na(ASTDY)) 1191 -> 1146 (was 1120)
dropped 45 rows, e.g. 01-701-1111 / 1; 01-701-1111 / 4
45 rows: ASTDY >= 1 | is.na(ASTDY) is FALSE, e.g. 01-701-1111 / 1 (ASTDY = -4); 01-701-1111 / 4 (ASTDY = -4); 01-701-1111 / 3 (ASTDY = -60) @try[3]
(rows matched by USUBJID+AESEQ)
4 select(USUBJID, AESEQ, AEDECOD, AESTDTC, ASTDY, TRT01A) 1146 -> 1146 (was 1120) ...
The file is not changed. The edit: programs/23_1_lost.R:12: filter(ASTDY >= 1) -> filter(ASTDY >= 1 | is.na(ASTDY)); the data after each step: @try[k]
RESULT: CLEAN | TRY: CLEAN | step 3 tried: at the end 1146 rows (was 1120) (data after step k: @pipe[k]; rows step k dropped: @pipe[k]:dropped)

Now the 26 records stay, to be reviewed or imputed. In the Pipeline view, click try a fix on a step, edit its code and press Enter. Propose the edit… shows the change of the file as a diff; you choose Apply or Discard. Nothing changes the file without your click.

Records can also appear. This program adds the disposition (DS) to ADSL:

programs/23_2_copies.R
# Chapter 23: add the disposition to ADSL, and the join copies the subjects
library(dplyr)
adsl <- pharmaverseadam::adsl |>
filter(SAFFL == "Y") |>
select(USUBJID, TRT01A)
adsl_ds <- adsl |>
left_join(pharmaversesdtm::ds, by = "USUBJID") |>
select(USUBJID, TRT01A, DSCAT, DSDECOD)
adsl_ds |>
count(TRT01A)
#> # A tibble: 3 x 2
#> TRT01A n
#> <chr> <int>
#> 1 Placebo 263
#> 2 Xanomeline High Dose 225
#> 3 Xanomeline Low Dose 310

The safety population has 86 subjects on placebo, but the table says 263. The verdict of the run is CHECK, with rows went from 254 to 798 in a join. The pipe debugger says which key copied the rows:

$ dsr pipe programs/23_2_copies.R:8
Pipeline at programs/23_2_copies.R:8 (adsl_ds <- ...): 2 steps
0 adsl 254 rows
1 left_join(pharmaversesdtm::ds, by = "USUBJID") 254 -> 798 +STUDYID +DOMAIN +DSSEQ +DSSPID +DSTERM +DSDECOD +DSCAT +VISITNUM +VISIT +DSDTC +DSSTDTC +DSSTDY
left join with pharmaversesdtm (850 rows) by USUBJID
52 rows of pharmaversesdtm with no match: not used, e.g. 01-701-1057; 01-701-1145; 01-701-1162 @pipe[1]:unmatched:y
254 -> 798: USUBJID 01-701-1023 is 4 times in pharmaversesdtm (also 01-701-1033 4 times, 01-701-1047 4 times): 544 rows more @pipe[1]:multiplied
added 544 rows
2 select(USUBJID, TRT01A, DSCAT, DSDECOD) 798 -> 798 -STUDYID -DOMAIN -DSSEQ -DSSPID -DSTERM -VISITNUM -VISIT -DSDTC -DSSTDTC -DSSTDY
RESULT: CLEAN (data after step k: @pipe[k]; rows step k dropped: @pipe[k]:dropped)

(The tool names the table by its package here: pharmaversesdtm means pharmaversesdtm::ds, with 850 rows.) DS has several records per subject: the randomization, the disposition event, and other events. A subject with 4 DS records comes out 4 times. The 52 records without a match belong to the screen failures, who are not in the safety population. Follow the running example:

$ dsr pipe programs/23_2_copies.R:8 --follow 01-701-1015
...
Follow USUBJID == "01-701-1015":
0 adsl 1 row: 01-701-1015
1 left_join(pharmaversesdtm::ds, by = "USUBJID 1 -> 3
copied: 01-701-1015 is 3 times now (was 1)
+STUDYID = "CDISCPILOT01"
+DOMAIN = "DS"
+DSSEQ = 1 (01-701-1015 / 1), 2 (01-701-1015 / 2), 3 (01-701-1015 / 3)
+DSSPID = NA
+DSTERM = "RANDOMIZED" (01-701-1015 / 1), "PROTOCOL COMPLETED" (01-701-1015 / 2), "FINAL LAB VISIT" (01-701-1015 / 3)
+DSDECOD = "RANDOMIZED" (01-701-1015 / 1), "COMPLETED" (01-701-1015 / 2), "FINAL LAB VISIT" (01-701-1015 / 3)
2 select(USUBJID, TRT01A, DSCAT, DSDECOD) 3 -> 3
RESULT: CLEAN | FOLLOW: 1 row of USUBJID == "01-701-1015" at the start, 3 at the end (data after step k: @pipe[k]; rows step k dropped: @pipe[k]:dropped)

The fix is to join only the records you need: one per subject (exercise 23-3).

A long program has many pipelines. Which one loses subject 01-701-1239? --stop-on-drop runs the program and stops at the first step where a record of that subject leaves:

$ dsr run programs/23_1_lost.R --stop-on-drop 01-701-1239
4 NOTE: adsl has 254 rows and 3 columns.
7 NOTE: ae has 1191 rows and 35 columns.
7 NOTE: Empty columns in ae (missing on every row): AELLTCD, AEPTCD, AEHLTCD, AEHLGTCD, AEBDSYCD, AESOCCD, AEACN. A bug until you know why.
=> pipeline programs/23_1_lost.R:12
10 inner_join(adsl, by = "USUBJID") |>
11 mutate(ASTDT = as.Date(AESTDTC), ASTDY = as.numeric(ASTDT - TRTSDT) + 1) |>
=> 12 filter(ASTDY >= 1) |>
13 select(USUBJID, AESEQ, AEDECOD, AESTDTC, ASTDY, TRT01A)
14
RESULT: PRY | AT: programs/23_1_lost.R:12: stopped: a record left the pipeline: step 3 of the pipeline at programs/23_1_lost.R:9 (filter(ASTDY >= 1)): 2 records of USUBJID == "01-701-1239" left: 01-701-1239 / 10 (ASTDY >= 1 is NA: ASTDY = NA); 01-701-1239 / 9 (ASTDY >= 1 is NA: ASTDY = NA) (0.09 s) (live: dsr pry next | step | finish | continue | quit)

The run waits at the stop. It is a pry stop (chapter 24): . is the data after the step, before the data before it, and dropped the records that left. Look, then go on:

$ dsr pry eval 'dropped[c("USUBJID", "AESEQ", "AEDECOD", "AESTDTC")]'
USUBJID AESEQ AEDECOD AESTDTC
127 01-701-1239 10 HORDEOLUM 2014-04
126 01-701-1239 9 FATIGUE 2014-03
RESULT: CLEAN (0 s)
$ dsr pry continue
9 NOTE: teae has 1120 rows and 6 columns.
...
RESULT: CLEAN (0.3 s)

--stop-on-dup stops where a key stops being unique. On the DS join:

$ dsr run programs/23_2_copies.R --stop-on-dup USUBJID
4 NOTE: adsl has 254 rows and 2 columns.
=> pipeline programs/23_2_copies.R:9
7
8 adsl_ds <- adsl |>
=> 9 left_join(pharmaversesdtm::ds, by = "USUBJID") |>
10 select(USUBJID, TRT01A, DSCAT, DSDECOD)
11
RESULT: PRY | AT: programs/23_2_copies.R:9: stopped: a key is no longer unique: step 1 of the pipeline at programs/23_2_copies.R:8 (left_join(pharmaversesdtm::ds, by = "USUBJID")): USUBJID is no longer unique: 254 keys more than once, e.g. 01-701-1023 (4 times) (0.08 s) (live: dsr pry next | step | finish | continue | quit)
$ dsr pry quit
PRY: programs/23_2_copies.R:9: the run ended at this stop (quit); the lines after it did not run
Workspace: new: adsl (tbl_df 254 x 2)
RESULT: STOPPED | AT: programs/23_2_copies.R:9: the run ended at this stop (quit); the lines after it did not run (0.19 s)

At this stop, dups holds the rows with a duplicate key. Always end a stop with dsr pry continue or dsr pry quit, so that the session is free again.

A pipeline debugger looks at data. Sometimes the problem is a function that gives a wrong value. Trace runs one call and lists every line that ran, the variables each line changed, and each if with its result. This function computes the study day:

programs/23_3_study_day.R
# Chapter 23: a study day function, and the day of the first dose comes out wrong
study_day <- function(date, ref) {
days <- as.numeric(date - ref)
if (days > 0) {
days + 1
} else {
days
}
}
study_day(as.Date("2014-01-03"), as.Date("2014-01-02"))
#> [1] 2
study_day(as.Date("2014-01-02"), as.Date("2014-01-02"))
#> [1] 0

The day after the first dose is day 2, which is right. But the day of the first dose gives 0. In CDISC there is no day 0: the day of the first dose is day 1. Put the cursor on study_day and choose Trace Function Call; a form asks for the arguments. Or in a terminal:

$ dsr trace programs/23_3_study_day.R --call 'study_day(as.Date("2014-01-02"), as.Date("2014-01-02"))'
programs/23_3_study_day.R:3 days <- as.numeric(date - ref)
days = 0
programs/23_3_study_day.R:4 if (days > 0) {
-> days > 0 [FALSE] days = 0
programs/23_3_study_day.R:7 days
[1] 0
Value: 0
RESULT: CLEAN (0.03 s)

Line 4 is the bug: days > 0 is FALSE when days is 0, so the else branch returns 0. The test must be days >= 0. You found it without a stop and without a print() in the function. The file argument (programs/23_3_study_day.R) only defines the functions; the rest of the file does not run. --into g also traces a function g that the call uses.

  • dsr pipe file.R:line (or Debug pipeline) shows each step of a pipeline: rows before and after, columns added and removed.
  • For a filter, it gives the reason of each dropped record: the condition was FALSE or NA. filter() drops NA rows without a word.
  • For a join, it gives the records without a match on each side and the keys that copied rows.
  • The sets @pipe[k], @pipe[k]:dropped:na, @pipe[k]:multiplied work in the data viewer and in every data tool.
  • --follow follows one subject; --try k '<code>' tries a fix and leaves the file as it is.
  • dsr run --stop-on-drop S and --stop-on-dup KEY stop a long program where a record leaves or a key stops being unique.
  • dsr trace (Trace Function Call) lists every line of one call, with its values and each if.

Exercise 23-1. In 23_1_lost.R, how many records does the filter drop because the condition is FALSE, and how many because it is NA? Which of them are a problem?

Exercise 23-2. Follow subject 01-701-1015 through the pipeline of 23_1_lost.R. Which of its records leave, and what are their study days?

Exercise 23-3. Change 23_2_copies.R so that it joins only the disposition event (DSCAT == "DISPOSITION EVENT"). How many subjects are in each arm now?

Exercise 23-4. Trace study_day() for an event on 26 December 2013, with the first dose on 2 January 2014. What is the value, and is it right? Then fix line 4 and check the day of the first dose again.

The answers are in Answers, Part VI.

Found an error or have a question? Write to datastep.studio@gmail.com· Every program runs in DataStep Studio for R, the R extension for VS Code and Positron.

Data: the CDISC pilot study (cdiscpilot01), from the R packagespharmaversesdtm andpharmaverseadam (Apache License 2.0).

R is a project of the R Foundation. SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. This book is not affiliated with or endorsed by SAS Institute Inc. or the R Foundation. CDISC, SDTM and ADaM are trademarks of the Clinical Data Interchange Standards Consortium.

© 2026 DataStep Labs. All rights reserved.