Debug a pipeline
A pipeline is a chain of steps: ae |> inner_join() |> mutate() |> filter(). You see what goes in and what comes out. You do not see what each step did in between. When a count is wrong, the question is: which step lost the records, and why? The pipe debugger answers it, step by step and record by record.
The pipe debugger and trace are in the Professional edition, free during Early Access until 17 February 2027. The keys and commands are on the cheat sheet.
23.1 A clean run that loses records
Section titled “23.1 A clean run that loses records”This program makes the treatment-emergent adverse events from SDTM: an event that starts on or after the first dose day.
# Chapter 23: treatment-emergent adverse events from SDTM, and some records get lost on the waylibrary(dplyr)
adsl <- pharmaverseadam::adsl |> filter(SAFFL == "Y") |> select(USUBJID, TRT01A, TRTSDT)ae <- pharmaversesdtm::ae
teae <- ae |> inner_join(adsl, by = "USUBJID") |> mutate(ASTDT = as.Date(AESTDTC), ASTDY = as.numeric(ASTDT - TRTSDT) + 1) |> filter(ASTDY >= 1) |> select(USUBJID, AESEQ, AEDECOD, AESTDTC, ASTDY, TRT01A)
teae |> summarise(records = n(), subjects = n_distinct(USUBJID))#> # A tibble: 1 x 2#> records subjects#> <int> <int>#> 1 1120 218The log of the run is clean:
$ dsr run programs/23_1_lost.R 4 NOTE: adsl has 254 rows and 3 columns. 7 NOTE: ae has 1191 rows and 35 columns. 7 NOTE: Empty columns in ae (missing on every row): AELLTCD, AEPTCD, AEHLTCD, AEHLGTCD, AEBDSYCD, AESOCCD, AEACN. A bug until you know why. 9 NOTE: teae has 1120 rows and 6 columns. # A tibble: 1 × 2 records subjects <int> <int> 1 1120 218Workspace: new: adsl (tbl_df 254 x 3), ae (tbl_df 1191 x 35), teae (tbl_df 1120 x 6)RESULT: CLEAN (0.02 s)1191 records went in, 1120 came out. Some events started before the first dose, so a drop is expected. But how many, and are they the right ones? The log sees only the result of the pipeline at line 9, not its steps.
23.2 The rows at each step: dsr pipe
Section titled “23.2 The rows at each step: dsr pipe”Put the cursor in the pipeline and choose Debug the Pipeline at the Cursor, or click the code lens Debug pipeline above it. The Pipeline view opens. In a terminal, give the file and the line of the pipeline:
$ dsr pipe programs/23_1_lost.R:9Pipeline at programs/23_1_lost.R:9 (teae <- ...): 4 steps 0 ae 1191 rows 1 inner_join(adsl, by = "USUBJID") 1191 -> 1191 +TRT01A +TRTSDT inner join with adsl (254 rows) by USUBJID 29 rows of adsl with no match: not used, e.g. 01-701-1033; 01-701-1234; 01-701-1345 @pipe[1]:unmatched:y 2 mutate(ASTDT = as.Date(AESTDTC), ASTDY = as.numeric(ASTDT - 1191 -> 1191 +ASTDT +ASTDY 3 filter(ASTDY >= 1) 1191 -> 1120 dropped 71 rows, e.g. 01-701-1111 / 1; 01-701-1111 / 4 45 rows: ASTDY >= 1 is FALSE, e.g. 01-701-1111 / 1 (ASTDY = -4); 01-701-1111 / 4 (ASTDY = -4); 01-701-1111 / 3 (ASTDY = -60) @pipe[3]:dropped:false 26 rows: ASTDY >= 1 is NA (filter() drops NA without a word), e.g. 01-701-1118 / 1 (ASTDY = NA); 01-701-1148 / 8 (ASTDY = NA); 01-701-1180 / 4 (ASTDY = NA) @pipe[3]:dropped:na (rows matched by USUBJID+AESEQ) 4 select(USUBJID, AESEQ, AEDECOD, AESTDTC, ASTDY, TRT01A) 1120 -> 1120 -STUDYID -DOMAIN -AESPID -AETERM ...RESULT: CLEAN (data after step k: @pipe[k]; rows step k dropped: @pipe[k]:dropped)(The list of the columns that step 4 removes is cut here.) Each step has the rows before and after it, and the columns it added (+) or removed (-). A record is named by its key, USUBJID / AESEQ.
- Step 1, the join, kept all 1191 records. 29 subjects of the safety population have no adverse event; the join does not use them.
- Step 3, the filter, dropped 71 records, for two reasons:
- 45 rows:
ASTDY >= 1is FALSE. These events started before the first dose. Dropping them is right. - 26 rows:
ASTDY >= 1is NA.filter()drops a row whose condition isNA, and it says nothing.
- 45 rows:
23.3 Why NA: look at the dropped records
Section titled “23.3 Why NA: look at the dropped records”Each line of the answer ends with a name such as @pipe[3]:dropped:na: the set of those records. The data viewer and every data tool take it. In the Pipeline view, click show them. In a terminal:
$ dsr query '@pipe[3]:dropped:na' --vars 'USUBJID AESEQ AEDECOD AESTDTC TRTSDT' --n 30@pipe[3]:dropped:na: 26 of 26 rows, rows 1-26 USUBJID AESEQ AEDECOD AESTDTC TRTSDT 01-701-1118 1 COUGH 2003 2014-03-12 01-701-1148 8 DYSPEPSIA 2012-02 2013-08-23 01-701-1180 4 HEADACHE 2002 2013-02-12 01-701-1192 4 COUGH 2010-06 2012-07-22 01-701-1192 9 COUGH 2010-06 2012-07-22 01-701-1239 9 FATIGUE 2014-03 2014-01-11 01-701-1239 10 HORDEOLUM 2014-04 2014-01-11 01-701-1363 2 HEADACHE 1986 2013-05-30... 01-716-1418 5 HEADACHE 2013-07 2013-05-05 01-716-1418 7 HEADACHE 2013-07 2013-05-05 01-716-1418 6 VISION BLURRED 2013-07 2013-05-05 01-716-1418 8 VISION BLURRED 2013-07 2013-05-05 01-717-1004 1 ENURESIS 2013-05 2014-01-14 01-717-1357 1 DIZZINESS 1994-04 2013-05-01 01-718-1355 3 DERMATITIS ATOPIC 1982 2013-02-28(Rows 9 to 19 are cut here.) All 26 start dates are partial: only a year, or a year and a month. as.Date() cannot make a date from 2014-03, so ASTDY is NA. Many of these events are years before the study. But look at subject 01-701-1239: FATIGUE in March 2014 and HORDEOLUM in April 2014, after the first dose on 11 January 2014. These two are treatment-emergent, and the program lost them.
The fix is not a change of the filter alone. A partial date needs a rule: an imputation (a completed date, with a flag), as the statistical analysis plan says. ADaM keeps the imputed date in ASTDT and the flag in ASTDTF (chapter 11).
23.4 Follow one subject
Section titled “23.4 Follow one subject”Where does each record of this subject leave the pipeline, and why? Type the subject in the Follow field of the Pipeline view, or click a key such as 01-701-1239 / 9. In a terminal, --follow prints the steps (as above), then:
$ dsr pipe programs/23_1_lost.R:9 --follow 01-701-1239...Follow USUBJID == "01-701-1239": 0 ae 10 rows: 01-701-1239 / 1, 01-701-1239 / 10, 01-701-1239 / 2, 01-701-1239 / 3, 01-701-1239 / 4, 01-701-1239 / 5, ... 1 inner_join(adsl, by = "USUBJID") 10 -> 10 +TRT01A = "Xanomeline High Dose" +TRTSDT = "2014-01-11" 2 mutate(ASTDT = as.Date(AESTDTC), ASTDY = as. 10 -> 10 +ASTDT = "2014-01-12" (01-701-1239 / 1), NA (01-701-1239 / 10), "2014-01-12" (01-701-1239 / 2), "2014-01-13" (01-701-1239 / 3), "2014-01-20" (01-701-1239 / 4), "2014-01-20" (01-701-1239 / 5), "2014-01-12" (01-701-1239 / 6), "2014-02-20" (01-701-1239 / 7) +ASTDY = 2 (01-701-1239 / 1), NA (01-701-1239 / 10), 2 (01-701-1239 / 2), 3 (01-701-1239 / 3), 10 (01-701-1239 / 4), 10 (01-701-1239 / 5), 2 (01-701-1239 / 6), 41 (01-701-1239 / 7) 3 filter(ASTDY >= 1) 10 -> 8 left: 01-701-1239 / 10 (ASTDY >= 1 is NA: ASTDY = NA) left: 01-701-1239 / 9 (ASTDY >= 1 is NA: ASTDY = NA) 4 select(USUBJID, AESEQ, AEDECOD, AESTDTC, AST 8 -> 8RESULT: CLEAN | FOLLOW: 10 rows of USUBJID == "01-701-1239" at the start, 8 at the end (left at step 3: 2) (data after step k: @pipe[k]; rows step k dropped: @pipe[k]:dropped)The subject has 10 events. The join adds the arm and the first dose date, the mutate() adds the study day, and the filter removes the two events with partial dates. --follow also takes a full key ('01-701-1239 / 9') or a condition ('USUBJID == "01-701-1239" & AESEQ == 9').
23.5 Try a fix without a change of the file
Section titled “23.5 Try a fix without a change of the file”What would each later step give with other code in step 3? --try runs the pipeline again from that step, on the data that step 3 had:
$ dsr pipe programs/23_1_lost.R:9 --try 3 'filter(ASTDY >= 1 | is.na(ASTDY))'...Try step 3: filter(ASTDY >= 1 | is.na(ASTDY)) instead of filter(ASTDY >= 1) 3 filter(ASTDY >= 1 | is.na(ASTDY)) 1191 -> 1146 (was 1120) dropped 45 rows, e.g. 01-701-1111 / 1; 01-701-1111 / 4 45 rows: ASTDY >= 1 | is.na(ASTDY) is FALSE, e.g. 01-701-1111 / 1 (ASTDY = -4); 01-701-1111 / 4 (ASTDY = -4); 01-701-1111 / 3 (ASTDY = -60) @try[3] (rows matched by USUBJID+AESEQ) 4 select(USUBJID, AESEQ, AEDECOD, AESTDTC, ASTDY, TRT01A) 1146 -> 1146 (was 1120) ...The file is not changed. The edit: programs/23_1_lost.R:12: filter(ASTDY >= 1) -> filter(ASTDY >= 1 | is.na(ASTDY)); the data after each step: @try[k]RESULT: CLEAN | TRY: CLEAN | step 3 tried: at the end 1146 rows (was 1120) (data after step k: @pipe[k]; rows step k dropped: @pipe[k]:dropped)Now the 26 records stay, to be reviewed or imputed. In the Pipeline view, click try a fix on a step, edit its code and press Enter. Propose the edit… shows the change of the file as a diff; you choose Apply or Discard. Nothing changes the file without your click.
23.6 A join that copies records
Section titled “23.6 A join that copies records”Records can also appear. This program adds the disposition (DS) to ADSL:
# Chapter 23: add the disposition to ADSL, and the join copies the subjectslibrary(dplyr)
adsl <- pharmaverseadam::adsl |> filter(SAFFL == "Y") |> select(USUBJID, TRT01A)
adsl_ds <- adsl |> left_join(pharmaversesdtm::ds, by = "USUBJID") |> select(USUBJID, TRT01A, DSCAT, DSDECOD)
adsl_ds |> count(TRT01A)#> # A tibble: 3 x 2#> TRT01A n#> <chr> <int>#> 1 Placebo 263#> 2 Xanomeline High Dose 225#> 3 Xanomeline Low Dose 310The safety population has 86 subjects on placebo, but the table says 263. The verdict of the run is CHECK, with rows went from 254 to 798 in a join. The pipe debugger says which key copied the rows:
$ dsr pipe programs/23_2_copies.R:8Pipeline at programs/23_2_copies.R:8 (adsl_ds <- ...): 2 steps 0 adsl 254 rows 1 left_join(pharmaversesdtm::ds, by = "USUBJID") 254 -> 798 +STUDYID +DOMAIN +DSSEQ +DSSPID +DSTERM +DSDECOD +DSCAT +VISITNUM +VISIT +DSDTC +DSSTDTC +DSSTDY left join with pharmaversesdtm (850 rows) by USUBJID 52 rows of pharmaversesdtm with no match: not used, e.g. 01-701-1057; 01-701-1145; 01-701-1162 @pipe[1]:unmatched:y 254 -> 798: USUBJID 01-701-1023 is 4 times in pharmaversesdtm (also 01-701-1033 4 times, 01-701-1047 4 times): 544 rows more @pipe[1]:multiplied added 544 rows 2 select(USUBJID, TRT01A, DSCAT, DSDECOD) 798 -> 798 -STUDYID -DOMAIN -DSSEQ -DSSPID -DSTERM -VISITNUM -VISIT -DSDTC -DSSTDTC -DSSTDYRESULT: CLEAN (data after step k: @pipe[k]; rows step k dropped: @pipe[k]:dropped)(The tool names the table by its package here: pharmaversesdtm means pharmaversesdtm::ds, with 850 rows.) DS has several records per subject: the randomization, the disposition event, and other events. A subject with 4 DS records comes out 4 times. The 52 records without a match belong to the screen failures, who are not in the safety population. Follow the running example:
$ dsr pipe programs/23_2_copies.R:8 --follow 01-701-1015...Follow USUBJID == "01-701-1015": 0 adsl 1 row: 01-701-1015 1 left_join(pharmaversesdtm::ds, by = "USUBJID 1 -> 3 copied: 01-701-1015 is 3 times now (was 1) +STUDYID = "CDISCPILOT01" +DOMAIN = "DS" +DSSEQ = 1 (01-701-1015 / 1), 2 (01-701-1015 / 2), 3 (01-701-1015 / 3) +DSSPID = NA +DSTERM = "RANDOMIZED" (01-701-1015 / 1), "PROTOCOL COMPLETED" (01-701-1015 / 2), "FINAL LAB VISIT" (01-701-1015 / 3) +DSDECOD = "RANDOMIZED" (01-701-1015 / 1), "COMPLETED" (01-701-1015 / 2), "FINAL LAB VISIT" (01-701-1015 / 3) 2 select(USUBJID, TRT01A, DSCAT, DSDECOD) 3 -> 3RESULT: CLEAN | FOLLOW: 1 row of USUBJID == "01-701-1015" at the start, 3 at the end (data after step k: @pipe[k]; rows step k dropped: @pipe[k]:dropped)The fix is to join only the records you need: one per subject (exercise 23-3).
23.7 Stop the run where it happens
Section titled “23.7 Stop the run where it happens”A long program has many pipelines. Which one loses subject 01-701-1239? --stop-on-drop runs the program and stops at the first step where a record of that subject leaves:
$ dsr run programs/23_1_lost.R --stop-on-drop 01-701-1239 4 NOTE: adsl has 254 rows and 3 columns. 7 NOTE: ae has 1191 rows and 35 columns. 7 NOTE: Empty columns in ae (missing on every row): AELLTCD, AEPTCD, AEHLTCD, AEHLGTCD, AEBDSYCD, AESOCCD, AEACN. A bug until you know why.=> pipeline programs/23_1_lost.R:12 10 inner_join(adsl, by = "USUBJID") |> 11 mutate(ASTDT = as.Date(AESTDTC), ASTDY = as.numeric(ASTDT - TRTSDT) + 1) |>=> 12 filter(ASTDY >= 1) |> 13 select(USUBJID, AESEQ, AEDECOD, AESTDTC, ASTDY, TRT01A) 14RESULT: PRY | AT: programs/23_1_lost.R:12: stopped: a record left the pipeline: step 3 of the pipeline at programs/23_1_lost.R:9 (filter(ASTDY >= 1)): 2 records of USUBJID == "01-701-1239" left: 01-701-1239 / 10 (ASTDY >= 1 is NA: ASTDY = NA); 01-701-1239 / 9 (ASTDY >= 1 is NA: ASTDY = NA) (0.09 s) (live: dsr pry next | step | finish | continue | quit)The run waits at the stop. It is a pry stop (chapter 24): . is the data after the step, before the data before it, and dropped the records that left. Look, then go on:
$ dsr pry eval 'dropped[c("USUBJID", "AESEQ", "AEDECOD", "AESTDTC")]' USUBJID AESEQ AEDECOD AESTDTC 127 01-701-1239 10 HORDEOLUM 2014-04 126 01-701-1239 9 FATIGUE 2014-03RESULT: CLEAN (0 s)$ dsr pry continue 9 NOTE: teae has 1120 rows and 6 columns....RESULT: CLEAN (0.3 s)--stop-on-dup stops where a key stops being unique. On the DS join:
$ dsr run programs/23_2_copies.R --stop-on-dup USUBJID 4 NOTE: adsl has 254 rows and 2 columns.=> pipeline programs/23_2_copies.R:9 7 8 adsl_ds <- adsl |>=> 9 left_join(pharmaversesdtm::ds, by = "USUBJID") |> 10 select(USUBJID, TRT01A, DSCAT, DSDECOD) 11RESULT: PRY | AT: programs/23_2_copies.R:9: stopped: a key is no longer unique: step 1 of the pipeline at programs/23_2_copies.R:8 (left_join(pharmaversesdtm::ds, by = "USUBJID")): USUBJID is no longer unique: 254 keys more than once, e.g. 01-701-1023 (4 times) (0.08 s) (live: dsr pry next | step | finish | continue | quit)$ dsr pry quit PRY: programs/23_2_copies.R:9: the run ended at this stop (quit); the lines after it did not runWorkspace: new: adsl (tbl_df 254 x 2)RESULT: STOPPED | AT: programs/23_2_copies.R:9: the run ended at this stop (quit); the lines after it did not run (0.19 s)At this stop, dups holds the rows with a duplicate key. Always end a stop with dsr pry continue or dsr pry quit, so that the session is free again.
23.8 Trace one function call
Section titled “23.8 Trace one function call”A pipeline debugger looks at data. Sometimes the problem is a function that gives a wrong value. Trace runs one call and lists every line that ran, the variables each line changed, and each if with its result. This function computes the study day:
# Chapter 23: a study day function, and the day of the first dose comes out wrongstudy_day <- function(date, ref) { days <- as.numeric(date - ref) if (days > 0) { days + 1 } else { days }}
study_day(as.Date("2014-01-03"), as.Date("2014-01-02"))#> [1] 2study_day(as.Date("2014-01-02"), as.Date("2014-01-02"))#> [1] 0The day after the first dose is day 2, which is right. But the day of the first dose gives 0. In CDISC there is no day 0: the day of the first dose is day 1. Put the cursor on study_day and choose Trace Function Call; a form asks for the arguments. Or in a terminal:
$ dsr trace programs/23_3_study_day.R --call 'study_day(as.Date("2014-01-02"), as.Date("2014-01-02"))'programs/23_3_study_day.R:3 days <- as.numeric(date - ref) days = 0programs/23_3_study_day.R:4 if (days > 0) { -> days > 0 [FALSE] days = 0programs/23_3_study_day.R:7 days [1] 0Value: 0RESULT: CLEAN (0.03 s)Line 4 is the bug: days > 0 is FALSE when days is 0, so the else branch returns 0. The test must be days >= 0. You found it without a stop and without a print() in the function. The file argument (programs/23_3_study_day.R) only defines the functions; the rest of the file does not run. --into g also traces a function g that the call uses.
23.9 Summary
Section titled “23.9 Summary”dsr pipe file.R:line(or Debug pipeline) shows each step of a pipeline: rows before and after, columns added and removed.- For a filter, it gives the reason of each dropped record: the condition was FALSE or NA.
filter()dropsNArows without a word. - For a join, it gives the records without a match on each side and the keys that copied rows.
- The sets
@pipe[k],@pipe[k]:dropped:na,@pipe[k]:multipliedwork in the data viewer and in every data tool. --followfollows one subject;--try k '<code>'tries a fix and leaves the file as it is.dsr run --stop-on-drop Sand--stop-on-dup KEYstop a long program where a record leaves or a key stops being unique.dsr trace(Trace Function Call) lists every line of one call, with its values and eachif.
23.10 Exercises
Section titled “23.10 Exercises”Exercise 23-1. In 23_1_lost.R, how many records does the filter drop because the condition is FALSE, and how many because it is NA? Which of them are a problem?
Exercise 23-2. Follow subject 01-701-1015 through the pipeline of 23_1_lost.R. Which of its records leave, and what are their study days?
Exercise 23-3. Change 23_2_copies.R so that it joins only the disposition event (DSCAT == "DISPOSITION EVENT"). How many subjects are in each arm now?
Exercise 23-4. Trace study_day() for an event on 26 December 2013, with the first dose on 2 January 2014. What is the value, and is it right? Then fix line 4 and check the day of the first dose again.
The answers are in Answers, Part VI.
Found an error or have a question? Write to datastep.studio@gmail.com· Every program runs in DataStep Studio for R, the R extension for VS Code and Positron.
Data: the CDISC pilot study (cdiscpilot01), from the R packagespharmaversesdtm andpharmaverseadam (Apache License 2.0).
R is a project of the R Foundation. SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. This book is not affiliated with or endorsed by SAS Institute Inc. or the R Foundation. CDISC, SDTM and ADaM are trademarks of the Clinical Data Interchange Standards Consortium.
© 2026 DataStep Labs. All rights reserved.