Coding Workshop Part 1

Computational Reproducibility

Karissa Whiting
June 1st, 2026

Summer Training Series
Memorial Sloan Kettering

Agenda

Computational Reproducibility Overview
Project Setup
Writing Good Code
Reproducible Reporting
AI Guidelines for Coding

What is Reproducibility?

A data analysis is reproducible if all the information (data, files, etc.) needed to compute results is available for someone else to re-do your entire analysis and get the same results.

All data processing steps from raw data to cleaned data are available and documented
All analysis decisions made are documented and available in code
Results don’t depend on your specific computational environment (e.g. no hard coded file paths, seeds set for stochastic computations)

Why is Reproducibility Important?

Allows you to show evidence of your results
Encourages transparency about decisions made during analysis
Enables others to check and use/extend your methods and results
Enables FUTURE YOU to check and use/extend your methods and results

“You mostly collaborate with yourself, and me-from-two-months-ago never responds to email”: Dr. Mark Holder, Computational Biologist

Why is Reproducibility Important?

Dangers of writing code that is hard to double-check or confirm:

The New York Times

Journal Code-Sharing Requirements

Increasingly, journals and funders require code to be submitted. You may be asked to provide your cleaned analysis data (and possibly code) at time of publication.

Journal / Publisher	Requirement
NIH	As of Jan 2023, all grants require a Data Management Plan; sharing of data and code may be required at publication or grant end.
The BMJ	Submissions ≥ May 2024: code used to analyze data must be submitted
PLOS Biology/Medicine	Submissions ≥ Jan 2026: author-generated code directly related to findings must be available publicly
Nature / Springer Nature	Code Availability Statement required; sharing is a condition of publication

Tip

Requirements vary by journal, article type, and whether code is essential to reproduce findings. Even when not required, sharing code is increasingly expected.

Five Pillars Of Reproducibility

How Do We Ensure Our Code is Reproducible?

Compute Environment Control
- Virtual environments, avoid absolute file paths (e.g. ~/Users/Whiting/Projects...)
Code Version Control
- Document changes you make, or use git/Github
Documentation
- Comment and document your code
- Invest in a good README.md
Data Integrity - more details later
Literate Programming
- Have a clear project structure, avoid ‘by hand’ steps

Project Setup

Good Project Setup First = Hours Saved Later

The goal is to set yourself up with a project that is easy to navigate & self-explanatory without needing to hold everything in your head
🔮✨ A little upfront effort in organizing your code and files will save you an enormous amount of time and frustration later

Project Setup: Anatomy of a Project

📁 my-project/
├── 📁 raw-data/
├── 📁 data/
├── 📁 scripts/
│   ├── 📄 01-clean.R
│   └── 📄 02-analysis.R
├── 📁 outputs/
├── 📁 admin/
├── 📄 README.md
└── 📄 my-project.Rproj

Keep raw and processed data separate (raw-data vs. data)
Folder for scripts, ordered or labelled descriptively
Optionally have admin for project notes and outputs for final reports and figures
README — text file that introduces and explains a project (usethis::use_readme_md())
R Project (.Rproj file) — tells RStudio all your files belong to one project and sets the working directory for the entire project

Compute Environment Control: Virtual Environments

Make it easy for others (and future you) to re-run your code on any machine

Isolate package versions per project so code runs the same everywhere

Python — venv or conda:

python -m venv .venv && source .venv/bin/activate
pip freeze > requirements.txt   # save
pip install -r requirements.txt # restore

R — renv:

You can commit & track renv.lock / requirements.txt / environment.yml

Compute Environment Control: Avoid Absolute File Paths

Hard-coded paths break on every other machine:

# ❌ breaks on anyone else's computer
read.csv("~/Users/Whiting/Projects/data.csv")

# ✅ relative path from project root
read.csv("data/raw/data.csv")

# ✅ R: use here::here() for safety
read.csv(here::here("data", "raw", "data.csv"))

Always use paths relative to the project root
here::here() (R) and pyprojroot.here() (Python) find the project root automatically

Setup Version Control

Git is version control software that runs locally on your computer

Tracks every change made to your files over time, so you can revert to any previous version
Work on experimental changes without breaking working code (branches)

GitHub is a cloud platform for hosting Git repositories

Back up your code and git history remotely and collaborate with others on the same codebase

Tip

Think of Git as “track changes” for your entire project, and GitHub as Google Drive — but for code and those documented changes

GitHub & GitHub Enterprise

GitHub

www.github.com
Public website
Individual repositories can be made public or private (only you or invited collaborators can see repo)

GitHub Enterprise (MSK)

www.github.mskcc.org
Behind the MSK firewall (need to be on VPN if not on MSK network)
Login via MSK credentials
Should be used for MSK projects
Otherwise identical to github.com

Set Up `.gitignore` Before Your First Commit

Warning

Deleting a file does not remove it from Git history. If PHI or sensitive data is ever committed, it is a violation — even if you delete it in the next commit.

A .gitignore file tells git which files to NOT track. Set it up before you commit anything.
ßUse usethis::use_git_ignore() in R or GitHub’s gitignore templates to get started quickly (but check where data resides and add as needed)

`.gitignore` Example

# Data — never commit raw or sensitive data
data/
raw-data/
*.csv
*.xlsx

# Credentials and config
.env
*.key

# R-specific
.Rhistory
.RData
.Rproj.user/

# Python virtual environment (use requirements.txt instead)
.venv/
venv/

# Compiled bytecode
__pycache__/
*.pyc

# Jupyter notebook checkpoints
.ipynb_checkpoints/

# OS clutter
.DS_Store

Writing Good Code

Analysis Plan First

Write the plan before you write the code

Why it matters:

Prevents p-hacking — decisions made after seeing results inflate false positive rates
Forces clarity on the research questions and makes collaboration easier
Makes coding easier!!!!

What to include:

Research question, hypotheses, endpoints
Cohort definition
Statistical methods and model specifications (e.g how to handle missing data)
Planned sensitivity analyses, expected outputs/tables/figures

Literate Programming

Avoid ‘by hand’ steps used in the analysis

Don’t clean by hand in Excel. All analysis steps should be done in code and saved in scripts. If any ‘non-scriptable’ steps are unavoidable, document those steps very clearly.
Write code comments and maintain a good README
Consider efficiency / readibility trade-off
DNR (Do Not Repeat) - if you do it more than 3 times, consider writing a function
Use reproducible reporting practices for analyses (e.g. Rmd, quarto, Jupyter notebook, inline text stats)

📁 my-project/
├── 📁 raw-data/
├── 📁 data/
├── 📁 scripts/
│   ├── 📄 01-clean.R
│   └── 📄 02-analysis.R
├── 📁 outputs/
├── 📁 admin/
├── 📄 README.md
└── 📄 my-project.Rproj

Reproducible Reporting

R Markdown, Quarto and Jupyter are tools for integrating code and narrative text into a single executable document
Can be rendered into various output formats (HTML, PDF, Word, and slides)
Detailed code and data analysis steps are included in one document, encouraging transparency and providing a complete record of the research process
Documents automatically update when data or code changes, reducing errors and maintaining consistency.
Version-control compatible

Quarto vs. Jupyter Notebooks

Quarto (.qmd)

Native R, Python, Julia, support
Plain-text Markdown + code instructions — no stored outputs
Git-friendly — diffs are clean and meaningful
Easy to turn analysis into formal report: Renders to HTML, PDF, Word, slides, websites, etc

Jupyter (.ipynb)

Primarily Python-first; widely used in data science and ML
JSON container storing code, text, and outputs — Results are visible on reopen
Cell execution order can be non-linear (a reproducibility risk)
Git diffs can be messy

Tip

The hidden risk of ipynb: Jupyter lets you run cells out of order. Results can depend on state you forgot you created earlier. The notebook looks coherent, but a fresh top-to-bottom run may not reproduce the same story. Quarto rendering enforces clean execution and exposes that quickly.

Quarto Features: Callouts and Comments

Sometimes you need to draw attention to something in your report. You can do this using {.callout-note}

::: {.callout-note}
Note that there are five types of callouts, including:
`note`, `warning`, `important`, `tip`, and `caution`.
:::

Note

Note that there are five types of callouts, including: note, warning, important, tip, and caution.

::: {.callout-warning}
Here is an example of a warning
:::

Warning

Here is an example of a warning

Quarto Features: Tabs

{gtsummary}

{gtsummary} - Tools to create publication-ready analytical and summary tables using the R programming language.
Summarizes data sets, regression models, and more, using sensible defaults with highly customizable capabilities.

{gtsummary} overview

Create tabular summaries including:
- “Table 1”
- Cross-tabulation
- Regression models summaries
- Survival data summaries
Report statistics from {gtsummary} tables inline in R Markdown
Stack or merge any table type
Use themes to standardize across tables
Choose from different print engines

Basic tbl_summary()

sm_trial <- trial %>% 
  select(trt, age, grade, response)

sm_trial %>%
  select(-trt) %>%
  tbl_summary()

Characteristic	N = 200¹
Age	47 (38, 57)
Unknown	11
Grade
I	68 (34%)
II	68 (34%)
III	64 (32%)
Tumor Response	61 (32%)
Unknown	7
¹ Median (Q1, Q3); n (%)

Four types of summaries: continuous, continuous2, categorical, and dichotomous
Variables coded 0/1, TRUE/FALSE, Yes/No treated as dichotomous
Statistics are median (IQR) for continuous, n (%) for categorical/dichotomous
Lists NA values under “Unknown”
Label attributes are printed automatically

Advanced Tips: tbl_uvregression() with formula

formula argument is powerful! You can adjust for variables, or pass mixed model formats (e.g. "{y} ~ {x} + (1 | gear)")
Additionally, add_global_p() can be useful

tbl_uvreg <- sm_trial %>% 
  tbl_uvregression(
    method = glm,
    y = response,
    method.args = list(family = binomial),
    formula = "{y} ~ {x} + age",
    include = -c(age), 
    exponentiate = TRUE
  ) %>%
  bold_labels() %>%
  add_global_p()

Error in `add_global_p()`:
! There was an error running `anova_fun` for variable "trt". See message
  below.
✖ The package "parameters" (>= 0.20.2) is required.

tbl_uvreg

Error:
! object 'tbl_uvreg' not found

Advanced Tip: gtsummary Themes

Themes control many aspects of how a table is printed. Function defaults can be controlled with themes, as well as other aspects that are not modifiable with function arguments.
The {gtsummary} package comes with a few themes, and we welcome user-contributed themes as well!
Most commonly used theme: gtsummary::theme_gtsummary_compact()
More info: https://www.danieldsjoberg.com/gtsummary/articles/themes.html

Other Customizations

Many more customization available!

See the documentation at http://www.danieldsjoberg.com/gtsummary/reference/index.html

And a detailed tbl_summary() vignette at http://www.danieldsjoberg.com/gtsummary/articles/tbl_summary.html

Report Reproducible Statistics with gtsummary::inline_text()

Tables are important, but we often need to report results in-line in a report.
Any statistic reported in a {gtsummary} table can be extracted and reported in-line in an R Markdown document with the inline_text() function.
The pattern of what is reported can be modified with the pattern = argument.
Default is pattern = "{estimate} ({conf.level*100}% CI {conf.low}, {conf.high}; {p.value})"

Report Reproducbile Statistics with gtsummary::inline_text()

library(gtsummary)

tbl_uvreg <- sm_trial %>%
  tbl_uvregression(
    method = glm,
    y = response,
    method.args = list(family = binomial),
    exponentiate = TRUE
  ) %>%
  bold_labels()

tbl_uvreg

Characteristic	N	OR	95% CI	p-value
Chemotherapy Treatment	193
Drug A		—	—
Drug B		1.21	0.66, 2.24	0.5
Age	183	1.02	1.00, 1.04	0.10
Grade	193
I		—	—
II		0.95	0.45, 2.00	0.9
III		1.10	0.52, 2.29	0.8
Abbreviations: CI = Confidence Interval, OR = Odds Ratio

In Code: The odds ratio for age is ‘inline_text(tbl_uvreg, variable = age)’

In Report: The odds ratio for age is 1.02 (95% CI 1.00, 1.04; p=0.10)

{lubridate}

We work with a LOT of dates
{lubridate} helps parse dates from strings, and improves functional operations on date-times
Data cleaning training will cover this in more depth or see R for Data Science: https://r4ds.had.co.nz/dates-and-times.html

library(lubridate)

bday <- dmy("14/10/1940")
month(bday)

[1] 10

wday(bday, label = TRUE)

[1] Mon Levels: Sun < Mon < Tue < Wed < Thu < Fri < Sat

year(bday) <- 2016
wday(bday, label = TRUE)

[1] Fri Levels: Sun < Mon < Tue < Wed < Thu < Fri < Sat

Data Versioning

Five Pillars Of Reproducibility

Data Versioning

How data versions are managed is still highly depending on what service and data types you work with
For genomic or imaging data, try to use a standardized pipeline
For clinical data, try to establish a workflow with your collaborators.
Avoid making changes to excel yourself
Use the README to track

AI Best Practices

AI in Coding: Opportunities & Risks

Pros

Helps scaffold - Speeds up boilerplate code and complex code pipelines
Helps translate between R and Python
Debugging
Documentation - Generates documentation and inline comments
Code review

Cons

PHI risk
Hallucinations - AI confidently generates plausible but incorrect code
Reproducibility gap — if you can’t explain what the code does, you can’t defend the analysis
Outdated training data - limited knowledge of niche packages

Warning

MSK policy: Use only approved institutional tools. These are currently web & browser based only.

Tips for Coding with AI ⚠️

AI Tips: Stay Close to Code During Data QA

Know your data. AI cannot catch errors it doesn’t know to look for.

❌ Don’t : Let an LLM take over QA. Don’t let it write code on unchecked variables.

✅ Do : Check variable types and levels, missingness, and ranges yourself before handing off to an AI tool. Make sure you understand every variable needed to execute the analysis plan

AI Tips: Don’t Outsource Your Statistical Judgment

❌ Don’t : Let an LLM write a statistical plan for you.

✅ Do : Let an LLM clean up and format a statistical plan you’ve already written. Ask it to review it for gaps or inconsistencies, but make sure you understand and can defend every decision in the plan yourself.

AI Tips: Don’t Outsource Your Statistical Judgment

❌ Don’t : Let an LLM pick the most appropriate statistical test or models for your analysis.

✅ Do: Get help with syntax, but make the decisions yourself. If you don’t understand why a test or model is appropriate for your data and question, see if the LLM can help explain it to you in simple terms (I’ve had varying success with this last point).

AI Tips: Think Systematically

❌ Don’t : Tell an LLM to produce a figure based on a dataset you input.

✅ Do : Provide context on the structure of your data (which variables you want to use), and the specifications of the plot you would like to make. Then, ask it to generate readable and well-commented code to produce the plot, and explain the code to you line by line.

AI Tips: Think Systematically — Example

❌ Vague prompt:

“Here is my dataset. Make a survival curve.”

LLM has to guess variable names, event coding, grouping variable
Output may use wrong packages or defaults
Hard to verify or adapt the code you get back

✅ Specific prompt:

“I have a data frame trial with: ttdeath (time to death in months), death (1 = event, 0 = censored), trt (‘Drug A’ / ‘Drug B’). Using {survival} and {ggsurvfit}, write well-commented R code to plot Kaplan-Meier curves by treatment group with a risk table. Then explain each step.”

LLM produces code you can read and verify
Explanation helps you catch mistakes and learn
You stay in control of the analytical decisions

AI Tips: Don’t Let Your Code Base Get Away From You

❌ Don’t : Let AI take over your codebase without understanding and verifying every line of code it produces.

✅ Do : Use AI to scaffold, debug, and explain code. Use it to help organize, functionalize and modularize your code (eg. helping with the ‘do not repeat 3 times’ rule)

Warning

You are still responsible for understanding and maintaining this codebase. Stay in control — treat AI suggestions like code review, not ground truth.

Example: AI-Assisted Pipeline Review

Ask AI to review your code systematically — bugs, performance, and structure all at once

Tip

Prompt: “Review this pipeline for bugs, efficiency and parameter consistency across scripts. Suggest areas to speed up the code. List findings by priority and suggest specific fixes with file and line references.”

# Pipeline Review: run-pipeline.R and Related Scripts

## Findings

### Bugs / Errors

- Sequential bootstrap loop — parallel cluster is set up but never used
  Fix: Replace `for` loop in `run_bootstrap()` with `foreach %dopar%`

- `prep-data.R` uses hardcoded parameters duplicated from `run-pipeline.R`
  Fix: Source shared parameters from a single config file

- Pipeline assumes `data/model-data.RData` exists — fails with cryptic error if missing
  Fix: Check if file exists; source `prep-data.R` first if not

- Potential syntax bug in `preprocess-functions.R` line ~57: `[, , drop = FALSE]`
  Fix: Verify intent and replace with proper indexing

### Performance Improvements

1. Parallelize the bootstrap loop (biggest gain) — swap `for` → `foreach %dopar%`
2. Batch file writes — collect rows during loop, write once at the end
3. Replace `do.call(cbind, ...)` with `dplyr::bind_cols()` — faster for wide data

### Verification Plan

- Run local mode with NBOOT=5, confirm no errors end-to-end
- Confirm parallelized and sequential bootstrap give same AUC distributions (same seed)
- Unit test the indexing fix on a small data slice

Other AI Tips

✅ Do : Use AI to write documentation
✅ Do : Use AI for code review and improving code efficiency
✅ Do : Use context/project/skill files to specify style, preferred packages, compute constraints, etc.
✅ Do : Input cluster guidelines as context for cluster computing scripts
✅ Do : Use Plan mode and iterate several times for larger code refactors
✅ Do : Use it to build/organize larger pipelines, but make sure you understand which code/functions depend on which scripts, etc.

Thank You!!!

Questions?

Resources

{biostaR} - https://github.mskcc.org/pages/datadojo/biostatR/index.html
{gtsummary} - https://www.danieldsjoberg.com/gtsummary/
{bstfun} - https://www.danieldsjoberg.com/bstfun/index.html
Departmental Resource Guide - https://rconnect.mskcc.org/resource-guide/
Quarto Docs - https://quarto.org/docs/guide/
Quarto Blog Post by Alison Hill - https://www.apreshill.com/blog/2022-04-we-dont-talk-about-quarto/

Coding Workshop Part 1

Agenda

What is Reproducibility?

Why is Reproducibility Important?

Why is Reproducibility Important?

Journal Code-Sharing Requirements

Five Pillars Of Reproducibility

How Do We Ensure Our Code is Reproducible?

Project Setup

Good Project Setup First = Hours Saved Later

Project Setup: Anatomy of a Project

Compute Environment Control: Virtual Environments

Compute Environment Control: Avoid Absolute File Paths

Setup Version Control

GitHub & GitHub Enterprise

Set Up .gitignore Before Your First Commit

.gitignore Example

Writing Good Code

Analysis Plan First

Literate Programming

Reproducible Reporting

Reproducible Reporting

Quarto vs. Jupyter Notebooks

Quarto Features: Callouts and Comments

Quarto Features: Tabs

{gtsummary}

{gtsummary} overview

Basic tbl_summary()

Advanced Tips: tbl_uvregression() with formula

Advanced Tip: gtsummary Themes

Other Customizations

Report Reproducible Statistics with gtsummary::inline_text()

Report Reproducbile Statistics with gtsummary::inline_text()

{lubridate}

Data Versioning

Five Pillars Of Reproducibility

Data Versioning

AI Best Practices

AI in Coding: Opportunities & Risks

Tips for Coding with AI ⚠️

AI Tips: Stay Close to Code During Data QA

AI Tips: Don’t Outsource Your Statistical Judgment

AI Tips: Don’t Outsource Your Statistical Judgment

AI Tips: Think Systematically

AI Tips: Think Systematically — Example

AI Tips: Don’t Let Your Code Base Get Away From You

Example: AI-Assisted Pipeline Review

Other AI Tips

Thank You!!!

Resources

Set Up `.gitignore` Before Your First Commit

`.gitignore` Example