DS 5030 is a graduate course in the UVA School of Data Science that picks up the threads of students’ math and probability education and builds a foundation for later work in machine learning and AI, grounded in data. Topics include linear algebra for data science, random variables and expectation, representing distributions (ECDFs, CDFs/PDFs, kernel density estimates), order statistics and extreme events, sampling distributions, the bootstrap and the CLT, joint and conditional distributions, conditional expectation, Markov chains, maximum likelihood models (failure times, counts, regression, and binary outcomes), and Bayesian inference.

The second half of the course builds standard modeling frameworks (Markov chains, maximum likelihood, Bayes), but every lab follows the same arc: explore the data non-parametrically first, then estimate the model, then do inference on its parameters, then predict. The bootstrap from Week 6 comes back in every assignment as the default way to measure how precisely a parameter or prediction is estimated.

Each week has a Tuesday math day (concept-driven lecture and quiz) and a Thursday lab day (a pre-lab bridge notebook, then a coding lab that starts in class). Students fork a template repository and pull new lecture notebooks, labs, and datasets each week; the datasets for every notebook below are in that repo.

Course roadmap

• Roadmap with UU and Roadmap at SDS

Getting set up

• Pre-Class Video: Environments and VS Code [Video]
• Setup checklist: Anaconda, VS Code, Git, and forking the course repo [PDF]
• Day 1 Python test notebook [Notebook] [.ipynb]

Week 1: Introduction

• Pre-Class Video: Git and GitHub basics [Video] [Notebook] [.ipynb]
• Pre-Class Video: Data wrangling basics [Video] [Notebook] [.ipynb]
• Lecture: Introduction to the course [Slides]
• Lecture: Introduction to Python and GitHub [Slides]
• Pre-Lab Bridge: Exploratory data analysis [Notebook] [.ipynb] [In-class version] [.ipynb]
  • Interactive: What the sample mean minimizes
  • Interactive: Why the sample variance undershoots (Bessel’s correction)
• Lab 01: Data wrangling and EDA [Notebook] [.ipynb] [Dataset catalog] [Data download script]

Week 2: Linear algebra

• Pre-Class Video: Vectors and matrices [Video] [Slides] [Notebook] [.ipynb] [Basics notebook] [.ipynb]
• Lecture: Inner product and orthogonality [Slides] [Notebook] [.ipynb]
• Problem Set 1: Vectors and matrices [Notebook] [.ipynb] [Optional linear algebra references]
• Pre-Lab Bridge: From inner product to cosine similarity [Notebook] [.ipynb] [In-class version] [.ipynb]
  • Interactive: Cosine similarity clock
• Lab 02: Spotify song recommendations with vector similarity [Notebook] [.ipynb] [In-class version] [.ipynb]

Week 3: Probability

• Pre-Class Video: Probability basics [Video] [Slides] [Notebook] [.ipynb] [Basics notebook] [.ipynb]
• Lecture: Random variables and expectation [Slides] [Notebook] [.ipynb]
• Problem Set 2: Probability [PDF]
• Pre-Lab Bridge: Variance of dice rolls, Bayes’ rule, and the base rate fallacy [Notebook] [.ipynb]
• Lab 03: Differential diagnosis and conditional probability with emergency department visits [Notebook] [.ipynb] [Codebook]

Week 4: Representing distributions

• Pre-Class Video: Random number generation [Video] [Notebook] [.ipynb]
• Lecture: Categorical variables and the ECDF [Slides] [Notebook] [.ipynb]
• Lecture: Expectations and densities: the CDF and PDF [Slides] [Notebook] [.ipynb]
• Problem Set 3: Categorical variables and ECDFs [PDF]
• Lab 04: Generating CDFs from simulations [PDF]
  • Interactive: Generating CDFs

Week 5: Density estimation and order statistics

• Lecture: Kernel density estimation [Notebook] [.ipynb]
  • Interactive: Density vs. cumulative
• Lecture: Transcendentals and the Gaussian function [Notebook] [.ipynb]
• Problem Set 4: Kernel density estimation [Problems]
• Lab 05: Order statistics and extreme events: distributions of the sample maximum and minimum in market returns and rainfall [Notebook] [.ipynb]

Week 6: Sampling, bootstrapping, and the CLT

• Pre-Class (Optional): Sampling distributions and estimator properties
  • Interactive: One sample, one point [Skewed population] [Estimating variance]
  • Interactive: How bad is my estimator? (Standard error)
  • Interactive: Unbiased, but never consistent
• Lecture: Bootstrapping [Notebook] [.ipynb] [Bootstrap game]
• Pre-Lab Bridge: A/B testing with the bootstrap: D.A.R.E. and Scared Straight [Notebook] [.ipynb]
• Lab 06: A/B testing and randomized controlled trials (ISCHEMIA-CKD and intra-aortic balloon pump trials) [Notebook] [.ipynb]

Week 7: Midterm

• Midterm study guide [PDF] [Key]

Week 8: Standard errors, the CLT, and conditional distributions

• Lecture: From the bootstrap to asymptotic inference: standard errors, the CLT, confidence intervals, and tests [Slides]
• Background Resources, Claude-generated: Standard errors, the CLT, confidence intervals, and hypothesis tests (built from Wasserman’s All of Statistics and Efron & Hastie’s Computer Age Statistical Inference) [Study guide]
  • Interactive (go deeper): §1 Sampling distributions · §2 LLN, CLT, delta method · §3 Five ways to get an SE · §4 Is the SE uncertain? · §5 Confidence intervals · §6 Tests and p-values · §7 Many tests · §8 Regression inference · §9 When it breaks
• Tuesday: Conditional densities and distributions from joint and marginal distributions (cross tabs, then densities) · Problem set
• Thursday: Conditional expectation: regression as the conditional expectation function, local constant least squares · Lab: TBA
  • Pre-Lab: Confidence intervals and testing (Day 2) [Slides]
• Lab 07: Pricing under uncertainty: market tests, standard errors, and confidence intervals for the optimal price (run in Colab) [Notebook] [.ipynb] [Open in Colab]
• In class challenge: reading uncertainty plots: vote true or false on each plot [In class challenge]

Coming up (tentative)

Materials will be linked here as each week is posted. Lab topics marked TBA are still being finalized.

Week 9: Markov chains

• Tuesday: Markov transition matrices · Lab: NYC taxicabs, estimating the transition matrix
  • Interactive: Markov chain walk: type in a transition matrix or joint table and watch a random walk
• Thursday: Simulating a Markov chain · Lab: NYC taxicabs, simulating a cab

Week 10: Forecasting with Markov chains

• Tuesday: Forecasting a Markov chain · Lab: NYC taxicabs, expected fares and travel times
• Thursday: Conditional expectation with numeric state spaces · Lab: a second Markov chain application (TBA)

Week 11: Maximum likelihood

• Tuesday: Introduction to maximum likelihood estimation (Normal, Lognormal) · Problem set
• Thursday: Failure models (Exponential, Erlang) · Lab: jet engine replacement

Week 12: Count data and parameterized models

• Tuesday: Working with count data (Poisson, Negative Binomial) · Lab: soccer goals and basketball field goals
• Thursday: Parameterized MLE models: replacing a distribution’s parameters with functions of the data (OLS as a Normal likelihood, Poisson regression) · Lab: imputation

Week 13: Binary outcomes and standard errors

• Tuesday: Binary outcomes (logistic regression, multinomial logit) · Lab: consumer demand
• Thursday: Standard errors from MLE models · Problem set

Week 14: Bayes’ theorem

• Tuesday: Bayes’ theorem · Lab: naive Bayes classifier
• Thursday: No class (Thanksgiving)

Week 15: Bayesian inference

• Tuesday: Conjugate priors · Problem set
• Thursday: Regression and posterior sampling with PyMC · Lab: comparing MLE and Bayesian estimates

Week 16: Wrap-up

• Tuesday: Bayesian credible intervals and prediction intervals