Understanding Uncertainty · Roadmap

Roadmap for this course and Roadmap within SDS

Why are we teaching these concepts in an MSDS program?

Understanding Uncertainty · Fall 2026

Underlined concepts open a concept box when you hover or tap them: its definition, why you'll use it, where you'll see it, and an example.

Connections to other classes

As a data scientist, you will generally do these steps:

Have a question you want to answer
Gather data
Put that data somewhere
Make a guess about the underlying population based on your data (your sample) [estimation]
Figure out how accurate you can expect your guess to be [inference]
Predict things based on the more accurate estimates [predictive modeling, ML]
Sometimes we will go back and check our model [inference]
Present these predictions to someone who has more responsibilities than you, or shareholders, or make a decision yourself based on the predictions; or deploying the model for use by others

This process goes wrong if:

You don’t have a good question or aren’t asking the right question
You gather data in a biased way that doesn’t actually address your question or is skewed from the true population
You store the data improperly
You didn’t find a good estimate (it’s biased, it doesn’t fit the data you were given, etc)
Your estimate has a lot of uncertainty (variance)
Your predictive model is based on these estimates that are not accurate or certain
You misrepresent the results of the model (for example, by being overconfident in your conclusions and predictions).

Real companies have lost money at every step. Successful companies do at least one of these steps really well. Data scientists who miss a few steps will make bad predictions, mislead people, lose money, and not truly learn about the world. Data scientists who can do all of the steps (or some of them really well, while understanding the others) are successful.

In UU, we are most interested in steps 4, 5, and 7. Of course, the interplay between inference and predictive modeling will relate what you are learning in ML and this class—sometimes you will do inference first and model after you have quantified your uncertainty; sometimes you will do predictive modeling first and then check how certain you are about the parameters in the model. But to be effective as a data scientist, you need to be able to do both. Predictive modeling without inference leads to overconfidence in our models [see Zillow Offers iBuying program] and not understanding the why behind the model. Inference without predictive modeling feels less satisfying because you may understand something about the world better (or understand that you don’t understand), but you haven’t done anything with that information. There are data scientists that do entirely one side. Our goal is to get you to do, or at least appreciate, both.

Inference requires a deep understanding of probability and statistics. There is no way around that. Predictive modeling requires a deep understanding of machine learning algorithms (and therefore, more indirectly, probability and statistics).

Roadmap for this course

How are all of the concepts in UU connected? The course has two main halves. The first half is building up your statistical toolkit. These ideas build on each other and relate, and they will keep coming back over and over. The early statistical ideas we will start with are important for both frequentist and Bayesian approaches (for example, everyone agrees on Bayes’ rule), but we will quickly dive into frequentist ideas for non-parametric inference and parametric inference. This includes sampling distributions and resampling our data (bootstrapping) to calculate how uncertain our estimates are. Then we'll move to Fisherian inference, a subset of frequentism. This includes hypothesis testing and confidence intervals. We'll detour into Markov chains to understand how conditional probabilities work in practice and can model temporal data. Then, we'll focus on several implementations of another Fisherian approach that needs conditional probabilities: maximum likelihood estimation, which is a key method for fitting statistical models. When you fit a statistical model with Python, many standard packages will solve an MLE problem under the hood (including statsmodels, scipy, and scikit-learn). Finally, we'll end the course with a survey of Bayesian inference methods, and use PyMC (a package relying on Markov Chain Monte Carlo) to implement Bayesian inference.

Click a box to jump to that section.

Probability key concepts

Basic set theory
Kolmogorov’s axioms
Probability spaces, events, and outcomes
Independence
Conditional probability
Bayes’ theorem

Linear algebra key concepts

Vectors
Matrices
Dot Product
Cosine Similarity
Correlation
Orthogonality
Covariance

Data wrangling

One-hot encoding
Type casting
Missing values
Groupby
Filtering

Representing distributions and densities

PMF
PDF
KDE
ECDF
CDF
iid
binomial variables & bernoulli trials
indicator function

Estimation

Sampling and the bootstrap

Example: the bootstrap is …

Standard errors

Example: a standard error is …

The CLT

Confidence intervals & hypothesis tests

hypothesis testing
confidence intervals

Conditional probability/expectation

Likelihood

Markov chains

Markov chains

Maximum likelihood

maximum likelihood estimation

Example: the likelihood is …

Bayesian inference

Bayesian inference

Example: the posterior is …

learning something about your data when you don’t want to make assumptions

This gives us the tools to talk about when two or more events happen together (A and B, the intersection), when at least one of them happens (A or B, the union), and when an event does not happen (not A, the complement). A set is a collection of outcomes, and these three operations combine sets into new ones.

How do we talk about two sets of outcomes (or more)?

Independence, conditional probability, the probability axioms

P(A and B) = P(A) × P(B) if A and B are independent

Fancy name for three rules of probability: every probability is non-negative; the probability of anything happening at all is 1; and the probability of A, B, C, etc., if A, B, and C don’t overlap, is equal to the sum of the individual probabilities. A result of these is the complement rule: the probabilities of A happening and A not happening add to 1.

everyone agrees on these axioms as basic rules of probability.

indirectly, when we calculate any probabilities.

a probability space is the full setup for a random experiment, made of three pieces: the sample space, which is the set of every possible outcome (for one die roll, {1, 2, 3, 4, 5, 6}); the events, which are the collections of outcomes we might ask about (“roll an even number” is the event {2, 4, 6}); and a probability for each event, assigned so that Kolmogorov’s axioms hold. An outcome is one single result of the experiment; an event is a set of outcomes.

When you get a random variable, you need a probability space to assign probabilities to its values.

After that, we can sort of pretend the probability space doesn't exist anymore, but it's in the background of any problem you do.

Two events A and B are independent when knowing that one of them happened does not change the probability of the other. In symbols, \(P(A \text{ and } B) = P(A)\,P(B)\): the probability that both happen is the product of their separate probabilities.

The probability that A happens given that we already know B happened, written \(P(A \mid B)\) and read “the probability of A given B.” Knowing B happened shrinks the set of possible outcomes down to B, so we ask what fraction of B is also A: \(P(A \mid B) = P(A \text{ and } B)\,/\,P(B)\), as long as \(P(B) > 0\).

A rule for flipping a conditional probability around. It turns \(P(B \mid A)\), which is often easy to know, into \(P(A \mid B)\), which is often what we want: \(P(A \mid B) = P(B \mid A)\,P(A)\,/\,P(B)\). It follows from writing the definition of conditional probability both ways, since \(P(A \text{ and } B) = P(A \mid B)\,P(B) = P(B \mid A)\,P(A)\).

You only ever get one dataset, so you only ever get one estimate. A frequentist asks: if I could go back and collect the data again, and again, and again, how much would my estimate jump around? Each rerun would give a slightly different number. How spread out those numbers are (and whether they center on the truth) is how a frequentist says how much to trust the one estimate you actually have. That’s where bias, variance, and standard errors come from. In CASI’s words, frequentists ask “What would I see if I reran the same situation again?” (Efron & Hastie, Computer Age Statistical Inference, Ch. 2.)