Markov Chain Course

Lesson 1 • Introduction to Stochastic Processes

Defines stochastic processes as indexed families of random variables. Positions Markov chains as a special, tractable subclass.

Lesson 2 • Random Variables and Distributions

Introduces discrete and continuous random variables with key distributions. Connects distributional thinking to state-based modelling.

Lesson 3 • Core Probability Concepts Review

Covers sample spaces, events, and probability axioms. Establishes notation and reasoning skills used throughout the course.

Lesson 1 • Chapman-Kolmogorov Equations

Derives the Chapman-Kolmogorov equations and applies them to multi-step transition probabilities. Connects matrix multiplication to probabilistic reasoning.

Lesson 2 • State Spaces and Transition Probabilities

Defines finite and countably infinite state spaces and one-step transition probabilities. Builds intuition for how chains move between states.

Lesson 3 • The Markov Property

States the memoryless condition formally and contrasts it with non-Markovian dependence. Grounds the property in conditional probability notation.

Lesson 4 • Initial Distributions and Chain Evolution

Combines initial state distributions with transition matrices to compute multi-step distributions. Introduces matrix exponentiation for chain evolution.

Lesson 1 • Ergodic Chains

Defines ergodic chains as irreducible, positive recurrent, and aperiodic. Establishes conditions guaranteeing unique stationary distributions.

Lesson 2 • Periodicity of States

Defines the period of a state and identifies aperiodic chains. Explains how periodicity affects convergence to steady state.

Lesson 3 • Communicating States and Classes

Defines accessibility and communication between states and partitions chains into communicating classes. Identifies irreducible chains.

Lesson 4 • Absorbing States and Chains

Identifies absorbing states and analyses chains with absorbing barriers. Prepares students for absorption time and probability calculations.

Lesson 1 • Long-Run Averages and Ergodic Theorem

Applies the ergodic theorem to compute time-average quantities from stationary distributions. Connects theory to simulation-based estimation.

Lesson 2 • Stationary Distribution Definition

Defines stationary distributions as fixed points of the transition operator. Derives the balance equations used for computation.

Lesson 3 • Detailed Balance and Reversibility

Introduces detailed balance equations as a sufficient condition for stationarity. Identifies reversible chains and their computational advantages.

Lesson 4 • Computing Stationary Distributions

Applies linear algebra techniques to solve balance equations for finite chains. Covers both direct and iterative solution methods.

Lesson 1 • Expected Absorption Times

Derives expected time to absorption using the fundamental matrix. Applies results to gambler's ruin and inventory models.

Lesson 2 • Occupation Times and Visit Counts

Computes expected number of visits to each state before absorption. Connects occupation measures to reward and cost modelling.

Lesson 3 • Absorption Probabilities

Computes the probability of being absorbed into each absorbing state from any transient state. Applies the fundamental matrix of absorbing chains.

Lesson 4 • First Passage Time Analysis

Defines first passage times and derives their distributions using first-step analysis. Connects passage times to recurrence classification.

Lesson 1 • Generator Matrix and Kolmogorov Equations

Defines the infinitesimal generator matrix Q and derives forward and backward Kolmogorov equations. Connects Q to transition rate structure.

Lesson 2 • Exponential Holding Times

Establishes the exponential distribution as the unique memoryless continuous distribution. Derives holding time properties for continuous-time chains.

Lesson 3 • Uniformisation and Embedded Chains

Introduces uniformisation to convert continuous-time chains to discrete-time equivalents. Enables numerical computation of transition probabilities.

Lesson 4 • Birth-Death Chains

Analyses birth-death chains as a tractable continuous-time subclass. Derives closed-form stationary distributions using detailed balance.

Lesson 1 • Inventory and Supply Chain Models

Formulates inventory level dynamics as a Markov chain with demand and replenishment transitions. Optimises reorder policies using stationary analysis.

Lesson 2 • Markov Models in Finance

Applies Markov chains to credit rating migration and regime-switching models. Computes default probabilities and expected portfolio transitions.

Lesson 3 • Biological and Epidemiological Models

Models population dynamics and disease spread using Markov chains. Analyses endemic equilibria and extinction probabilities.

Lesson 4 • Queuing Systems as Markov Chains

Models single-server and multi-server queues as birth-death chains. Derives performance metrics including throughput, utilisation, and mean queue length.

Lesson 1 • Metropolis-Hastings Algorithm

Derives the Metropolis-Hastings acceptance rule from detailed balance. Implements the algorithm and analyses proposal distribution choices.

Lesson 2 • MCMC Foundations and Motivation

Explains why MCMC uses Markov chains to sample from complex distributions. Connects stationary distribution theory to sampling correctness.

Lesson 3 • Gibbs Sampling

Introduces Gibbs sampling as a special case of Metropolis-Hastings with unit acceptance. Applies it to multivariate posterior distributions.

Lesson 4 • Advanced MCMC Techniques

Surveys Hamiltonian Monte Carlo and parallel tempering for challenging posteriors. Compares efficiency and applicability across methods.