Understanding Information Theory Course

Lesson 1 • Probability Review for Information Theory

Refreshes essential probability concepts needed to derive information measures. Connects random variables and distributions to uncertainty quantification.

Lesson 2 • Surprise and Self-Information

Defines self-information as the surprise of a single outcome. Motivates the logarithmic measure and its intuitive properties.

Lesson 3 • What Is Information Theory

Traces the origins of information theory from communication engineering to modern data science. Establishes why quantifying information matters across disciplines.

Lesson 4 • Axiomatic Foundations of Entropy

Presents Shannon's axioms that uniquely characterize entropy. Reinforces why entropy is the correct measure of uncertainty.

Lesson 1 • KL Divergence and Relative Entropy

Measures the cost of assuming the wrong distribution. Students apply KL divergence to model comparison and hypothesis testing.

Lesson 2 • Information Diagrams and Relationships

Visualizes relationships among entropy, mutual information, and divergence using Venn-style diagrams. Consolidates all measures into a unified framework.

Lesson 3 • Cross-Entropy and Its Applications

Defines cross-entropy as the average code length under a mismatched model. Links cross-entropy to KL divergence and loss functions in machine learning.

Lesson 4 • Joint and Conditional Entropy

Defines entropy for pairs of variables and the residual uncertainty given side information. Builds the chain rule for entropy.

Lesson 1 • Codes and Code Properties

Introduces symbol codes, uniquely decodable codes, and prefix-free codes. Establishes the Kraft inequality as a necessary condition for efficiency.

Lesson 2 • Dictionary-Based and Universal Coding

Covers LZ-family algorithms that adapt to unknown source statistics. Connects universal coding to the concept of entropy rate for stationary sources.

Lesson 3 • Arithmetic Coding

Encodes entire messages as intervals to approach entropy more closely than Huffman coding. Covers encoding, decoding, and precision issues.

Lesson 4 • Shannon's Source Coding Theorem

Proves that entropy is the fundamental limit of lossless compression. Students interpret the theorem's achievability and converse.

Lesson 1 • Channel Capacity Computation Methods

Applies the Blahut-Arimoto algorithm to compute capacity numerically. Students iterate the algorithm and verify convergence on example channels.

Lesson 2 • Gaussian Channels and Bandwidth

Extends capacity analysis to additive white Gaussian noise channels. Derives the Shannon-Hartley formula relating bandwidth, power, and capacity.

Lesson 3 • Discrete Memoryless Channels

Defines the discrete memoryless channel via transition probability matrices. Students compute output distributions and identify channel symmetry.

Lesson 4 • Shannon's Channel Coding Theorem

States that reliable communication is possible at any rate below capacity. Covers achievability via random coding and the converse argument.

Lesson 1 • Convolutional Codes and Viterbi Decoding

Encodes streams using shift-register circuits and decodes with the Viterbi algorithm. Students trace trellis diagrams to find maximum-likelihood paths.

Lesson 2 • Cyclic Codes and Reed-Solomon Codes

Exploits algebraic structure for efficient encoding and burst-error correction. Covers polynomial representation and Reed-Solomon applications.

Lesson 3 • Modern Codes: Turbo and LDPC

Introduces capacity-approaching codes and iterative belief-propagation decoding. Connects modern code performance to Shannon limits.

Lesson 4 • Error Detection and Correction Basics

Introduces Hamming distance, error detection, and correction capabilities. Connects code parameters to the channel noise model.

Lesson 1 • Rate-Distortion Function

Defines the rate-distortion function as the minimum rate for a given distortion level. Students derive it for Gaussian and binary sources.

Lesson 2 • Quantization Theory

Applies rate-distortion principles to scalar and vector quantization design. Students analyze quantization noise and high-resolution approximations.

Lesson 3 • Practical Lossy Compression Standards

Connects rate-distortion theory to transform coding used in audio and image compression. Analyzes how standards approach theoretical limits.

Lesson 4 • Shannon's Rate-Distortion Theorem

Proves that any rate above the rate-distortion function is achievable. Covers the converse showing rates below the function are impossible.

Lesson 1 • Mutual Information in Feature Selection

Uses mutual information to rank and select informative features for classification. Students apply information gain and compare it to correlation-based methods.

Lesson 2 • Variational Inference and Information Theory

Frames variational inference as KL divergence minimization. Students connect the evidence lower bound to rate-distortion and mutual information.

Lesson 3 • Minimum Description Length Principle

Frames model selection as a compression problem using the MDL principle. Connects MDL to Bayesian model comparison and Occam's razor.

Lesson 4 • Hypothesis Testing and Divergence

Links KL divergence to error exponents in binary hypothesis testing. Students derive the Chernoff-Stein lemma and interpret its operational meaning.

Lesson 1 • Information-Theoretic Security

Defines perfect secrecy and the wiretap channel model. Students compute secrecy capacity and compare information-theoretic to computational security.

Lesson 2 • Information Theory in Neuroscience

Applies entropy and mutual information to neural coding and sensory systems. Students analyze spike train data using information-theoretic metrics.

Lesson 3 • Emerging Frontiers in Information Theory

Surveys active research areas including coded distributed computing and semantic communication. Students identify open problems and research directions.

Lesson 4 • Network Information Theory

Extends single-channel results to multi-user networks including broadcast and multiple-access channels. Students compute capacity regions for two-user cases.