Evaluate LLMs: Test and Prove Significance Course

Lesson 1 • What LLM Evaluation Means

Defines evaluation in the context of language models and distinguishes it from general ML benchmarking. Anchors the chapter's scope and vocabulary.

Lesson 2 • Evaluation Lifecycle Overview

Maps evaluation activities across model development, deployment, and monitoring phases. Shows where each course topic fits in a real workflow.

Lesson 3 • Core Quality Dimensions

Introduces the primary axes—accuracy, fluency, safety, and utility—used to judge model outputs. Provides the taxonomy applied throughout the course.

Lesson 1 • Avoiding Evaluation Pitfalls

Identifies leakage, selection bias, and label noise as threats to test set validity. Equips learners to audit and remediate flawed test sets.

Lesson 2 • Sourcing and Curating Prompts

Teaches methods for collecting, filtering, and annotating prompts from real and synthetic sources. Directly feeds the test sets used in subsequent evaluation exercises.

Lesson 3 • Principles of Test Set Design

Covers sampling strategies, coverage requirements, and contamination risks. Establishes the quality bar for all test data created in later chapters.

Lesson 4 • Reference Answer Construction

Explains how to create gold-standard reference answers and when references are unnecessary. Connects to metric selection in the next chapter.

Lesson 1 • Metric Reliability and Validity

Examines correlation with human judgment, metric sensitivity, and systematic biases. Enables learners to justify metric choices to technical and non-technical audiences.

Lesson 2 • Semantic Similarity Metrics

Introduces embedding-based metrics such as BERTScore and semantic textual similarity measures. Shows when semantic metrics outperform lexical ones.

Lesson 3 • Combining Multiple Metrics

Teaches composite scoring, metric ensembles, and trade-off visualization. Prepares learners to build dashboards used in applied evaluation workflows.

Lesson 4 • Lexical Overlap Metrics

Covers BLEU, ROUGE, METEOR, and related n-gram metrics, including their assumptions and failure modes. Grounds learners in the most widely reported scores.

Lesson 1 • Annotator Recruitment and Training

Addresses sourcing, screening, and calibrating human annotators for LLM evaluation tasks. Directly impacts the reliability of judgments collected in later sections.

Lesson 2 • Preference and Ranking Studies

Explains pairwise preference, best-of-N ranking, and Elo-style rating systems for comparing models. Connects to model comparison workflows in Chapter 5.

Lesson 3 • Human Evaluation Study Design

Covers rating scales, evaluation protocols, and task framing for human judges. Establishes the methodological rigor required for credible human evaluation.

Lesson 4 • Inter-Annotator Agreement

Teaches Cohen's kappa, Krippendorff's alpha, and percent agreement for measuring rater consistency. Provides the statistical foundation for validating human evaluation data.

Lesson 1 • Principles of LLM-as-Judge

Explains the rationale, assumptions, and known failure modes of using LLMs to evaluate LLM outputs. Sets expectations for when this approach is and is not appropriate.

Lesson 2 • Bias Detection in LLM Judges

Identifies verbosity, self-preference, and positional biases specific to LLM judges. Extends human evaluation bias concepts from Chapter 4 to automated judges.

Lesson 3 • Calibrating Judge Models

Teaches alignment of judge scores to human ratings through calibration datasets and fine-tuning. Ensures judge outputs are trustworthy before deployment.

Lesson 4 • Prompt Engineering for Judges

Covers rubric-based prompts, chain-of-thought scoring, and reference-guided evaluation prompts. Directly determines the quality of automated judgments produced.

Lesson 1 • Multiple Comparisons and Corrections

Addresses Bonferroni, Benjamini-Hochberg, and family-wise error rate control when testing many models. Prevents false discovery in large-scale evaluation campaigns.

Lesson 2 • Effect Size and Practical Significance

Teaches Cohen's d, odds ratios, and relative improvement metrics to quantify the magnitude of differences. Bridges statistical and business significance for stakeholder reporting.

Lesson 3 • Parametric Significance Tests

Covers paired t-tests and ANOVA for comparing model scores under normality assumptions. Connects to metric data collected in Chapters 3 and 4.

Lesson 4 • Non-Parametric Significance Tests

Introduces Wilcoxon signed-rank, McNemar's test, and permutation tests for non-normal evaluation data. Expands the learner's toolkit beyond parametric assumptions.

Lesson 1 • Fairness Criteria and Trade-offs

Explains demographic parity, equalized odds, and calibration as competing fairness definitions. Equips learners to select and justify fairness criteria for specific use cases.

Lesson 2 • Demographic Bias Measurement

Teaches counterfactual data augmentation, stereotype benchmarks, and disparity metrics for detecting bias. Connects harm taxonomy to quantitative bias measurement.

Lesson 3 • Red-Teaming Methodologies

Covers structured adversarial probing, jailbreak testing, and automated red-teaming pipelines. Builds practical skills for uncovering model vulnerabilities before deployment.

Lesson 4 • Safety Evaluation Reporting

Structures safety findings into model cards, risk registers, and executive summaries. Translates technical safety results into actionable guidance for decision-makers.

Lesson 1 • Continuous Evaluation and Monitoring

Designs ongoing evaluation loops for detecting model drift, data shift, and emerging failure modes. Extends one-time evaluation into a sustainable operational practice.

Lesson 2 • Communicating Evaluation Results

Teaches visualization, narrative framing, and stakeholder-specific reporting for evaluation outcomes. Ensures findings drive decisions rather than sit in technical reports.

Lesson 3 • Evaluation Infrastructure and Tooling

Covers experiment tracking, result storage, and reproducibility tooling for evaluation pipelines. Ensures evaluation workflows are scalable and auditable.

Lesson 4 • Selecting and Combining Methods

Provides a decision framework for choosing among automated, human, and LLM-as-judge methods. Synthesizes all prior chapters into a unified method-selection process.