LLM Benchmarking and Evaluation Training

Lesson 1 • Evaluation Lifecycle Overview

Traces evaluation from research prototyping through production monitoring. Shows how each lifecycle stage demands different methods covered later.

Lesson 2 • What LLM Evaluation Means

Defines evaluation in the context of generative AI and distinguishes it from traditional ML metrics. Anchors the chapter by establishing shared terminology.

Lesson 3 • Taxonomy of LLM Capabilities

Maps the landscape of skills LLMs are evaluated on, from language understanding to reasoning. Provides a reference taxonomy used throughout the course.

Lesson 1 • Static vs. Dynamic Benchmarks

Contrasts fixed dataset benchmarks with adaptive and evolving ones. Explains trade-offs in stability, contamination risk, and coverage.

Lesson 2 • Domain-Specific Benchmarks

Examines benchmarks targeting specialized domains such as medicine, law, and code. Highlights domain-specific validity challenges.

Lesson 3 • Task Format Taxonomy

Categorizes benchmarks by task format: multiple choice, open generation, ranking, and structured output. Links format choice to scoring feasibility.

Lesson 4 • Benchmark Suites and Aggregation

Explores composite benchmark suites that aggregate multiple tasks into a single score. Discusses aggregation methods and their interpretive limits.

Lesson 1 • Metric Reliability and Correlation

Evaluates how well automated metrics correlate with human judgments and where they diverge. Prepares students to justify metric choices to stakeholders.

Lesson 2 • Reference-Based Text Metrics

Covers n-gram overlap metrics such as BLEU, ROUGE, and METEOR and their statistical foundations. Establishes baseline scoring knowledge for generation tasks.

Lesson 3 • Perplexity and Likelihood Metrics

Explains perplexity as a measure of language model fluency and its relationship to cross-entropy loss. Connects perplexity to benchmark design for language modeling tasks.

Lesson 4 • Embedding-Based Similarity Metrics

Introduces semantic similarity metrics including BERTScore and cosine similarity over dense embeddings. Explains when semantic metrics outperform lexical ones.

Lesson 1 • Rater Recruitment and Training

Addresses sourcing expert vs. crowd raters, onboarding, and quality control. Connects rater quality to downstream evaluation validity.

Lesson 2 • Human Evaluation Study Design

Covers experimental design choices: rating scales, comparison paradigms, and task framing. Grounds study design in the evaluation goals established earlier.

Lesson 3 • Annotation Guideline Development

Teaches how to write clear, unambiguous annotation guidelines that reduce rater variance. Directly impacts the reliability of human evaluation data.

Lesson 4 • Bias and Cognitive Effects in Rating

Identifies anchoring, order effects, and verbosity bias that distort human ratings. Provides mitigation strategies to improve evaluation integrity.

Lesson 1 • Multi-Judge and Ensemble Approaches

Explores using multiple judge models or prompts to reduce variance and increase robustness. Extends single-judge methods to production-grade pipelines.

Lesson 2 • Prompt Engineering for Judges

Covers techniques for crafting judge prompts that elicit consistent, calibrated scores. Directly affects the reliability of LLM-as-judge pipelines.

Lesson 3 • Bias in LLM Judges

Catalogs known biases: self-preference, verbosity, and position bias in pairwise judging. Teaches detection and mitigation strategies.

Lesson 4 • Principles of LLM-as-Judge

Explains the rationale for using LLMs as judges and the conditions under which it is valid. Builds on human evaluation concepts to frame automated judging.

Lesson 1 • Alignment Evaluation Frameworks

Introduces frameworks for measuring instruction following, value alignment, and refusal behavior. Connects alignment metrics to deployment readiness decisions.

Lesson 2 • Defining Safety in LLM Outputs

Establishes a working taxonomy of harmful output categories including toxicity, misinformation, and privacy violations. Frames safety as a measurable evaluation dimension.

Lesson 3 • Bias and Fairness Benchmarks

Surveys benchmarks that measure demographic, occupational, and representational bias. Explains how bias metrics connect to fairness definitions.

Lesson 4 • Hallucination Detection and Measurement

Covers methods for detecting factual errors and unsupported claims in model outputs. Provides quantitative frameworks for hallucination rate estimation.

Lesson 1 • Data Collection and Sourcing

Covers sourcing strategies: web scraping, expert authoring, synthetic generation, and licensing. Addresses data quality and provenance requirements.

Lesson 2 • Contamination Prevention and Detection

Addresses the risk of benchmark items appearing in model training data and methods to detect and prevent it. Critical for maintaining benchmark validity over time.

Lesson 3 • Requirements and Scope Definition

Translates evaluation goals into concrete benchmark requirements including task types and coverage targets. Establishes the design contract for subsequent construction steps.

Lesson 4 • Item Writing and Quality Control

Teaches principles of writing unambiguous, discriminative benchmark items with clear answer keys. Quality control processes ensure item validity before release.

Lesson 1 • Designing an Evaluation Plan

Guides students through scoping, method selection, and resource allocation for a complete evaluation project. Integrates all prior methods into a unified workflow.

Lesson 2 • Continuous Evaluation and Monitoring

Establishes frameworks for ongoing post-deployment evaluation including drift detection and regression testing. Closes the evaluation lifecycle loop.

Lesson 3 • Evaluation Result Visualization

Teaches chart types and dashboards suited to communicating benchmark results clearly. Connects visualization choices to audience and decision context.

Lesson 4 • Comparative Model Evaluation

Covers head-to-head model comparison including statistical significance testing and effect size reporting. Enables defensible model selection decisions.