Evaluate and Optimize LLM Performance Course

Lesson 1 • Evaluation Goals and Stakeholders

Maps evaluation objectives to business, research, and compliance stakeholder needs. Clarifies how goal alignment shapes metric selection and reporting.

Lesson 2 • Taxonomy of Evaluation Methods

Surveys human, automated, and hybrid evaluation approaches at a high level. Sets the stage for deep dives in later chapters.

Lesson 3 • How LLMs Generate Text

Covers token prediction, sampling strategies, and temperature effects on output variability. Grounds all subsequent evaluation work in model behavior mechanics.

Lesson 1 • Annotation and Labeling Workflows

Covers annotation guidelines, inter-annotator agreement, and quality control processes. Produces labeled ground-truth data required for supervised evaluation metrics.

Lesson 2 • Dataset Versioning and Governance

Establishes practices for versioning, documenting, and auditing evaluation datasets over time. Ensures reproducibility and traceability across model iterations.

Lesson 3 • Adversarial and Stress Testing

Introduces prompt injection, jailbreak attempts, and boundary-condition inputs as stress tests. Prepares students to expose model vulnerabilities systematically.

Lesson 4 • Input Diversity and Edge Cases

Techniques for generating diverse prompts, including rare domains, ambiguous queries, and multilingual inputs. Ensures evaluation surfaces real-world failure modes.

Lesson 1 • Reference-Free and Model-Based Metrics

Explores perplexity, coherence scores, and LLM-as-judge approaches that require no gold reference. Highlights when reference-free evaluation is necessary.

Lesson 2 • Metric Aggregation and Reporting

Teaches composite scoring, confidence intervals, and statistical significance testing for metric results. Enables defensible, reproducible performance reporting.

Lesson 3 • Lexical Overlap Metrics

Covers BLEU, ROUGE, METEOR, and CIDEr as n-gram-based similarity measures. Explains their assumptions, strengths, and well-documented limitations.

Lesson 4 • Task-Specific Automated Metrics

Covers accuracy, F1, exact match, and code execution pass rates for classification, QA, and coding tasks. Aligns metric choice to task structure.

Lesson 1 • Annotator Recruitment and Training

Addresses annotator qualification criteria, onboarding, and calibration exercises. Reduces variance introduced by annotator background and interpretation differences.

Lesson 2 • Inter-Rater Reliability Analysis

Applies Cohen's kappa, Krippendorff's alpha, and Fleiss' kappa to measure annotator agreement. Guides decisions on when to adjudicate or discard disagreements.

Lesson 3 • Human Evaluation Study Design

Covers rating scales, pairwise comparison, and ranking paradigms for collecting human judgments. Connects study design choices to statistical power and reliability.

Lesson 4 • Scaling Human Evaluation Efficiently

Covers active learning sampling, stratified annotation, and hybrid human-automated pipelines to reduce cost. Maintains quality while scaling to large output sets.

Lesson 1 • Bias and Fairness Failures

Covers demographic, representational, and stereotyping biases detectable in model outputs. Connects bias diagnosis to downstream harm and mitigation planning.

Lesson 2 • Instruction-Following Failures

Analyzes cases where models ignore constraints, format requirements, or task specifications. Develops targeted test suites to measure instruction adherence.

Lesson 3 • Root Cause Attribution

Maps observed failures to training data, prompt design, or decoding parameter causes. Enables targeted remediation rather than trial-and-error fixes.

Lesson 4 • Reasoning and Logic Errors

Identifies multi-step reasoning failures, arithmetic errors, and logical inconsistencies in LLM outputs. Applies chain-of-thought probing to surface reasoning breakdowns.

Lesson 1 • Prompt Sensitivity and Robustness

Measures how small prompt variations cause large output changes and applies robustness testing. Produces stable prompts that perform consistently across paraphrases.

Lesson 2 • Systematic Prompt Experimentation

Applies A/B testing, factorial design, and automated prompt search to optimize prompt variables. Connects experimentation rigor to reproducible performance improvements.

Lesson 3 • Prompt Structure and Anatomy

Breaks down system instructions, context, examples, and output format directives as prompt components. Establishes a structured template for consistent prompt construction.

Lesson 4 • Few-Shot and Chain-of-Thought Prompting

Covers example selection, ordering effects, and reasoning elicitation through chain-of-thought patterns. Demonstrates measurable accuracy gains on complex tasks.

Lesson 1 • Post-Optimization Evaluation

Applies pre/post evaluation comparisons, regression testing, and capability preservation checks after optimization. Ensures improvements in target areas do not degrade other capabilities.

Lesson 2 • Reinforcement Learning from Human Feedback

Explains reward model training, PPO optimization, and preference data collection for RLHF pipelines. Aligns model behavior with human quality judgments at scale.

Lesson 3 • Parameter-Efficient Fine-Tuning Methods

Introduces LoRA, prefix tuning, and adapter layers as low-cost alternatives to full fine-tuning. Enables optimization under compute and data constraints.

Lesson 4 • Direct Preference Optimization

Covers DPO as a simpler alternative to RLHF that uses preference pairs without a separate reward model. Compares DPO and RLHF on stability, cost, and output quality.

Lesson 1 • Data and Concept Drift Detection

Applies statistical tests and embedding-space monitoring to detect input distribution and output quality drift. Triggers timely retraining or prompt updates before degradation compounds.

Lesson 2 • Automated Quality Gates and Alerts

Implements threshold-based and anomaly-detection quality gates that block or flag degraded outputs. Reduces mean time to detection for production quality incidents.

Lesson 3 • Continuous Improvement Workflows

Establishes evaluation-driven retraining cycles, prompt update cadences, and model versioning governance. Operationalizes a sustainable loop from monitoring signal to deployed improvement.

Lesson 4 • User Feedback Collection and Analysis

Designs thumbs-up/down, explicit rating, and implicit behavioral feedback mechanisms. Converts user signals into actionable evaluation data for continuous improvement.