Testing and Refining LLM Applications Course

Lesson 1 • Anatomy of an LLM Application

Breaks down the components of a typical LLM-powered app: prompt, model, retrieval, and output. Clarifies what each layer contributes to testable behavior.

Lesson 2 • Core Quality Dimensions for LLMs

Introduces accuracy, relevance, safety, and latency as primary quality axes. Provides a shared evaluation vocabulary used throughout the course.

Lesson 3 • Testing Mindset and Failure Taxonomy

Builds a structured way to anticipate and categorize LLM failures before writing a single test. Prepares students to design tests with clear failure hypotheses.

Lesson 1 • Test Case Anatomy and Structure

Defines the components of a well-formed LLM test case: input, context, expected behavior, and pass criteria. Establishes a consistent format for the course.

Lesson 2 • Adversarial and Stress Test Design

Introduces prompt injection, jailbreak attempts, and overload inputs as deliberate test strategies. Prepares students to harden applications against misuse.

Lesson 3 • Equivalence Partitioning for Prompts

Adapts classical equivalence partitioning to prompt design, grouping inputs by behavioral class. Reduces redundant tests while maximizing coverage.

Lesson 4 • Scenario-Based and User-Journey Tests

Designs multi-turn and task-completion scenarios that mirror real user workflows. Connects unit-level prompt tests to end-to-end application behavior.

Lesson 1 • Metric Aggregation and Dashboards

Combines multiple metrics into composite scores and visualizes trends over time. Enables teams to track quality regressions across model or prompt changes.

Lesson 2 • Reference-Based String Metrics

Covers BLEU, ROUGE, and exact-match metrics for outputs with known correct answers. Explains when these metrics are reliable and when they mislead.

Lesson 3 • Semantic Similarity Metrics

Introduces embedding-based similarity scores that capture meaning beyond surface tokens. Bridges the gap between string overlap and human judgment.

Lesson 1 • Principles of LLM-as-Judge

Explains why LLM judges work, where they fail, and how to scope their use responsibly. Grounds the technique in its theoretical strengths and known biases.

Lesson 2 • Calibrating and Validating Judges

Measures judge agreement with human raters using correlation and kappa statistics. Ensures judge prompts are trustworthy before deploying them at scale.

Lesson 3 • Pairwise and Reference-Free Evaluation

Covers head-to-head comparison and absolute scoring without reference answers. Expands evaluation to open-ended tasks where ground truth is unavailable.

Lesson 4 • Designing Judge Prompts

Teaches rubric construction, scoring scale design, and chain-of-thought elicitation for judge prompts. Directly determines the reliability of automated judgments.

Lesson 1 • Measuring Inter-Rater Reliability

Applies Cohen's kappa, Krippendorff's alpha, and percent agreement to annotation data. Quantifies label consistency and identifies ambiguous criteria.

Lesson 2 • Annotation Guideline Development

Builds detailed annotation schemas with examples, edge-case rules, and decision trees. Reduces annotator confusion and improves label consistency.

Lesson 3 • Crowdsourcing vs. Expert Annotation

Compares crowd platforms with domain-expert panels for different evaluation tasks. Guides the choice of annotator type based on task complexity and budget.

Lesson 1 • Tracking and Versioning Prompt Changes

Establishes a prompt registry with version history, evaluation scores, and change rationale. Prevents regression and enables rollback when new prompts underperform.

Lesson 2 • Diagnosing Prompt Failures

Uses evaluation outputs to classify prompt failures by root cause. Connects failure taxonomy from Chapter 1 to actionable prompt edits.

Lesson 3 • Convergence Criteria and Stopping Rules

Defines when a prompt is good enough using statistical thresholds and business targets. Prevents endless iteration and focuses effort on high-value improvements.

Lesson 4 • Prompt Engineering Techniques

Covers few-shot examples, chain-of-thought, role assignment, and output formatting. Provides a toolkit of techniques to apply during refinement cycles.

Lesson 1 • Monitoring Post-Deployment Quality

Extends testing into production with sampling, shadow evaluation, and drift detection. Closes the feedback loop between live usage and the test suite.

Lesson 2 • CI/CD Pipeline Architecture for LLMs

Maps LLM evaluation steps onto standard CI/CD stages: lint, test, evaluate, deploy. Adapts software delivery pipelines to handle probabilistic test outcomes.

Lesson 3 • Regression Test Suite Design

Selects and maintains a curated regression set that catches known failure modes efficiently. Balances coverage with runtime cost for fast feedback loops.

Lesson 1 • Harmful Output Detection

Covers toxicity classifiers, keyword filters, and LLM-based safety judges for detecting harmful content. Establishes a layered detection strategy for production systems.

Lesson 2 • Safety Evaluation Reporting

Structures findings into a safety report with severity ratings, evidence, and remediation steps. Communicates risk clearly to technical and non-technical stakeholders.

Lesson 3 • Bias Measurement Across Demographics

Applies counterfactual input testing and disparity metrics to surface demographic bias. Quantifies differential treatment across protected attribute groups.

Lesson 4 • Fairness Criteria and Trade-offs

Introduces group fairness, individual fairness, and calibration as competing criteria. Helps students choose and justify fairness definitions for their application context.