Selecting the Right LLM with Hugging Face Course

Lesson 1 • What LLMs Are and How They Work

Covers transformer architecture, tokenization, and next-token prediction at a conceptual level. Establishes the technical baseline required for informed model selection.

Lesson 2 • Open vs. Proprietary Model Landscape

Contrasts open-weight and proprietary models across licensing, customizability, and cost. Sets context for why Hugging Face is central to open-model selection.

Lesson 3 • Key LLM Capability Dimensions

Maps the core capability axes—reasoning, instruction-following, coding, multilingual support—that differentiate models. Provides the evaluation framework used throughout the course.

Lesson 1 • Identifying Task Type and Modality

Maps common business objectives to NLP task types: classification, generation, summarization, QA, and more. Prevents misalignment between task needs and model capabilities.

Lesson 2 • Defining Quality and Performance Thresholds

Establishes minimum acceptable accuracy, latency, and throughput targets tied to business impact. Creates measurable criteria that make model comparison objective.

Lesson 3 • Mapping Constraints: Budget, Infrastructure, and Compliance

Captures hardware limits, cost ceilings, and regulatory constraints that eliminate non-viable models early. Reduces wasted evaluation effort on unsuitable candidates.

Lesson 4 • Stakeholder and User Requirements

Gathers end-user expectations, domain vocabulary needs, and output format requirements from stakeholders. Ensures the selected model serves real users, not just benchmark scores.

Lesson 1 • Hugging Face Hub Architecture

Introduces the Hub's structure: model cards, datasets, spaces, and organizations. Orients students to the primary interface for all model discovery tasks.

Lesson 2 • Searching and Filtering Models Effectively

Teaches tag-based filtering, task categories, and sort options to narrow thousands of models to a relevant shortlist. Directly enables the evaluation workflow in later chapters.

Lesson 3 • Hugging Face Libraries and Tooling

Surveys Transformers, Datasets, Evaluate, and PEFT libraries and their roles in the selection pipeline. Connects library knowledge to hands-on evaluation tasks ahead.

Lesson 4 • Reading and Interpreting Model Cards

Decodes model card sections: intended use, limitations, training data, and evaluation results. Builds critical reading skills that prevent misapplication of models.

Lesson 1 • Using the Hugging Face Evaluate Library

Demonstrates computing standard metrics programmatically using the Evaluate library on custom data. Enables reproducible, automated metric computation in later evaluation labs.

Lesson 2 • Standard NLP Benchmarks Explained

Covers widely used benchmarks for reasoning, language understanding, math, and coding tasks. Teaches what each benchmark actually measures and where it falls short.

Lesson 3 • Benchmark Limitations and Gaming Risks

Identifies data contamination, benchmark saturation, and overfitting risks that inflate reported scores. Builds critical skepticism essential for trustworthy model selection.

Lesson 4 • Matching Benchmarks to Use Case Requirements

Applies the requirements spec to select a benchmark subset that reflects actual deployment conditions. Bridges abstract metrics to the practical evaluation pipeline.

Lesson 1 • Running Comparative Evaluation Experiments

Executes side-by-side inference runs across candidate models and collects metric scores. Produces the raw data needed for the selection decision in the next chapter.

Lesson 2 • Analyzing and Visualizing Results

Aggregates scores, builds comparison tables, and visualizes trade-offs across accuracy, speed, and cost. Transforms raw numbers into decision-ready insights.

Lesson 3 • Setting Up the Evaluation Environment

Configures Python environment, GPU drivers, and Hugging Face credentials for reproducible evaluation runs. Removes setup friction so students focus on evaluation logic.

Lesson 4 • Loading and Running Candidate Models

Loads models and tokenizers from the Hub using the Transformers pipeline and AutoModel APIs. Establishes the standard loading pattern used across all evaluation experiments.

Lesson 1 • Latency Profiling and Benchmarking

Measures time-to-first-token, tokens-per-second, and end-to-end latency under realistic load. Produces latency data that feeds directly into the requirements-vs-model comparison.

Lesson 2 • Quantization Techniques for Efficiency

Covers INT8, INT4, and GPTQ quantization methods and their accuracy-vs-size trade-offs. Expands the set of viable models by reducing hardware requirements.

Lesson 3 • Cost Modeling for Inference at Scale

Builds a cost-per-query model accounting for hardware, cloud pricing, and request volume. Enables financially grounded model selection decisions.

Lesson 4 • Serving Frameworks and Inference Servers

Compares vLLM, TGI, and ONNX Runtime for production serving and explains when to use each. Connects deployment architecture to the model selection criteria.

Lesson 1 • Parameter-Efficient Fine-Tuning Methods

Covers LoRA, QLoRA, and prefix tuning as low-cost alternatives to full fine-tuning. Expands viable model options by reducing the compute required for adaptation.

Lesson 2 • Data Requirements for Effective Fine-Tuning

Specifies minimum dataset size, quality standards, and format requirements for successful fine-tuning. Ensures students assess data readiness before committing to a base model.

Lesson 3 • When Fine-Tuning Is and Is Not Needed

Distinguishes scenarios where prompting suffices from those requiring fine-tuning for quality or cost reasons. Prevents unnecessary fine-tuning investment when simpler approaches work.

Lesson 4 • Evaluating Fine-Tuned Model Quality

Applies task-specific metrics and human evaluation to assess fine-tuned model improvement over the base. Closes the loop between selection, adaptation, and validation.

Lesson 1 • Governance, Monitoring, and Re-Evaluation Triggers

Establishes ongoing monitoring metrics and criteria for triggering a model re-evaluation cycle. Embeds the selection process into a continuous improvement governance framework.

Lesson 2 • Scoring and Ranking Candidate Models

Applies a weighted scoring matrix to rank candidates across performance, cost, and risk dimensions. Converts multi-dimensional evidence into a single, auditable ranking.

Lesson 3 • Risk Assessment and Mitigation Planning

Identifies model-specific risks—bias, hallucination, vendor lock-in—and maps mitigation strategies. Ensures the selection decision accounts for operational and reputational risks.

Lesson 4 • Writing the Model Selection Report

Structures a concise report covering requirements, evaluation methodology, results, and recommendation. Produces the primary deliverable for stakeholder communication and audit trails.