Parallel Programming Course

Lesson 1 • Hardware Architecture Overview

Surveys CPU cores, caches, memory buses, and GPU streaming multiprocessors. Connects hardware topology to software design decisions throughout the course.

Lesson 2 • Measuring Parallel Performance

Teaches Amdahl's Law, Gustafson's Law, and efficiency metrics. Students gain quantitative tools used to evaluate every program they write.

Lesson 3 • Concurrency vs. Parallelism

Distinguishes concurrent interleaving from true simultaneous execution. Clarifies terminology used in all subsequent chapters.

Lesson 4 • Parallel Programming Models

Introduces shared-memory, message-passing, and data-parallel models. Provides a taxonomy students apply when selecting tools in later chapters.

Lesson 1 • Atomic Operations and Memory Models

Explains hardware atomics, compare-and-swap, and memory ordering guarantees. Prepares students for lock-free data structures in later chapters.

Lesson 2 • Mutual Exclusion and Locks

Teaches mutexes, spinlocks, and lock scoping to protect shared state. Directly addresses the data-race hazards introduced in Chapter 1.

Lesson 3 • Thread Fundamentals

Covers thread creation, joining, and detachment using POSIX-style APIs. Grounds students in the execution model before introducing synchronization.

Lesson 4 • Condition Variables and Barriers

Introduces condition variables for producer-consumer coordination and barriers for bulk synchronization. Extends locking to event-driven thread coordination.

Lesson 1 • Data Scoping and Reduction

Teaches private, shared, firstprivate, and reduction clauses to control variable visibility. Prevents data races without manual locking.

Lesson 2 • Parallel Loops and Work Sharing

Covers parallel-for, sections, and single constructs for distributing loop iterations. Directly applies the data-parallel model from Chapter 1.

Lesson 3 • OpenMP Synchronization Constructs

Covers barrier, critical, atomic, and flush directives for fine-grained control. Complements the mutex and atomic knowledge from Chapter 2.

Lesson 4 • OpenMP Programming Model

Introduces the fork-join execution model and compiler directive syntax. Establishes the mental model students use for all OpenMP constructs.

Lesson 1 • MPI Execution Model

Covers the SPMD model, communicators, and rank identification. Establishes the distributed execution context for all MPI programs.

Lesson 2 • Point-to-Point Communication

Teaches blocking and non-blocking send and receive operations with tag matching. Builds the communication primitives underlying all higher-level patterns.

Lesson 3 • Collective Communication Operations

Covers broadcast, scatter, gather, reduce, and all-to-all collectives. Replaces manual point-to-point patterns with optimized library calls.

Lesson 4 • MPI Performance and Scalability

Analyzes latency, bandwidth, and communication-to-computation overlap strategies. Applies Amdahl's and Gustafson's Laws from Chapter 1 to distributed programs.

Lesson 1 • Profiling and Debugging CUDA Programs

Covers profiler-driven optimization workflows and common GPU bug patterns. Equips students to diagnose and fix performance and correctness issues.

Lesson 2 • CUDA Memory Hierarchy

Covers global, shared, constant, and register memory with access patterns. Efficient memory use is the primary lever for GPU performance.

Lesson 3 • Kernel Optimization Techniques

Teaches coalesced memory access, occupancy tuning, and warp divergence reduction. Directly improves throughput of kernels written in earlier sections.

Lesson 4 • CUDA Programming Model

Introduces grids, blocks, threads, and the SIMT execution model. Connects GPU hardware from Chapter 1 to the CUDA software abstraction.

Lesson 1 • Lock-Free Data Structures

Designs lock-free stacks, queues, and hash maps using CAS operations. Applies atomic primitives from Chapter 2 to high-concurrency data structures.

Lesson 2 • Parallel Graph Algorithms

Covers BFS, SSSP, and connected components using parallel graph frameworks. Applies collective communication and task parallelism to irregular workloads.

Lesson 3 • Work and Span Analysis

Introduces work-span model, parallelism ratio, and Brent's theorem. Provides the analytical framework for evaluating all algorithms in this chapter.

Lesson 4 • Parallel Prefix and Scan

Covers inclusive and exclusive scan algorithms and their applications. Scan is a foundational primitive used in sorting, compaction, and graph algorithms.

Lesson 1 • Profiling Parallel Applications

Covers hardware performance counters, sampling profilers, and trace-based tools. Establishes the measurement foundation for all optimisation decisions.

Lesson 2 • Roofline Model and Bottleneck Analysis

Applies the roofline model to classify compute-bound vs. memory-bound kernels. Guides students to the highest-impact optimization for any given program.

Lesson 3 • Communication Overhead Reduction

Teaches message aggregation, overlap, and topology-aware routing for MPI programmes. Extends MPI performance concepts from Chapter 4 with optimisation techniques.

Lesson 4 • Load Balancing Strategies

Covers static partitioning, dynamic work stealing, and guided scheduling. Resolves the load imbalance that limits scalability in real applications.

Lesson 1 • End-to-End Parallel Application Design

Guides students through requirements analysis, algorithm selection, and performance validation for a complete parallel application. Synthesizes all course competencies.

Lesson 2 • Pipeline and Wavefront Patterns

Covers software pipeline stages, wavefront computation, and throughput analysis. Extends pipelining concepts from Chapter 1 to multi-stage parallel programs.

Lesson 3 • Hybrid MPI and OpenMP Programming

Designs hybrid programmes combining MPI ranks with OpenMP threads per node. Integrates Chapters 3 and 4 into a unified multi-level parallelism strategy.

Lesson 4 • Divide-and-Conquer Parallelism

Teaches recursive task decomposition, cutoff thresholds, and span optimisation. Builds on OpenMP tasks and work-span analysis from earlier chapters.