Introduction to Long Short-Term Memory (LSTM) Training

Lesson 1 • Motivation for Gated Architectures

Surveys historical attempts to fix RNN memory issues and introduces the gating concept. Sets the stage for the LSTM cell introduced in the next chapter.

Lesson 2 • Recurrent Neural Network Basics

Introduces the RNN hidden state and the recurrence equation. Connects the concept of shared weights across time steps to the chapter's core theme.

Lesson 3 • Backpropagation Through Time

Explains how gradients flow backward through unrolled RNN steps. Reveals the vanishing and exploding gradient problems that LSTM was designed to solve.

Lesson 4 • Feedforward Networks and Their Limits

Reviews how feedforward networks process fixed-length inputs and why they cannot model variable-length sequences. Motivates the need for memory in neural architectures.

Lesson 1 • The Output Gate and Hidden State

Covers how the output gate controls what portion of the cell state becomes the hidden state. Ties the hidden state to downstream predictions and next-step inputs.

Lesson 2 • Full Forward Pass Walkthrough

Traces a complete forward pass through one LSTM cell step by step with concrete numbers. Reinforces gate interactions and prepares students for backpropagation.

Lesson 3 • The LSTM Cell at a Glance

Presents the dual-state design of the LSTM cell and contrasts it with the vanilla RNN. Orients students to the cell state highway and the hidden state output.

Lesson 4 • The Forget Gate

Details how the forget gate selectively erases cell state content using a sigmoid activation. Demonstrates its role in preventing irrelevant past information from persisting.

Lesson 1 • Sequence Windowing and Batching

Explains sliding window extraction and how to form fixed-length input sequences from raw streams. Covers batch construction for efficient GPU utilisation.

Lesson 2 • Train, Validation, and Test Splitting

Applies time-aware splitting strategies to prevent data leakage in sequential datasets. Covers walk-forward validation for realistic performance estimation.

Lesson 3 • Text Tokenization and Embedding

Prepares text sequences for LSTM input through tokenization, vocabulary building, and embedding layers. Connects embedding output shape to LSTM input requirements.

Lesson 4 • Handling Missing Values in Sequences

Addresses imputation strategies specific to sequential data, including forward fill and interpolation. Discusses masking as an alternative to imputation.

Lesson 1 • Parameter Gradients and Weight Updates

Computes gradients for all weight matrices and bias vectors inside the LSTM cell. Connects gradient computation to standard optimizer update rules.

Lesson 2 • Loss Functions for Sequence Tasks

Surveys loss functions suited to sequence prediction, classification, and generation tasks. Establishes the scalar loss needed to initiate backpropagation.

Lesson 3 • Optimisation Algorithms for LSTMs

Compares SGD, momentum, RMSProp, and Adam for LSTM training. Highlights adaptive learning rate benefits for sparse gradient landscapes in sequences.

Lesson 4 • Backpropagation Through Time for LSTMs

Derives gradient flow through the cell state and each gate using the chain rule. Shows how the cell state highway mitigates vanishing gradients.

Lesson 1 • Early Stopping and Model Checkpointing

Implements early stopping based on validation loss patience to halt training at the optimal point. Covers saving and restoring the best model checkpoint.

Lesson 2 • Dropout for Recurrent Networks

Distinguishes standard dropout from recurrent dropout and variational dropout for LSTMs. Explains why naive dropout on recurrent connections harms gradient flow.

Lesson 3 • Weight Regularisation Techniques

Applies L1 and L2 penalties to LSTM weight matrices to reduce overfitting. Discusses activity regularisation as a complementary approach.

Lesson 4 • Systematic Hyperparameter Search

Compares grid search, random search, and Bayesian optimisation for LSTM tuning. Introduces efficient search strategies that minimise compute cost.

Lesson 1 • Stacking LSTM Layers

Explains how hidden states from one LSTM layer feed as inputs to the next. Covers the return-sequences setting required for intermediate layers.

Lesson 2 • Architecture Selection Guidelines

Provides decision criteria for choosing between single-layer, stacked, and bidirectional LSTMs. Balances model capacity against dataset size and latency constraints.

Lesson 3 • Encoder-Decoder LSTM Sequence-to-Sequence

Builds the encoder-decoder framework for variable-length input-to-output mapping. Connects the encoder's final state to the decoder's initial state.

Lesson 4 • Residual Connections in Deep LSTMs

Applies skip connections between LSTM layers to ease gradient flow in very deep stacks. Discusses when residual connections provide measurable benefit.

Lesson 1 • Dot-Product and Scaled Dot-Product Attention

Presents the Luong dot-product attention and its scaled variant for numerical stability. Compares computational cost and expressiveness with additive attention.

Lesson 2 • Training and Evaluating Attention Models

Covers loss computation, gradient flow through attention weights, and alignment quality metrics. Introduces attention entropy as a diagnostic for attention sharpness.

Lesson 3 • The Context Vector Bottleneck Problem

Diagnoses information loss when long sequences are compressed into a single context vector. Motivates attention as a dynamic, content-based retrieval mechanism.

Lesson 4 • Self-Attention in LSTM Encoders

Applies self-attention within the encoder to capture intra-sequence dependencies. Discusses how self-attention complements recurrent processing.

Lesson 1 • Monitoring Models in Production

Establishes data drift detection and prediction monitoring for deployed LSTM models. Defines retraining triggers based on performance degradation signals.

Lesson 2 • Sequence Classification Tasks

Applies LSTMs to sentiment analysis and activity recognition as classification benchmarks. Covers pooling strategies to convert sequence outputs to class logits.

Lesson 3 • Model Export and Serving

Exports trained LSTM models to portable formats and serves them via REST APIs. Addresses stateful inference challenges unique to recurrent models.

Lesson 4 • Time Series Forecasting with LSTMs

Builds a complete univariate and multivariate forecasting pipeline from raw data to predictions. Evaluates models with MAE, RMSE, and MAPE on held-out test sets.