Sequence Models, Attention, and the Road to Transformers

Some data has a natural order: text, time-series, audio, events. The position of each element relative to others carries information that a standard feedforward network ignores. Recurrent Neural Networks (RNNs) - and their successors LSTMs and GRUs - process sequences by maintaining a hidden state that accumulates context as each element is processed. While Transformers have largely replaced RNNs for language tasks, RNNs remain relevant for time-series forecasting, streaming inference, and understanding why Transformers were designed the way they were.

The Core RNN Idea

At each timestep t, an RNN takes the current input xₜ and the previous hidden state hₜ₋₁, and produces a new hidden state hₜ:

hₜ = tanh(Wₓ · xₜ + Wₕ · hₜ₋₁ + b)

The same weight matrices Wₓ and Wₕ are applied at every timestep - the network shares parameters across time.

pythonCopy
import torch
import torch.nn as nn

class SimpleRNN(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim):
        super().__init__()
        self.rnn = nn.RNN(input_dim, hidden_dim, batch_first=True)
        self.fc = nn.Linear(hidden_dim, output_dim)

    def forward(self, x):
        # x: [batch, seq_len, input_dim]
        output, h_n = self.rnn(x)
        # output: [batch, seq_len, hidden_dim]  -  hidden state at each timestep
        # h_n: [1, batch, hidden_dim]  -  final hidden state
        return self.fc(h_n.squeeze(0))

# Sequence classification: use final hidden state
# Sequence labeling (e.g., NER): use output at each timestep

batch_first=True means the input shape is [batch, seq_len, features]. The default is [seq_len, batch, features] - always be explicit.

The Vanishing Gradient Problem

Standard RNNs fail to learn long-range dependencies. The gradient of the loss with respect to early timesteps is the product of many Jacobians - and if each factor is < 1 (which happens with tanh), the product vanishes exponentially fast. By the time the gradient reaches timestep 1 in a sequence of length 100, it is effectively zero.

This is why LSTMs were invented.

LSTM: Learning What to Remember

The Long Short-Term Memory (LSTM) adds a cell state cₜ alongside the hidden state hₜ, controlled by three learned gates:

• Forget gate: what to erase from cell state
• Input gate: what new information to write to cell state
• Output gate: what to expose as hidden state

pythonCopy
class LSTMClassifier(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim,
                 num_layers=2, dropout=0.3):
        super().__init__()
        self.lstm = nn.LSTM(
            input_dim, hidden_dim,
            num_layers=num_layers,
            batch_first=True,
            dropout=dropout,          # between LSTM layers (not applied on last layer)
            bidirectional=True        # see both past and future context
        )
        self.dropout = nn.Dropout(dropout)
        # Bidirectional → 2x hidden_dim
        self.fc = nn.Linear(hidden_dim * 2, output_dim)

    def forward(self, x):
        lstm_out, (h_n, c_n) = self.lstm(x)
        # h_n: [num_layers * 2, batch, hidden_dim] for bidirectional
        # Take last layer's forward and backward final states
        h_forward  = h_n[-2]   # last layer, forward direction
        h_backward = h_n[-1]   # last layer, backward direction
        h_combined = torch.cat([h_forward, h_backward], dim=1)
        return self.fc(self.dropout(h_combined))

Bidirectional LSTM: processes the sequence in both directions and concatenates the results. Standard for tasks where you have the full sequence available (text classification, NER). Do not use bidirectional for autoregressive generation or real-time streaming (you don't have future tokens).

GRU: Simpler Than LSTM, Often as Good

The Gated Recurrent Unit (GRU) merges the forget and input gates into a single update gate, and merges cell and hidden state. Fewer parameters, often similar performance to LSTM, faster to train.

pythonCopy
self.gru = nn.GRU(input_dim, hidden_dim, batch_first=True, bidirectional=True)

Rule of thumb: Try GRU first. Use LSTM if you need to model very long-range dependencies or if GRU underperforms.

Time-Series Forecasting with LSTMs

pythonCopy
import pandas as pd
import numpy as np

def make_sequences(series: np.ndarray, seq_len: int, horizon: int):
    """Convert a univariate time series into (input_seq, target) pairs."""
    X, y = [], []
    for i in range(len(series) - seq_len - horizon + 1):
        X.append(series[i : i + seq_len])
        y.append(series[i + seq_len : i + seq_len + horizon])
    return np.array(X)[..., np.newaxis], np.array(y)  # X: [N, seq_len, 1]

# Example: predict next 7 days from past 30 days
series = df['daily_revenue'].values
X, y = make_sequences(series, seq_len=30, horizon=7)

class LSTMForecaster(nn.Module):
    def __init__(self, input_dim=1, hidden_dim=64, horizon=7):
        super().__init__()
        self.lstm = nn.LSTM(input_dim, hidden_dim, num_layers=2, batch_first=True, dropout=0.2)
        self.fc = nn.Linear(hidden_dim, horizon)

    def forward(self, x):
        _, (h_n, _) = self.lstm(x)
        return self.fc(h_n[-1])  # use final hidden state to predict all horizon steps

Critical for time-series: always split chronologically. The test set must be temporally after the training set. Never shuffle before splitting.

Handling Variable-Length Sequences

Real-world sequences (sentences, event logs) have different lengths. Padding pads shorter sequences with zeros; masking tells the model which positions to ignore.

pythonCopy
from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence

class LSTMWithPacking(nn.Module):
    def __init__(self, vocab_size, embed_dim, hidden_dim, num_classes):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embed_dim, padding_idx=0)
        self.lstm = nn.LSTM(embed_dim, hidden_dim, batch_first=True)
        self.fc = nn.Linear(hidden_dim, num_classes)

    def forward(self, x, lengths):
        # x: [batch, max_seq_len] (padded token ids)
        embedded = self.embedding(x)
        # Pack: skip padding positions during LSTM computation
        packed = pack_padded_sequence(embedded, lengths, batch_first=True, enforce_sorted=False)
        _, (h_n, _) = self.lstm(packed)
        return self.fc(h_n[-1])

Why Transformers Replaced RNNs for NLP

RNNs process sequences step-by-step - parallelization across timesteps is impossible during training. For a sequence of length 512, you need 512 sequential steps. This makes RNNs slow to train on modern hardware designed for parallel computation.

Transformers process all positions simultaneously using attention - every token attends to every other token in a single parallel operation. For long sequences, they are dramatically faster to train and capture longer dependencies more reliably.

However, RNNs retain advantages for:

• Streaming inference: process one token at a time with constant memory (no need to materialize the full attention matrix).
• Short sequences with temporal structure: short time-series where efficiency matters.
• On-device ML: smaller memory footprint than attention mechanisms.

Common Mistakes and Bad Instincts

Not normalizing input features for LSTM time-series. LSTMs, like all gradient-based models, are sensitive to input scale. Normalize your time-series (per-feature z-score or min-max scaling) before feeding to the LSTM.

Using bidirectional LSTM for generation. Bidirectional models have access to future tokens during training. Using them for autoregressive tasks (predicting the next word) leaks information from the future and produces a model that cannot be used for generation.

Forgetting to detach the hidden state between batches. If you reuse hidden state across batches (for stateful RNN training), call h = h.detach() to prevent gradients from propagating back through all previous batches.

Not using enforce_sorted=False in pack_padded_sequence. Older PyTorch required sequences to be sorted by length. Modern PyTorch handles unsorted sequences when you set this flag.

Treating RNN hidden state as a general-purpose representation. The hidden state at the final timestep is the only state you typically use for sequence classification, but it may not be the best representation for long sequences. Attention pooling over all hidden states often outperforms just the final state.

Where to Go Next

• Module 20 (Transformers and Attention) is the direct successor - it explains why attention mechanisms replaced RNNs for most sequence tasks.
• Module 22 (Working with LLMs) covers how to use pretrained language models (which are transformer-based) for production NLP tasks without training from scratch.