RNN, LSTM & GRU

Sequential Data and Recurrence

Many real-world problems involve ordered sequences where history matters:

• Natural language ("The bank by the river" vs "The bank approved the loan")
• Time series (stock prices, sensor readings, weather)
• Audio and speech
• DNA sequences

MLPs and CNNs process fixed-size inputs independently. Recurrent Neural Networks (RNNs) maintain a hidden state that carries information across timesteps.

Vanilla RNN

At each timestep $t$, the RNN takes the current input $x_t$ and the previous hidden state $h_{t-1}$:

$h_t = \tanh(W_{hh} h_{t-1} + W_{xh} x_t + b_h)$ $y_t = W_{hy} h_t + b_y$

The same weights $W_{hh}$, $W_{xh}$, $W_{hy}$ are shared across all timesteps.

The Vanishing Gradient Problem

Backpropagation Through Time (BPTT) computes gradients by unrolling the RNN:

$\frac{\partial \mathcal{L}}{\partial h_0} = \frac{\partial \mathcal{L}}{\partial h_T} \prod_{t=1}^{T} \frac{\partial h_t}{\partial h_{t-1}}$

Each $\frac{\partial h_t}{\partial h_{t-1}}$ involves the Jacobian of tanh, whose singular values are < 1. For long sequences, the product vanishes exponentially → the network can't learn long-range dependencies.

Vanilla RNNs in practice: effective for sequences of length ≤ 10–20 steps. Anything longer requires LSTM or GRU.

LSTM: Long Short-Term Memory

LSTM (Hochreiter & Schmidhuber, 1997) solves vanishing gradients by introducing a cell state $C_t$ — a "memory highway" — with three learned gates that control information flow:

The Four Equations

Forget gate — what to erase from memory: $f_t = \sigma(W_f [h_{t-1}, x_t] + b_f)$

Input gate — what new information to store: $i_t = \sigma(W_i [h_{t-1}, x_t] + b_i)$ $\tilde{C}_t = \tanh(W_C [h_{t-1}, x_t] + b_C)$

Cell state update — update the memory: $C_t = f_t \odot C_{t-1} + i_t \odot \tilde{C}_t$

Output gate — what to output based on current memory: $o_t = \sigma(W_o [h_{t-1}, x_t] + b_o)$ $h_t = o_t \odot \tanh(C_t)$

Why LSTM Works

The cell state $C_t$ can propagate through time with additive updates (not multiplicative), allowing gradients to flow unchanged over hundreds of steps.

GRU: Gated Recurrent Unit

GRU (Cho et al., 2014) simplifies LSTM by merging the cell state and hidden state, using only two gates:

Reset gate — how much past to forget: $r_t = \sigma(W_r [h_{t-1}, x_t])$

Update gate — how much to update state: $z_t = \sigma(W_z [h_{t-1}, x_t])$

Candidate hidden state: $\tilde{h}_t = \tanh(W [r_t \odot h_{t-1}, x_t])$

Final hidden state: $h_t = (1 - z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t$

LSTM vs GRU

	LSTM	GRU
Gates	3 (forget, input, output)	2 (reset, update)
Parameters	More	~25% fewer
Performance	Slightly better on long sequences	Comparable, often similar
Speed	Slower	Faster
Best for	Long-range dependencies, complex patterns	Shorter sequences, limited compute

Rule of thumb: start with GRU (faster to iterate); switch to LSTM if performance is insufficient.

import torch
import torch.nn as nn
from torch.utils.data import DataLoader, TensorDataset
import numpy as np

# Minimal text classification setup
# Assume: sequences padded to max_len=200, vocab_size=10000

class SentimentLSTM(nn.Module):
    def __init__(self, vocab_size, embed_dim, hidden_dim, n_layers, dropout=0.3):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embed_dim, padding_idx=0)
        self.lstm = nn.LSTM(
            embed_dim, hidden_dim,
            num_layers=n_layers,
            batch_first=True,
            dropout=dropout if n_layers > 1 else 0,
            bidirectional=True,     # process sequence in both directions
        )
        self.dropout = nn.Dropout(dropout)
        self.fc = nn.Linear(hidden_dim * 2, 1)  # *2 for bidirectional

    def forward(self, x, lengths=None):
        embedded = self.dropout(self.embedding(x))  # (B, T, E)

        # Pack for efficient computation with variable-length sequences
        if lengths is not None:
            packed = nn.utils.rnn.pack_padded_sequence(
                embedded, lengths.cpu(), batch_first=True, enforce_sorted=False
            )
            output, (h_n, _) = self.lstm(packed)
        else:
            output, (h_n, _) = self.lstm(embedded)

        # Concatenate last hidden state from both directions
        h_last = torch.cat([h_n[-2], h_n[-1]], dim=1)  # (B, hidden*2)
        return self.fc(self.dropout(h_last)).squeeze(1)

model = SentimentLSTM(
    vocab_size=10000, embed_dim=128, hidden_dim=256, n_layers=2, dropout=0.3
)

# Time-series forecasting with GRU
class GRUForecast(nn.Module):
    def __init__(self, input_size, hidden_size, n_layers, forecast_steps):
        super().__init__()
        self.gru = nn.GRU(input_size, hidden_size, n_layers,
                          batch_first=True, dropout=0.2)
        self.fc = nn.Linear(hidden_size, forecast_steps)

    def forward(self, x):
        out, _ = self.gru(x)           # (B, T, H)
        return self.fc(out[:, -1, :])  # use last timestep hidden state

# x shape: (batch, seq_len, n_features)
# y shape: (batch, forecast_steps)
gru_model = GRUForecast(input_size=5, hidden_size=128, n_layers=2, forecast_steps=7)
x = torch.randn(32, 30, 5)     # 32 samples, 30 timesteps, 5 features
y_pred = gru_model(x)
print(f"Forecast shape: {y_pred.shape}")   # (32, 7)

Bidirectional RNNs and Beyond

Bidirectional LSTM

Processes the sequence in both forward and backward directions. The final representation is the concatenation of both hidden states:

$\overrightarrow{h_t} = \text{LSTM}(x_t, \overrightarrow{h_{t-1}})$ $\overleftarrow{h_t} = \text{LSTM}(x_t, \overleftarrow{h_{t+1}})$ $h_t = [\overrightarrow{h_t}; \overleftarrow{h_t}]$

Useful when the full context (past + future) is available at inference time — classification, tagging, NER. Not applicable for generation/forecasting.

Stacked RNNs

Layer the output of one RNN as input to the next. 2–4 layers typically sufficient; more than 4 rarely helps.

RNNs vs Transformers (2024)

Transformers have largely replaced RNNs for NLP tasks due to parallelizability and better long-range modeling. However, RNNs still excel for:

• Real-time streaming (online, low-latency inference)
• Long time series where O(n²) attention is prohibitive
• Edge/embedded devices with strict memory budgets

Summary

• Vanilla RNNs process sequences via a shared hidden state but suffer from vanishing gradients on long sequences
• LSTM introduces a cell state with forget/input/output gates — the gold standard for long-range dependencies
• GRU simplifies LSTM to two gates with fewer parameters — often equally effective and faster
• Bidirectional RNNs capture both past and future context — great for classification and tagging
• Transformers now dominate NLP, but RNNs remain relevant for streaming, time series, and edge devices

Next: Model Evaluation Metrics — how to measure if your model is actually good.