Deep Learning Essentials: Recurrent Neural Networks (RNNs)

RNNs are neural nets with baggage — and honestly, that’s their superpower.

You’ve already hung out with Activation Functions (the emotional regulators of neurons) and CNNs (the detectives of spatial patterns). Cute. Now we’re stepping into the drama of time: sequences. Text, audio, stock prices, sensor readings — data that arrives like a TV series, not a photo album.

If CNNs are great at understanding a single image, RNNs are the friend who remembers what you said three messages ago and uses it against you — for predictions.

---

Why RNNs? Because Order Matters

Imagine you read: "I did not say he stole the money." Depending on which word you stress, the meaning changes. This is sequence data: order matters, context matters, and the present is shaped by the past. Traditional feedforward networks treat each input as isolated. Not ideal for language, music, or any data with temporal dependencies.

• Problem: How do we let a neural network remember what happened before?
• Solution: Give it a hidden state — a tiny memory it updates every time step.

RNNs are to time what CNNs are to space. Both reuse parameters, but RNNs reuse them across time steps instead of across pixels.

---

The Core Idea: A Loop with Memory

An RNN processes a sequence one element at a time, carrying forward a hidden state.

At time t:

• Input: x_t (e.g., a word embedding)
• Hidden state from the past: h_{t-1}
• Update: combine x_t and h_{t-1} to produce h_t
• Output: y_t (optional, depends on the task)

Minimal math (don’t flinch):

h_t = tanh(W_x x_t + W_h h_{t-1} + b_h)
y_t = softmax(W_y h_t + b_y)   # e.g., next-word probabilities

Linking to our Activation Functions friends:

• tanh keeps the hidden state in a nice range (−1 to 1). It’s the mellow one.
• sigmoid (σ) often gates information (more on that with LSTMs/GRUs).
• ReLU in vanilla RNNs? Risky. Can blow up or die. Tanh/sigmoid are the classics here.

Visual vibe (unrolled through time):

h0 -> [RNN cell] --h1--> [RNN cell] --h2--> [RNN cell] --h3-->
       ^  x1             ^  x2             ^  x3

One set of parameters, used at every time step. Economical. Like wearing one good outfit multiple ways.

---

What Can RNNs Do?

Different shapes of mapping between sequences and outputs:

• One-to-One: a regular feedforward task (baseline, not RNN-specific)
• One-to-Many: e.g., generate music from a starting note
• Many-to-One: e.g., sentiment classification of a sentence
• Many-to-Many: e.g., machine translation, speech recognition

Mapping	Example	Output timing
One-to-Many	Image captioning (CNN → RNN)	Outputs over time
Many-to-One	Sentiment of a review	Final time step only
Many-to-Many	Translation	Each time step
Many-to-Many*	Seq labeling (POS tags)	Aligned with inputs

*Same-length input and output.

---

Training RNNs: Backpropagation Through Time (BPTT)

BPTT is just backprop that unrolls the RNN over time.

• Forward: roll through the sequence, accumulating states and losses.
• Backward: gradients flow back through each time step.

Two big issues show up like uninvited guests:

• Vanishing gradients: early time steps barely get any learning signal.
• Exploding gradients: gradients go cosmic, destabilizing training.

Fixes you’ll actually use:

• Gradient clipping (e.g., clip norm to 1.0)
• Careful initialization
• Truncated BPTT (only backprop through, say, 50 time steps)
• Better architectures: LSTM/GRU with gates

If vanilla RNNs are goldfish, LSTMs and GRUs are elephants with calendars.

---

Meet the Gated Crew: LSTM and GRU

The idea: control what to remember, what to forget, and what to output using gates with sigmoid activations. Sigmoid outputs numbers between 0 and 1 — perfect for filtering.

LSTM (Long Short-Term Memory)

• Keeps a cell state c_t (the long-term memory highway)
• Uses three gates: input, forget, output

Core vibe (simplified):

f_t = σ(W_f [h_{t-1}, x_t])   # forget gate
 i_t = σ(W_i [h_{t-1}, x_t])   # input gate
 g_t = tanh(W_g [h_{t-1}, x_t])# candidate content
c_t = f_t ⊙ c_{t-1} + i_t ⊙ g_t
 o_t = σ(W_o [h_{t-1}, x_t])   # output gate
h_t = o_t ⊙ tanh(c_t)

Interpretation: decide what to erase, what new info to write, and how much to reveal.

GRU (Gated Recurrent Unit)

• Simpler: merges cell and hidden state, uses update and reset gates

Simplified flow:

z_t = σ(W_z [h_{t-1}, x_t])     # update gate
r_t = σ(W_r [h_{t-1}, x_t])     # reset gate
h̃_t = tanh(W_h [r_t ⊙ h_{t-1}, x_t])
h_t = (1 - z_t) ⊙ h_{t-1} + z_t ⊙ h̃_t

LSTM vs GRU: Cheat Sheet

Model	Pros	Cons	When to try
Vanilla RNN	Simple, fast	Forgets long-term stuff, unstable	Short sequences, teaching basics
LSTM	Best at long dependencies	Heavier (more params)	Long texts, complex sequences
GRU	Almost-as-good memory, fewer params	Slightly less expressive than LSTM	Great default for speed/accuracy

---

Embeddings, Masking, and Friends

• Word embeddings: turn tokens into dense vectors. The RNN eats embeddings, not one-hot chaos.
• Padding and masking: make batches of different-length sequences. Mask so the model ignores padding.
• Bidirectional RNNs: run forward and backward, then concat states. Super for tasks like tagging when future context helps.
• Regularization: dropout on inputs and recurrent connections (use the library’s built-in variational dropout), early stopping.
• Optimization: Adam works well; still clip gradients.
• Teacher forcing (in sequence generation): during training, feed the true previous token; at inference, feed the model’s own outputs.

---

RNNs vs CNNs vs Transformers: The Friendly Roast

• CNNs: local spatial patterns; weight sharing across space; great for images and also 1D signals.
• RNNs: temporal dependencies; weight sharing across time; naturally sequential.
• Transformers: parallelize across time with attention; became the cool kids for long-range dependencies.

But understanding RNNs unlocks intuition about sequence modeling, gating, and the origins of attention. It’s like learning manual transmission before driving stick-assisted space Teslas.

---

Tiny Worked Example: Sentiment from Text

Task: many-to-one classification. Input sequence of word embeddings; output: positive/negative.

Pseudocode:

for batch in data:
    h = zeros(batch_size, hidden_dim)
    for t in range(T):
        h = RNNcell(x[t], h)        # LSTM/GRU preferred
    logits = W_out @ h + b
    loss = cross_entropy(logits, labels)
    loss.backward()
    clip_grad_norm_(model.parameters(), 1.0)
    optimizer.step(); optimizer.zero_grad()

Evaluation tip: use accuracy, F1 for imbalanced sets. For language modeling, use perplexity.

---

Common Misunderstandings (We See You)

• “RNNs memorize everything forever.” No — without gates and careful training, they forget quickly.
• “Just make sequences longer.” Truncated BPTT exists for sanity; too-long sequences cause vanishing gradients and slow training.
• “We can skip embeddings.” Please don’t. Embeddings are the context blender your model craves.
• “Dropout is the same everywhere.” Recurrent dropout needs special handling; use the framework’s built-in options.

---

Quick Design Checklist

• Start with GRU or LSTM, hidden size 64–256 for beginner projects.
• Use embeddings (pretrained like GloVe/fastText or learned end-to-end).
• Clip gradients; use Adam; try learning rates around 1e-3.
• Pad and mask your sequences correctly.
• Consider bidirectional layers for classification/tagging.
• For generation, implement teacher forcing and scheduled sampling.

---

TL;DR and Big Mood Insight

• RNNs add memory via hidden states, making them perfect for sequential data.
• Parameter sharing over time lets them generalize across positions, just like CNNs generalize across space.
• BPTT trains them, but gradients can vanish/explode; LSTM/GRU gates fix a lot of that.
• Practical success hinges on embeddings, masking, gradient clipping, and the right architecture choice.

The present is never just the present in sequence modeling. Your model’s next prediction is a remix of everything it’s felt so far.

Keep this energy as we keep climbing: you now understand how networks read, listen, and remember. Next, we’ll flirt with attention and Transformers — where remembering isn’t just sequential, it’s selective and global. Bring snacks.