RNNs and LSTMs¶
Recurrent networks come up in interviews for NLP, time series, and speech roles. Even as Transformers have displaced RNNs in many applications, understanding why RNNs exist, where they fail, and what LSTMs fixed is essential for demonstrating genuine depth of understanding.
Q1: Why do standard feedforward networks struggle with sequences?¶
Show answer
Feedforward networks take a fixed-size input and produce a fixed-size output. They have no mechanism to handle:
- Variable-length inputs: a sentence of 5 words and a sentence of 50 words have different lengths. A fixed input layer cannot handle both.
- Sequential dependencies: the meaning of "bank" in "river bank" vs "bank account" depends on surrounding context. A feedforward network treats each input position independently with no memory of previous positions.
- Temporal ordering: a feedforward network sees all inputs simultaneously. The order in which elements are presented carries no information to the network.
Flattening a sequence into a fixed-size vector and feeding it to an MLP throws away the temporal structure and scales poorly to long sequences.
What we need: a network architecture that processes one element at a time, maintains state from previous elements, and applies the same transformation at each step — regardless of sequence length. That is an RNN.
Q2: What is an RNN and what does the hidden state represent?¶
Show answer
An RNN (Recurrent Neural Network) processes sequences one element at a time, maintaining a hidden state that acts as a compressed memory of everything seen so far.
At each time step t:
xₜis the input at stepthₜ₋₁is the hidden state from the previous stephₜis the new hidden state — the network's summary of the sequence up to stept- The same weight matrices (Wₕ, Wₓ) are shared across all time steps
import tensorflow as tf
# SimpleRNN in Keras
model = tf.keras.Sequential([
tf.keras.layers.SimpleRNN(
units=64,
return_sequences=True, # return hidden state at every step
input_shape=(20, 10) # sequences of length 20, 10 features each
),
tf.keras.layers.SimpleRNN(64), # return only the final hidden state
tf.keras.layers.Dense(1)
])
model.summary()
Parameter sharing across time steps: the same weights are applied at every step. This gives RNNs two important properties: - Handle variable-length sequences (any number of steps with the same weights) - Generalise patterns across time positions (a pattern learned at step 3 is also recognised at step 15)
The hidden state is a fixed-size representation that must compress all relevant history — this compression is its strength and its limitation.
Q3: What is the vanishing gradient problem in RNNs and why is it worse than in feedforward networks?¶
Show answer
Backpropagation through time (BPTT) unrolls the RNN across all time steps and applies standard backprop. The gradient of the loss at time step T must flow back through every step from T to 1.
At each step, the gradient is multiplied by the Jacobian of tanh(Wₕ · hₜ₋₁ + ...) with respect to hₜ₋₁. The tanh derivative is at most 1 and is often much smaller (especially when the hidden state is saturated). Over T steps:
This product of T matrices typically has eigenvalues much less than 1 → the gradient vanishes exponentially with T. For sequences of length 50–100, gradient from step 50 barely reaches step 1.
Why it's worse in RNNs than feedforward nets: in a feedforward net, depth is fixed (e.g., 20 layers). In an RNN processing a 500-token sequence, the effective depth is 500. Gradients must flow 500 steps back. The vanishing gradient problem is proportional to sequence length, not just architecture depth.
Consequence: standard RNNs can only learn short-range dependencies. They forget information from more than ~10–20 steps ago. This motivated the design of LSTMs.
Warning
Never claim RNNs "can't handle long sequences." They process them fine at inference time. The problem is during training: they cannot learn to depend on information from far-back time steps because that information's gradient vanishes before reaching the relevant weights.
Q4: What is an LSTM and what do the three gates do?¶
Show answer
An LSTM (Long Short-Term Memory) introduces a cell state cₜ — a separate, carefully controlled memory that carries information across many time steps. Gates regulate what information enters, leaves, and passes through.
Three gates (each is a sigmoid → outputs between 0 and 1):
Forget gate: decides what to erase from the cell state.
Values near 0 → forget. Values near 1 → remember.Input gate (+ candidate): decides what new information to write to the cell state.
Cell state update:
Output gate: decides what to output to the hidden state.
import tensorflow as tf
model = tf.keras.Sequential([
tf.keras.layers.LSTM(
units=128,
return_sequences=True,
dropout=0.2,
recurrent_dropout=0.2,
input_shape=(50, 64) # sequences of 50 steps, 64 features
),
tf.keras.layers.LSTM(64),
tf.keras.layers.Dense(1, activation='sigmoid')
])
Why the cell state solves vanishing gradients: the cell state update is additive (cₜ = fₜ ⊙ cₜ₋₁ + ...). Addition, unlike multiplication, allows gradients to flow back through time steps without systematic shrinkage — analogous to residual connections in feedforward networks. When the forget gate is near 1 and the input gate is near 0, the cell state is preserved unchanged across many steps: the gradient flows directly back through these steps unimpeded.
Q5: What is a GRU and when should you prefer it over an LSTM?¶
Show answer
A GRU (Gated Recurrent Unit) simplifies the LSTM by combining the forget and input gates into a single update gate, and merging the cell state and hidden state.
zₜ = σ(Wz · [hₜ₋₁, xₜ]) # update gate: how much of old state to keep
rₜ = σ(Wr · [hₜ₋₁, xₜ]) # reset gate: how much old state to use for candidate
h̃ₜ = tanh(W · [rₜ ⊙ hₜ₋₁, xₜ]) # candidate hidden state
hₜ = (1 - zₜ) ⊙ hₜ₋₁ + zₜ ⊙ h̃ₜ
import tensorflow as tf
model = tf.keras.Sequential([
tf.keras.layers.GRU(
units=128,
return_sequences=True,
dropout=0.2,
input_shape=(50, 64)
),
tf.keras.layers.GRU(64),
tf.keras.layers.Dense(10, activation='softmax')
])
GRU vs LSTM:
| GRU | LSTM | |
|---|---|---|
| Gates | 2 (update, reset) | 3 (forget, input, output) |
| States | 1 (hidden state only) | 2 (hidden + cell state) |
| Parameters | ~25% fewer | More |
| Training speed | Faster | Slower |
| Performance | Often comparable | Slightly better on complex tasks |
When to prefer GRU: - Small to medium datasets — fewer parameters reduce overfitting risk - When training speed matters - When LSTM and GRU are both reasonable choices — GRU is simpler and often sufficient
When to prefer LSTM: - Long sequences with complex long-range dependencies - Tasks where the explicit separation of cell state and hidden state matters (e.g., some translation tasks) - When you have enough data to benefit from the extra capacity
Q6: What is a bidirectional RNN and when does context from both directions matter?¶
Show answer
A standard RNN processes sequences left-to-right — at step t, it only has access to past context (steps 1 to t-1). For tasks where future context matters, this is a limitation.
A bidirectional RNN runs two independent RNNs on the sequence: one left-to-right and one right-to-left. Their outputs (hidden states) are concatenated at each time step, giving the model access to both past and future context.
import tensorflow as tf
model = tf.keras.Sequential([
tf.keras.layers.Bidirectional(
tf.keras.layers.LSTM(64, return_sequences=True),
input_shape=(100, 128)
),
# Each time step now has 128-dim output (64 forward + 64 backward)
tf.keras.layers.Bidirectional(
tf.keras.layers.LSTM(32)
),
tf.keras.layers.Dense(5, activation='softmax')
])
When both directions matter: - Named Entity Recognition: "Apple" in "Apple sued Samsung" (company) vs "Apple is a fruit" (fruit) — need right context to disambiguate - Part-of-speech tagging: "saw" is a noun in "I have a saw" and a verb in "I saw him" — right context is essential - Text classification and sentiment analysis
When bidirectional is NOT appropriate: - Language modelling: predicting the next word — you don't have access to future tokens at inference time. Using bidirectional would be cheating during training. - Real-time speech recognition: future audio hasn't been heard yet. - Any autoregressive generation task.
Q7: What is teacher forcing?¶
Show answer
In sequence-to-sequence models (e.g., machine translation), the decoder generates one output token at a time, feeding each generated token as input to the next step.
The problem at training time: if the decoder generates a wrong token at step t, all subsequent steps are conditioned on that wrong token. Errors compound — training becomes unstable.
Teacher forcing: during training, instead of feeding the decoder's own predictions as input to the next step, feed the ground-truth token from the target sequence.
Inference: decoder input at step t+1 = decoder output at step t (its own prediction)
Teacher forcing: decoder input at step t+1 = target[t] (the correct token)
# Conceptual illustration of teacher forcing in training
for t in range(target_length):
decoder_input = target_tokens[:, t] # ground truth, not predicted
decoder_output, hidden_state = decoder(decoder_input, hidden_state)
loss += criterion(decoder_output, target_tokens[:, t+1])
Advantages: faster, more stable training. The decoder always sees correctly-formatted input.
Disadvantage — exposure bias: at inference, the decoder sees its own (imperfect) predictions, but at training it always saw correct inputs. This mismatch degrades performance on long sequences. The decoder was never trained to recover from its own mistakes.
Mitigation — Scheduled Sampling: gradually reduce teacher forcing ratio over training. Start with 100% teacher forcing and decay toward 0%, so the model learns to handle its own outputs.
Q8: What is a sequence-to-sequence model?¶
Show answer
A seq2seq model maps an input sequence of variable length to an output sequence of variable (and different) length. The canonical application is machine translation: "How are you?" → "Comment allez-vous?".
Architecture: - Encoder: reads the input sequence and compresses it into a context vector (the final hidden state) - Decoder: conditioned on the context vector, generates the output sequence one token at a time
import tensorflow as tf
# Encoder
encoder_inputs = tf.keras.Input(shape=(None, input_vocab_size))
encoder = tf.keras.layers.LSTM(256, return_state=True)
_, state_h, state_c = encoder(encoder_inputs)
encoder_states = [state_h, state_c]
# Decoder
decoder_inputs = tf.keras.Input(shape=(None, target_vocab_size))
decoder_lstm = tf.keras.layers.LSTM(256, return_sequences=True, return_state=True)
decoder_outputs, _, _ = decoder_lstm(decoder_inputs, initial_state=encoder_states)
decoder_dense = tf.keras.layers.Dense(target_vocab_size, activation='softmax')
decoder_outputs = decoder_dense(decoder_outputs)
model = tf.keras.Model([encoder_inputs, decoder_inputs], decoder_outputs)
The bottleneck problem: compressing an entire input sequence into a single fixed-size context vector loses information, especially for long sequences. Attention mechanisms — which allow the decoder to look back at all encoder hidden states at each decoding step — were developed to solve this. The Transformer architecture replaced the entire encoder-decoder LSTM with attention mechanisms and became dominant.
Q9: When are RNNs still preferred over Transformers?¶
Show answer
Transformers have outperformed RNNs on most NLP benchmarks, but RNNs remain preferable in specific scenarios:
Edge inference with limited compute: A Transformer's attention mechanism scales quadratically with sequence length. An RNN processes one step at a time with constant memory regardless of sequence length. For on-device inference (smartphones, microcontrollers, embedded systems), an LSTM or GRU can be dramatically smaller and faster.
Streaming / real-time applications: RNNs are inherently causal — they process input one step at a time and produce output immediately. Transformers with full self-attention require the entire input before producing any output. For real-time speech recognition or live text processing, latency matters.
Very long sequences on limited memory: Attention memory scales as O(n²). A 10,000-token sequence requires 100M attention weights. LSTMs process this in O(n) memory. For very long documents on memory-constrained hardware, RNNs remain viable.
Interpretability of recurrent dynamics: The hidden state of an LSTM is a single vector that can be tracked over time, making certain analyses (e.g., what the model is "tracking") more accessible than inspecting 12 attention heads across multiple layers.
In practice: for production NLP on standard hardware with sequence lengths < 2,048, Transformers win. For streaming, on-device, or unusually long sequences, RNNs and their variants remain competitive.
Q10: What are common applications of RNNs?¶
Show answer
Time series forecasting: Stock prices, sensor readings, energy demand, weather. The temporal ordering and sequential dependencies make RNNs a natural fit. LSTMs are used widely in industry for multivariate time series.
import tensorflow as tf
import numpy as np
# Univariate time series: predict next value from last 30 steps
model = tf.keras.Sequential([
tf.keras.layers.LSTM(64, return_sequences=True, input_shape=(30, 1)),
tf.keras.layers.LSTM(32),
tf.keras.layers.Dense(1)
])
model.compile(optimizer='adam', loss='mse')
Text generation: Character-level or word-level language models that generate text one token at a time. Each prediction is conditioned on all previous tokens.
Speech recognition (acoustic model): Bidirectional LSTMs process audio feature sequences (mel spectrograms) and output character or phoneme sequences. Still used in production systems alongside newer architectures.
Machine translation (before Transformers): Seq2seq with attention was the dominant approach from 2015 to 2017. The attention mechanism developed for RNN-based MT was the direct precursor to the Transformer.
Sentiment analysis and text classification: Bidirectional LSTMs that read a sequence and output a class label for the full sequence.
Anomaly detection in time series: Train an LSTM autoencoder on normal data. High reconstruction error at inference signals an anomaly — useful for detecting equipment failures, fraud, and intrusions.
Q11: What is the difference between return_sequences=True and return_sequences=False in Keras?¶
Show answer
This controls what the LSTM layer outputs:
-
return_sequences=False(default): outputs only the hidden state from the final time step. Shape:(batch, units). Use when you need one output for the entire sequence (classification, regression). -
return_sequences=True: outputs the hidden state at every time step. Shape:(batch, timesteps, units). Use when the next layer also processes sequences (stacked LSTMs, seq2seq decoder, CRF layer).
import tensorflow as tf
# Stacked LSTMs — intermediate layers must return sequences
model = tf.keras.Sequential([
tf.keras.layers.LSTM(128, return_sequences=True, input_shape=(50, 20)),
tf.keras.layers.LSTM(64, return_sequences=True), # feeds into third LSTM
tf.keras.layers.LSTM(32), # final: returns only last step
tf.keras.layers.Dense(1)
])
# Sequence labelling (e.g., NER) — need output at every step
labelling_model = tf.keras.Sequential([
tf.keras.layers.Bidirectional(
tf.keras.layers.LSTM(64, return_sequences=True),
input_shape=(100, 50)
),
tf.keras.layers.TimeDistributed(
tf.keras.layers.Dense(n_tags, activation='softmax')
)
])
Common mistake: using return_sequences=False in an intermediate LSTM layer and then stacking another LSTM on top — this fails because the second LSTM expects a 3D sequence input, not a 2D final state.