Training and Optimization

Training mechanics are where most deep learning projects succeed or fail. Interviewers probe these topics to separate practitioners who have debugged real training runs from those who have only read documentation.

---

Q1: What is backpropagation and what does it actually compute?

Show answer

Backpropagation is an efficient algorithm for computing the gradient of the loss function with respect to every parameter in the network. It does not update weights itself — that is the job of the optimiser. Backprop computes the gradients; the optimiser decides how to use them.

How it works: 1. Run the forward pass and store all intermediate activations 2. Compute the loss at the output 3. Starting from the output, use the chain rule to propagate the gradient backwards through each layer, computing ∂L/∂W and ∂L/∂b for each layer's parameters

The key insight: the gradient at any layer depends on the gradient from the layer above and the locally stored activations from the forward pass. This is why you must store intermediate activations — they are needed during the backward pass.

import tensorflow as tf

x = tf.constant([[1.0, 2.0]])
y_true = tf.constant([[1.0]])

model = tf.keras.Sequential([
    tf.keras.layers.Dense(4, activation='relu'),
    tf.keras.layers.Dense(1)
])

with tf.GradientTape() as tape:
    y_pred = model(x, training=True)
    loss = tf.keras.losses.mse(y_true, y_pred)

gradients = tape.gradient(loss, model.trainable_variables)

for var, grad in zip(model.trainable_variables, gradients):
    print(f"{var.name}: grad shape {grad.shape}, norm {tf.norm(grad):.4f}")

What makes backprop efficient: it reuses intermediate computations. Without it, you would need one forward pass per parameter to estimate gradients numerically — millions of forward passes for a modern network.

---

Q2: What is the chain rule and why does backpropagation depend on it?

Show answer

The chain rule from calculus states: if L is a function of y which is a function of x, then:

dL/dx = (dL/dy) · (dy/dx)

In a neural network, the loss is a composition of many functions:

L = loss(softmax(W₃ · relu(W₂ · relu(W₁x + b₁) + b₂) + b₃))

Backpropagation applies the chain rule repeatedly from output to input:

∂L/∂W₁ = (∂L/∂a₃) · (∂a₃/∂z₃) · (∂z₃/∂a₂) · (∂a₂/∂z₂) · (∂z₂/∂a₁) · (∂a₁/∂z₁) · (∂z₁/∂W₁)

Each term is the local gradient at that operation. Because we multiply these terms across layers, the gradient at early layers is a product of many values — which is precisely why gradients can vanish (if many terms < 1) or explode (if many terms > 1).

Automatic differentiation in modern frameworks (TensorFlow, PyTorch) implements this symbolically, so you never write these derivatives by hand. The framework builds a computation graph during the forward pass and traverses it in reverse to compute gradients.

---

Q3: What is gradient descent and what are its main variants?

Show answer

Gradient descent updates model parameters in the direction that reduces the loss, by following the negative gradient:

θ ← θ - α · ∇θL

where α is the learning rate and ∇θL is the gradient of the loss with respect to parameters.

Batch Gradient Descent: computes gradient over the entire dataset per update. Accurate but slow and memory-heavy. Not used in practice for large datasets.

Stochastic Gradient Descent (SGD): one sample per update. Fast but very noisy. The noise can help escape local minima but makes convergence unstable.

Mini-batch SGD: a batch of N samples per update. The standard in deep learning. Balances noise and efficiency; leverages GPU parallelism.

import tensorflow as tf

model = tf.keras.Sequential([
    tf.keras.layers.Dense(128, activation='relu', input_shape=(784,)),
    tf.keras.layers.Dense(10, activation='softmax')
])

model.compile(
    optimizer=tf.keras.optimizers.SGD(learning_rate=0.01),
    loss='sparse_categorical_crossentropy',
    metrics=['accuracy']
)
model.fit(X_train, y_train, batch_size=64, epochs=20)

---

Q4: What does momentum add to SGD and why does it help?

Show answer

Vanilla SGD computes each update using only the current gradient. Momentum augments this by accumulating a velocity vector — an exponentially weighted moving average of past gradients:

v ← β·v + α·∇θL
θ ← θ - v

where β is typically 0.9.

Why it helps: - In directions where the gradient consistently points the same way, momentum builds up speed - In directions where the gradient oscillates, past gradients partially cancel out and reduce oscillation - Helps navigate "ravines" — elongated regions in the loss landscape where the gradient points mainly sideways rather than toward the minimum

Geometric analogy: a ball rolling downhill accumulates velocity in the downhill direction and doesn't get thrown sideways by small surface irregularities.

optimizer = tf.keras.optimizers.SGD(learning_rate=0.01, momentum=0.9)

Nesterov momentum improves on standard momentum by computing the gradient at the lookahead position (θ - β·v) rather than the current position. This gives a more accurate gradient estimate and often converges faster — it is the default when using momentum > 0 in practice.

optimizer = tf.keras.optimizers.SGD(learning_rate=0.01, momentum=0.9, nesterov=True)

---

Q5: How does Adam work and why is it the default optimiser for most tasks?

Show answer

Adam (Adaptive Moment Estimation) combines momentum with adaptive per-parameter learning rates. It maintains two exponentially weighted moving averages:

• m = first moment (mean of gradients) — like momentum
• v = second moment (mean of squared gradients) — tracks gradient scale per parameter

m ← β₁·m + (1-β₁)·g        # momentum term
v ← β₂·v + (1-β₂)·g²       # scale term
m̂ = m / (1 - β₁ᵗ)           # bias correction (important early in training)
v̂ = v / (1 - β₂ᵗ)           # bias correction
θ ← θ - α · m̂ / (√v̂ + ε)   # update

Default values: β₁=0.9, β₂=0.999, ε=1e-8, α=1e-3.

Why it's the default: - Parameters with large gradients get smaller effective learning rates - Parameters with small gradients get larger effective learning rates - Robust across different parameter scales — works well with sparse gradients (embeddings) and dense gradients alike - Requires minimal manual tuning

Adam's weakness: can converge to sharper minima than SGD+momentum, sometimes generalising slightly worse on image classification. For vision benchmarks, SGD+momentum+schedule is often preferred. For everything else (NLP, tabular, early experimentation), Adam is the practical default.

optimizer = tf.keras.optimizers.Adam(learning_rate=1e-3)
model.compile(optimizer=optimizer, loss='sparse_categorical_crossentropy')

---

Q6: How does the learning rate affect training and what is learning rate scheduling?

Show answer

The learning rate (α) controls the size of each parameter update. It is the most impactful hyperparameter in deep learning.

• Too high: updates overshoot the minimum. Loss oscillates or diverges. May produce NaN.
• Too low: convergence is extremely slow. Model may get stuck in poor local optima.
• Just right: loss decreases smoothly toward a good minimum.

Learning rate scheduling — reducing the learning rate over training — consistently improves final accuracy because a large rate is useful for fast progress early on, while a small rate is needed for fine-grained convergence later.

import tensorflow as tf

# Step decay
def step_decay(epoch):
    initial_lr = 0.01
    drop = 0.1
    epochs_drop = 30
    return initial_lr * (drop ** (epoch // epochs_drop))

lr_scheduler = tf.keras.callbacks.LearningRateScheduler(step_decay)

# Cosine annealing — smooth decay to near zero
cosine_decay = tf.keras.optimizers.schedules.CosineDecay(
    initial_learning_rate=1e-3,
    decay_steps=10000
)

# ReduceLROnPlateau — reduce when validation loss stops improving
reduce_lr = tf.keras.callbacks.ReduceLROnPlateau(
    monitor='val_loss',
    factor=0.5,
    patience=5,
    min_lr=1e-6
)

Linear scaling rule: when doubling batch size, double the learning rate. This approximately preserves training dynamics but breaks down at very large batch sizes.

Learning rate finder: train for one epoch with LR increasing from 1e-7 to 10. Plot loss vs LR. Choose the LR just before the loss starts rising rapidly.

---

Q7: What are the tradeoffs between large and small batch sizes?

Show answer

Batch size affects gradient noise, generalisation, memory, and training speed simultaneously.

Property	Small batch (8–64)	Large batch (512–4096)
Gradient estimate	Noisy	Accurate
GPU utilisation	Low	High
Steps per epoch	Many	Fewer
Generalisation	Often better	Often worse
Time per epoch	Slower	Faster
Memory	Low	High

The generalisation gap: large-batch training consistently converges to sharper minima (narrow valleys in the loss landscape). Sharp minima generalise poorly — a small change in weights causes a large increase in loss. Small-batch training's noisy gradients act as implicit regularisation, steering the optimiser toward flatter, wider minima that generalise better to new data.

# Small batch — more regularising, typically better generalisation
model.fit(X_train, y_train, batch_size=32, epochs=50)

# Large batch — faster epochs, may need LR adjustment and extra regularisation
model.fit(X_train, y_train, batch_size=512, epochs=50)

Practical guidance: start with batch size 32–128. Scale up only if training is the bottleneck. When scaling up, apply the linear scaling rule for LR and monitor validation metrics carefully — large batches often need additional regularisation (warmup, extra dropout).

---

Q8: What is overfitting in neural networks and how do you address it?

Show answer

Overfitting occurs when the network memorises training data rather than learning generalisable patterns. The diagnostic signal: training loss decreases while validation loss plateaus or rises.

Dropout: During each training step, randomly zeroes a fraction of neurons. Forces the network to learn redundant representations — it cannot rely on any single path. At inference, all neurons are active (outputs are scaled to match expected values).

model = tf.keras.Sequential([
    tf.keras.layers.Dense(256, activation='relu'),
    tf.keras.layers.Dropout(0.5),
    tf.keras.layers.Dense(128, activation='relu'),
    tf.keras.layers.Dropout(0.3),
    tf.keras.layers.Dense(10, activation='softmax')
])

L2 Weight Decay: Adds λ · Σw² to the loss. Shrinks weights toward zero, discouraging the model from relying on any single large weight.

tf.keras.layers.Dense(
    128,
    activation='relu',
    kernel_regularizer=tf.keras.regularizers.l2(1e-4)
)

Early Stopping: Stop training when validation performance stops improving; restore the best checkpoint.

early_stop = tf.keras.callbacks.EarlyStopping(
    monitor='val_loss',
    patience=10,
    restore_best_weights=True
)

Data Augmentation: Artificially expand the training set with transformed versions of existing samples. For images: random flips, rotations, crops, colour jitter. Prevents memorisation of specific training examples.

data_augmentation = tf.keras.Sequential([
    tf.keras.layers.RandomFlip('horizontal'),
    tf.keras.layers.RandomRotation(0.1),
    tf.keras.layers.RandomZoom(0.1),
])

Rule of thumb: if training loss is much lower than validation loss, add regularisation. If both are high, the model is underfitting — increase capacity or train longer.

---

Q9: What is batch normalisation, where should it go, and why does it help?

Show answer

Batch Normalisation (BatchNorm) normalises the pre-activation inputs to each layer across the current mini-batch, then applies a learned scale (γ) and shift (β):

μ_B = mean of batch
σ²_B = variance of batch
x̂ = (x - μ_B) / √(σ²_B + ε)     # normalise
output = γ · x̂ + β                # scale and shift (learned)

Why it helps: - Keeps activations in a stable range — reduces the risk of saturation and dying ReLU - Allows higher learning rates — layers are less sensitive to the scale of parameters in previous layers - Acts as mild regularisation: batch statistics add noise similar to dropout - Reduces sensitivity to weight initialisation

Where to place it: place BatchNorm between the linear transformation and the activation:

Dense → BatchNorm → ReLU

model = tf.keras.Sequential([
    tf.keras.layers.Dense(128),
    tf.keras.layers.BatchNormalization(),
    tf.keras.layers.Activation('relu'),
    tf.keras.layers.Dense(64),
    tf.keras.layers.BatchNormalization(),
    tf.keras.layers.Activation('relu'),
    tf.keras.layers.Dense(10, activation='softmax')
])

Critical training/inference difference: during training, BatchNorm uses batch mean and variance. During inference, it uses running statistics accumulated during training. Always pass training=True during the training step; frameworks handle this automatically with model.fit().

Limitations: performs poorly with batch size < 8 (noisy statistics). Not suitable for recurrent networks. Alternatives: Layer Norm (used in Transformers), Group Norm (used in object detection with small batches).

---

Q10: What is gradient clipping and when is it necessary?

Show answer

Gradient clipping caps the magnitude of gradients before the weight update to prevent the exploding gradient problem — where gradients grow exponentially through deep networks or long sequences.

Norm clipping (preferred): if the global gradient norm exceeds a threshold, scale down the entire gradient vector proportionally. This preserves the direction of the gradient while limiting its magnitude.

if ||g|| > threshold:
    g ← g · (threshold / ||g||)

# Built into the optimiser (recommended)
optimizer = tf.keras.optimizers.Adam(
    learning_rate=1e-3,
    clipnorm=1.0
)

# Manual clipping in a custom training loop
with tf.GradientTape() as tape:
    loss = model(x, training=True)

gradients = tape.gradient(loss, model.trainable_variables)
gradients, _ = tf.clip_by_global_norm(gradients, clip_norm=1.0)
optimizer.apply_gradients(zip(gradients, model.trainable_variables))

When to use it: - RNNs, LSTMs, GRUs: gradients propagate through many time steps and frequently explode - Transformer training: useful during early training when gradients can be large - Any run where you see loss becoming NaN or inf

A clip norm of 1.0–5.0 is standard. If you need aggressive clipping (norm < 0.1), investigate the root cause — likely a learning rate that is too high.

---

Q11: What is transfer learning and what is the fine-tuning strategy?

Show answer

Transfer learning reuses weights from a model trained on a large dataset (e.g., ImageNet, large text corpora) as the starting point for a different but related task. Features learned at scale are broadly useful — fine-tuning is cheaper and more data-efficient than training from scratch.

Why features transfer: early layers of deep networks learn generic features (edges, textures for vision; morphology, syntax for NLP). These representations are task-agnostic and useful across many downstream tasks.

Strategy:

import tensorflow as tf

base_model = tf.keras.applications.ResNet50(
    weights='imagenet',
    include_top=False,
    input_shape=(224, 224, 3)
)

# Phase 1: Freeze base, train only new classification head
base_model.trainable = False

inputs = tf.keras.Input(shape=(224, 224, 3))
x = base_model(inputs, training=False)
x = tf.keras.layers.GlobalAveragePooling2D()(x)
x = tf.keras.layers.Dense(256, activation='relu')(x)
outputs = tf.keras.layers.Dense(5, activation='softmax')(x)
model = tf.keras.Model(inputs, outputs)

model.compile(optimizer=tf.keras.optimizers.Adam(1e-3),
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])
model.fit(X_train, y_train, epochs=10, validation_data=(X_val, y_val))

# Phase 2: Unfreeze and fine-tune with a much lower learning rate
base_model.trainable = True
model.compile(optimizer=tf.keras.optimizers.Adam(1e-5),
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])
model.fit(X_train, y_train, epochs=10, validation_data=(X_val, y_val))

What to freeze: early layers (generic features) → freeze. Late layers (task-specific features) → unfreeze for fine-tuning. If your task is very different from the pre-training task, unfreeze more layers. If your dataset is small, keep most layers frozen to avoid overfitting.

Why lower learning rate in phase 2? The pre-trained weights encode hard-won representations. A large learning rate will overwrite them. Fine-tuning nudges the weights gently rather than retraining from scratch.

---

Q12: What is the difference between an epoch and a training step?

Show answer

• Step (iteration): one forward-backward pass on a single mini-batch. Weights are updated once.
• Epoch: one complete pass through the entire training dataset.

steps_per_epoch = ceil(n_training_samples / batch_size)
total_steps = epochs × steps_per_epoch

When to stop training: avoid choosing a fixed epoch count upfront. Use early stopping based on validation loss instead.

history = model.fit(
    X_train, y_train,
    batch_size=64,
    epochs=200,
    validation_split=0.2,
    callbacks=[
        tf.keras.callbacks.EarlyStopping(
            monitor='val_loss',
            patience=15,
            restore_best_weights=True
        ),
        tf.keras.callbacks.ReduceLROnPlateau(
            monitor='val_loss',
            factor=0.5,
            patience=5,
            min_lr=1e-7
        )
    ]
)
best_epoch = len(history.history['val_loss'])
print(f"Training stopped at epoch {best_epoch}")

Comparing experiments: when batch sizes differ across experiments, comparing by epoch count is misleading. Comparing by total gradient steps is a fairer basis.

---

Q13: What is He initialisation and why is it preferred over random normal for ReLU networks?

Show answer

He initialisation draws weights from a distribution with mean 0 and variance 2 / fan_in, where fan_in is the number of inputs to the layer.

Why the factor of 2? ReLU zeroes out all negative pre-activations — roughly half the inputs. If you use Xavier/Glorot initialisation (variance 2 / (fan_in + fan_out)), the effective output variance of each layer is halved. Over many layers, activations shrink toward zero: the network suffers from a weak form of vanishing activations before training even begins.

He initialisation compensates by doubling the variance, keeping activations well-scaled from the start.

import numpy as np
import tensorflow as tf

# Manual demonstration
fan_in = 256
std_he = np.sqrt(2.0 / fan_in)
std_xavier = np.sqrt(2.0 / (fan_in + 128))

print(f"He std: {std_he:.4f}, Xavier std: {std_xavier:.4f}")

# In Keras
layer_relu = tf.keras.layers.Dense(
    128,
    activation='relu',
    kernel_initializer='he_normal'   # He for ReLU
)

layer_tanh = tf.keras.layers.Dense(
    128,
    activation='tanh',
    kernel_initializer='glorot_uniform'  # Xavier for tanh/sigmoid
)

Summary: He for ReLU and its variants (Leaky ReLU, ELU). Xavier/Glorot for tanh, sigmoid, and linear activations. Keras defaults to Glorot — override to He when using ReLU.

---