Training Neural Networks

A randomly initialized neural network produces random outputs. Training is the process of systematically adjusting every weight in the network so those outputs move closer to the correct answers. Understanding how that adjustment works — the mechanics of loss, gradients, and backpropagation — is what allows you to diagnose why a network is not learning and know what to change.

Learning Objectives

• Define what a loss function measures and choose the right one for a given task
• Explain gradient descent using the rolling-downhill intuition
• Describe backpropagation as "blame propagation" through the network
• Explain why learning rate is the most important hyperparameter to tune first
• Distinguish between epochs, steps, and batches, and explain mini-batch gradient descent
• Explain why Adam is the default optimizer and when you might choose something else

---

The Loss Function: Quantifying How Wrong You Are

Before you can improve, you need a number that measures how wrong the current predictions are. That is the loss function. It takes the network's predictions and the true labels, and returns a single scalar — lower is better.

Choosing the Right Loss

Task	Output activation	Loss function	Keras name
Regression	None (linear)	Mean Squared Error	"mse"
Regression (robust)	None (linear)	Mean Absolute Error	"mae"
Binary classification	Sigmoid	Binary cross-entropy	"binary_crossentropy"
Multiclass (one-hot labels)	Softmax	Categorical cross-entropy	"categorical_crossentropy"
Multiclass (integer labels)	Softmax	Sparse categorical cross-entropy	"sparse_categorical_crossentropy"

import numpy as np

# Binary cross-entropy computed manually
# Shows what Keras is doing under the hood

y_true = np.array([1, 0, 1, 1])       # actual labels
y_pred = np.array([0.9, 0.2, 0.8, 0.4])  # predicted probabilities

# BCE: -mean(y * log(p) + (1-y) * log(1-p))
epsilon = 1e-8  # avoid log(0)
bce = -np.mean(
    y_true * np.log(y_pred + epsilon) +
    (1 - y_true) * np.log(1 - y_pred + epsilon)
)
print(f"Binary cross-entropy loss: {bce:.4f}")
# Output: Binary cross-entropy loss: 0.2853

# Lower is better. Perfect predictions would give loss ≈ 0.

Info

Cross-entropy penalizes confident wrong predictions severely. If the true label is 1 and the network predicts 0.01, the loss term is -log(0.01) ≈ 4.6. If it predicts 0.99, the loss is -log(0.99) ≈ 0.01. This asymmetry is what drives the network to be confidently correct, not vaguely correct.

---

Gradient Descent: Rolling Downhill to the Minimum

Imagine the loss function as a hilly landscape. The height at any point is the loss. Your goal is to find the lowest valley. Gradient descent finds the direction that slopes downward most steeply from your current position, then takes a small step in that direction.

Formally, for each weight w:

w = w - learning_rate * (dLoss/dw)

The derivative dLoss/dw is the gradient — it tells you which direction increases the loss. Subtracting it moves you downhill.

# Gradient descent on a simple 1D loss landscape
import numpy as np
import matplotlib.pyplot as plt

# Loss function: L(w) = (w - 3)^2  (minimum at w=3)
def loss(w):
    return (w - 3) ** 2

def gradient(w):
    return 2 * (w - 3)

# Start at w = -1, learning rate = 0.1
w = -1.0
lr = 0.1
history = [w]

for step in range(20):
    grad = gradient(w)
    w = w - lr * grad
    history.append(w)

print(f"Final w: {w:.4f}")  # Should be close to 3.0
# Output: Final w: 2.9998

# Plot the optimization trajectory
w_range = np.linspace(-2, 5, 200)
plt.figure(figsize=(8, 4))
plt.plot(w_range, loss(w_range), label="Loss landscape")
plt.scatter(history, [loss(h) for h in history],
            color="red", zorder=5, label="Gradient descent steps")
plt.xlabel("Weight value (w)")
plt.ylabel("Loss")
plt.title("Gradient Descent Rolling Downhill")
plt.legend()
plt.show()

Warning

Gradient descent finds a local minimum, not necessarily the global minimum. For neural networks this is usually fine — the loss landscape has many local minima, but most of them generalize similarly well in practice. Saddle points (where the gradient is zero but you are not at a minimum) are a bigger practical concern than local minima.

---

Backpropagation: Blaming the Right Weights

Backpropagation answers a specific question: given that the network made an error, which weights are most responsible, and by how much?

The Intuition

Think of the network as a chain of decisions. The final prediction was wrong. Backprop traces backward through each layer, asking: "How much did this weight contribute to the error?"

The answer comes from the chain rule of calculus:

dLoss/dw1 = dLoss/dOutput × dOutput/dLayer2 × dLayer2/dLayer1 × dLayer1/dw1

Each term is a local gradient that is easy to compute. Multiplying them together gives the total gradient of the loss with respect to w1 — how much changing w1 by a tiny amount would change the final loss.

[Input] → [Layer 1] → [Layer 2] → [Output] → [Loss]
                                                 ↑
                    Backprop flows loss signal ←←←

This is not magic — it is the same calculus you used in high school, applied systematically across thousands of weights simultaneously.

Blame Propagation in Code

import numpy as np

# Minimal two-layer network: forward and backward pass from scratch
np.random.seed(42)

# Data: 4 samples, 2 features
X = np.array([[0.1, 0.2], [0.3, 0.4], [0.5, 0.6], [0.7, 0.8]])
y = np.array([[1], [0], [1], [0]])

# Initialize weights
W1 = np.random.randn(2, 3) * 0.1
b1 = np.zeros((1, 3))
W2 = np.random.randn(3, 1) * 0.1
b2 = np.zeros((1, 1))

lr = 0.1

for epoch in range(5):
    # --- Forward pass ---
    z1 = X @ W1 + b1
    a1 = np.maximum(0, z1)          # ReLU

    z2 = a1 @ W2 + b2
    a2 = 1 / (1 + np.exp(-z2))     # Sigmoid

    # Loss: binary cross-entropy
    eps = 1e-8
    loss = -np.mean(y * np.log(a2 + eps) + (1 - y) * np.log(1 - a2 + eps))

    # --- Backward pass (backpropagation) ---
    # Gradient of loss w.r.t. sigmoid output
    dA2 = -(y / (a2 + eps)) + (1 - y) / (1 - a2 + eps)
    dZ2 = dA2 * a2 * (1 - a2)       # sigmoid derivative: σ(1-σ)

    dW2 = a1.T @ dZ2 / len(X)       # blame W2
    db2 = np.mean(dZ2, axis=0, keepdims=True)

    dA1 = dZ2 @ W2.T
    dZ1 = dA1 * (z1 > 0)            # ReLU derivative: 1 if z>0, else 0

    dW1 = X.T @ dZ1 / len(X)        # blame W1
    db1 = np.mean(dZ1, axis=0, keepdims=True)

    # --- Weight update ---
    W2 -= lr * dW2
    b2 -= lr * db2
    W1 -= lr * dW1
    b1 -= lr * db1

    print(f"Epoch {epoch+1}, Loss: {loss:.4f}")

# Output:
# Epoch 1, Loss: 0.6939
# Epoch 2, Loss: 0.6883
# Epoch 3, Loss: 0.6829
# Epoch 4, Loss: 0.6778
# Epoch 5, Loss: 0.6728

Loss is decreasing — the network is learning. In practice, you never write this yourself. Keras and TensorFlow handle backprop automatically via automatic differentiation (tf.GradientTape). But understanding what is happening underneath makes you a better debugger.

Success

Backpropagation is not a mysterious algorithm — it is systematic application of the chain rule, flowing the error signal backward through the network and computing each weight's share of the blame. The key insight: local gradients are easy to compute, and the chain rule lets you multiply them to get the global gradient for any weight.

---

Learning Rate: The Most Important Hyperparameter

The learning rate controls how large a step to take in the downhill direction after each gradient computation.

import numpy as np
import matplotlib.pyplot as plt

def loss(w):
    return (w - 3) ** 2

def gradient(w):
    return 2 * (w - 3)

fig, axes = plt.subplots(1, 3, figsize=(14, 4))
settings = [
    (0.01, "Too small — slow convergence"),
    (0.3,  "Good — converges in ~20 steps"),
    (1.1,  "Too large — diverges"),
]

for ax, (lr, title) in zip(axes, settings):
    w = -1.0
    history = [w]
    for _ in range(30):
        w = w - lr * gradient(w)
        history.append(w)
        if abs(w) > 20:
            break

    w_range = np.linspace(-5, 8, 200)
    ax.plot(w_range, loss(w_range), color="steelblue")
    ax.scatter(history, [loss(min(max(h, -5), 8)) for h in history],
               color="red", s=20, zorder=5)
    ax.set_title(f"lr = {lr}\n{title}", fontsize=10)
    ax.set_xlim(-5, 8)
    ax.set_ylim(0, 50)

plt.tight_layout()
plt.show()

Learning rate	Symptom
Too small (< 0.0001)	Loss decreases glacially; training takes forever
Too large (> 0.1 for SGD)	Loss oscillates or explodes; NaN values appear
Good range	Loss decreases smoothly, stabilizes after some epochs

Tip

When training diverges (loss becomes NaN or shoots upward), the first thing to try is cutting the learning rate by 10x. When training is stable but slow, try multiplying the learning rate by 3x. The Adam optimizer (below) largely handles this automatically.

---

Epochs, Steps, and Batches

These three terms describe how data flows through the training loop.

Dataset: 10,000 samples
Batch size: 32
Steps per epoch: 10,000 / 32 = 313 steps
Epochs: 50

Total weight updates: 313 × 50 = 15,625

# What model.fit() is doing internally (simplified)
import numpy as np

def training_loop(X, y, model, optimizer, loss_fn,
                  epochs=10, batch_size=32):
    n = len(X)
    for epoch in range(epochs):
        # Shuffle data at the start of each epoch
        indices = np.random.permutation(n)
        X_shuffled = X[indices]
        y_shuffled = y[indices]

        epoch_loss = 0
        steps = 0

        for start in range(0, n, batch_size):
            # Extract mini-batch
            X_batch = X_shuffled[start:start + batch_size]
            y_batch = y_shuffled[start:start + batch_size]

            # Forward pass + loss + backward + update
            predictions = model.forward(X_batch)
            batch_loss = loss_fn(y_batch, predictions)
            gradients = model.backward(batch_loss)
            optimizer.update(model.weights, gradients)

            epoch_loss += batch_loss
            steps += 1

        print(f"Epoch {epoch+1}: avg loss = {epoch_loss / steps:.4f}")

Info

Why not use the whole dataset in one step? Computing the gradient on the full dataset (batch gradient descent) is stable but slow and memory-intensive. Computing it on one sample at a time (stochastic gradient descent) is fast but noisy. Mini-batch gradient descent — the standard — balances both: fast updates with enough averaging to reduce noise. Typical batch sizes: 32, 64, 128.

---

The Adam Optimizer: Why It Is the Default

Vanilla gradient descent uses a single global learning rate for all weights. Adam (Adaptive Moment Estimation) maintains a separate adaptive learning rate for each weight, adjusted based on the history of gradients.

The two ideas behind Adam:

1. Momentum: accumulate a running average of past gradients. Weights that have been consistently pointing in the same direction get a larger step (like a ball rolling and building speed downhill).
2. Adaptive learning rates: weights with historically large gradients get smaller steps; weights with historically small gradients get larger steps. This prevents any single weight from dominating the update.

import tensorflow as tf
from tensorflow import keras

# Adam with default hyperparameters — correct for ~90% of problems
model.compile(
    optimizer=keras.optimizers.Adam(learning_rate=0.001),
    loss="binary_crossentropy",
    metrics=["accuracy"]
)

# Common alternatives and when to use them:
# SGD with momentum — image classification with careful lr scheduling
# RMSprop — recurrent networks
# AdamW — when L2 regularization matters (Adam + weight decay, more theoretically sound)

sgd_with_momentum = keras.optimizers.SGD(learning_rate=0.01, momentum=0.9)
rmsprop = keras.optimizers.RMSprop(learning_rate=0.001)
adamw = keras.optimizers.AdamW(learning_rate=0.001, weight_decay=0.01)

Success

Start with Adam at learning_rate=0.001. It works out of the box on the vast majority of problems. Only switch to something else if you have a specific reason (e.g., SGD + cosine annealing for ResNet-style image classifiers, RMSprop for LSTMs).

---

Putting It Together: The Training Loop in Keras

import tensorflow as tf
from tensorflow import keras
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

# Generate a synthetic binary classification problem
X, y = make_classification(
    n_samples=2000, n_features=10, n_informative=6,
    random_state=42
)

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Always scale inputs before feeding to a neural network
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test  = scaler.transform(X_test)

# Build model
model = keras.Sequential([
    keras.layers.Dense(64, activation="relu", input_shape=(10,)),
    keras.layers.Dense(32, activation="relu"),
    keras.layers.Dense(1,  activation="sigmoid"),
])

# Compile: attach loss, optimizer, and metrics
model.compile(
    optimizer=keras.optimizers.Adam(learning_rate=0.001),
    loss="binary_crossentropy",
    metrics=["accuracy"]
)

# Train: each call to model.fit runs the full training loop
history = model.fit(
    X_train, y_train,
    validation_split=0.2,   # hold out 20% of training data for validation
    epochs=30,
    batch_size=32,
    verbose=1               # print progress each epoch
)

# Output (last few lines):
# Epoch 28/30: loss=0.2841 accuracy=0.8844 val_loss=0.3102 val_accuracy=0.8625
# Epoch 29/30: loss=0.2789 accuracy=0.8906 val_loss=0.3089 val_accuracy=0.8688
# Epoch 30/30: loss=0.2771 accuracy=0.8906 val_loss=0.3094 val_accuracy=0.8688

---

Interview Questions

Q1: What is backpropagation and why is it necessary?

Show answer

Backpropagation is the algorithm that computes the gradient of the loss with respect to every weight in the network by applying the chain rule backward from the output to the input. It is necessary because direct computation of each weight's gradient would be prohibitively expensive without the chain rule's ability to reuse intermediate computations. In plain terms: it figures out how much each weight is "to blame" for the prediction error.

Q2: What happens if you set the learning rate too high?

Show answer

A learning rate that is too high causes the weight updates to overshoot the minimum. The loss oscillates or diverges — you will see it bouncing or shooting toward NaN. Practically: cut the learning rate by 10x and retrain. With Adam this is less common because of its adaptive step sizes, but still possible.

Q3: What is the difference between an epoch and a batch?

Show answer

A batch is a subset of the training data used for a single weight update. An epoch is one complete pass through the entire training dataset. With 1,000 samples and a batch size of 32, one epoch involves about 31 weight updates (steps). Training typically runs for tens to hundreds of epochs.

Q4: Why is Adam usually the best optimizer to start with?

Show answer

Adam combines momentum (accumulated gradient history to smooth updates) with adaptive per-weight learning rates. This means it handles sparse gradients, scales well across different network architectures, and converges faster than plain SGD without requiring careful learning rate tuning. Its default hyperparameters (lr=0.001, beta1=0.9, beta2=0.999) work well across a wide range of tasks.

---

---

What's Next

You've covered backpropagation as systematic chain-rule gradient computation, the learning rate's effect on convergence and divergence, mini-batch gradient descent and the epoch/step/batch relationship, and Adam's momentum and adaptive learning rate mechanisms. Next up: 03-keras-quickstart — where you'll build and train binary classification, multiclass classification, and regression networks using the Keras Sequential API, learn the correct output activation and loss function for each task type, and save and load models for deployment.

Optional Deep Dive

Read "Dive into Deep Learning" (d2l.ai) Chapter 5 (Multilayer Perceptrons) — it provides both the mathematical derivation and the Python implementation of the forward and backward pass, helping you understand exactly what Keras's model.fit() is doing under the hood and why the training loop is structured the way it is.

Back: Neural Network Intuition | Next: Keras Quickstart