Overfitting and Regularization¶
A neural network with enough parameters can memorize the training data — including its noise. On the training set it scores perfectly; on new data it fails. This is overfitting, and it is the most common failure mode in deep learning. Knowing how to detect it early and which tools to apply is the difference between a model that works in production and one that only works on the data you already have.
Learning Objectives¶
- Identify the overfitting signature from training curves
- Apply Dropout and explain what it does mechanically
- Apply L1 and L2 weight regularization
- Use BatchNormalization to stabilize training
- Configure EarlyStopping and ModelCheckpoint callbacks
- Know which tool to reach for first and why
Recognizing Overfitting¶
The training curve tells you everything. Plot loss over epochs for both the training and validation sets.
import numpy as np
import tensorflow as tf
from tensorflow import keras
import matplotlib.pyplot as plt
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
# Create a small dataset — small datasets overfit more easily
X, y = make_classification(
n_samples=500, n_features=20, n_informative=5, random_state=42
)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42
)
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)
# Deliberately overparameterized model — will overfit on 400 samples
overfit_model = keras.Sequential([
keras.layers.Dense(256, activation="relu", input_shape=(20,)),
keras.layers.Dense(256, activation="relu"),
keras.layers.Dense(256, activation="relu"),
keras.layers.Dense(1, activation="sigmoid"),
])
overfit_model.compile(optimizer="adam", loss="binary_crossentropy",
metrics=["accuracy"])
history = overfit_model.fit(
X_train, y_train,
validation_split=0.2,
epochs=100,
batch_size=32,
verbose=0,
)
# Plot the signature
plt.figure(figsize=(10, 4))
plt.plot(history.history["loss"], label="Train loss")
plt.plot(history.history["val_loss"], label="Val loss", linestyle="--")
plt.xlabel("Epoch")
plt.ylabel("Loss")
plt.title("Overfitting: Train loss falls, Val loss rises")
plt.legend()
plt.show()
The overfitting signature:
- Training loss continues to fall epoch after epoch
- Validation loss falls initially, then flattens or begins rising
- The gap between train and val loss grows over time
Warning
Many beginners only look at training accuracy and conclude their model is working. Always track both training and validation metrics. A model with 99% train accuracy and 65% val accuracy is not a good model — it has memorized the training set.
Tool 1: EarlyStopping¶
The most practical regularization tool. Stop training when the validation loss stops improving, and restore the weights from the best epoch.
early_stop = keras.callbacks.EarlyStopping(
monitor="val_loss", # watch validation loss
patience=10, # stop after 10 epochs with no improvement
restore_best_weights=True, # rewind weights to the best epoch
verbose=1,
)
model = keras.Sequential([
keras.layers.Dense(256, activation="relu", input_shape=(20,)),
keras.layers.Dense(256, activation="relu"),
keras.layers.Dense(1, activation="sigmoid"),
])
model.compile(optimizer="adam", loss="binary_crossentropy",
metrics=["accuracy"])
history = model.fit(
X_train, y_train,
validation_split=0.2,
epochs=200, # set high — early stopping will stop us
batch_size=32,
callbacks=[early_stop],
verbose=1,
)
# Output:
# Epoch 38: val_loss did not improve from 0.47231
# Restoring model weights from the end of the best epoch: 28.
# Training stopped at epoch 38. Best model was from epoch 28.
Tip
Set patience based on how noisy your validation loss is. If it fluctuates a lot, use patience=15 or more. If it is smooth, patience=5 is enough. With restore_best_weights=True, you do not need to save the model separately — EarlyStopping restores it automatically.
Tool 2: Dropout¶
Dropout randomly sets a fraction of neuron outputs to zero during each training step. Each batch sees a different random subset of the network. This forces the network to learn redundant representations — no single neuron can become essential to the prediction.
At inference time, Dropout is turned off automatically by Keras. No adjustment is needed.
# Adding Dropout layers after each hidden layer
model_with_dropout = keras.Sequential([
keras.layers.Dense(256, activation="relu", input_shape=(20,)),
keras.layers.Dropout(0.4), # zero out 40% of outputs randomly
keras.layers.Dense(256, activation="relu"),
keras.layers.Dropout(0.4),
keras.layers.Dense(1, activation="sigmoid"),
])
model_with_dropout.compile(optimizer="adam", loss="binary_crossentropy",
metrics=["accuracy"])
history_do = model_with_dropout.fit(
X_train, y_train,
validation_split=0.2,
epochs=100,
batch_size=32,
callbacks=[early_stop],
verbose=0,
)
Comparing With and Without Dropout¶
def plot_comparison(h1, h2, labels=("No Dropout", "With Dropout")):
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 4))
for h, label in [(h1, labels[0]), (h2, labels[1])]:
ax1.plot(h.history["loss"], label=f"{label} — Train")
ax1.plot(h.history["val_loss"], label=f"{label} — Val",
linestyle="--")
ax2.plot(h.history["accuracy"], label=f"{label} — Train")
ax2.plot(h.history["val_accuracy"], label=f"{label} — Val",
linestyle="--")
ax1.set_title("Loss"); ax1.set_xlabel("Epoch"); ax1.legend()
ax2.set_title("Accuracy"); ax2.set_xlabel("Epoch"); ax2.legend()
plt.tight_layout()
plt.show()
plot_comparison(history, history_do)
Info
Dropout rates between 0.2 and 0.5 are typical for fully-connected layers. Do not apply Dropout to the output layer. Do not use very high rates (>0.6) — it makes training slow and unstable. A common pattern: higher rates for larger layers, lower rates for smaller ones.
Warning
Dropout makes training loss artificially high because neurons are being randomly disabled during the forward pass. You will notice that validation loss (computed with Dropout off) can sometimes be lower than training loss during training. This is expected and normal — it disappears after training completes.
Tool 3: L1 and L2 Regularization¶
Weight regularization adds a penalty term to the loss that discourages large weights.
- L2 (Ridge): penalizes the sum of squared weights. Pushes all weights toward zero but rarely to exactly zero. The most common choice.
- L1 (Lasso): penalizes the sum of absolute weights. Can push weights exactly to zero (produces sparse networks).
- L1+L2: both penalties combined.
from tensorflow.keras import regularizers
model_l2 = keras.Sequential([
keras.layers.Dense(
256,
activation="relu",
input_shape=(20,),
kernel_regularizer=regularizers.l2(0.001), # penalty on weights
),
keras.layers.Dense(
256,
activation="relu",
kernel_regularizer=regularizers.l2(0.001),
),
keras.layers.Dense(1, activation="sigmoid"),
])
model_l2.compile(optimizer="adam", loss="binary_crossentropy",
metrics=["accuracy"])
# L1 — produces sparse weights
model_l1 = keras.Sequential([
keras.layers.Dense(64, activation="relu", input_shape=(20,),
kernel_regularizer=regularizers.l1(0.001)),
keras.layers.Dense(1, activation="sigmoid"),
])
# L1 + L2 combined (ElasticNet equivalent)
model_l1l2 = keras.Sequential([
keras.layers.Dense(64, activation="relu", input_shape=(20,),
kernel_regularizer=regularizers.l1_l2(l1=0.0005, l2=0.001)),
keras.layers.Dense(1, activation="sigmoid"),
])
Tip
L2 regularization is the default choice. Start with l2=0.001. If the model still overfits, try 0.01. Combine L2 with Dropout for strong regularization on large models. L1 is useful when you want interpretability — it zeros out unimportant weight connections.
Tool 4: BatchNormalization¶
BatchNormalization normalizes the activations within each mini-batch to have mean 0 and standard deviation 1, then applies a learned scale and shift. It was originally designed to stabilize training (reducing sensitivity to initialization and learning rate), but it also acts as a mild regularizer.
model_bn = keras.Sequential([
keras.layers.Dense(256, activation=None, input_shape=(20,)),
keras.layers.BatchNormalization(), # normalize before activation
keras.layers.Activation("relu"),
keras.layers.Dense(256, activation=None),
keras.layers.BatchNormalization(),
keras.layers.Activation("relu"),
keras.layers.Dense(1, activation="sigmoid"),
])
model_bn.compile(
optimizer=keras.optimizers.Adam(learning_rate=0.01), # can use higher lr with BN
loss="binary_crossentropy",
metrics=["accuracy"],
)
history_bn = model_bn.fit(
X_train, y_train,
validation_split=0.2,
epochs=60,
batch_size=32,
verbose=0,
)
Info
BatchNorm before or after activation? The original paper places it before the activation. In practice, both orderings work. Some practitioners use BatchNorm after activation for simplicity (Dense(relu) → BatchNorm). Either is acceptable; consistency within a model matters more than the exact choice.
Warning
Do not use BatchNormalization and Dropout together on the same layer without understanding the interaction. They can cancel each other out or produce unexpected behavior. If you need strong regularization, pick one. BatchNormalization alone is often enough for medium-sized networks.
Tool 5: ModelCheckpoint¶
EarlyStopping with restore_best_weights=True handles most cases, but ModelCheckpoint gives you more control — saving the model to disk at every improvement.
checkpoint = keras.callbacks.ModelCheckpoint(
filepath="best_model.keras", # save location
monitor="val_loss",
save_best_only=True, # only overwrite when val_loss improves
verbose=1,
)
reduce_lr = keras.callbacks.ReduceLROnPlateau(
monitor="val_loss",
factor=0.5, # multiply lr by 0.5 when val_loss plateaus
patience=5,
min_lr=1e-6,
verbose=1,
)
model.fit(
X_train, y_train,
validation_split=0.2,
epochs=100,
batch_size=32,
callbacks=[early_stop, checkpoint, reduce_lr],
verbose=1,
)
# Load best model from disk
best_model = keras.models.load_model("best_model.keras")
Putting It All Together: Regularized Training Pipeline¶
import numpy as np
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import regularizers
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
import matplotlib.pyplot as plt
# Data
X, y = make_classification(
n_samples=800, n_features=20, n_informative=6, random_state=42
)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42
)
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)
# Regularized model
model = keras.Sequential([
keras.layers.Dense(128, activation="relu", input_shape=(20,),
kernel_regularizer=regularizers.l2(0.001)),
keras.layers.Dropout(0.3),
keras.layers.Dense(64, activation="relu",
kernel_regularizer=regularizers.l2(0.001)),
keras.layers.Dropout(0.3),
keras.layers.Dense(1, activation="sigmoid"),
])
model.compile(
optimizer=keras.optimizers.Adam(learning_rate=0.001),
loss="binary_crossentropy",
metrics=["accuracy"],
)
callbacks = [
keras.callbacks.EarlyStopping(
monitor="val_loss", patience=10, restore_best_weights=True
),
keras.callbacks.ReduceLROnPlateau(
monitor="val_loss", factor=0.5, patience=5, min_lr=1e-6
),
]
history = model.fit(
X_train, y_train,
validation_split=0.2,
epochs=200,
batch_size=32,
callbacks=callbacks,
verbose=1,
)
test_loss, test_acc = model.evaluate(X_test, y_test, verbose=0)
print(f"\nTest accuracy: {test_acc:.4f}")
# Visualize
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 4))
ax1.plot(history.history["loss"], label="Train loss")
ax1.plot(history.history["val_loss"], label="Val loss", linestyle="--")
ax1.set_title("Loss")
ax1.legend()
ax2.plot(history.history["accuracy"], label="Train accuracy")
ax2.plot(history.history["val_accuracy"], label="Val accuracy", linestyle="--")
ax2.set_title("Accuracy")
ax2.legend()
plt.tight_layout()
plt.show()
Decision Guide: Which Tool to Use First¶
| Situation | First tool to try |
|---|---|
| Model overfits on medium-sized dataset | EarlyStopping, then Dropout |
| Large model, training is unstable | BatchNormalization |
| Very small dataset (< 1,000 samples) | L2 regularization + Dropout |
| Overfitting persists after Dropout | Reduce model size (fewer neurons) |
| Training is fine, need to recover best checkpoint | ModelCheckpoint |
| Learning plateaus mid-training | ReduceLROnPlateau |
Success
The most reliable regularization workflow: (1) Set up EarlyStopping with patience=10 and restore_best_weights=True on every training run. (2) If you still overfit, add Dropout(0.3) after each hidden layer. (3) If you still overfit, reduce model size. Adding complexity — more layers, more neurons — should only come after confirming the model is underfitting.
Interview Questions¶
Q1: What is the overfitting signature you look for in training curves?
Show answer
Training loss continues to decrease while validation loss plateaus or begins to rise. The gap between training and validation metrics grows over epochs. A model with perfect training accuracy and poor validation accuracy has memorized the training data — it has not learned generalizable patterns.
Q2: What does Dropout do mechanically, and why does it help with overfitting?
Show answer
During each training step, Dropout randomly sets a fraction of neuron outputs to zero. The network cannot rely on any single neuron being present, so it must learn redundant, distributed representations of the same information. At inference time, Dropout is turned off — all neurons are active. This prevents co-adaptation of neurons (where neurons become dependent on each other in ways that only work for the training data).
Q3: What is the difference between L1 and L2 regularization?
Show answer
Both add a penalty to the loss function that discourages large weights. L2 penalizes the sum of squared weights — it pushes all weights toward zero but rarely exactly to zero, producing small distributed weights. L1 penalizes the sum of absolute weights — it can push weights exactly to zero, producing sparse networks where unimportant connections are eliminated. L2 is the default choice; L1 is useful when interpretability or feature selection matters.
Q4: What does restore_best_weights=True do in EarlyStopping?
Show answer
Without it, when training stops (because val_loss stopped improving), the model retains the weights from the final epoch — which may be worse than the best epoch seen during training. With restore_best_weights=True, Keras automatically rewinds the model weights to the epoch where val_loss was lowest. This means you always end up with the best model, not the last model.
Q5: Why might validation loss be lower than training loss during training?
Show answer
When Dropout is active, it randomly disables neurons during the forward pass at training time, artificially inflating training loss. At validation time, Dropout is disabled, so all neurons are active and the model performs at its full capacity. This can make val_loss temporarily lower than train_loss. It is expected behavior and disappears after training when you evaluate the model with Dropout off.