Skip to content

Ridge, Lasso, and ElasticNet

Linear regression is optimal when its assumptions hold and the number of features is small relative to the number of observations. In the real world, neither condition is usually satisfied. Datasets frequently have dozens or hundreds of features, many of which are correlated with each other or irrelevant to the target. Regularisation is how we make linear models robust to these conditions — by adding a penalty that discourages large or numerous coefficients, and forces the model to be more conservative.

Learning Objectives

  • Explain why ordinary linear regression fails with many correlated features
  • Describe the geometric intuition behind L2 (Ridge) and L1 (Lasso) penalties
  • Predict when Lasso will drive coefficients to exactly zero and why
  • Choose the right regularisation method based on the structure of your data
  • Use cross-validated alpha selection with RidgeCV and LassoCV
  • Understand ElasticNet as a principled compromise between Ridge and Lasso

Why Regularisation Exists

Consider a dataset with 500 observations and 200 features. Ordinary linear regression will find a set of coefficients that perfectly (or near-perfectly) minimises training error. But it has so many parameters relative to data points that it will memorise noise — the coefficients will be large, unstable, and will not generalise.

The same problem appears with fewer features when those features are correlated. If total_rooms and total_bedrooms move together, the OLS solution may assign a coefficient of +50,000 to one and -48,000 to the other — essentially cancelling each other in noise. The coefficients are technically correct on the training data but are wildly sensitive to small changes in the training set.

Info

Multicollinearity occurs when two or more features are highly correlated. In the presence of multicollinearity, the matrix XᵀX becomes near-singular (nearly non-invertible), and the OLS coefficient estimates become extremely unstable. Ridge regression directly addresses this by adding a small constant to the diagonal of XᵀX before inverting — making the matrix well-conditioned regardless of multicollinearity.

Regularisation adds a penalty term to the loss function:

Regularised Loss = MSE + λ × Penalty(coefficients)

The penalty discourages large coefficients. The hyperparameter λ (called alpha in scikit-learn) controls how hard the penalty pushes back against large weights. Larger alpha = stronger regularisation = simpler model.

Ridge Regression — L2 Penalty

Ridge adds the sum of squared coefficients to the loss:

Ridge Loss = MSE + α × Σβᵢ²

The Geometry

Imagine the space of all possible coefficient vectors as a 2D plane (for two features). The ordinary least squares solution sits at the bottom of an elliptical bowl — the point that minimises MSE. Ridge adds a circular constraint: the solution must stay within a circle of radius 1/α centred at the origin. The optimal Ridge solution is where the ellipse first touches the circle.

Because the circle is smooth (no corners), the Ridge solution shrinks all coefficients toward zero, but rarely to exactly zero. Every feature contributes, just with diminished influence.

When to Use Ridge

  • You believe most features are genuinely relevant, but some are noisy
  • Features are correlated — Ridge handles this gracefully
  • You want to keep all features in the model (for interpretability or downstream use)
  • The number of features is large relative to observations
import numpy as np
import pandas as pd
from sklearn.datasets import fetch_california_housing
from sklearn.linear_model import Ridge, RidgeCV
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.metrics import mean_absolute_error, r2_score

housing = fetch_california_housing(as_frame=True)
X, y = housing.data, housing.target

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Manual alpha selection
ridge_pipe = Pipeline([
    ("scaler", StandardScaler()),
    ("model", Ridge(alpha=1.0))
])
ridge_pipe.fit(X_train, y_train)
y_pred_ridge = ridge_pipe.predict(X_test)

print(f"Ridge (alpha=1.0)")
print(f"  MAE: {mean_absolute_error(y_test, y_pred_ridge):.3f}")
print(f"  R²:  {r2_score(y_test, y_pred_ridge):.3f}")
# Output:
# Ridge (alpha=1.0)
#   MAE: 0.533
#   R²:  0.576

Cross-Validated Alpha Selection with RidgeCV

Choosing alpha by hand is guesswork. RidgeCV tries multiple alpha values using cross-validation on the training set and picks the best one automatically:

from sklearn.linear_model import RidgeCV

# Try alphas across several orders of magnitude
alphas_to_try = np.logspace(-3, 4, 50)  # 0.001 to 10,000

ridge_cv_pipe = Pipeline([
    ("scaler", StandardScaler()),
    ("model", RidgeCV(alphas=alphas_to_try, cv=5))
])
ridge_cv_pipe.fit(X_train, y_train)
y_pred_cv = ridge_cv_pipe.predict(X_test)

best_alpha = ridge_cv_pipe.named_steps["model"].alpha_
print(f"Best alpha found by CV: {best_alpha:.4f}")
print(f"MAE: {mean_absolute_error(y_test, y_pred_cv):.3f}")
print(f"R²:  {r2_score(y_test, y_pred_cv):.3f}")
# Output:
# Best alpha found by CV: 0.1778
# MAE: 0.533
# R²:  0.576

Tip

Always search alpha on a log scale (np.logspace), not a linear scale. The effect of regularisation is multiplicative — the difference between alpha=0.01 and alpha=0.1 is far more significant than between alpha=100 and alpha=101. Searching 0.001, 0.01, 0.1, 1, 10, 100, 1000 covers the relevant range far better than 1, 2, 3, 4, 5.

Lasso Regression — L1 Penalty

Lasso adds the sum of absolute values of coefficients to the loss:

Lasso Loss = MSE + α × Σ|βᵢ|

The Geometry — Why Lasso Sets Coefficients to Zero

The L1 constraint region is a diamond (in 2D), with sharp corners at the axes. The optimal Lasso solution is where the MSE ellipse first touches the diamond. Geometrically, the corners of the diamond sit exactly on the coordinate axes — meaning that when the ellipse's minimum is close to an axis, the solution is pushed directly onto a corner, where one coordinate is zero.

This is not a coincidence or a threshold trick — it is the inevitable consequence of the L1 geometry. Lasso performs automatic feature selection by driving irrelevant coefficients to exactly zero.

When to Use Lasso

  • You suspect many features are irrelevant and want the model to identify which ones
  • You want a sparse model (fewer non-zero coefficients) for interpretability or deployment efficiency
  • Features are not strongly correlated with each other (correlated features cause Lasso to pick one arbitrarily)
  • High-dimensional data (more features than observations)
from sklearn.linear_model import Lasso, LassoCV

# Cross-validated Lasso
lasso_cv_pipe = Pipeline([
    ("scaler", StandardScaler()),
    ("model", LassoCV(cv=5, max_iter=10_000, random_state=42))
])
lasso_cv_pipe.fit(X_train, y_train)
y_pred_lasso = lasso_cv_pipe.predict(X_test)

lasso_model = lasso_cv_pipe.named_steps["model"]
best_alpha_lasso = lasso_model.alpha_

# Count non-zero coefficients (feature selection in action)
non_zero = np.sum(lasso_model.coef_ != 0)
total_features = X.shape[1]

print(f"Best alpha: {best_alpha_lasso:.4f}")
print(f"Non-zero coefficients: {non_zero} / {total_features}")
print(f"MAE: {mean_absolute_error(y_test, y_pred_lasso):.3f}")
print(f"R²:  {r2_score(y_test, y_pred_lasso):.3f}")
# Output:
# Best alpha: 0.0011
# Non-zero coefficients: 8 / 8  (all kept on this dataset — it's a small one)
# MAE: 0.533
# R²:  0.576

Seeing Lasso's feature selection on a synthetic high-dimensional example:

from sklearn.datasets import make_regression
from sklearn.linear_model import LassoCV
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split

# 100 features, but only 10 are truly informative
X_syn, y_syn = make_regression(
    n_samples=300,
    n_features=100,
    n_informative=10,
    noise=20,
    random_state=42
)

X_syn_train, X_syn_test, y_syn_train, y_syn_test = train_test_split(
    X_syn, y_syn, test_size=0.2, random_state=42
)

scaler = StandardScaler()
X_syn_train_scaled = scaler.fit_transform(X_syn_train)

lasso = LassoCV(cv=5, max_iter=10_000, random_state=42)
lasso.fit(X_syn_train_scaled, y_syn_train)

non_zero_lasso = np.sum(lasso.coef_ != 0)
print(f"Features retained by Lasso: {non_zero_lasso} / 100")
# Output:
# Features retained by Lasso: 12 / 100
# Lasso found 12 relevant features — close to the true 10

Warning

Lasso is unstable when features are correlated. If feature_A and feature_B are highly correlated and both genuinely relevant, Lasso tends to pick one and drive the other to zero — essentially arbitrarily. The one it keeps can change entirely if you resample the data. In this situation, use ElasticNet or Ridge instead.

Coefficient Comparison: Ridge vs. Lasso

Visualising how coefficients change with different alpha values is one of the best ways to build intuition:

import matplotlib.pyplot as plt
from sklearn.linear_model import ridge_regression, lasso_path
from sklearn.datasets import make_regression

X_demo, y_demo = make_regression(
    n_samples=200, n_features=10, n_informative=5,
    noise=15, random_state=0
)
X_demo_scaled = StandardScaler().fit_transform(X_demo)

# Ridge coefficient paths
alphas = np.logspace(-2, 4, 100)
ridge_coefs = []
for alpha in alphas:
    r = Ridge(alpha=alpha)
    r.fit(X_demo_scaled, y_demo)
    ridge_coefs.append(r.coef_)
ridge_coefs = np.array(ridge_coefs)

# Lasso coefficient paths
lasso_alphas, lasso_coefs, _ = lasso_path(X_demo_scaled, y_demo)

fig, axes = plt.subplots(1, 2, figsize=(14, 5))

for i in range(ridge_coefs.shape[1]):
    axes[0].plot(np.log10(alphas), ridge_coefs[:, i])
axes[0].axhline(0, color="black", linewidth=0.5)
axes[0].set_xlabel("log₁₀(alpha)")
axes[0].set_ylabel("Coefficient value")
axes[0].set_title("Ridge: All coefficients shrink gradually toward zero")

for i in range(lasso_coefs.shape[0]):
    axes[1].plot(-np.log10(lasso_alphas + 1e-10), lasso_coefs[i, :])
axes[1].axhline(0, color="black", linewidth=0.5)
axes[1].set_xlabel("Regularisation strength →")
axes[1].set_ylabel("Coefficient value")
axes[1].set_title("Lasso: Coefficients drop to exactly zero")

plt.tight_layout()
plt.show()

The contrast is stark: Ridge coefficients glide smoothly toward zero, approaching but never reaching it. Lasso coefficients hit zero at a specific alpha and stay there — a clean, interpretable feature removal.

ElasticNet — The Middle Ground

ElasticNet combines L1 and L2 penalties:

ElasticNet Loss = MSE + α × [l1_ratio × Σ|βᵢ| + (1 - l1_ratio) × Σβᵢ²]

The l1_ratio parameter controls the blend: - l1_ratio = 1.0 → pure Lasso - l1_ratio = 0.0 → pure Ridge - l1_ratio = 0.5 → equal blend

When to Use ElasticNet

ElasticNet is the right choice when:

  • Features are correlated AND you want sparsity. Lasso picks one arbitrarily from a group of correlated features. ElasticNet's Ridge component groups correlated features and selects or drops them together more consistently.
  • You have more features than observations. Lasso can select at most n_samples features. ElasticNet does not have this limitation.
  • You are unsure whether Lasso or Ridge is better and want cross-validation to figure it out
from sklearn.linear_model import ElasticNet, ElasticNetCV

# ElasticNetCV searches over both alpha and l1_ratio
elastic_cv_pipe = Pipeline([
    ("scaler", StandardScaler()),
    ("model", ElasticNetCV(
        l1_ratio=[0.1, 0.3, 0.5, 0.7, 0.9, 0.95, 1.0],
        cv=5,
        max_iter=10_000,
        random_state=42
    ))
])
elastic_cv_pipe.fit(X_train, y_train)
y_pred_elastic = elastic_cv_pipe.predict(X_test)

elastic_model = elastic_cv_pipe.named_steps["model"]
print(f"Best alpha:    {elastic_model.alpha_:.4f}")
print(f"Best l1_ratio: {elastic_model.l1_ratio_:.2f}")
print(f"MAE: {mean_absolute_error(y_test, y_pred_elastic):.3f}")
print(f"R²:  {r2_score(y_test, y_pred_elastic):.3f}")
# Output (approximate):
# Best alpha:    0.0018
# Best l1_ratio: 0.50
# MAE: 0.533
# R²:  0.576

Scaling is Not Optional for Regularised Models

Ridge, Lasso, and ElasticNet penalise coefficient magnitude. If income is measured in dollars (range: 20,000 – 150,000) and n_rooms is measured in integers (range: 1 – 15), an unpenalised income coefficient of 0.003 and a room coefficient of 5.0 might represent the same true effect — but the regulariser will penalise the room coefficient far more heavily because it is numerically larger. This is wrong. Scale first, always.

# Correct: scale before applying regularised model
correct_pipe = Pipeline([
    ("scaler", StandardScaler()),
    ("model", Ridge(alpha=1.0))
])

# Wrong: applying Ridge without scaling
# wrong_model = Ridge(alpha=1.0)
# wrong_model.fit(X_train, y_train)  # features on different scales → biased regularisation

Success

Decision rule in practice:

  • Features are many and likely most are relevant → Ridge
  • Features are many and you suspect most are noise → Lasso
  • Features are correlated AND you want some dropped → ElasticNet
  • You are unsure → ElasticNet with CV over l1_ratio (lets cross-validation pick the blend)

The computational cost of searching over l1_ratio is low. When in doubt, use ElasticNetCV and let the data decide.

Model Comparison on One Dataset

from sklearn.linear_model import LinearRegression, Ridge, Lasso, ElasticNet

models = {
    "LinearRegression": LinearRegression(),
    "Ridge (alpha=0.1)": Ridge(alpha=0.1),
    "Ridge (alpha=10)":  Ridge(alpha=10.0),
    "Lasso (alpha=0.01)": Lasso(alpha=0.01, max_iter=10_000),
    "Lasso (alpha=0.1)":  Lasso(alpha=0.1, max_iter=10_000),
    "ElasticNet":         ElasticNet(alpha=0.05, l1_ratio=0.5, max_iter=10_000),
}

results = []
for name, regressor in models.items():
    pipe = Pipeline([("scaler", StandardScaler()), ("model", regressor)])
    pipe.fit(X_train, y_train)
    y_pred_m = pipe.predict(X_test)
    results.append({
        "Model": name,
        "MAE": round(mean_absolute_error(y_test, y_pred_m), 3),
        "R²":  round(r2_score(y_test, y_pred_m), 3),
    })

comparison = pd.DataFrame(results).set_index("Model")
print(comparison)
# Output:
#                        MAE     R²
# LinearRegression     0.533  0.576
# Ridge (alpha=0.1)    0.533  0.576
# Ridge (alpha=10)     0.535  0.575
# Lasso (alpha=0.01)   0.534  0.575
# Lasso (alpha=0.1)    0.564  0.550
# ElasticNet           0.538  0.573

On the California Housing dataset (8 features, no extreme multicollinearity), regularisation provides minimal benefit — the dataset is well-conditioned. Regularisation earns its keep on high-dimensional or noisy data.

What's Next

You've covered Ridge L2 regularisation, Lasso L1 regularisation and its feature selection property, the coefficient path visualisation that reveals how each penalises differently, ElasticNet for correlated-feature situations, and why scaling is mandatory before any regularised model. Next up: 04-tree-based-regression — where you'll move beyond linear models to decision trees, Random Forests, and Gradient Boosting, learning how ensemble methods combine many weak trees to achieve the high R² scores that linear models cannot reach on nonlinear data.

Optional Deep Dive

Read the original Tibshirani (1996) Lasso paper "Regression Shrinkage and Selection via the Lasso" (available via JSTOR) — it is surprisingly readable and shows the constraint-form derivation that explains exactly why L1 produces sparse solutions while L2 does not. This is the intuition interviewers expect you to have.

02-linear-regression | 04-tree-based-regression