Regression Overview¶
Every time a business wants to know how much — how much revenue next quarter, how many units will sell, what price a house will fetch — the answer is a regression problem. Regression is the branch of supervised learning concerned with predicting a continuous numeric value, and it is often the first tool a data scientist reaches for precisely because its predictions are interpretable and its failure modes are diagnosable.
Learning Objectives¶
- Define regression and distinguish it from classification
- Identify real-world problems that are regression problems
- Understand the concept of a loss function and why minimising it is the central goal
- Know the landscape of regression models and when each makes sense
- Describe the standard regression workflow from raw data to evaluated model
What Regression Actually Does¶
A regression model learns a function f such that:
Where X is a matrix of input features and y_hat is the predicted numeric output. The goal is for y_hat to be as close as possible to the true value y across all training examples — and, critically, to remain close on data the model has never seen.
The word "regression" was coined by Francis Galton in 1886 when he noticed that children of very tall parents tended to be tall, but not as tall as their parents — they "regressed toward the mean." The statistical technique that explained this phenomenon kept the name.
Info
Regression predicts continuous values (house price, temperature, revenue). Classification predicts discrete categories (spam/not spam, churn/no churn). The line between them can blur — predicting probability of churn is technically regression — but the distinction matters for choosing loss functions and evaluation metrics.
Real-World Regression Problems¶
Regression shows up constantly across industries:
| Domain | Target Variable | Features |
|---|---|---|
| Real estate | Sale price ($) | Square footage, location, age, bedrooms |
| E-commerce | Order value ($) | User history, session behaviour, device |
| Energy | Power consumption (kWh) | Temperature, time of day, building size |
| Finance | Stock return (%) | Macro indicators, earnings, sentiment |
| Healthcare | Hospital stay (days) | Age, diagnosis, comorbidities |
| Logistics | Delivery time (hours) | Distance, traffic, weather, carrier |
The unifying thread: the output is a number on a continuous scale, and small differences in prediction accuracy translate directly into business impact.
The Loss Function: The Heart of Every Regression Model¶
A regression model does not guess randomly — it optimises. Before training begins, you choose a loss function: a mathematical measure of how wrong the model's predictions are. Training is the process of adjusting the model's parameters to make the loss as small as possible.
The most common regression loss is Mean Squared Error (MSE):
Squaring the errors does two things: it makes all errors positive (so positive and negative errors do not cancel), and it penalises large errors disproportionately more than small ones.
import numpy as np
y_true = np.array([300_000, 450_000, 200_000, 550_000])
y_pred = np.array([310_000, 440_000, 230_000, 500_000])
mse = np.mean((y_true - y_pred) ** 2)
rmse = np.sqrt(mse)
print(f"MSE: {mse:,.0f}")
print(f"RMSE: {rmse:,.0f}")
# Output:
# MSE: 787,500,000
# RMSE: 28,061
The RMSE of 28,061 means the model is off by roughly $28,000 on average — a number that lives in the same units as the target and is immediately interpretable.
Tip
Think of the loss function as the model's scorecard during training. The algorithm's only job is to find the parameter settings that produce the lowest possible score on that card. Changing the loss function changes what the model considers "a good prediction" — which changes the model's behaviour fundamentally.
Regression as Optimisation¶
Training a regression model means solving an optimisation problem: find the parameters that minimise the loss function over the training data. Depending on the model:
- Linear regression has a closed-form solution (the Normal Equations) — you can find the exact minimum analytically
- Regularised linear models (Ridge, Lasso) are still convex — gradient descent finds the global minimum reliably
- Tree-based models use greedy search — they find good splits locally but cannot guarantee a globally optimal tree
Understanding that training is optimisation explains why:
- Scaling features matters for gradient-based methods (unscaled features create uneven loss surfaces)
- Hyperparameters like alpha in Ridge control the shape of the loss surface
- Overfitting happens when the model optimises the training loss so well that it no longer generalises
The Standard Regression Workflow¶
Every regression project follows roughly the same arc:
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, r2_score
# 1. Load data
housing = pd.read_csv("housing.csv")
# 2. Separate features and target
feature_cols = [c for c in housing.columns if c != "sale_price"]
X = housing[feature_cols]
y = housing["sale_price"]
# 3. Split — always do this before any preprocessing
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42
)
# 4. Preprocess (fit only on train, transform both)
# ... scaling, encoding, imputation
# 5. Train
model = LinearRegression()
model.fit(X_train, y_train)
# 6. Predict
y_pred = model.predict(X_test)
# 7. Evaluate
print(f"MAE: {mean_absolute_error(y_test, y_pred):,.0f}")
print(f"R²: {r2_score(y_test, y_pred):.3f}")
# 8. Inspect residuals
residuals = y_test - y_pred
# plot residuals vs predicted values — next file covers this in depth
Warning
The single most common mistake in regression projects is fitting the preprocessor on the full dataset before splitting. If your StandardScaler has seen the test set's values, your evaluation is optimistic. Always split first, then fit preprocessing on training data only. Use sklearn Pipelines to enforce this mechanically.
Regression Model Landscape¶
Before diving into each model, here is the map:
| Model | When to Reach For It |
|---|---|
| Linear Regression | Baseline; when interpretability is paramount; when relationships are roughly linear |
| Ridge | Linear baseline with multicollinearity or many correlated features |
| Lasso | You want automatic feature selection; many features but few are truly relevant |
| ElasticNet | Features are correlated AND sparse — Lasso alone is unstable |
| Decision Tree | Quick nonlinear model; useful for understanding splits; standalone use is rare |
| Random Forest | Strong general-purpose model; robust, needs minimal tuning |
| Gradient Boosting | When you need the highest accuracy and have time to tune |
Success
The model hierarchy in practice: start with linear regression as a sanity check. If it performs well, interpret it. If it falls short, move to regularised variants. If the data has clear nonlinear structure, move to tree-based ensembles. The linear model's performance gives you a floor to beat.
What's Next¶
You've covered the regression task definition, the end-to-end sklearn regression workflow (load, split, preprocess, train, evaluate, diagnose), and the model landscape from linear regression through gradient boosting — including when to use each. Next up: 02-linear-regression — where you'll go deep on ordinary least squares, coefficient interpretation, assumptions and diagnostics, multicollinearity detection with VIF, and the residuals-vs-fitted plot that is the single most important regression diagnostic.
Optional Deep Dive
Read "An Introduction to Statistical Learning" (ISLR) by James, Witten, Hastie, and Tibshirani, Chapter 3 (free PDF at statlearning.com) — it provides the mathematical derivation of OLS, the assumptions underlying linear regression, and the diagnostic tests that distinguish whether a linear model is appropriate for your data.