Logistic Regression — Probabilities, Thresholds, and Interpretation¶
Logistic Regression is not a regression algorithm. It is a classifier that outputs a probability via the sigmoid function. The name is historical — it comes from the logistic function at its core. Do not let the name mislead you. On structured tabular data, it is often the first model a practitioner trains: fast, interpretable, and a reliable indicator of whether your features actually carry signal.
Learning Objectives¶
- Explain how the sigmoid function maps any number to a probability between 0 and 1
- Interpret log-odds and what the model coefficients mean
- Tune the decision threshold beyond the default 0.5
- Use regularisation (the
Cparameter) to control overfitting - Handle multiclass problems with OvR and softmax strategies
- Know when Logistic Regression is the right tool and when to reach for something else
The Sigmoid Function — Why It Works¶
A linear model produces outputs that range from negative infinity to positive infinity. That is fine for regression, but useless for probability. Probabilities must live between 0 and 1.
The sigmoid function solves this:
$$\sigma(z) = \frac{1}{1 + e^{-z}}$$
Where z is the linear combination of your features: z = w₀ + w₁x₁ + w₂x₂ + ...
Feed any number into sigmoid and you get a value between 0 and 1. Feed it a very large positive number and you get close to 1. Feed it a very large negative number and you get close to 0.
import numpy as np
import matplotlib.pyplot as plt
def sigmoid(z):
return 1 / (1 + np.exp(-z))
z_values = np.linspace(-10, 10, 300)
probabilities = sigmoid(z_values)
plt.figure(figsize=(8, 4))
plt.plot(z_values, probabilities, color='teal', linewidth=2)
plt.axhline(0.5, color='gray', linestyle='--', alpha=0.7, label='threshold = 0.5')
plt.axvline(0, color='gray', linestyle='--', alpha=0.7)
plt.xlabel("z (linear combination of features)")
plt.ylabel("Predicted probability")
plt.title("Sigmoid Function")
plt.legend()
plt.tight_layout()
plt.savefig("sigmoid.png", dpi=150)
# A few reference points:
print(sigmoid(0)) # Output: 0.5
print(sigmoid(2)) # Output: 0.880
print(sigmoid(-2)) # Output: 0.119
print(sigmoid(10)) # Output: 0.99995
Info
The decision boundary of Logistic Regression is where z = 0, i.e., where the predicted probability equals 0.5. In 2D feature space this is a straight line. In higher dimensions it is a hyperplane. This is why Logistic Regression is called a linear classifier.
Log-Odds — What the Coefficients Actually Mean¶
The model does not learn probabilities directly. It learns log-odds (the logit):
$$\log\left(\frac{p}{1-p}\right) = w_0 + w_1 x_1 + w_2 x_2 + \ldots$$
A coefficient w_i means: for a one-unit increase in feature x_i, the log-odds of the positive class increase by w_i, holding everything else constant.
Exponentiating the coefficient gives you an odds ratio, which is more interpretable:
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
import pandas as pd
import numpy as np
cancer = load_breast_cancer(as_frame=True)
X_train, X_test, y_train, y_test = train_test_split(
cancer.data, cancer.target,
test_size=0.2, random_state=42, stratify=cancer.target
)
log_reg_pipeline = Pipeline([
("scaler", StandardScaler()),
("model", LogisticRegression(max_iter=1000, C=1.0))
])
log_reg_pipeline.fit(X_train, y_train)
# Extract coefficients from the trained model
coefficients = log_reg_pipeline.named_steps["model"].coef_[0]
feature_names = cancer.feature_names
coef_df = pd.DataFrame({
"feature": feature_names,
"coefficient": coefficients,
"odds_ratio": np.exp(coefficients)
}).sort_values("coefficient", ascending=False)
print(coef_df.head(5).to_string(index=False))
# Output (approximate):
# feature coefficient odds_ratio
# mean concavity 1.823 6.188
# worst perimeter 1.561 4.764
# worst concave points 1.398 4.047
# mean texture -1.271 0.280
# ...
A positive coefficient means that feature pushes the prediction toward the positive class (benign in this dataset). A negative coefficient pushes toward the negative class. After scaling, the magnitude tells you relative importance.
Tip
After scaling features, the coefficients are directly comparable in magnitude. A coefficient of 1.8 has roughly double the influence of one of 0.9. Without scaling, comparing coefficients is meaningless because features live on different scales.
Training and Evaluating a Full Pipeline¶
from sklearn.metrics import accuracy_score, classification_report
# Already trained above as log_reg_pipeline
y_pred = log_reg_pipeline.predict(X_test)
y_proba = log_reg_pipeline.predict_proba(X_test)
print(f"Accuracy: {accuracy_score(y_test, y_pred):.4f}")
# Output: Accuracy: 0.9737
print(classification_report(y_test, y_pred, target_names=["malignant", "benign"]))
# Output:
# precision recall f1-score support
# malignant 0.98 0.95 0.96 43
# benign 0.97 0.99 0.98 71
# accuracy 0.97 114
# macro avg 0.97 0.97 0.97 114
# weighted avg 0.97 0.97 0.97 114
The Decision Threshold — 0.5 Is Rarely Right¶
The default threshold of 0.5 means: if the model thinks there is more than a 50% chance this is positive, predict positive. That split is arbitrary. In real problems, the cost of a false negative and a false positive are almost never equal.
In medical diagnosis, missing a cancer (false negative) is far more costly than an unnecessary follow-up test (false positive). You want to catch every positive you can. Lower the threshold.
In spam detection, a legitimate email landing in spam (false positive) is disruptive. You want high precision. Raise the threshold.
positive_proba = log_reg_pipeline.predict_proba(X_test)[:, 1]
thresholds = [0.3, 0.4, 0.5, 0.6, 0.7]
from sklearn.metrics import precision_score, recall_score, f1_score
print(f"{'Threshold':>12} {'Precision':>12} {'Recall':>10} {'F1':>8}")
print("-" * 46)
for thresh in thresholds:
y_pred_thresh = (positive_proba >= thresh).astype(int)
prec = precision_score(y_test, y_pred_thresh)
rec = recall_score(y_test, y_pred_thresh)
f1 = f1_score(y_test, y_pred_thresh)
print(f"{thresh:>12.1f} {prec:>12.3f} {rec:>10.3f} {f1:>8.3f}")
# Output:
# Threshold Precision Recall F1
# ----------------------------------------------
# 0.3 0.944 1.000 0.971
# 0.4 0.958 1.000 0.979
# 0.5 0.972 0.986 0.979
# 0.6 0.986 0.972 0.979
# 0.7 1.000 0.944 0.971
Warning
Never tune the threshold using your test set. You will overfit to it. Use a validation set or cross-validation to find the threshold, then apply it once to the held-out test set.
Regularisation — The C Parameter¶
Logistic Regression in sklearn includes L2 regularisation by default. Regularisation penalises large coefficients, which helps prevent overfitting on training data.
The regularisation strength is controlled by C, which is the inverse of the regularisation penalty (i.e., C = 1/λ):
- Small C (e.g. 0.01): strong regularisation, coefficients are shrunk aggressively, simpler model
- Large C (e.g. 100): weak regularisation, model can fit the training data more closely, risk of overfitting
from sklearn.model_selection import cross_val_score
C_values = [0.001, 0.01, 0.1, 1.0, 10.0, 100.0]
print(f"{'C':>10} {'Mean CV Accuracy':>20} {'Std':>10}")
print("-" * 44)
for c in C_values:
pipeline_c = Pipeline([
("scaler", StandardScaler()),
("model", LogisticRegression(max_iter=2000, C=c))
])
scores = cross_val_score(pipeline_c, cancer.data, cancer.target, cv=5, scoring="accuracy")
print(f"{c:>10.3f} {scores.mean():>20.4f} {scores.std():>10.4f}")
# Output (approximate):
# C Mean CV Accuracy Std
# --------------------------------------------
# 0.001 0.9419 0.0178
# 0.010 0.9596 0.0132
# 0.100 0.9701 0.0168
# 1.000 0.9736 0.0157
# 10.000 0.9719 0.0168
# 100.000 0.9719 0.0120
Info
You can also use penalty='l1' (Lasso) for sparse models — L1 drives some coefficients to exactly zero, effectively doing feature selection. Use solver='liblinear' or solver='saga' for L1 regularisation.
Multiclass Logistic Regression¶
from sklearn.datasets import load_iris
iris = load_iris(as_frame=True)
X_train_iris, X_test_iris, y_train_iris, y_test_iris = train_test_split(
iris.data, iris.target, test_size=0.2, random_state=42, stratify=iris.target
)
# One-vs-Rest (OvR): trains 3 binary classifiers, picks the most confident
ovr_model = Pipeline([
("scaler", StandardScaler()),
("model", LogisticRegression(multi_class="ovr", max_iter=1000))
])
ovr_model.fit(X_train_iris, y_train_iris)
# Softmax (multinomial): trains one model over all classes jointly
softmax_model = Pipeline([
("scaler", StandardScaler()),
("model", LogisticRegression(multi_class="multinomial", solver="lbfgs", max_iter=1000))
])
softmax_model.fit(X_train_iris, y_train_iris)
print(f"OvR accuracy: {ovr_model.score(X_test_iris, y_test_iris):.4f}")
print(f"Softmax accuracy: {softmax_model.score(X_test_iris, y_test_iris):.4f}")
# Output:
# OvR accuracy: 0.9667
# Softmax accuracy: 0.9667
For most multiclass problems the difference between OvR and softmax is small. Softmax is preferred when classes are mutually exclusive and you want probabilities that sum to exactly 1.
When to Use Logistic Regression¶
Use it when:
- You need an interpretable model with explainable coefficients
- You need a fast, reliable baseline before trying more complex models
- Features are roughly linearly related to the log-odds of the outcome
- You are building a binary or multiclass classifier on structured tabular data
- You need calibrated probabilities (LogReg probabilities are well-calibrated out of the box)
Look elsewhere when:
- Relationships between features and target are highly non-linear
- Features have complex interactions that are not captured by linear combinations
- You have very high-dimensional sparse data without meaningful feature engineering
- You need the best possible predictive performance on a leaderboard
Success
Logistic Regression is not glamorous, but it is the model that gets deployed in many high-stakes financial and clinical systems because practitioners can explain every coefficient to a regulator or a clinician. Interpretability has real business value.
Interview Questions¶
Q: Why is Logistic Regression a classifier and not a regressor?
Show answer
Logistic Regression predicts a probability (a number between 0 and 1) and then applies a threshold to output a class label. The output is categorical, making it a classifier. The word "regression" refers to the internal linear function it fits — specifically, it models the log-odds of the outcome as a linear function of the features.
Q: What does the C parameter control?
Show answer
C is the inverse of the regularisation strength (C = 1/λ). A smaller C applies stronger regularisation, shrinking coefficients toward zero and producing a simpler model. A larger C relaxes regularisation and allows the model to fit the training data more closely. The default is C=1.0.
Q: When would you lower the decision threshold below 0.5?
Show answer
When false negatives are more costly than false positives. In medical diagnosis (e.g., cancer screening), missing a positive case is more dangerous than an unnecessary follow-up test. Lowering the threshold means you flag more cases as positive, increasing recall at the expense of precision.
Q: Why must you scale features before training Logistic Regression?
Show answer
Two reasons: (1) The gradient descent optimisation converges faster when features are on a similar scale. (2) Regularisation penalises the magnitude of coefficients — without scaling, features on larger scales will have smaller coefficients just because of their units, not because they are less important. Scaling ensures regularisation is applied fairly.
What's Next¶
You've covered logistic regression's sigmoid function and log-odds foundation, coefficient interpretation, threshold tuning for precision/recall tradeoffs, the C regularisation parameter with cross-validation, and one-vs-rest vs softmax for multiclass problems. Next up: 03-knn-and-naive-bayes — where you'll learn two fundamentally different classifiers: KNN, which makes predictions purely from proximity in feature space, and Naive Bayes, which applies Bayes' theorem with a conditional independence assumption to produce a fast generative classifier.
Optional Deep Dive
Read "Logistic Regression: A Self-Learning Text" by Kleinbaum and Klein, Chapter 1–2 — it provides the epidemiological and clinical interpretation of logistic regression coefficients (odds ratios, confidence intervals) that are essential when working in healthcare, clinical trials, or any domain where the model outputs need to be communicated to non-technical stakeholders.
Previous: Classification Overview | Next: KNN and Naive Bayes