Classification — What It Is and How to Think About It

Every time a bank declines your card, a classifier ran in milliseconds and decided your transaction looked like fraud. Every spam filter, every medical diagnosis assist tool, every content moderation system — these are all classifiers at their core. Understanding what a classifier actually does, and where it can silently fail, is the foundation for everything in this session.

Learning Objectives

• Distinguish binary classification, multiclass, and multilabel problems
• Explain what a model output means: class label vs probability
• Describe what a decision boundary is and why it matters
• Use a dummy baseline classifier to set a minimum performance floor
• Match a classification algorithm to a problem type

---

What a Classifier Actually Outputs

A classifier learns a function that maps input features to output categories. But there are two distinct things it can give you:

A class label — the predicted category directly. "This email is spam." "This transaction is fraud."

A probability — a score between 0 and 1 representing confidence. "There is a 0.87 probability this email is spam."

The probability is almost always more useful. It lets you choose where to draw the line.

Info

The line between "positive" and "negative" is called the decision threshold. The default in most sklearn classifiers is 0.5, but that is rarely the right choice for real problems. You will learn how to tune it in 05-classification-metrics.md.

from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

cancer = load_breast_cancer(as_frame=True)
X_train, X_test, y_train, y_test = train_test_split(
    cancer.data, cancer.target, test_size=0.2, random_state=42, stratify=cancer.target
)

model = LogisticRegression(max_iter=1000)
model.fit(X_train, y_train)

# Class label prediction (applies 0.5 threshold internally)
labels = model.predict(X_test)
print(labels[:10])
# Output: [1 0 0 1 1 0 0 0 1 1]

# Probability output — far more informative
probabilities = model.predict_proba(X_test)
print(probabilities[:5])
# Output: [[0.02 0.98]
#           [0.94 0.06]
#           [0.91 0.09]
#           [0.08 0.92]
#           [0.03 0.97]]
# Column 0 = probability of class 0 (malignant), Column 1 = probability of class 1 (benign)

---

Binary vs Multiclass vs Multilabel

Type	Classes	Example	Typical Output
Binary	2	Churn yes/no	Single probability for positive class
Multiclass	3+	Low/medium/high risk	Probability per class, must sum to 1
Multilabel	Multiple at once	Article tagged "python" + "ML"	Independent probability per label

Most algorithms handle binary natively. For multiclass, sklearn uses One-vs-Rest (OvR) by default for many algorithms — it trains one binary classifier per class and picks the most confident one. Some algorithms (like softmax Logistic Regression) handle multiclass directly.

from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression

iris = load_iris(as_frame=True)
X_train_iris, X_test_iris, y_train_iris, y_test_iris = train_test_split(
    iris.data, iris.target, test_size=0.2, random_state=42, stratify=iris.target
)

# multi_class='multinomial' uses softmax directly instead of OvR
multiclass_model = LogisticRegression(multi_class='multinomial', max_iter=1000)
multiclass_model.fit(X_train_iris, y_train_iris)

proba_per_class = multiclass_model.predict_proba(X_test_iris)
print(proba_per_class[:3])
# Output: [[0.002 0.341 0.657]
#           [0.941 0.058 0.001]
#           [0.003 0.067 0.930]]
# Each row sums to 1.0

---

The Decision Boundary

The decision boundary is the line (or surface in higher dimensions) that separates the classes in feature space. Different algorithms learn very different shaped boundaries:

Algorithm	Boundary Shape
Logistic Regression	Linear (straight line or hyperplane)
KNN	Irregular, follows local data density
Decision Tree	Axis-aligned rectangular regions
Random Forest / Boosting	Complex, non-linear, high fidelity
SVM (RBF kernel)	Curved, margin-maximising

Tip

When your classes are linearly separable — meaning a straight line could reasonably divide them — start with Logistic Regression. It is fast, interpretable, and trains in seconds. When classes are interleaved or have complex structure, move to trees.

---

The Baseline Classifier — Your Minimum Bar

Before training any real model, calculate what you get for free. A dummy classifier that always predicts the majority class is your baseline. Any model that cannot beat it is useless.

from sklearn.dummy import DummyClassifier
from sklearn.metrics import accuracy_score

# Simulated churn dataset: 90% no-churn, 10% churn
import numpy as np
np.random.seed(42)
y_imbalanced = np.random.choice([0, 1], size=1000, p=[0.90, 0.10])
X_dummy = np.random.randn(1000, 5)

X_tr, X_te, y_tr, y_te = train_test_split(
    X_dummy, y_imbalanced, test_size=0.2, random_state=42
)

baseline = DummyClassifier(strategy="most_frequent")
baseline.fit(X_tr, y_tr)
baseline_acc = accuracy_score(y_te, baseline.predict(X_te))
print(f"Baseline accuracy: {baseline_acc:.2%}")
# Output: Baseline accuracy: 91.00%

Warning

A model with 91% accuracy on a 90/10 imbalanced dataset is doing nothing useful — it is predicting "no churn" for every single customer. This is why accuracy alone is a misleading metric. You will cover better metrics in 05-classification-metrics.md.

---

The Classification Workflow

Every classification project follows this structure. The steps are always the same; the choices within each step are what require judgement.

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report

# Step 1: Define features and target
df = pd.read_csv("churn.csv")
feature_cols = [c for c in df.columns if c != "churn"]
X = df[feature_cols]
y = df["churn"]

# Step 2: Split with stratification
# stratify=y ensures both train and test have the same class ratio
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# Step 3: Build a pipeline (preprocessing + model)
clf_pipeline = Pipeline([
    ("scaler", StandardScaler()),
    ("classifier", RandomForestClassifier(n_estimators=200, random_state=42))
])

# Step 4: Train
clf_pipeline.fit(X_train, y_train)

# Step 5: Evaluate
y_pred = clf_pipeline.predict(X_test)
print(classification_report(y_test, y_pred))

# Step 6: Get probabilities for threshold tuning
y_proba = clf_pipeline.predict_proba(X_test)[:, 1]

Success

Always use stratify=y when splitting classification datasets. Without it, a small class might end up severely underrepresented in the test set, making your evaluation noisy.

---

Choosing the Right Algorithm

There is no universally best classifier. Here is the decision logic practitioners actually use:

Start here:

1. Train a DummyClassifier → baseline
2. Train Logistic Regression → fast interpretable baseline
3. Train Random Forest → usually beats LogReg, still interpretable via feature importance
4. If you need top performance: try XGBoost or LightGBM

Then ask these questions:

Question	If Yes
Do you need to explain every prediction?	Logistic Regression or shallow Decision Tree
Is the dataset small (<5k rows)?	KNN or Naive Bayes can work well
Are your features text?	Multinomial Naive Bayes or TF-IDF + LogReg
Is the relationship clearly non-linear?	Random Forest or Gradient Boosting
Do you need the best possible AUC?	XGBoost / LightGBM with tuned hyperparameters

Info

In industry, Random Forest is almost always the first serious model you train. It handles mixed feature types, is robust to outliers, gives you feature importance for free, and almost always outperforms Logistic Regression on a first pass without any tuning.

---

---

What's Next

You've covered the classification task definition, the class imbalance problem and why accuracy misleads, the stratified train/test split pattern, a working pipeline template for any classification project, and the algorithm selection decision table. Next up: 02-logistic-regression — where you'll go deep on how logistic regression maps a linear combination of features to a probability, why the sigmoid function matters, coefficient interpretation, threshold tuning, and L1/L2 regularisation via the C parameter.

Optional Deep Dive

Read "An Introduction to Statistical Learning" (ISLR) Chapter 4 (Classification) — it covers logistic regression, linear discriminant analysis, and K-nearest neighbours from first principles, building the statistical foundations that underpin all the classification algorithms covered in this session.

Previous: Agenda | Next: Logistic Regression