Model Selection and Tuning

Hyperparameter tuning is the step where practitioners most often fool themselves. It is easy to run a grid search, report the best validation score, and call it done — without noticing that the best score was cherry-picked from hundreds of trials on the same validation folds. Tuning is a search process, and every search has a risk of overfitting to the search space itself.

Learning Objectives

• Explain the bias-variance tradeoff and identify underfitting and overfitting from learning curves
• Compare models using cross-validated scores rather than single-split scores
• Use GridSearchCV and RandomizedSearchCV correctly
• Understand when GridSearch is overkill and when RandomizedSearch is the better tool
• Build tuning pipelines that prevent data leakage
• Read and use best_params_, best_score_, and best_estimator_

---

The Bias-Variance Tradeoff

Every model has two sources of error beyond irreducible noise:

Bias — systematic error from assumptions baked into the model. A linear model fit to a non-linear relationship will always underfit, regardless of how much data you give it.

Variance — sensitivity to the specific training data. A very deep decision tree memorises training data and produces wildly different predictions when trained on a slightly different sample.

Problem	Symptom	Cause	Fix
Underfitting (high bias)	Training score ≈ validation score, both poor	Model too simple	More complex model, add features
Overfitting (high variance)	Training score high, validation score much lower	Model too complex	Regularise, prune, get more data
Well-fitted	Training score ≈ validation score, both good	Model matched to problem	Deploy

Learning Curves

A learning curve plots training and validation performance as a function of training set size. It is the most informative diagnostic tool for understanding bias and variance.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import fetch_california_housing
from sklearn.ensemble import RandomForestRegressor
from sklearn.linear_model import Ridge
from sklearn.model_selection import learning_curve
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

housing = fetch_california_housing()
X_housing, y_housing = housing.data, housing.target

# Underfitting model: linear with default settings
underfitting_model = Pipeline([
    ("scaler", StandardScaler()),
    ("ridge", Ridge(alpha=1000))   # heavy regularisation forces underfitting
])

# Well-fitted model
good_model = Pipeline([
    ("scaler", StandardScaler()),
    ("rf", RandomForestRegressor(n_estimators=100, max_depth=10, random_state=42))
])

# Overfitting model
overfitting_model = Pipeline([
    ("scaler", StandardScaler()),
    ("rf", RandomForestRegressor(n_estimators=100, max_depth=None, random_state=42))
])

fig, axes = plt.subplots(1, 3, figsize=(16, 5))
models_to_plot = [
    (underfitting_model, "High Bias (Underfitting)", axes[0]),
    (good_model,         "Well-Fitted",             axes[1]),
    (overfitting_model,  "High Variance (Overfitting)", axes[2]),
]

train_sizes = np.linspace(0.1, 1.0, 10)

for model, title, ax in models_to_plot:
    sizes, train_scores, val_scores = learning_curve(
        model,
        X_housing[:2000],   # subset for speed
        y_housing[:2000],
        train_sizes=train_sizes,
        cv=5,
        scoring="neg_mean_absolute_error",
        n_jobs=-1
    )
    train_mean = -train_scores.mean(axis=1)
    val_mean   = -val_scores.mean(axis=1)

    ax.plot(sizes, train_mean, label="Training MAE",   color="steelblue")
    ax.plot(sizes, val_mean,   label="Validation MAE", color="darkorange")
    ax.fill_between(sizes,
                    train_mean - train_scores.std(axis=1),
                    train_mean + train_scores.std(axis=1),
                    alpha=0.1, color="steelblue")
    ax.set_title(title)
    ax.set_xlabel("Training set size")
    ax.set_ylabel("MAE")
    ax.legend()

plt.tight_layout()
plt.savefig("learning_curves.png", dpi=150)
plt.show()

Reading the curves:

• High bias: both curves converge to a high error. Adding more data barely helps — the model needs more capacity.
• High variance: training error is low, validation error is high, large gap between the two. More data helps close the gap; regularisation also helps.
• Good fit: both curves converge to a low error, small gap.

---

GridSearchCV — Exhaustive Search

GridSearchCV tries every combination of hyperparameters you specify. For a grid of 3 values × 3 values × 2 values = 18 combinations with cv=5, it trains and evaluates 18 × 5 = 90 models.

from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

cancer_data = load_breast_cancer()
X_cancer, y_cancer = cancer_data.data, cancer_data.target

X_train, X_test, y_train, y_test = train_test_split(
    X_cancer, y_cancer, test_size=0.2, random_state=42, stratify=y_cancer
)

# Build a pipeline — this prevents data leakage during CV
cancer_pipeline = Pipeline([
    ("scaler", StandardScaler()),
    ("rf", RandomForestClassifier(random_state=42))
])

# Note the double underscore: step_name__parameter_name
param_grid = {
    "rf__n_estimators": [50, 100, 200],
    "rf__max_depth":    [None, 5, 10],
    "rf__min_samples_split": [2, 5]
}

grid_search = GridSearchCV(
    cancer_pipeline,
    param_grid,
    cv=5,
    scoring="f1",
    refit=True,              # refit the best model on all training data
    n_jobs=-1,
    verbose=1
)

grid_search.fit(X_train, y_train)

print(f"Best parameters:      {grid_search.best_params_}")
print(f"Best CV F1:           {grid_search.best_score_:.3f}")
print(f"Test set F1:          {grid_search.score(X_test, y_test):.3f}")
# Output: Best parameters:      {'rf__max_depth': None, 'rf__min_samples_split': 2, 'rf__n_estimators': 200}
# Output: Best CV F1:           0.982
# Output: Test set F1:          0.986

After GridSearchCV completes:

# The best estimator is ready to use for predictions
best_model = grid_search.best_estimator_
y_pred = best_model.predict(X_test)

# Inspect the full results
import pandas as pd
cv_results = pd.DataFrame(grid_search.cv_results_)
print(cv_results[["params", "mean_test_score", "std_test_score", "rank_test_score"]]
      .sort_values("rank_test_score")
      .head(5)
      .to_string(index=False))

Warning

Always use a Pipeline inside GridSearchCV. If you scale your features outside the pipeline and then run CV, the scaler has seen the validation data during fit_transform. This is data leakage and will inflate your CV scores. The Pipeline ensures that preprocessing is refit on training folds only.

When GridSearch makes sense: - Small parameter spaces (fewer than ~50 combinations) - Cheap models (linear models, shallow trees) - You have a strong prior on which ranges matter and want to test all combinations

When GridSearch is overkill: - Large parameter spaces (deep learning, XGBoost with 10+ parameters) - Expensive models (anything that takes more than a few seconds per fit) - When you do not know the right ranges — a coarse random search is better exploration

---

RandomizedSearchCV — Sampled Search

RandomizedSearchCV samples n_iter random combinations from distributions you define. With 100 combinations and 5 folds, it trains 500 models — but it can cover a much larger search space than GridSearch does with the same budget.

from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import randint, uniform
from sklearn.ensemble import GradientBoostingClassifier

gb_pipeline = Pipeline([
    ("scaler", StandardScaler()),
    ("gb", GradientBoostingClassifier(random_state=42))
])

# Use distributions instead of fixed lists
param_distributions = {
    "gb__n_estimators":     randint(50, 500),        # sample integers from 50 to 499
    "gb__max_depth":        randint(2, 8),
    "gb__learning_rate":    uniform(0.01, 0.29),     # uniform from 0.01 to 0.30
    "gb__subsample":        uniform(0.6, 0.4),       # uniform from 0.6 to 1.0
    "gb__min_samples_leaf": randint(1, 20)
}

random_search = RandomizedSearchCV(
    gb_pipeline,
    param_distributions,
    n_iter=50,         # try 50 random combinations
    cv=5,
    scoring="f1",
    refit=True,
    n_jobs=-1,
    random_state=42,
    verbose=1
)

random_search.fit(X_train, y_train)

print(f"Best parameters: {random_search.best_params_}")
print(f"Best CV F1:      {random_search.best_score_:.3f}")
print(f"Test set F1:     {random_search.score(X_test, y_test):.3f}")
# Output: Best parameters: {'gb__learning_rate': 0.12, 'gb__max_depth': 3, ...}
# Output: Best CV F1:      0.979
# Output: Test set F1:     0.986

Tip

Research by Bergstra & Bengio (2012) showed that random search outperforms grid search for most hyperparameter problems, because most hyperparameters have low importance — you waste budget on them in a grid. Random search allocates budget more efficiently across the space. Start with random search unless you have a specific reason to use grid search.

---

Bayesian Optimisation — a Brief Note

Grid and random search are "memoryless" — each trial is independent of what came before. Bayesian optimisation builds a model of which hyperparameter regions are promising based on past results, and samples from high-probability regions.

The most practical Python library for this is Optuna.

# Conceptual example — requires: pip install optuna
import optuna
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

def objective(trial):
    n_estimators  = trial.suggest_int("n_estimators", 50, 500)
    max_depth     = trial.suggest_int("max_depth", 2, 15)
    min_samples_split = trial.suggest_int("min_samples_split", 2, 10)

    model = Pipeline([
        ("scaler", StandardScaler()),
        ("rf", RandomForestClassifier(
            n_estimators=n_estimators,
            max_depth=max_depth,
            min_samples_split=min_samples_split,
            random_state=42
        ))
    ])

    scores = cross_val_score(model, X_train, y_train, cv=5, scoring="f1")
    return scores.mean()

study = optuna.create_study(direction="maximize")
study.optimize(objective, n_trials=50, show_progress_bar=True)

print(f"Best F1:    {study.best_value:.3f}")
print(f"Best params: {study.best_params}")

Info

Bayesian optimisation becomes meaningfully better than random search when each trial is expensive (minutes to hours). For most scikit-learn models, random search with 50–100 iterations is sufficient. Use Bayesian optimisation for deep learning, XGBoost on large datasets, or when compute time is constrained.

---

Comparing Multiple Models

When you want to select between different model types, use cross-validated scores — not single-split scores.

from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

candidate_models = {
    "Logistic Regression": Pipeline([
        ("scaler", StandardScaler()),
        ("clf", LogisticRegression(max_iter=1000, random_state=42))
    ]),
    "Decision Tree": Pipeline([
        ("scaler", StandardScaler()),
        ("clf", DecisionTreeClassifier(random_state=42))
    ]),
    "Random Forest": Pipeline([
        ("scaler", StandardScaler()),
        ("clf", RandomForestClassifier(n_estimators=100, random_state=42))
    ]),
    "Gradient Boosting": Pipeline([
        ("scaler", StandardScaler()),
        ("clf", GradientBoostingClassifier(n_estimators=100, random_state=42))
    ]),
}

print(f"{'Model':<25} {'CV F1 Mean':>12} {'CV F1 Std':>12}")
print("-" * 52)

for model_name, pipeline in candidate_models.items():
    scores = cross_val_score(pipeline, X_train, y_train, cv=5, scoring="f1", n_jobs=-1)
    print(f"{model_name:<25} {scores.mean():>12.3f} {scores.std():>12.3f}")

# Output:
# Model                      CV F1 Mean    CV F1 Std
# ----------------------------------------------------
# Logistic Regression             0.961        0.016
# Decision Tree                   0.924        0.024
# Random Forest                   0.982        0.011
# Gradient Boosting               0.979        0.012

Success

When two models have overlapping mean ± std intervals, they are not meaningfully different. Choose the simpler one. A logistic regression at 0.961 ± 0.016 is not clearly worse than a gradient boosting model at 0.979 ± 0.012 — and the logistic regression is faster, more interpretable, and easier to debug.

---

The Golden Rules of Tuning

These rules exist to prevent subtle forms of overfitting that are easy to do accidentally:

1. The test set is touched exactly once. Every decision — which model, which hyperparameters, which threshold — is made on training data or cross-validation. The test set reports the final answer. Not an intermediate checkpoint.

2. Always use a Pipeline. Preprocessing steps (scaling, imputation, encoding) must be inside the pipeline that goes into GridSearchCV or cross_val_score. Otherwise, the preprocessing sees validation data during fit, which is leakage.

3. Tune on CV score, report on test score. The best CV score is slightly optimistic because you selected the model that happened to do best on those folds. The test score is your honest estimate.

4. Match the scoring parameter to the business metric. scoring="accuracy" is wrong for imbalanced problems. scoring="f1" tunes the model to balance precision and recall. scoring="roc_auc" tunes for discrimination. Pick the one that matches what the business actually cares about.

5. Don't chase tiny gains. A 0.002 improvement in CV F1 is almost certainly noise. If you tune long enough, you will find settings that overfit to your validation folds. Stop when improvements are within the standard deviation of your CV estimate.

Warning

Running GridSearchCV and then evaluating on the test set is only valid if you run it once. If you look at the test score, adjust your parameter grid, and run again, you have used the test set for tuning — even indirectly. This is one of the most common sources of optimistic test scores in practice.

---

The Scoring Parameter Reference

Common values for the scoring parameter in cross-validation and search:

Task	Metric	Scoring string
Classification (balanced)	Accuracy	"accuracy"
Classification (imbalanced, positive class)	F1	"f1"
Classification (multiclass)	F1 macro	"f1_macro"
Classification (ranking)	ROC-AUC	"roc_auc"
Regression	MAE	"neg_mean_absolute_error"
Regression	RMSE	"neg_root_mean_squared_error"
Regression	R²	"r2"

Info

Regression scoring strings return negative values (e.g., neg_mean_absolute_error) because scikit-learn's optimisation convention is "higher is better." Negate them when reporting: best_score = -search.best_score_.

---

Previous: Classification Evaluation | Next: Exercises