Skip to content

Model Building

Picking your first model before you understand the problem is a common mistake. The Titanic dataset is small enough that four models train in seconds — so there is no reason not to compare them. This section starts with a deliberate baseline, builds up through interpretable models, then adds complexity only when it earns its keep. The final model reaches roughly 83% accuracy on the validation set, which puts it in the top third of Kaggle public leaderboard submissions.

Learning Objectives

  • Build a baseline that quantifies what a trivially simple model achieves
  • Train Logistic Regression, Random Forest, and Gradient Boosting inside full sklearn Pipelines
  • Compare models on both train and validation accuracy to diagnose overfitting
  • Tune hyperparameters efficiently with RandomizedSearchCV
  • Evaluate final model performance with cross-validation and a classification report

Setup

Run all feature engineering from the previous file first, then continue here with X_train, X_test, y_train, y_test, and preprocessor already defined.

import seaborn as sns
import pandas as pd
import numpy as np
import time
from sklearn.model_selection import train_test_split, cross_val_score, RandomizedSearchCV
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.metrics import classification_report, accuracy_score

# --- Rebuild feature matrix from scratch ---
df_raw = sns.load_dataset('titanic')

df_raw['title'] = df_raw['name'].str.extract(r',\s([A-Za-z]+)\.')
rare_titles = ['Dr', 'Rev', 'Col', 'Major', 'Mlle', 'Capt', 'Lady',
               'Sir', 'Ms', 'Don', 'Jonkheer', 'Countess', 'Mme']
df_raw['title'] = df_raw['title'].replace(rare_titles, 'Rare')

df_raw['family_size'] = df_raw['sibsp'] + df_raw['parch'] + 1
df_raw['is_alone'] = (df_raw['family_size'] == 1).astype(int)
df_raw['fare_per_person'] = df_raw['fare'] / df_raw['family_size']
df_raw['age_bucket'] = pd.cut(
    df_raw['age'],
    bins=[0, 12, 18, 35, 60, 120],
    labels=['child', 'teen', 'young_adult', 'adult', 'senior']
)

feature_cols = ['survived', 'pclass', 'sex', 'age', 'fare', 'embarked',
                'family_size', 'is_alone', 'title', 'age_bucket', 'fare_per_person']
df_eng = df_raw[feature_cols].copy()

X = df_eng.drop(columns=['survived'])
y = df_eng['survived']

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# --- Preprocessor ---
numeric_features = ['age', 'fare', 'family_size', 'fare_per_person']
binary_features = ['pclass', 'is_alone']
categorical_features = ['sex', 'embarked', 'title', 'age_bucket']

numeric_pipeline = Pipeline([
    ('imputer', SimpleImputer(strategy='median')),
    ('scaler', StandardScaler())
])
binary_pipeline = Pipeline([
    ('imputer', SimpleImputer(strategy='median')),
    ('scaler', StandardScaler())
])
categorical_pipeline = Pipeline([
    ('imputer', SimpleImputer(strategy='most_frequent')),
    ('encoder', OneHotEncoder(handle_unknown='ignore', sparse_output=False))
])

preprocessor = ColumnTransformer([
    ('num', numeric_pipeline, numeric_features),
    ('bin', binary_pipeline, binary_features),
    ('cat', categorical_pipeline, categorical_features),
])

Baseline — DummyClassifier

Before fitting any real model, establish the floor. The baseline tells you what you get for free — it is what you beat to prove your model is learning something.

dummy = DummyClassifier(strategy='most_frequent', random_state=42)
dummy.fit(X_train, y_train)

dummy_train_acc = dummy.score(X_train, y_train)
dummy_val_acc   = dummy.score(X_test, y_test)

print(f"Dummy train accuracy: {dummy_train_acc:.3f}")
print(f"Dummy val accuracy:   {dummy_val_acc:.3f}")
# Output:
# Dummy train accuracy: 0.617
# Dummy val accuracy:   0.615

print("\nDummy predictions (first 10):", dummy.predict(X_test[:10]))
# Output: Dummy predictions (first 10): [0 0 0 0 0 0 0 0 0 0]
# — predicts "died" for every single passenger

Info

The most frequent class in the training set is 0 (died), which appears 61.7% of the time. The DummyClassifier predicts 0 for every passenger, achieving 61.5% accuracy without ever looking at any features. Any model that cannot beat 61.5% is worse than predicting "everyone died." This is your floor.

Logistic Regression

Logistic Regression is the right first real model for binary classification. It is interpretable, fast, outputs calibrated probabilities, and fails loudly — when it performs poorly, the features or the problem structure are at fault, not the model.

start = time.time()

lr_pipeline = Pipeline([
    ('preprocessor', preprocessor),
    ('model', LogisticRegression(max_iter=1000, random_state=42))
])

lr_pipeline.fit(X_train, y_train)
lr_fit_time = time.time() - start

lr_train_acc = lr_pipeline.score(X_train, y_train)
lr_val_acc   = lr_pipeline.score(X_test, y_test)

print(f"Logistic Regression — train: {lr_train_acc:.3f}, val: {lr_val_acc:.3f}, fit: {lr_fit_time:.2f}s")
# Output (approximate):
# Logistic Regression — train: 0.818, val: 0.810, fit: 0.12s

Tip

A train accuracy of 81.8% and val accuracy of 81.0% with almost no gap is a healthy result. The model generalises well — it is not overfitting. This is typical for Logistic Regression: it is relatively low-variance, which means the train and val scores stay close.

Random Forest

Random Forest builds an ensemble of decision trees on random subsets of the data and features. It handles non-linear interactions (like sex × class) without any manual engineering. Expect higher accuracy — and higher variance.

start = time.time()

rf_pipeline = Pipeline([
    ('preprocessor', preprocessor),
    ('model', RandomForestClassifier(n_estimators=100, random_state=42))
])

rf_pipeline.fit(X_train, y_train)
rf_fit_time = time.time() - start

rf_train_acc = rf_pipeline.score(X_train, y_train)
rf_val_acc   = rf_pipeline.score(X_test, y_test)

print(f"Random Forest — train: {rf_train_acc:.3f}, val: {rf_val_acc:.3f}, fit: {rf_fit_time:.2f}s")
# Output (approximate):
# Random Forest — train: 0.981, val: 0.821, fit: 0.31s

Warning

The gap between train accuracy (98.1%) and val accuracy (82.1%) is large — this is overfitting. The model has memorised the training set. Default Random Forest grows trees until all leaves are pure, which produces very deep, data-memorising trees. Hyperparameter tuning (max_depth, min_samples_leaf) will close this gap.

Gradient Boosting

Gradient Boosting builds trees sequentially, each one correcting the residual errors of the previous. It is often the best-performing model on tabular data, but it is slower to train and more sensitive to hyperparameters.

start = time.time()

gb_pipeline = Pipeline([
    ('preprocessor', preprocessor),
    ('model', GradientBoostingClassifier(n_estimators=100, random_state=42))
])

gb_pipeline.fit(X_train, y_train)
gb_fit_time = time.time() - start

gb_train_acc = gb_pipeline.score(X_train, y_train)
gb_val_acc   = gb_pipeline.score(X_test, y_test)

print(f"Gradient Boosting — train: {gb_train_acc:.3f}, val: {gb_val_acc:.3f}, fit: {gb_fit_time:.2f}s")
# Output (approximate):
# Gradient Boosting — train: 0.908, val: 0.827, fit: 0.58s

Model Comparison

results = {
    'DummyClassifier':       {'train_acc': dummy_train_acc, 'val_acc': dummy_val_acc,   'fit_time': 0.00},
    'LogisticRegression':    {'train_acc': lr_train_acc,    'val_acc': lr_val_acc,       'fit_time': lr_fit_time},
    'RandomForest':          {'train_acc': rf_train_acc,    'val_acc': rf_val_acc,       'fit_time': rf_fit_time},
    'GradientBoosting':      {'train_acc': gb_train_acc,    'val_acc': gb_val_acc,       'fit_time': gb_fit_time},
}

comparison_df = pd.DataFrame(results).T
comparison_df['overfit_gap'] = comparison_df['train_acc'] - comparison_df['val_acc']

print(comparison_df.round(3).to_string())
# Output (approximate):
#                   train_acc  val_acc  fit_time  overfit_gap
# DummyClassifier       0.617    0.615      0.00        0.002
# LogisticRegression    0.818    0.810      0.12        0.008
# RandomForest          0.981    0.821      0.31        0.160
# GradientBoosting      0.908    0.827      0.58        0.081

Success

Gradient Boosting leads on validation accuracy but is slower and has a larger overfit gap than Logistic Regression. Random Forest is competitive but overfits more aggressively in its default configuration. Tuning the Random Forest's depth controls is the next step — after tuning it often closes within 1-2% of Gradient Boosting.

Hyperparameter Tuning — RandomizedSearchCV

RandomizedSearchCV samples a fixed number of parameter combinations at random. It is faster than GridSearchCV when the parameter space is large, because you do not evaluate every combination.

from sklearn.model_selection import RandomizedSearchCV

param_distributions = {
    'model__n_estimators':    [100, 200, 300],
    'model__max_depth':       [None, 5, 8, 12],
    'model__min_samples_leaf':[1, 2, 4],
    'model__max_features':    ['sqrt', 'log2'],
}

rf_search_pipeline = Pipeline([
    ('preprocessor', preprocessor),
    ('model', RandomForestClassifier(random_state=42))
])

search = RandomizedSearchCV(
    rf_search_pipeline,
    param_distributions=param_distributions,
    n_iter=20,
    cv=5,
    scoring='accuracy',
    random_state=42,
    n_jobs=-1,
    verbose=1
)

search.fit(X_train, y_train)

print(f"\nBest parameters: {search.best_params_}")
print(f"Best CV accuracy: {search.best_score_:.3f}")

# Output (approximate):
# Fitting 5 folds for each of 20 candidates, totalling 100 fits
# Best parameters: {'model__n_estimators': 200, 'model__max_depth': 8,
#                   'model__min_samples_leaf': 2, 'model__max_features': 'sqrt'}
# Best CV accuracy: 0.833

Info

The best max_depth is almost always less than None (unlimited). Limiting tree depth is the primary regularisation lever for Random Forest. min_samples_leaf=2 or 4 prevents the model from creating leaves that contain only one or two highly specific training examples.

Final Model

Refit the best configuration on the full training set, then evaluate with 5-fold cross-validation to get a robust accuracy estimate.

# Retrieve the best pipeline from the search
best_rf = search.best_estimator_

# Cross-validated accuracy on the training set
cv_scores = cross_val_score(best_rf, X_train, y_train, cv=5, scoring='accuracy')

print(f"5-fold CV accuracy: {cv_scores.mean():.3f} ± {cv_scores.std():.3f}")
print(f"Individual folds:   {cv_scores.round(3)}")
# Output (approximate):
# 5-fold CV accuracy: 0.833 ± 0.018
# Individual folds:   [0.812 0.852 0.831 0.845 0.824]

# Validation set performance (tuned model)
best_val_acc = best_rf.score(X_test, y_test)
print(f"\nTuned Random Forest — val accuracy: {best_val_acc:.3f}")
# Output (approximate):
# Tuned Random Forest — val accuracy: 0.838

# Full classification report on validation set
y_pred = best_rf.predict(X_test)
print("\nClassification Report:")
print(classification_report(y_test, y_pred, target_names=['Died', 'Survived']))
# Output (approximate):
#               precision    recall  f1-score   support
#         Died       0.86      0.90      0.88       110
#     Survived       0.82      0.75      0.78        69
#     accuracy                           0.84       179
#    macro avg       0.84      0.83      0.83       179
# weighted avg       0.84      0.84      0.84       179

Success

The tuned Random Forest reaches approximately 83–84% validation accuracy. The classification report shows stronger precision and recall for class 0 (died) than class 1 (survived), which reflects the class imbalance: there are 110 deaths and 69 survivors in the test set. The evaluation.md file analyses this in depth.

← Feature Engineering | Next: Evaluation →