Evaluation and Interpretation

A model's accuracy score tells you how often it is right. It does not tell you what kind of errors it makes, which features drive its decisions, or whether those decisions make sense given the historical record. This file works through the full evaluation stack: test set performance, confusion matrix, ROC curve, feature importances, and a manual look at where the model goes wrong.

Learning Objectives

• Evaluate a trained model on a truly held-out test set and interpret each metric
• Read a confusion matrix and map each quadrant to a concrete real-world consequence
• Plot and interpret an ROC curve with AUC
• Use permutation importance to identify which features the model actually relies on
• Analyse misclassified passengers to understand the model's systematic blind spots

---

Setup

Continue from model-building.md — best_rf, X_train, X_test, y_train, y_test must be defined. Run the full setup block from model-building first, then add the imports below.

import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
import pandas as pd
from sklearn.metrics import (
    classification_report, confusion_matrix,
    roc_curve, roc_auc_score
)
from sklearn.inspection import permutation_importance

---

Test Set Performance

The test set has been untouched since the train-test split. This is the only honest estimate of how the model performs on data it has never seen.

y_pred      = best_rf.predict(X_test)
y_pred_prob = best_rf.predict_proba(X_test)[:, 1]  # probability of surviving

test_accuracy = (y_pred == y_test).mean()
print(f"Test set accuracy: {test_accuracy:.3f}")
# Output (approximate):
# Test set accuracy: 0.838

print("\nClassification Report:")
print(classification_report(y_test, y_pred, target_names=['Died (0)', 'Survived (1)']))
# Output (approximate):
#                precision    recall  f1-score   support
#     Died (0)       0.86      0.91      0.88       110
# Survived (1)       0.82      0.74      0.78        69
#     accuracy                           0.84       179
#    macro avg       0.84      0.82      0.83       179
# weighted avg       0.85      0.84      0.84       179

Reading each metric:

Metric	Class 0 (Died)	Class 1 (Survived)	Interpretation
Precision	0.86	0.82	When the model predicts this class, how often is it right?
Recall	0.91	0.74	Of all passengers in this class, what fraction did the model find?
F1-score	0.88	0.78	Harmonic mean of precision and recall
Support	110	69	Number of test passengers in this class

Info

Recall for survivors (0.74) is lower than recall for deaths (0.91). The model misses 26% of actual survivors — it predicts they died when they actually lived. This is because survivors are the minority class: there are only 69 survivors vs 110 deaths in the test set. The model is slightly biased toward predicting the majority class.

---

Confusion Matrix

cm = confusion_matrix(y_test, y_pred)

fig, ax = plt.subplots(figsize=(6, 5))
sns.heatmap(
    cm,
    annot=True,
    fmt='d',
    cmap='Blues',
    xticklabels=['Predicted: Died', 'Predicted: Survived'],
    yticklabels=['Actual: Died', 'Actual: Survived'],
    linewidths=0.5,
    ax=ax
)
ax.set_xlabel('Predicted', fontsize=12)
ax.set_ylabel('Actual', fontsize=12)
ax.set_title('Confusion Matrix — Titanic Test Set', fontsize=13)
plt.tight_layout()
plt.savefig('confusion_matrix.png', dpi=150)
plt.show()

print(f"\nConfusion matrix values:")
print(f"  True Negatives  (TN): {cm[0,0]} — correctly predicted deaths")
print(f"  False Positives (FP): {cm[0,1]} — deaths incorrectly flagged as survivors")
print(f"  False Negatives (FN): {cm[1,0]} — survivors we missed")
print(f"  True Positives  (TP): {cm[1,1]} — correctly predicted survivors")
# Output (approximate):
#   True Negatives  (TN): 100 — correctly predicted deaths
#   False Positives (FP):  10 — deaths incorrectly flagged as survivors
#   False Negatives (FN):  18 — survivors we missed
#   True Positives  (TP):  51 — correctly predicted survivors

Quadrant interpretation:

• TN (top-left): Passengers who died and the model correctly predicted death. The majority cell.
• FP (top-right): Passengers who died but the model predicted survival. The model was overconfident — it saw features associated with survival (perhaps a woman in third class) but that passenger did not make it.
• FN (bottom-left): Passengers who survived but the model predicted death. These are the model's most consequential errors — it failed to identify a survivor. This is the largest error category for the minority class.
• TP (bottom-right): Passengers who survived and the model correctly predicted survival.

Tip

Whether FP or FN is more costly depends on the application. In historical analysis, both error types are equally important. In a hypothetical rescue prioritisation scenario, FN errors (missed survivors) would be far more costly than FP errors.

---

ROC Curve

The ROC curve shows the trade-off between true positive rate (recall for survivors) and false positive rate across every possible classification threshold. AUC summarises this in a single number: 1.0 is perfect, 0.5 is random.

fpr, tpr, thresholds = roc_curve(y_test, y_pred_prob)
auc_score = roc_auc_score(y_test, y_pred_prob)

fig, ax = plt.subplots(figsize=(7, 6))
ax.plot(fpr, tpr, color='#0D9488', lw=2, label=f'Random Forest (AUC = {auc_score:.3f})')
ax.plot([0, 1], [0, 1], color='#94A3B8', lw=1, linestyle='--', label='Random classifier (AUC = 0.500)')
ax.fill_between(fpr, tpr, alpha=0.1, color='#0D9488')
ax.set_xlabel('False Positive Rate', fontsize=12)
ax.set_ylabel('True Positive Rate (Recall)', fontsize=12)
ax.set_title('ROC Curve — Titanic Survival Prediction', fontsize=13)
ax.legend(loc='lower right', fontsize=11)
ax.set_xlim([0.0, 1.0])
ax.set_ylim([0.0, 1.05])
plt.tight_layout()
plt.savefig('roc_curve.png', dpi=150)
plt.show()

print(f"AUC score: {auc_score:.3f}")
# Output (approximate):
# AUC score: 0.872

Success

An AUC of ~0.87 means that if you randomly pick one survivor and one death from the test set, the model ranks the survivor as more likely to survive 87% of the time. AUC is threshold-independent — it measures how well the model separates the two classes, regardless of where you draw the classification line.

---

Feature Importance — Permutation Importance

Permutation importance measures how much model accuracy drops when a single feature is randomly shuffled. A feature is important if shuffling it hurts performance; it is unimportant if the model ignores it.

perm_result = permutation_importance(
    best_rf, X_test, y_test,
    n_repeats=20,
    random_state=42,
    scoring='accuracy'
)

# Get feature names from the preprocessor
feature_names_out = (
    numeric_features
    + binary_features
    + list(best_rf.named_steps['preprocessor']
           .named_transformers_['cat']
           .named_steps['encoder']
           .get_feature_names_out(categorical_features))
)

importance_df = pd.DataFrame({
    'feature':    feature_names_out,
    'importance': perm_result.importances_mean,
    'std':        perm_result.importances_std
}).sort_values('importance', ascending=False).head(15)

fig, ax = plt.subplots(figsize=(9, 6))
ax.barh(
    importance_df['feature'][::-1],
    importance_df['importance'][::-1],
    xerr=importance_df['std'][::-1],
    color='#0D9488',
    edgecolor='white',
    alpha=0.85
)
ax.set_xlabel('Mean Accuracy Decrease', fontsize=12)
ax.set_title('Top 15 Features — Permutation Importance', fontsize=13)
plt.tight_layout()
plt.savefig('feature_importance.png', dpi=150)
plt.show()

print(importance_df[['feature', 'importance']].to_string(index=False))
# Output (approximate, top features):
#               feature  importance
#           sex_female        0.094
#         title_Master        0.041
#           sex_male         0.038
#               pclass        0.035
#                 fare        0.028
#      fare_per_person        0.021
#          family_size        0.019
#          title_Miss         0.017
#            title_Mr         0.014
#             is_alone        0.012

Info

sex_female is the single most important feature — shuffling it alone drops accuracy by ~9.4 percentage points. title_Master (male children) ranks second because it captures survival signal for a group where sex alone is insufficient. pclass and fare appear next, confirming the class hierarchy effect. Features like embarked port tend to land near the bottom — their survival correlation was mostly confounded by class composition.

Interpretation: The permutation importance ranking matches the historical record almost perfectly. The "women and children first" protocol is quantitatively captured: sex_female and title_Master (children) dominate. Class and fare (a proxy for class) come next. Embarkation port matters little once class is accounted for.

---

Error Analysis

Look at the passengers the model got wrong. Misclassification patterns reveal what the model fails to learn.

# Reconstruct the original test rows alongside predictions
# We need the raw dataframe aligned with X_test
df_raw = sns.load_dataset('titanic')
df_raw['title'] = df_raw['name'].str.extract(r',\s([A-Za-z]+)\.')
rare_titles = ['Dr', 'Rev', 'Col', 'Major', 'Mlle', 'Capt', 'Lady',
               'Sir', 'Ms', 'Don', 'Jonkheer', 'Countess', 'Mme']
df_raw['title'] = df_raw['title'].replace(rare_titles, 'Rare')
df_raw['family_size'] = df_raw['sibsp'] + df_raw['parch'] + 1
df_raw['is_alone'] = (df_raw['family_size'] == 1).astype(int)
df_raw['fare_per_person'] = df_raw['fare'] / df_raw['family_size']
df_raw['age_bucket'] = pd.cut(
    df_raw['age'],
    bins=[0, 12, 18, 35, 60, 120],
    labels=['child', 'teen', 'young_adult', 'adult', 'senior']
)

feature_cols = ['survived', 'pclass', 'sex', 'age', 'fare', 'embarked',
                'family_size', 'is_alone', 'title', 'age_bucket', 'fare_per_person']
df_eng = df_raw[feature_cols].copy()

# Re-align predictions with test indices
misclassified_mask = (y_pred != y_test.values)
misclassified_idx  = X_test.index[misclassified_mask]

errors = df_eng.loc[misclassified_idx, ['sex', 'pclass', 'age', 'fare', 'title', 'survived']].copy()
errors['predicted'] = y_pred[misclassified_mask]

print(f"Total misclassified: {len(errors)}")
print("\nSample misclassifications:")
print(errors.head(10).to_string())
# Output (approximate):
# Total misclassified: 28
#     sex  pclass   age      fare    title  survived  predicted
# 1  female       1  38.0   71.283      Mrs         1          0
# ...

# Summarise error patterns
print("\nMisclassified — breakdown by sex:")
print(errors.groupby('sex')['survived'].value_counts())

print("\nMisclassified — breakdown by pclass:")
print(errors.groupby('pclass')['survived'].value_counts())

print("\nMisclassified — mean age and fare:")
print(errors.groupby('survived')[['age', 'fare']].mean().round(1))
# Output (approximate):
# Misclassified — mean age and fare:
#           age   fare
# survived
# 0        35.2   72.4   <- wealthy men the model expected to survive
# 1        26.1    9.8   <- poor women the model expected to die

Patterns in the errors:

• Wealthy men who survived: High fare, first class, male — the model correctly assigns these passengers low survival probability, but some of them actually survived (perhaps luck, proximity to a lifeboat). These are inherently difficult to predict.
• Poor women who died: Third-class women had roughly a 50% survival rate in EDA, much worse than first or second class women. The model learns the average and misclassifies the unlucky half.

Warning

Some misclassifications are irreducible — they are caused by factors the dataset does not capture (which lifeboat was nearby, which crew member helped, individual luck). Do not over-tune trying to eliminate every error. At some point, the remaining errors reflect the limits of the available features, not the model.

---

Business Interpretation

Three plain-English takeaways from this model that a non-technical audience can act on:

1. Survival was strongly structured by sex and passenger class. Women and first-class passengers survived at dramatically higher rates — not by chance, but because of deliberate evacuation protocols and physical proximity to lifeboats. The model captures both effects quantitatively.

2. The model encodes "women and children first" as a data pattern. The title feature identifies male children (Masters) as a high-survival group even when their age is missing. This is the feature engineering insight that lifts the model from 78% to 83% accuracy.

3. Fare is largely a proxy for class and cabin deck position. Passengers who paid more were housed closer to the boat deck, giving them faster lifeboat access. The fare coefficient effectively encodes cabin location — information that the deck column is too sparse to provide directly.

---

What Would You Do Next

• Engineer cabin/deck features: The deck column was dropped because 77% of values are missing. A deck_known binary flag and survival analysis on the 23% of passengers with known decks could add a meaningful signal.

• Try a stacking ensemble: Combine the predictions of Logistic Regression, Random Forest, and Gradient Boosting with a meta-learner. Stacking often adds 0.5–1% accuracy when the base models make different types of errors.

• Submit to Kaggle: The competition uses a separate 418-passenger test set with no survival labels. Submitting gives you an objective leaderboard benchmark. A model at ~83% local validation typically scores around 78–80% on the Kaggle public leaderboard (the leaderboard test set has a different class balance).

• Calibrate probabilities: Use CalibratedClassifierCV to ensure that the predicted probability of survival is well-calibrated. A passenger the model rates at 70% should survive about 70% of the time across many similar passengers.

---

← Model Building | Next: Interview Questions →