EDA — Exploring the Titanic Dataset¶
Exploratory Data Analysis is not a checklist exercise. The goal is to build a mental model of your data before you touch any modelling code. Every plot and summary statistic here should answer a specific question: who survived, and why? Your EDA findings will directly drive the feature engineering decisions in the next file.
Learning Objectives¶
- Compute and visualise survival rates across multiple demographic slices
- Identify the three missing value patterns and choose an appropriate treatment for each
- Spot redundant columns before they contaminate the feature matrix
- Form hypotheses about which engineered features will be predictive
Load and First Look¶
import seaborn as sns
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
df = sns.load_dataset('titanic')
print(df.shape)
# Output: (891, 15)
print(df.head())
# Output (first 5 rows — abbreviated):
# survived pclass sex age sibsp parch fare embarked class \
# 0 0 3 male 22.0 1 0 7.2500 S Third
# 1 1 1 female 38.0 1 0 71.2833 C First
# 2 1 3 female 26.0 0 0 7.9250 S Third
# 3 1 1 female 35.0 1 0 53.1000 S First
# 4 0 3 male 35.0 0 0 8.0500 S Third
print(df.info())
# Output shows dtypes: int64, float64, object, category, bool
# Non-null count reveals 714/891 age values and 203/891 deck values
Two columns jump out immediately: age has 714 non-null values (177 missing) and deck has only 203 non-null values (688 missing). Address these in feature engineering.
Survival Rate Overview¶
Start with the headline number, then break it down.
# Overall survival rate
overall_rate = df['survived'].mean()
print(f"Overall survival rate: {overall_rate:.1%}")
# Output: Overall survival rate: 38.4%
# Survival rate by sex
sex_survival = df.groupby('sex')['survived'].agg(['mean', 'count', 'sum'])
sex_survival.columns = ['survival_rate', 'total', 'survivors']
sex_survival['survival_rate'] = sex_survival['survival_rate'].map('{:.1%}'.format)
print("\nSurvival by sex:")
print(sex_survival)
# Output:
# Survival by sex:
# survival_rate total survivors
# sex
# female 74.2% 314 233
# male 18.9% 577 109
# Survival rate by passenger class
class_survival = df.groupby('pclass')['survived'].agg(['mean', 'count', 'sum'])
class_survival.columns = ['survival_rate', 'total', 'survivors']
class_survival['survival_rate'] = class_survival['survival_rate'].map('{:.1%}'.format)
print("\nSurvival by passenger class:")
print(class_survival)
# Output:
# Survival by passenger class:
# survival_rate total survivors
# pclass
# 1 62.9% 216 136
# 2 47.3% 184 87
# 3 24.2% 491 119
# Survival rate by embarkation port
port_survival = df.groupby('embarked')['survived'].mean().sort_values(ascending=False)
print("\nSurvival by embarkation port:")
print(port_survival.map('{:.1%}'.format))
# Output:
# Survival by embarkation port:
# embarked
# C 55.4%
# Q 38.9%
# S 33.7%
Interpretation: The sex gap is enormous — 74% vs 19%. This will be the single most important feature. Class also matters significantly: first-class passengers survived at more than double the rate of third-class passengers. The embarkation port difference is partly confounded by class composition (Cherbourg had more first-class passengers).
Visualise Survival by Key Groups¶
fig, axes = plt.subplots(1, 3, figsize=(15, 5))
# Plot 1: Survival by sex
sex_counts = df.groupby(['sex', 'survived']).size().unstack()
sex_counts.columns = ['Died', 'Survived']
sex_counts.plot(kind='bar', ax=axes[0], color=['#E57373', '#4DB6AC'], edgecolor='white')
axes[0].set_title('Survival by Sex', fontsize=13)
axes[0].set_xlabel('')
axes[0].set_xticklabels(['Female', 'Male'], rotation=0)
axes[0].set_ylabel('Passenger Count')
axes[0].legend()
# Plot 2: Survival by class
class_counts = df.groupby(['pclass', 'survived']).size().unstack()
class_counts.columns = ['Died', 'Survived']
class_counts.plot(kind='bar', ax=axes[1], color=['#E57373', '#4DB6AC'], edgecolor='white')
axes[1].set_title('Survival by Passenger Class', fontsize=13)
axes[1].set_xlabel('')
axes[1].set_xticklabels(['1st', '2nd', '3rd'], rotation=0)
axes[1].set_ylabel('Passenger Count')
axes[1].legend()
# Plot 3: Survival by sex and class (interaction)
pivot = df.groupby(['pclass', 'sex'])['survived'].mean().unstack()
pivot.plot(kind='bar', ax=axes[2], color=['#4DB6AC', '#5C6BC0'], edgecolor='white')
axes[2].set_title('Survival Rate: Sex × Class', fontsize=13)
axes[2].set_xlabel('Passenger Class')
axes[2].set_xticklabels(['1st', '2nd', '3rd'], rotation=0)
axes[2].set_ylabel('Survival Rate')
axes[2].set_ylim(0, 1)
axes[2].yaxis.set_major_formatter(plt.FuncFormatter(lambda y, _: f'{y:.0%}'))
axes[2].legend(title='Sex')
plt.tight_layout()
plt.savefig('survival_by_group.png', dpi=150)
plt.show()
Info
The sex × class interaction plot is the most revealing chart in this EDA. Almost all women in first and second class survived. Almost no men in any class survived. Third-class women had a roughly 50/50 chance. This interaction — sex AND class together — is more predictive than either feature alone.
Age Distribution¶
fig, axes = plt.subplots(1, 2, figsize=(13, 5))
# Age histogram split by survival
for survived, colour, label in [(0, '#E57373', 'Died'), (1, '#4DB6AC', 'Survived')]:
subset = df[df['survived'] == survived]['age'].dropna()
axes[0].hist(subset, bins=30, alpha=0.6, color=colour, label=label, edgecolor='white')
axes[0].set_title('Age Distribution by Survival', fontsize=13)
axes[0].set_xlabel('Age (years)')
axes[0].set_ylabel('Count')
axes[0].legend()
axes[0].axvline(16, color='gray', linestyle='--', alpha=0.7, label='Age 16')
# Median age by survival
median_ages = df.groupby('survived')['age'].median()
print("Median age by survival outcome:")
print(median_ages)
# Output:
# survived
# 0 28.0
# 1 28.0
# dtype: float64
# (Medians are close — but the distribution tails differ significantly)
# Survival rate by age group
df['age_group'] = pd.cut(df['age'], bins=[0, 16, 32, 48, 80],
labels=['Child (0-16)', 'Young Adult (17-32)',
'Adult (33-48)', 'Senior (49+)'])
age_survival = df.groupby('age_group', observed=True)['survived'].mean()
age_survival.plot(kind='bar', ax=axes[1], color='#4DB6AC', edgecolor='white')
axes[1].set_title('Survival Rate by Age Group', fontsize=13)
axes[1].set_xlabel('')
axes[1].set_xticklabels(age_survival.index, rotation=20, ha='right')
axes[1].set_ylabel('Survival Rate')
axes[1].set_ylim(0, 1)
axes[1].yaxis.set_major_formatter(plt.FuncFormatter(lambda y, _: f'{y:.0%}'))
# Drop temp column
df.drop(columns=['age_group'], inplace=True)
plt.tight_layout()
plt.savefig('age_distribution.png', dpi=150)
plt.show()
print("\nSurvival rate by age group:")
print(age_survival.map('{:.1%}'.format))
# Output (approximate):
# age_group
# Child (0-16) 54.1%
# Young Adult (17-32) 36.2%
# Adult (33-48) 40.4%
# Senior (49+) 34.4%
Children under 16 have a meaningfully higher survival rate — the "women and children first" protocol is visible in the data. This motivates creating an is_child flag or an age_bucket feature in feature engineering.
Fare Distribution¶
fig, axes = plt.subplots(1, 2, figsize=(13, 5))
# Raw fare distribution (right-skewed)
axes[0].hist(df['fare'], bins=50, color='#5C6BC0', edgecolor='white')
axes[0].set_title('Fare Distribution (Raw)', fontsize=13)
axes[0].set_xlabel('Fare (£)')
axes[0].set_ylabel('Count')
# After log transformation
axes[1].hist(np.log1p(df['fare']), bins=40, color='#4DB6AC', edgecolor='white')
axes[1].set_title('Fare Distribution (Log-Transformed)', fontsize=13)
axes[1].set_xlabel('log(1 + Fare)')
axes[1].set_ylabel('Count')
plt.tight_layout()
plt.savefig('fare_distribution.png', dpi=150)
plt.show()
# Fare statistics
print(df['fare'].describe())
# Output (approximate):
# count 891.000000
# mean 32.204208
# std 49.693429
# min 0.000000
# 25% 7.910400
# 50% 14.454200
# 75% 31.000000
# max 512.329200
# Fare by class
print("\nMedian fare by class:")
print(df.groupby('pclass')['fare'].median())
# Output:
# pclass
# 1 60.2875
# 2 14.2500
# 3 8.0500
Warning
Some fares are 0.0 — these are not free tickets. They likely represent crew members or passengers whose fare was not recorded. They are outliers that will distort scaling. When you create a fare_per_person feature, guard against division by zero and flag zero-fare passengers separately.
Fare is heavily right-skewed (max is £512, median is £14.45). Log-transforming fare with np.log1p() produces a near-normal distribution that is friendlier for linear models. Tree-based models do not need this transformation, but it does not hurt them either.
Missing Values¶
missing = df.isnull().sum()
missing_pct = (df.isnull().mean() * 100).round(1)
missing_df = pd.DataFrame({'Count': missing, 'Percentage': missing_pct})
missing_df = missing_df[missing_df['Count'] > 0].sort_values('Percentage', ascending=False)
print(missing_df)
# Output:
# Count Percentage
# deck 688 77.2
# age 177 19.9
# embark_town 2 0.2
# embarked 2 0.2
Each missing value pattern requires a different approach:
deck (77.2% missing): Missingness is not random — it correlates strongly with passenger class. Third-class passengers rarely had assigned cabins. Imputing 688 values from 203 non-null values would be fabricating data. Drop this column. Optionally, create a deck_known binary flag before dropping.
age (19.9% missing): Age is too important to drop. Dropping 177 rows loses 20% of the training set. Impute with the median age (28.0). Median is safer than mean here because the age distribution has a right tail (some passengers were in their 70s and 80s).
embarked (0.2% missing): Only 2 passengers. Impute with 'S' (Southampton) — the mode at 72% of passengers.
Warning
Fit your imputer on the training set only, then transform both train and test. Fitting on the full dataset — including test — causes data leakage. The median age of the test set is unknown at prediction time. Using sklearn's SimpleImputer inside a Pipeline handles this automatically.
Bivariate Analysis¶
# Survival rate as a function of family size
df['family_size'] = df['sibsp'] + df['parch'] + 1
family_survival = df.groupby('family_size')['survived'].agg(['mean', 'count'])
family_survival.columns = ['survival_rate', 'count']
print("Survival rate by family size:")
print(family_survival[family_survival['count'] >= 10].round(3))
# Output (approximate — only groups with ≥10 passengers):
# survival_rate count
# family_size
# 1 0.304 537
# 2 0.553 161
# 3 0.578 72
# 4 0.724 29
# 5 0.200 15
# 6 0.136 22
df.drop(columns=['family_size'], inplace=True)
# Key insight: small families (2-4) had higher survival than solo travellers
# Very large families (5+) had low survival — harder to evacuate as a group
The family size pattern is non-linear: travelling alone is bad, small families are better, large families are worse again. A simple numerical feature will not capture this. Consider bucketing into alone / small / large.
Correlation¶
# Select numeric columns for correlation analysis
numeric_cols = ['survived', 'pclass', 'age', 'sibsp', 'parch', 'fare']
corr_matrix = df[numeric_cols].corr()
fig, ax = plt.subplots(figsize=(8, 6))
sns.heatmap(
corr_matrix,
annot=True,
fmt='.2f',
cmap='coolwarm',
center=0,
square=True,
linewidths=0.5,
ax=ax
)
ax.set_title('Correlation Matrix — Numeric Features vs Survived', fontsize=13)
plt.tight_layout()
plt.savefig('correlation_heatmap.png', dpi=150)
plt.show()
print(corr_matrix['survived'].sort_values())
# Output (approximate):
# pclass -0.338
# age -0.077
# sibsp -0.035
# parch 0.082
# fare 0.257
# survived 1.000
Interpretation: pclass has the strongest negative correlation with survival (-0.34): lower class number (higher prestige) = higher survival. fare has a positive correlation (0.26) — but this is largely because fare is a proxy for class. age and family composition have weaker correlations in isolation, but become meaningful when combined with sex.
Info
Correlation only measures linear relationships between numeric features. The most important predictor — sex — is categorical and does not appear here. Do not judge feature importance from the correlation heatmap alone. After encoding, the model's feature importance scores are more reliable.
Key Findings¶
| Observation | Interpretation | Action |
|---|---|---|
| Women survived at 74%, men at 19% | "Women and children first" protocol dominated outcomes | sex is likely the single strongest feature; encode it as binary |
| First-class survival (63%) vs third-class (24%) | Deck proximity to lifeboats, crew enforcement of hierarchy | pclass is important as-is; also look at fare as a class proxy |
| Children under 16 survived at ~54% | Children were prioritised during evacuation | Create an age_bucket feature or is_child flag |
| Solo travellers survived at 30%; small families at 55–72% | Groups could coordinate; very large groups could not evacuate together | Create family_size and is_alone; bucket large families |
deck is 77% missing |
Missingness correlates with third-class (no cabin assigned) | Drop deck; optionally create deck_known binary flag |