Feature Engineering — Building Better Predictors

Raw data rarely arrives in a form that a model can use well. The Titanic dataset gives you a passenger's name — which seems useless — but hidden inside it is a title (Mr, Mrs, Miss, Master) that encodes sex, age group, and marital status simultaneously. Feature engineering is the skill of extracting latent signals from raw columns. On this dataset, the difference between a 78% and an 83% accuracy model is almost entirely explained by how well the features are engineered.

Learning Objectives

• Drop redundant columns with clear justification for each
• Impute missing values correctly, using training data statistics only
• Create new features that encode domain knowledge (family size, title, age bucket)
• Encode categorical features for sklearn compatibility
• Build a reproducible sklearn Pipeline that combines all preprocessing steps

---

Step 0: Load and Inspect

import seaborn as sns
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler, LabelEncoder
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder

df = sns.load_dataset('titanic')
print(f"Starting shape: {df.shape}")
# Output: Starting shape: (891, 15)

---

Step 1: Drop Redundant Columns

# Drop columns that duplicate information or cause leakage
columns_to_drop = [
    'class',        # duplicate of pclass (string version)
    'embark_town',  # duplicate of embarked (full name version)
    'alive',        # target leakage — string version of survived
    'who',          # derived from sex + age — no independent information
    'adult_male',   # derived from sex + age — no independent information
    'deck',         # 77% missing — not imputable reliably
]

df = df.drop(columns=columns_to_drop)
print(f"Shape after dropping redundant columns: {df.shape}")
# Output: Shape after dropping redundant columns: (891, 9)
print(df.columns.tolist())
# Output: ['survived', 'pclass', 'sex', 'age', 'sibsp', 'parch', 'fare', 'embarked', 'alone']

Info

alone is technically redundant (it equals sibsp + parch == 0) but we keep it for now and will replace it with the engineered is_alone feature. Drop the original alone after creating the new version to avoid duplication.

---

Step 2: Extract Title from Name

This step requires accessing the raw seaborn dataset before dropping columns. The name column is present in the seaborn dataset but we need to re-load it to access it.

# Reload to access name column
df_raw = sns.load_dataset('titanic')

# Extract title using regex
# Name format: "Last, Title. First Middle"
df_raw['title'] = df_raw['name'].str.extract(r',\s([A-Za-z]+)\.')

print(df_raw['title'].value_counts())
# Output:
# Mr          517
# Miss        182
# Mrs         125
# Master       40
# Dr            7
# Rev           6
# Col           2
# Mlle          2
# Major         2
# Capt          1
# Lady          1
# Sir           1
# Ms            1
# Don           1
# Jonkheer      1
# Countess      1
# Mme           1
# dtype: int64

# Consolidate rare titles
rare_titles = ['Dr', 'Rev', 'Col', 'Major', 'Mlle', 'Capt', 'Lady',
               'Sir', 'Ms', 'Don', 'Jonkheer', 'Countess', 'Mme']
df_raw['title'] = df_raw['title'].replace(rare_titles, 'Rare')

print("\nTitle survival rates:")
title_survival = df_raw.groupby('title')['survived'].agg(['mean', 'count'])
title_survival.columns = ['survival_rate', 'count']
print(title_survival.sort_values('survival_rate', ascending=False))
# Output (approximate):
#         survival_rate  count
# title
# Mrs              0.793    125
# Miss             0.700    182
# Rare             0.347     22
# Master           0.575     40
# Mr               0.157    517

# 'Master' is the title for male children — their elevated survival rate
# captures the 'children first' signal precisely

Master is the key discovery here. In the early 20th century, "Master" was used exclusively for boys under ~13. The model cannot learn "children survive more" from age alone when 20% of ages are missing — but title == Master flags children even when age is null.

Tip

This is why domain knowledge matters. Without knowing that "Master" is a title for young boys, you would never think to extract it. Always research what your raw columns actually represent before deciding which features to engineer.

---

Step 3: Create Family Size Features

# Create all new features on the raw dataframe, then subset what we need
df_raw['family_size'] = df_raw['sibsp'] + df_raw['parch'] + 1

# Binary flag: is the passenger travelling alone?
df_raw['is_alone'] = (df_raw['family_size'] == 1).astype(int)

# Family size bucket captures the non-linear relationship found in EDA
def family_bucket(size):
    if size == 1:
        return 'alone'
    elif size <= 4:
        return 'small'
    else:
        return 'large'

df_raw['family_bucket'] = df_raw['family_size'].apply(family_bucket)

print("Family size bucket survival rates:")
print(df_raw.groupby('family_bucket')['survived'].mean().sort_values(ascending=False).map('{:.1%}'.format))
# Output:
# family_bucket
# small    56.8%
# large    16.1%
# alone    30.4%

---

Step 4: Age Bucket

df_raw['age_bucket'] = pd.cut(
    df_raw['age'],
    bins=[0, 12, 18, 35, 60, 120],
    labels=['child', 'teen', 'young_adult', 'adult', 'senior']
)

print("Age bucket survival rates:")
print(df_raw.groupby('age_bucket', observed=True)['survived'].mean().map('{:.1%}'.format))
# Output (approximate):
# age_bucket
# child         59.4%
# teen          44.0%
# young_adult   37.5%
# adult         42.4%
# senior        32.1%

Info

The age bucket is particularly useful for passengers where age is missing — after imputation with the median, those passengers will all land in the young_adult or adult bucket. The bucket effectively softens the impact of the imputed values by grouping them with similar passengers rather than treating each imputed value as precise.

---

Step 5: Fare Per Person

# Some tickets were shared — the fare column lists the total ticket price
# Dividing by family size gives a per-person cost that is more comparable
df_raw['fare_per_person'] = df_raw['fare'] / df_raw['family_size']

# Passengers with zero fare need special handling
print(f"Zero-fare passengers: {(df_raw['fare'] == 0).sum()}")
# Output: Zero-fare passengers: 15

# Their fare_per_person will also be 0, which is fine — they form a distinct group

---

Step 6: Build the Final Feature Matrix

# Assemble the working dataframe with all engineered features
feature_cols = [
    'survived',       # target
    'pclass',         # ordinal — keep as integer
    'sex',            # binary after encoding
    'age',            # float — will impute then scale
    'fare',           # float — will scale
    'embarked',       # 3 categories — one-hot encode
    'family_size',    # integer — scale
    'is_alone',       # binary integer
    'title',          # 5 categories — one-hot encode
    'age_bucket',     # 5 categories — one-hot encode (handles missing age gracefully)
    'fare_per_person' # float — scale
]

df_eng = df_raw[feature_cols].copy()
print(f"Feature matrix shape: {df_eng.shape}")
# Output: Feature matrix shape: (891, 11)

print(df_eng.dtypes)
# Output:
# survived        int64
# pclass          int64
# sex            object
# age           float64
# fare          float64
# embarked       object
# family_size     int64
# is_alone        int64
# title          object
# age_bucket   category
# fare_per_person float64

---

Step 7: Train-Test Split (Before Any Fitting)

Always split before fitting any imputer or scaler. Fitting on the full dataset reveals test set statistics to your preprocessing steps — that is leakage.

X = df_eng.drop(columns=['survived'])
y = df_eng['survived']

X_train, X_test, y_train, y_test = train_test_split(
    X, y,
    test_size=0.2,
    random_state=42,
    stratify=y          # preserve class balance in both splits
)

print(f"Training set: {X_train.shape}")
print(f"Test set:     {X_test.shape}")
print(f"Train survival rate: {y_train.mean():.3f}")
print(f"Test survival rate:  {y_test.mean():.3f}")
# Output:
# Training set: (712, 10)
# Test set:     (179, 10)
# Train survival rate: 0.383
# Test survival rate:  0.385

---

Step 8: Build the Preprocessing Pipeline

from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

# Define column groups
numeric_features = ['age', 'fare', 'family_size', 'fare_per_person']
binary_features = ['pclass', 'is_alone']          # already numeric — just scale
categorical_features = ['sex', 'embarked', 'title', 'age_bucket']

# Numeric pipeline: impute missing values, then scale
numeric_pipeline = Pipeline([
    ('imputer', SimpleImputer(strategy='median')),
    ('scaler', StandardScaler())
])

# Categorical pipeline: impute missing categories, then one-hot encode
categorical_pipeline = Pipeline([
    ('imputer', SimpleImputer(strategy='most_frequent')),
    ('encoder', OneHotEncoder(handle_unknown='ignore', sparse_output=False))
])

# Binary/ordinal columns: only impute (already numeric, no encoding needed)
binary_pipeline = Pipeline([
    ('imputer', SimpleImputer(strategy='median')),
    ('scaler', StandardScaler())
])

# Combine with ColumnTransformer
preprocessor = ColumnTransformer([
    ('num', numeric_pipeline, numeric_features),
    ('bin', binary_pipeline, binary_features),
    ('cat', categorical_pipeline, categorical_features),
])

# Fit the preprocessor on TRAINING data only, then transform both sets
X_train_processed = preprocessor.fit_transform(X_train)
X_test_processed  = preprocessor.transform(X_test)

print(f"Processed training shape: {X_train_processed.shape}")
# Output: Processed training shape: (712, 20)  — exact number depends on one-hot cardinality

Warning

preprocessor.fit_transform(X_train) and then preprocessor.transform(X_test) — not fit_transform(X_test). Calling fit_transform on the test set re-fits the imputer and scaler to test set statistics. This makes your test set performance look better than it actually is. In deployment, you will only ever have transform available, not fit_transform. Use the Pipeline as your contract.

---

Final Feature Set

After preprocessing, the model receives these features:

Feature Group	Features	Encoding
Numeric (scaled)	age, fare, family_size, fare_per_person	StandardScaler after median imputation
Binary/Ordinal	pclass, is_alone	StandardScaler
Sex	sex (male/female)	OneHotEncoder → 2 binary columns
Embarkation	embarked (C/Q/S)	OneHotEncoder → 3 binary columns
Title	title (Mr/Mrs/Miss/Master/Rare)	OneHotEncoder → 5 binary columns
Age bucket	age_bucket (child/teen/young_adult/adult/senior)	OneHotEncoder → 5 binary columns

Total: approximately 20 features going into the model, all numeric, no missing values.

Success

The full preprocessing Pipeline can be saved and loaded with joblib. When you deploy this model, you pass raw passenger data through the same Pipeline — it handles all imputation and encoding automatically. This is why building a Pipeline from the start is the professional approach, not a convenience.

import joblib
# joblib.dump(preprocessor, 'titanic_preprocessor.pkl')
# loaded_preprocessor = joblib.load('titanic_preprocessor.pkl')

---

Previous: Exploratory Data Analysis | Next: Model Building