Feature Engineering Overview

In Kaggle competitions, the teams that win rarely do so by choosing a better algorithm. They win by building better features. In production ML systems, the same principle applies: the gap between a 78% and an 85% AUC model is almost always in the features, not the model class. Feature engineering is the craft of extracting signal from raw data — turning a timestamp into "is this a weekend?", a price column into "log-normalized spend", and a customer ID into "days since last purchase."

---

Learning Objectives

By the end of this note you will be able to:

• Explain why feature engineering typically has more impact than algorithm selection on tabular data
• Distinguish between feature engineering, feature extraction, and feature selection
• Describe the feature engineering workflow in order
• Define data leakage and identify the most common ways it occurs
• State the golden rule of feature engineering and apply it to a new scenario

---

What Feature Engineering Is

Feature engineering is the process of transforming raw data columns into representations that make the underlying patterns more accessible to a learning algorithm.

The word "representation" is the key idea. A model does not see your data the way you do. A linear model sees vectors of numbers and tries to find a weighted sum that separates classes. A tree sees threshold comparisons. When you hand a linear model a raw signup_date column, it sees a large integer (Unix timestamp) with no inherent meaning. When you hand it customer_tenure_days, it sees a number that directly encodes how long that customer has been around — a signal with a clear business interpretation.

Feature engineering is the translation layer between business reality and mathematical model.

Feature Engineering vs Feature Extraction vs Feature Selection

These three terms are related but distinct:

• Feature engineering — manually creating new features from raw columns using domain knowledge (e.g., extracting hour_of_day from a timestamp)
• Feature extraction — algorithmically deriving features from unstructured data, often with dimensionality reduction (e.g., PCA components, TF-IDF vectors, CNN embeddings)
• Feature selection — choosing which features to keep from an existing set (e.g., dropping columns with low variance or high correlation)

Today focuses on engineering and extraction. Feature selection is covered in Day 03 Part 2.

---

Why Features Matter More Than Algorithms

This claim deserves evidence, not just assertion.

Consider predicting customer churn. Your raw data has signup_date (a timestamp) and last_order_date (another timestamp). A decision tree trained on raw timestamps will split on arbitrary integer thresholds with no coherent meaning.

Now engineer two features: days_since_last_order and customer_tenure_days. These carry direct business meaning. A customer who has not ordered in 90 days is churning. A customer who joined 2 years ago and has not ordered in 30 days is more alarming than a new customer with the same gap. The model can now learn that combination because you gave it the right representation.

No amount of hyperparameter tuning on the raw-timestamp version will reach parity with the engineered version, because the signal that explains churn was not accessible in the original representation.

Pedro Domingos, "A Few Useful Things to Know About Machine Learning"

"Feature engineering is the key driver of predictive performance in practical ML. Coming up with features is difficult, time-consuming, requires expert knowledge, and is where most of the effort in a machine learning project goes."

The rise of deep learning has automated some feature extraction for images and text. For tabular data — the dominant format in industry — learned representations rarely beat careful domain-driven feature engineering. Tree-based models (XGBoost, LightGBM, CatBoost) plus strong features still win most Kaggle tabular competitions.

---

The Feature Engineering Workflow

Work through these steps in order. Skipping ahead — especially skipping step 1 — is where bad features come from.

Step 1 — Understand the business problem

Before touching any data, ask: what is being predicted, and what information would a human expert use to make that decision? If you are predicting loan default, a credit analyst would look at income-to-debt ratio, payment history length, and number of recent credit inquiries. Those concepts should become features.

Step 2 — Profile your data

Look at distributions, missing value rates, cardinality counts for categoricals, and temporal ranges. You cannot engineer good features without knowing what the raw data actually looks like.

import pandas as pd
import numpy as np

# Profile a DataFrame in one call
def quick_profile(df: pd.DataFrame) -> pd.DataFrame:
    profile = pd.DataFrame({
        "dtype": df.dtypes,
        "null_pct": df.isnull().mean().round(3),
        "nunique": df.nunique(),
        "sample_values": [df[c].dropna().iloc[:3].tolist() for c in df.columns]
    })
    return profile

# Output: a row per column with dtype, % missing, cardinality, and three example values

Step 3 — Identify feature types

Categorize every column before deciding what to do with it:

Type	Examples	Typical Transforms
Continuous numeric	age, price, duration	log, binning, scaling, polynomial
Count	orders, page_views	log1p (counts are right-skewed)
Ordinal categorical	risk_level, education	manual integer mapping
Nominal categorical	city, browser, product_id	OHE (low cardinality), target encoding (high cardinality)
Datetime	signup_date, order_date	extract components, compute deltas
Free text	review, description	length features, TF-IDF, embeddings
Boolean / flag	is_mobile, has_promo	leave as 0/1

Step 4 — Handle missing values

Imputation strategy depends on why the value is missing. A missing income field in a loan application is not random — it may indicate that the applicant did not disclose it, which is itself a signal. Add a income_was_missing binary flag before imputing.

# Add missingness indicator before imputing
df["income_was_missing"] = df["annual_income"].isnull().astype(int)
df["annual_income"] = df["annual_income"].fillna(df["annual_income"].median())

Step 5 — Create transformations

This is where most of the work happens. Covered in detail across the remaining notes: - Numeric: 02-numeric-features - Categorical: 03-categorical-features - Datetime and text: 04-datetime-and-text-features

Step 6 — Encode and scale

Encoding converts categories to numbers. Scaling adjusts numeric ranges. Both belong inside a sklearn Pipeline — never applied to the full dataset before the train/test split.

Step 7 — Validate with a model

Run a quick cross-validation score after each significant batch of new features. If the CV score does not improve, the feature is not helping — remove it. Unused features add noise and slow down training.

Step 8 — Check for leakage

Before finalizing any feature, ask: would this value be available at prediction time? If the answer is no, or if the value was computed using knowledge of the target, the feature is leaking. Covered in detail in 05-pipelines-and-leakage.

---

The Transformation Table

A quick reference of raw columns and what to do with them:

Raw Column	Problem	Engineered Feature
order_date (datetime)	Not numeric	order_month, order_day_of_week, is_weekend
signup_date (datetime)	Not numeric	customer_tenure_days (relative to reference date)
transaction_amount (right-skewed)	Outliers distort linear models	log1p(transaction_amount)
city (500 unique values)	Too many OHE columns	frequency encoding or target encoding
risk_level (Low/Med/High)	Ordered but text	manual map: {"Low": 1, "Medium": 2, "High": 3}
review_text (free text)	Cannot use raw strings	word_count, char_count, TF-IDF matrix
age (continuous)	Model may prefer groups	pd.cut(age, bins=[0, 25, 40, 60, 100])

---

The Golden Rule

The Golden Rule of Feature Engineering

Only use information that would be available at the time your model makes a prediction.

If your model predicts loan default at the moment of application, features computed from post-application events are leakage. If your model predicts next month's sales, features that include this month's actual sales are leakage.

This rule sounds obvious. It is violated constantly in practice. Violating it produces models that look great in evaluation and fail in production.

---

Key Takeaway

Feature engineering is not about adding more columns. It is about adding columns that encode signal your model cannot see in the raw data. One well-reasoned feature built from domain knowledge is worth more than ten automated polynomial combinations.

---

---

What's Next

You've covered the definition of feature engineering as structured signal creation, the difference between raw columns and engineered features, domain-knowledge-driven feature creation, interaction features, the feature engineering workflow sequence, and the leakage rule for time-based features. Next up: 02-numeric-features — where you'll apply log and Yeo-Johnson transforms to fix skewed distributions, winsorize outliers without losing rows, create ratio and polynomial interaction features, and choose the right scaler for each algorithm type.

Optional Deep Dive

Read "Feature Engineering for Machine Learning" by Alice Zheng and Amanda Casari (O'Reilly) — it is one of the few books dedicated entirely to this topic, covering numeric transformations, categorical encoding, text features, and interaction creation with practical examples from real datasets across multiple domains.

Back to Agenda | Next: Numeric Features