Categorical Features

Categorical columns are where most encoding mistakes happen. One-hot encoding a column with 500 unique values adds 500 columns to your DataFrame and almost certainly hurts your model — the signal gets diluted across hundreds of sparse binary columns. Target encoding exists for a reason. This note gives you a clear decision framework so you are not applying the wrong encoding and wondering why your model underperforms.

---

Learning Objectives

By the end of this note you will be able to:

• Choose between label encoding, one-hot encoding, target encoding, and frequency encoding based on column cardinality and data type
• Implement one-hot encoding with both pd.get_dummies and OneHotEncoder, and explain the dummy variable trap
• Implement target encoding safely with cross-validation to prevent leakage
• Apply frequency encoding for high-cardinality columns
• Handle rare categories and unseen categories in production

---

The Encoding Decision Framework

Before writing any code, categorise the column:

Column Type	Examples	Correct Encoding
Ordinal (has natural order)	risk_level, education, size (S/M/L)	Manual integer mapping
Nominal, low cardinality (≤15 unique)	browser, gender, region	One-hot encoding
Nominal, medium cardinality (15–50 unique)	product_category, department	One-hot with rare-category grouping
Nominal, high cardinality (50+ unique)	city, zip_code, product_id	Target encoding or frequency encoding
Identifier column	customer_id, transaction_id	Drop it — identifiers are not features

This framework will handle 95% of categorical columns you encounter in practice.

---

Label Encoding — For Ordinal Data Only

Label encoding assigns an integer to each category. It implies an ordering: 1 < 2 < 3. This is exactly what you want for ordinal data where that ordering exists in reality.

import pandas as pd

loan_applications = pd.DataFrame({
    "customer_id": [101, 102, 103, 104, 105],
    "credit_risk": ["Low", "High", "Medium", "Low", "High"],
    "education": ["Graduate", "Undergraduate", "PhD", "Graduate", "Undergraduate"]
})

# Correct: ordinal mapping preserves the meaningful order
credit_risk_map = {"Low": 1, "Medium": 2, "High": 3}
education_map = {"Undergraduate": 1, "Graduate": 2, "PhD": 3}

loan_applications["credit_risk_encoded"] = loan_applications["credit_risk"].map(credit_risk_map)
loan_applications["education_encoded"] = loan_applications["education"].map(education_map)

print(loan_applications[["credit_risk", "credit_risk_encoded", "education", "education_encoded"]])
# Output:
#   credit_risk  credit_risk_encoded        education  education_encoded
# 0         Low                    1         Graduate                  2
# 1        High                    3    Undergraduate                  1
# 2      Medium                    2              PhD                  3
# 3         Low                    1         Graduate                  2
# 4        High                    3    Undergraduate                  1

Label Encoding Nominal Data Is a Mistake

If you label-encode city as Delhi=0, Mumbai=1, Pune=2, you are telling the model that Pune is mathematically twice Mumbai. Linear models will treat these integer codes as meaningful magnitudes. For any column without a true order, use one-hot or target encoding instead.

---

One-Hot Encoding — For Low-Cardinality Nominal Data

One-hot encoding creates one binary column per unique category. The value is 1 if the row belongs to that category, 0 otherwise. It makes no assumptions about order between categories.

Using pd.get_dummies

Fast and convenient for exploratory work. Not suitable for production because it cannot handle unseen categories at inference time.

import pandas as pd

website_sessions = pd.DataFrame({
    "session_id": [1, 2, 3, 4, 5],
    "browser": ["Chrome", "Firefox", "Safari", "Chrome", "Edge"],
    "session_duration_sec": [120, 340, 89, 450, 210]
})

# Basic one-hot encoding
encoded = pd.get_dummies(website_sessions, columns=["browser"], dtype=int)

print(encoded)
# Output:
#    session_id  session_duration_sec  browser_Chrome  browser_Edge  browser_Firefox  browser_Safari
# 0           1                   120               1             0                0               0
# 1           2                   340               0             0                1               0
# 2           3                    89               0             0                0               1
# 3           4                   450               1             0                0               0
# 4           5                   210               0             1                0               0

# Drop one column to avoid the dummy variable trap
encoded_no_trap = pd.get_dummies(website_sessions, columns=["browser"], drop_first=True, dtype=int)
print(encoded_no_trap.columns.tolist())
# Output: ['session_id', 'session_duration_sec', 'browser_Firefox', 'browser_Safari', 'browser_Edge']

The Dummy Variable Trap

With 4 browser categories, you need only 3 binary columns to fully represent all possibilities. The 4th column is perfectly predictable from the other 3 (if Chrome=0, Firefox=0, Safari=0, then Edge=1). This perfect multicollinearity breaks linear models. Use drop_first=True in get_dummies or drop='first' in OneHotEncoder to avoid it.

Tree models are immune to the dummy variable trap — they split on individual features and the redundant column costs only a small amount of training time. Still, dropping it is good practice.

Using OneHotEncoder — Production-Safe

OneHotEncoder from sklearn is the correct choice for production systems. It stores the categories seen during training and can handle unseen categories at inference time via handle_unknown='ignore'.

import pandas as pd
import numpy as np
from sklearn.preprocessing import OneHotEncoder

train_sessions = pd.DataFrame({
    "browser": ["Chrome", "Firefox", "Safari", "Chrome", "Firefox"],
    "device": ["Mobile", "Desktop", "Mobile", "Desktop", "Mobile"],
    "converted": [1, 0, 1, 1, 0]
})

# Unseen category "Edge" appears at inference time
inference_session = pd.DataFrame({
    "browser": ["Chrome", "Edge"],  # Edge was not in training
    "device": ["Mobile", "Desktop"]
})

ohe = OneHotEncoder(drop="first", handle_unknown="ignore", sparse_output=False)
ohe.fit(train_sessions[["browser", "device"]])

# Training transform
train_encoded = ohe.transform(train_sessions[["browser", "device"]])
train_cols = ohe.get_feature_names_out(["browser", "device"])
print(pd.DataFrame(train_encoded, columns=train_cols).head())
# Output:
#    browser_Firefox  browser_Safari  device_Mobile
# 0              0.0             0.0            1.0
# 1              1.0             0.0            0.0
# 2              0.0             1.0            1.0
# 3              0.0             0.0            0.0
# 4              1.0             0.0            1.0

# Inference transform — Edge becomes all zeros (not an error)
inference_encoded = ohe.transform(inference_session[["browser", "device"]])
print(pd.DataFrame(inference_encoded, columns=train_cols))
# Output:
#    browser_Firefox  browser_Safari  device_Mobile
# 0              0.0             0.0            1.0
# 1              0.0             0.0            0.0   ← Edge → all zeros (treated as unknown)

handle_unknown='ignore' Is Non-Negotiable for Production

Your training data will not contain every possible category. New cities appear, new browsers launch, new product categories are added. Without handle_unknown='ignore', your pipeline will raise an error the first time it sees an unseen category. Set it on every OneHotEncoder you deploy.

---

Handling Rare Categories

High-frequency categories carry signal. A category that appears 3 times in 100,000 rows is noise — the model cannot learn anything meaningful from it, and it will overfit to those 3 specific rows.

Group rare categories into an "Other" bucket before encoding.

import pandas as pd
import numpy as np

np.random.seed(42)
# Simulate a city column with some high-frequency and many rare cities
cities = (["Mumbai"] * 3000 + ["Delhi"] * 2500 + ["Bangalore"] * 2000 +
          ["Chennai"] * 1500 + ["Pune"] * 1000 + ["Hyderabad"] * 800 +
          ["Kolkata"] * 600 + ["Ahmedabad"] * 400 +
          [f"City_{i}" for i in range(200)])  # 200 rare cities, 1 occurrence each

customer_data = pd.DataFrame({"city": cities})

# Count occurrences
city_counts = customer_data["city"].value_counts()
print(f"Total unique cities: {city_counts.shape[0]}")
# Output: Total unique cities: 208

# Keep only cities that appear more than 300 times
frequent_cities = city_counts[city_counts >= 300].index
customer_data["city_grouped"] = customer_data["city"].where(
    customer_data["city"].isin(frequent_cities), other="Other"
)

print(customer_data["city_grouped"].value_counts())
# Output:
# Mumbai       3000
# Delhi        2500
# Bangalore    2000
# Chennai      1500
# Pune         1000
# Hyderabad     800
# Other         600  ← all rare cities grouped
# Kolkata       600
# Ahmedabad     400

Threshold Choice

A common threshold is any category appearing in less than 1% of rows, or fewer than 50 occurrences, whichever is smaller for your dataset size. Compute it programmatically rather than eyeballing.

---

Target Encoding — For High-Cardinality Columns

Target encoding replaces each category with the mean of the target variable for that category. A city column with 500 unique values becomes a single numeric column encoding how often customers in each city converted.

This is powerful. It is also dangerous if done naively.

The Naive (Leaky) Version

import pandas as pd
import numpy as np

# DO NOT use this in production — shown only to illustrate what target encoding does
df = pd.DataFrame({
    "city": ["Mumbai", "Delhi", "Mumbai", "Pune", "Delhi", "Mumbai", "Pune", "Delhi"],
    "churn": [1, 0, 0, 1, 1, 1, 0, 0]
})

# Compute mean churn per city on the FULL dataset
city_churn_rate = df.groupby("city")["churn"].mean()
print(city_churn_rate)
# Output:
# city
# Delhi     0.333333
# Mumbai    0.666667
# Pune      0.500000

df["city_target_encoded"] = df["city"].map(city_churn_rate)
print(df[["city", "churn", "city_target_encoded"]])
# Output — city_target_encoded already incorporates the row's own target value

Naive Target Encoding Is Leakage

When you compute the mean churn rate for Mumbai using all rows including the row you are predicting, the encoding for that row already contains its own target value. The model learns to predict the target from a feature that was derived from the target. This inflates training performance and the model will fail in production.

Always apply target encoding using cross-validation or use category_encoders.TargetEncoder which handles this internally.

Safe Target Encoding with Cross-Validation

The safe approach: compute the target mean for each category using only the rows in the other folds. The row's own target value never contributes to its own encoding.

import pandas as pd
import numpy as np
from sklearn.model_selection import KFold

def cross_val_target_encode(df: pd.DataFrame,
                             col: str,
                             target: str,
                             n_splits: int = 5,
                             smoothing: float = 10.0) -> pd.Series:
    """
    Target-encode a categorical column using k-fold cross-validation.
    smoothing pulls rare category estimates toward the global mean.
    """
    global_mean = df[target].mean()
    encoded = pd.Series(np.nan, index=df.index, name=f"{col}_te")
    kf = KFold(n_splits=n_splits, shuffle=True, random_state=42)

    for train_idx, val_idx in kf.split(df):
        train_fold = df.iloc[train_idx]
        # Compute per-category mean on training fold only
        means = train_fold.groupby(col)[target].agg(["mean", "count"])
        # Smoothing: blend category mean with global mean weighted by count
        smoothed = (means["count"] * means["mean"] + smoothing * global_mean) / \
                   (means["count"] + smoothing)
        encoded.iloc[val_idx] = df.iloc[val_idx][col].map(smoothed).fillna(global_mean)

    return encoded


np.random.seed(42)
n = 1000
sales = pd.DataFrame({
    "product_category": np.random.choice(
        ["Electronics", "Clothing", "Food", "Books", "Toys", "Furniture",
         "Beauty", "Sports", "Automotive", "Garden"],
        size=n
    ),
    "purchase_value": np.random.uniform(10, 500, size=n),
    "converted": np.random.binomial(1, 0.3, size=n)
})

sales["product_category_te"] = cross_val_target_encode(
    sales, col="product_category", target="converted"
)

print(sales[["product_category", "converted", "product_category_te"]].head(8).round(3))
# Output: each category gets a smoothed mean conversion rate derived without its own row's target

category_encoders Library

The category_encoders library provides production-ready implementations of target encoding, leave-one-out encoding, and James-Stein encoding. Install with pip install category_encoders. For quick experiments: from category_encoders import TargetEncoder.

---

Frequency Encoding — A Leakage-Free Alternative for High Cardinality

Frequency encoding replaces each category with how often it appears in the dataset. It captures the idea that common categories may behave differently from rare ones, without using the target at all — so no leakage risk.

import pandas as pd
import numpy as np

np.random.seed(0)
user_events = pd.DataFrame({
    "user_id": np.random.randint(1, 500, size=2000),
    "product_id": np.random.choice(range(1, 150), size=2000, p=None),
    "purchased": np.random.binomial(1, 0.2, size=2000)
})

# Frequency encoding — fit on training data only
product_freq = user_events["product_id"].value_counts(normalize=True)
user_events["product_id_freq"] = user_events["product_id"].map(product_freq)

print(user_events[["product_id", "product_id_freq"]].head(6).round(4))
# Output:
#    product_id  product_id_freq
# 0         237           0.0110
# 1          34           0.0155
# 2         121           0.0130
# ...

Frequency encoding works well for tree-based models. It is less informative than target encoding but completely safe from leakage.

---

Encoding Comparison Summary

Method	Cardinality	Leakage Risk	Tree Models	Linear Models
Label (ordinal only)	Any	None	Good	Good if truly ordinal
One-hot	Low (≤15)	None	Fine	Good
One-hot + rare grouping	Medium (15–50)	None	Fine	Good
Frequency encoding	High (50+)	None	Good	Moderate
Target encoding (CV)	High (50+)	Low if done right	Excellent	Good
Target encoding (naive)	Any	High	Overfit	Overfit

---

Key Takeaway

Encoding strategy should be determined by cardinality and column type — not by habit. One-hot is not the default; it is the right choice for low-cardinality nominal columns. Target encoding is the right choice for high-cardinality columns, but only when applied with cross-validation or a dedicated library that handles leakage prevention internally.

---

---

What's Next

You've covered label encoding for ordinal columns, one-hot encoding with drop='first' for nominal columns, rare-category grouping for medium-cardinality columns, safe cross-validated target encoding to prevent leakage, and frequency encoding as a leakage-free high-cardinality alternative. Next up: 04-datetime-and-text-features — where you'll decompose datetime columns into calendar components, compute recency and duration features, apply cyclical encoding for time-of-day, and convert text into word count, TF-IDF, and statistical features that tree and linear models can use.

Optional Deep Dive

Read the category_encoders library documentation at https://contrib.scikit-learn.org/category_encoders/ — it provides production-ready implementations of 15+ categorical encoding methods including target encoding with smoothing, leave-one-out encoding, and James-Stein estimation, with sklearn-compatible API so they drop straight into your Pipelines.

Back: Numeric Features | Next: Datetime and Text Features