Dataset Guide

The California Housing dataset comes from the 1990 US Census. It describes census block groups — the smallest geographic unit for which the Census Bureau publishes sample data. Each row represents one block group, typically containing 600 to 3,000 people.

Info

A block group is not a single house. It is a neighborhood aggregate. When you see AveRooms = 6.2, that is the average across all households in that block group — not one house with 6.2 rooms.

Loading the Data

No CSV download, no Kaggle account. Scikit-learn ships this dataset.

from sklearn.datasets import fetch_california_housing
import pandas as pd
import numpy as np

# Load with as_frame=True to get pandas DataFrames
housing = fetch_california_housing(as_frame=True)

# housing.frame includes the target column
df = housing.frame

print(df.shape)
# (20640, 9)

print(df.dtypes)
# MedInc         float64
# HouseAge       float64
# AveRooms       float64
# AveBedrms      float64
# Population     float64
# AveOccup       float64
# Latitude       float64
# Longitude      float64
# MedHouseVal    float64

print(df.isnull().sum().sum())
# 0  — no missing values

Tip

housing.DESCR contains the full dataset description. Run print(housing.DESCR) once to read it. It gives context about the original paper (Pace & Barry, 1997) and how each variable was computed.

Column Reference

Column	Description	Units	Range
MedInc	Median household income in the block group	$10,000s	0.5 – 15.0
HouseAge	Median age of houses in the block group	Years	1 – 52
AveRooms	Average rooms per household	Rooms	0.8 – 141.9
AveBedrms	Average bedrooms per household	Bedrooms	0.3 – 34.1
Population	Total block group population	People	3 – 35,682
AveOccup	Average occupants per household	People	0.7 – 1,243.3
Latitude	Block group centroid latitude	Degrees N	32.5 – 41.9
Longitude	Block group centroid longitude	Degrees W (negative)	-124.4 – -114.3
MedHouseVal	Median house value — TARGET	$100,000s	0.15 – 5.0 (capped)

Summary Statistics

print(df.describe().round(2))
#         MedInc  HouseAge  AveRooms  AveBedrms  Population  AveOccup  Latitude  Longitude  MedHouseVal
# count  20640.0   20640.0   20640.0    20640.0     20640.0   20640.0   20640.0    20640.0      20640.0
# mean       3.87     28.64      5.43       1.10      1425.48      3.07     35.63     -119.57        2.07
# std        1.90     12.59      2.47       0.47      1132.46     10.39      2.14        2.00        1.15
# min        0.50      1.00      0.85       0.33         3.00      0.69     32.54     -124.35        0.15
# 25%        2.56     18.00      4.44       1.01       787.00      2.43     33.93     -121.80        1.20
# 50%        3.53     29.00      5.23       1.05      1166.00      2.82     34.26     -118.49        1.80
# 75%        4.74     37.00      6.05       1.10      1725.00      3.28     37.71     -118.01        2.65
# max       15.00     52.00    141.91      34.07     35682.00   1243.33     41.95     -114.31        5.00

The $500,000 Ceiling

This is the most important quirk in the dataset and the one most students miss.

# Check the ceiling
print(df['MedHouseVal'].max())
# 5.0

# Count how many blocks hit the ceiling exactly
capped = df['MedHouseVal'] == 5.0
print(f"Blocks capped at 5.0: {capped.sum()} ({capped.mean() * 100:.1f}%)")
# Blocks capped at 5.0: 965 (4.7%)

# Look at the top of the distribution
print(df['MedHouseVal'].value_counts().sort_index(ascending=False).head(10))
# 5.00    965
# 4.99      1
# 4.98      3
# 4.97      1
# 4.96      2
# ...

Warning

The jump from ~3 blocks at 4.98 to 965 blocks at exactly 5.0 is not natural. It is the data collection ceiling. Any block group whose true median house value exceeded $500,000 in 1990 was capped and recorded as 5.0.

This means your model is learning from corrupted labels for the most expensive neighborhoods. It will systematically underpredict values for high-income coastal blocks where the true value was above the cap.

What Can You Do About It?

Three practical options:

1. Nothing — acknowledge it in your evaluation and report metrics separately for capped and uncapped blocks. This is the most honest approach when you cannot get better data.
2. Exclude capped rows — remove all 965 rows where MedHouseVal == 5.0 before training. Your model will not learn to predict the highest tier, but its predictions for non-capped blocks will be cleaner.
3. Treat it as censored regression — use survival analysis techniques (Tobit model). This is statistically correct but beyond the scope of this project.

# Option 2: train on uncapped data only
df_uncapped = df[df['MedHouseVal'] < 5.0].copy()
print(df_uncapped.shape)
# (19675, 9)

Distributions at a Glance

import matplotlib.pyplot as plt

fig, axes = plt.subplots(3, 3, figsize=(14, 10))
axes = axes.flatten()

for i, col in enumerate(df.columns):
    axes[i].hist(df[col], bins=50, edgecolor='none', color='steelblue', alpha=0.8)
    axes[i].set_title(col, fontsize=10)
    axes[i].set_xlabel('')

plt.suptitle('California Housing — Feature Distributions', fontsize=13, y=1.01)
plt.tight_layout()
plt.show()

Key observations from the distributions:

• MedInc — right-skewed with a long tail toward high earners
• HouseAge — roughly uniform with a spike at 52 (another ceiling)
• AveRooms and AveBedrms — mostly tight but with extreme outliers above 20
• Population — strongly right-skewed; most blocks have under 2,000 people
• AveOccup — mostly between 2 and 4; extreme outliers above 20 (group quarters, dorms)
• Latitude / Longitude — bimodal; reflects the two population centers (LA and SF Bay Area)
• MedHouseVal — right-skewed with a clear spike at the 5.0 ceiling

Geographic Context

import matplotlib.pyplot as plt

plt.figure(figsize=(8, 10))
scatter = plt.scatter(
    df['Longitude'],
    df['Latitude'],
    c=df['MedHouseVal'],
    cmap='viridis',
    alpha=0.3,
    s=1
)
plt.colorbar(scatter, label='Median House Value ($100k)')
plt.xlabel('Longitude')
plt.ylabel('Latitude')
plt.title('California Block Groups — House Value by Location')
plt.tight_layout()
plt.show()

Info

The two dense clusters correspond to the Los Angeles metro area (southern, around 34°N) and the San Francisco Bay Area (northern, around 37–38°N). Coastal blocks are generally more expensive. Inland valleys and the Central Valley are cheaper. This geographic signal is strong enough that latitude and longitude alone are useful predictors.

---

← Back: Project Overview | Next: Exploratory Data Analysis →