Feature Engineering¶
The two recommendation approaches in this project need completely different feature representations. Content-based filtering needs movie features — something that describes what a movie is. Collaborative filtering needs the ratings matrix itself — something that captures how users behave. This file builds both, then adds the mean-centering transformation that makes user similarity more accurate.
Learning Objectives¶
- Encode movie genres as one-hot vectors and compute a movie-movie cosine similarity matrix
- Build a user taste profile as a weighted average of genre vectors
- Construct the user-item pivot table for collaborative filtering
- Mean-center ratings to correct for individual rating biases
- Compute user-user and item-item similarity matrices
- Hold out one rating per user for evaluation
Setup¶
import pandas as pd
import numpy as np
from sklearn.metrics.pairwise import cosine_similarity
rng = np.random.default_rng(42)
n_users, n_movies = 200, 50
movies = {
"movie_id": range(1, n_movies + 1),
"title": [
"The Matrix", "Inception", "Interstellar", "The Dark Knight", "Pulp Fiction",
"Forrest Gump", "The Shawshank Redemption", "Schindler's List", "Goodfellas", "Fight Club",
"The Silence of the Lambs", "Se7en", "The Usual Suspects", "Memento", "American Beauty",
"Gladiator", "Braveheart", "Saving Private Ryan", "Apocalypse Now", "Full Metal Jacket",
"Toy Story", "Finding Nemo", "The Lion King", "WALL-E", "Up",
"Home Alone", "The Princess Bride", "Groundhog Day", "Ferris Bueller's Day Off", "Clueless",
"The Terminator", "Alien", "Predator", "RoboCop", "Total Recall",
"Jurassic Park", "Indiana Jones", "Star Wars", "The Lord of the Rings", "Harry Potter",
"Titanic", "The Notebook", "Pretty Woman", "Sleepless in Seattle", "When Harry Met Sally",
"Parasite", "Crouching Tiger", "Cinema Paradiso", "Amélie", "Pan's Labyrinth"
],
"genre": [
"Sci-Fi", "Sci-Fi", "Sci-Fi", "Action", "Crime",
"Drama", "Drama", "Drama", "Crime", "Drama",
"Thriller", "Thriller", "Thriller", "Thriller", "Drama",
"Action", "Action", "War", "War", "War",
"Animation", "Animation", "Animation", "Animation", "Animation",
"Comedy", "Comedy", "Comedy", "Comedy", "Comedy",
"Action", "Sci-Fi", "Action", "Action", "Sci-Fi",
"Adventure", "Adventure", "Sci-Fi", "Fantasy", "Fantasy",
"Romance", "Romance", "Romance", "Romance", "Romance",
"Foreign", "Foreign", "Foreign", "Foreign", "Foreign"
]
}
movies_df = pd.DataFrame(movies)
user_factors = rng.normal(0, 1, (n_users, 5))
movie_factors = rng.normal(0, 1, (n_movies, 5))
base_ratings = user_factors @ movie_factors.T
base_ratings = (
(base_ratings - base_ratings.min())
/ (base_ratings.max() - base_ratings.min())
* 4 + 1
)
mask = rng.random((n_users, n_movies)) < 0.15
ratings_matrix_raw = np.where(
mask,
np.clip(base_ratings + rng.normal(0, 0.3, base_ratings.shape), 1, 5).round(1),
0
)
rows = []
for user_id in range(1, n_users + 1):
for movie_id in range(1, n_movies + 1):
if ratings_matrix_raw[user_id - 1, movie_id - 1] > 0:
rows.append({
"user_id": user_id,
"movie_id": movie_id,
"rating": ratings_matrix_raw[user_id - 1, movie_id - 1],
})
ratings_df = pd.DataFrame(rows)
Content-Based Features — Genre One-Hot Encoding¶
Each movie belongs to exactly one genre. One-hot encoding converts the genre label into a binary vector with 1 in the column for that genre and 0 everywhere else.
# One-hot encode genres — produces a 50 x 12 DataFrame
genre_dummies = pd.get_dummies(movies_df['genre'])
movie_features = pd.concat([movies_df[['movie_id', 'title']], genre_dummies], axis=1)
print(f"Shape: {movie_features.shape}") # (50, 14) — movie_id, title, + 12 genre columns
print(movie_features.head(6))
movie_id title Action Adventure Animation Comedy Crime Drama Fantasy Foreign Romance Sci-Fi Thriller War
0 1 The Matrix 0 0 0 0 0 0 0 0 0 1 0 0
1 2 Inception 0 0 0 0 0 0 0 0 0 1 0 0
2 3 Interstellar 0 0 0 0 0 0 0 0 0 1 0 0
3 4 The Dark Knight 1 0 0 0 0 0 0 0 0 0 0 0
4 5 Pulp Fiction 0 0 0 0 1 0 0 0 0 0 0 0
5 6 Forrest Gump 0 0 0 0 0 1 0 0 0 0 0 0
# Extract just the numeric feature matrix (50 x 12)
genre_cols = genre_dummies.columns.tolist()
movie_feature_matrix = movie_features[genre_cols].values # shape: (50, 12)
print(f"Movie feature matrix shape: {movie_feature_matrix.shape}")
Movie-Movie Cosine Similarity¶
Cosine similarity between two genre vectors measures how alike two movies are based on genre. Two Sci-Fi movies will have similarity 1.0; a Sci-Fi and a Romance will have similarity 0.0 (no shared genre dimensions).
movie_similarity = cosine_similarity(movie_feature_matrix) # shape: (50, 50)
# Wrap in a DataFrame for readable indexing
movie_sim_df = pd.DataFrame(
movie_similarity,
index=movies_df['title'],
columns=movies_df['title']
)
print(f"Movie similarity matrix shape: {movie_sim_df.shape}")
# Find top-5 most similar movies to "The Matrix"
similar_to_matrix = movie_sim_df['The Matrix'].sort_values(ascending=False)
print("\nMovies most similar to 'The Matrix':")
print(similar_to_matrix.head(8))
Movies most similar to 'The Matrix':
The Matrix 1.000000
Inception 1.000000
Interstellar 1.000000
Alien 1.000000
Total Recall 1.000000
Star Wars 1.000000
The Matrix 1.000000
The Dark Knight 0.000000
...
Info
All Sci-Fi movies are equally similar to each other because each has identical genre encodings. In a richer dataset you would add director, cast, plot keywords, or TF-IDF-weighted tags to get finer-grained similarity. Genre alone is a useful starting point.
User Profile Vector — Taste as a Weighted Genre Average¶
For each user, compute a 12-dimensional vector that represents their genre preferences. Movies they rated highly contribute more than movies they rated poorly.
def build_user_profile(user_id, ratings_df, movie_features, genre_cols):
"""
Weighted average of genre vectors for movies the user has rated.
Weight = the user's rating for that movie.
Returns a 1D array of length 12 (one value per genre).
"""
user_ratings = ratings_df[ratings_df['user_id'] == user_id]
if user_ratings.empty:
return np.zeros(len(genre_cols))
# Merge ratings with genre features
user_data = user_ratings.merge(
movie_features[['movie_id'] + genre_cols],
on='movie_id'
)
# Weighted sum: rating × genre_vector, then normalise by total weight
weights = user_data['rating'].values
genre_vectors = user_data[genre_cols].values # shape: (n_rated, 12)
profile = np.average(genre_vectors, axis=0, weights=weights)
return profile
# Build profiles for all users
user_profiles = {}
for uid in ratings_df['user_id'].unique():
user_profiles[uid] = build_user_profile(uid, ratings_df, movie_features, genre_cols)
# Convert to DataFrame: 200 rows × 12 genre columns
user_profiles_df = pd.DataFrame(user_profiles, index=genre_cols).T
user_profiles_df.index.name = 'user_id'
print(f"User profiles shape: {user_profiles_df.shape}")
print("\nProfile for user 1 (which genres do they prefer?):")
print(user_profiles_df.loc[1].sort_values(ascending=False).round(3))
The profile for user 1 shows which genres accumulated the most weighted rating mass. A high value in "Sci-Fi" means that user gave Sci-Fi movies good ratings on average. Content-based recommendations for this user will prioritise unrated Sci-Fi movies.
Collaborative Filtering Features — The Pivot Table¶
Collaborative filtering does not use movie metadata. The feature matrix IS the ratings pivot table: rows are users, columns are movies, values are ratings (0 where unrated).
# Pivot long-format ratings to wide matrix — NaN where unrated
ratings_matrix = ratings_df.pivot(
index='user_id',
columns='movie_id',
values='rating'
)
print(f"Ratings matrix shape: {ratings_matrix.shape}") # (200, 50)
print(f"Non-NaN cells: {ratings_matrix.notna().sum().sum()}")
# For similarity computations, fill NaN with 0
ratings_matrix_filled = ratings_matrix.fillna(0)
print(ratings_matrix_filled.iloc[:4, :6])
movie_id 1 2 3 4 5 6
user_id
1 0.0 0.0 3.2 0.0 0.0 0.0
2 4.7 0.0 0.0 0.0 0.0 0.0
3 0.0 2.1 0.0 3.8 0.0 0.0
4 0.0 0.0 0.0 0.0 2.6 0.0
Mean-Centering — Fixing the Harsh Rater Problem¶
Some users give everything 2–3 stars; others give everything 4–5 stars. A harsh 3 is actually a positive signal; an easy rater's 3 is a neutral one. Mean-centering subtracts each user's average rating so ratings become relative to that user's baseline.
# Each user's mean rating (computed only over rated movies)
user_means = ratings_matrix.mean(axis=1) # shape: (200,)
# Subtract user mean from each of their ratings
ratings_centered = ratings_matrix.sub(user_means, axis=0)
print("=== BEFORE mean-centering (user 1) ===")
user1_rated = ratings_matrix.loc[1].dropna()
print(user1_rated.values)
print(f"Mean: {user1_rated.mean():.2f}")
print("\n=== AFTER mean-centering (user 1) ===")
user1_centered = ratings_centered.loc[1].dropna()
print(user1_centered.values.round(2))
print(f"Mean: {user1_centered.mean():.4f}") # should be ~0
=== BEFORE mean-centering (user 1) ===
[3.2 4.1 2.8 3.7 1.9]
Mean: 3.14
=== AFTER mean-centering (user 1) ===
[ 0.06 0.96 -0.34 0.56 -1.24]
Mean: 0.0000
Warning
Fill the NaN (unrated) cells with 0 after mean-centering, not before. If you fill with 0 before subtracting the mean, those 0s pull the user's mean down, corrupting the centering. The correct order: center → then fill NaN with 0 for similarity computation.
# Correct order: center first, then fill NaN with 0 for matrix operations
ratings_centered_filled = ratings_centered.fillna(0)
print(f"Centered matrix shape: {ratings_centered_filled.shape}")
User-User Similarity Matrix¶
With mean-centered ratings, compute cosine similarity between every pair of users. The result is a 200×200 symmetric matrix where entry (i, j) is how similar user i and user j are in taste.
user_similarity = cosine_similarity(ratings_centered_filled) # shape: (200, 200)
user_sim_df = pd.DataFrame(
user_similarity,
index=ratings_matrix.index,
columns=ratings_matrix.index
)
print(f"User similarity matrix shape: {user_sim_df.shape}")
# Find the 5 users most similar to user 1
similar_to_user1 = user_sim_df[1].sort_values(ascending=False)
print("\nUsers most similar to user 1:")
print(similar_to_user1.iloc[1:6]) # skip the first entry (similarity with self = 1.0)
Tip
A user-user similarity score of 0.0 means no shared rated movies (or purely orthogonal preferences). Negative values mean the two users reliably disagree. For recommendations, only use users with positive similarity scores.
Item-Item Similarity Matrix¶
Transpose the ratings matrix and compute cosine similarity between movies. Two movies are similar if users who rated one tend to give similar scores to the other — regardless of genre.
# Transpose: now rows are movies, columns are users
item_similarity = cosine_similarity(ratings_centered_filled.T) # shape: (50, 50)
item_sim_df = pd.DataFrame(
item_similarity,
index=ratings_matrix.columns,
columns=ratings_matrix.columns
)
print(f"Item similarity matrix shape: {item_sim_df.shape}")
# Most similar movies to movie_id 1 (The Matrix)
similar_to_matrix_cf = item_sim_df[1].sort_values(ascending=False)
top_similar = similar_to_matrix_cf.iloc[1:6].reset_index()
top_similar = top_similar.merge(movies_df[['movie_id', 'title']], on='movie_id')
print("\nMovies most similar to 'The Matrix' (collaborative signal):")
print(top_similar[['title', 1]].rename(columns={1: 'similarity'}).to_string(index=False))
Info
Item-item similarity from collaborative filtering captures something content-based cannot: movies that users treat similarly regardless of genre. A War film and a Crime film might be rated similarly by the same audience — CF would detect this; content-based would not.
Train-Test Split for RecSys¶
Standard random splits do not work well for recommender evaluation. If you randomly hold out 15% of ratings, you might leave a user with no training data at all, or hold out a rating for a movie the user barely cares about.
The standard approach for offline evaluation: hold out the most recent rating per user (or, when timestamps are unavailable, a random single rating per user) as the test item.
# Hold out the last rating per user (by row order — no timestamps here)
# The held-out item is what we will try to recommend in evaluation.
train_rows = []
test_rows = []
for user_id, group in ratings_df.groupby('user_id'):
if len(group) < 2:
# Can't evaluate a user with only 1 rating — keep all in train
train_rows.append(group)
continue
# Last row per user goes to test
train_rows.append(group.iloc[:-1])
test_rows.append(group.iloc[[-1]])
train_df = pd.concat(train_rows).reset_index(drop=True)
test_df = pd.concat(test_rows).reset_index(drop=True)
print(f"Training ratings: {len(train_df)}")
print(f"Test ratings: {len(test_df)}")
print(f"\nSample test set (first 5 rows):")
print(test_df.head())
Training ratings: 1321
Test ratings: 197
Sample test set (first 5 rows):
user_id movie_id rating
0 1 12 2.8
1 2 19 3.3
2 3 4 3.8
3 4 28 1.4
4 5 33 4.2
Warning
This split means you train collaborative filtering models on train_df only, not the full ratings_df. Using the full dataset to train and then evaluating on a subset of it leads to data leakage — the model has already seen the test ratings.