Dataset Guide

Every real-world recommender relies on the same core data structure: a table of (user, item, rating) triples. Before you can build a recommendation engine, you need to understand the shape of that data — how sparse it is, how ratings are distributed, and what the cold start problem means in practice.

This guide walks you through generating a synthetic ratings dataset and understanding it at a structural level.

---

Learning Objectives

• Generate a realistic synthetic ratings dataset using latent factors
• Understand the two DataFrames: movie metadata and rating triples
• Measure and visualise sparsity
• Pivot long-format ratings into a user-item matrix
• Articulate the cold start problem

---

Generating the Dataset

The dataset is built from a latent-factor model. Each user has hidden taste preferences (their "factors"), each movie has hidden characteristics, and the rating a user gives a movie is determined by how well those factors align. Noise and random masking are added to make it realistic.

import pandas as pd
import numpy as np

rng = np.random.default_rng(42)

n_users = 200
n_movies = 50

movies = {
    "movie_id": range(1, n_movies + 1),
    "title": [
        "The Matrix", "Inception", "Interstellar", "The Dark Knight", "Pulp Fiction",
        "Forrest Gump", "The Shawshank Redemption", "Schindler's List", "Goodfellas", "Fight Club",
        "The Silence of the Lambs", "Se7en", "The Usual Suspects", "Memento", "American Beauty",
        "Gladiator", "Braveheart", "Saving Private Ryan", "Apocalypse Now", "Full Metal Jacket",
        "Toy Story", "Finding Nemo", "The Lion King", "WALL-E", "Up",
        "Home Alone", "The Princess Bride", "Groundhog Day", "Ferris Bueller's Day Off", "Clueless",
        "The Terminator", "Alien", "Predator", "RoboCop", "Total Recall",
        "Jurassic Park", "Indiana Jones", "Star Wars", "The Lord of the Rings", "Harry Potter",
        "Titanic", "The Notebook", "Pretty Woman", "Sleepless in Seattle", "When Harry Met Sally",
        "Parasite", "Crouching Tiger", "Cinema Paradiso", "Amélie", "Pan's Labyrinth"
    ],
    "genre": [
        "Sci-Fi", "Sci-Fi", "Sci-Fi", "Action", "Crime",
        "Drama", "Drama", "Drama", "Crime", "Drama",
        "Thriller", "Thriller", "Thriller", "Thriller", "Drama",
        "Action", "Action", "War", "War", "War",
        "Animation", "Animation", "Animation", "Animation", "Animation",
        "Comedy", "Comedy", "Comedy", "Comedy", "Comedy",
        "Action", "Sci-Fi", "Action", "Action", "Sci-Fi",
        "Adventure", "Adventure", "Sci-Fi", "Fantasy", "Fantasy",
        "Romance", "Romance", "Romance", "Romance", "Romance",
        "Foreign", "Foreign", "Foreign", "Foreign", "Foreign"
    ]
}
movies_df = pd.DataFrame(movies)

# 5 hidden taste dimensions per user and per movie
user_factors = rng.normal(0, 1, (n_users, 5))
movie_factors = rng.normal(0, 1, (n_movies, 5))

# Dot product gives a raw affinity score, then scale to 1-5
base_ratings = user_factors @ movie_factors.T  # shape: (200, 50)
base_ratings = (
    (base_ratings - base_ratings.min())
    / (base_ratings.max() - base_ratings.min())
    * 4 + 1
)

# Keep only ~15% of ratings (sparse, like real datasets)
mask = rng.random((n_users, n_movies)) < 0.15
ratings_matrix = np.where(
    mask,
    np.clip(base_ratings + rng.normal(0, 0.3, base_ratings.shape), 1, 5).round(1),
    0
)

# Reshape to long format: one row per (user, movie, rating)
rows = []
for user_id in range(1, n_users + 1):
    for movie_id in range(1, n_movies + 1):
        if ratings_matrix[user_id - 1, movie_id - 1] > 0:
            rows.append({
                "user_id": user_id,
                "movie_id": movie_id,
                "rating": ratings_matrix[user_id - 1, movie_id - 1],
            })

ratings_df = pd.DataFrame(rows)

print(f"movies_df shape: {movies_df.shape}")
print(f"ratings_df shape: {ratings_df.shape}")
print(ratings_df.head())

movies_df shape: (50, 3)
ratings_df shape: (1518, 3)   # approximately — depends on random seed
   user_id  movie_id  rating
0        1         3     3.2
1        1         7     4.1
2        1        12     2.8
3        2         1     4.7
4        2        19     3.3

---

Understanding the Two DataFrames

movies_df — Movie Metadata

Each row describes one movie.

print(movies_df.head(10))
print(movies_df['genre'].value_counts())

   movie_id                      title   genre
0         1                 The Matrix  Sci-Fi
1         2                  Inception  Sci-Fi
2         3               Interstellar  Sci-Fi
3         4            The Dark Knight  Action
4         5                Pulp Fiction   Crime
...

genre
Animation    5
Sci-Fi       8
Drama        5
Action       6
Comedy       5
Thriller     4
War          3
Adventure    2
Fantasy      2
Romance      5
Foreign      5

ratings_df — Rating Triples

Each row is one rating event: a specific user gave a specific movie a specific score.

print(f"Total ratings: {len(ratings_df)}")
print(f"Unique users who rated something: {ratings_df['user_id'].nunique()}")
print(f"Unique movies that received ratings: {ratings_df['movie_id'].nunique()}")
print(f"\nRating statistics:")
print(ratings_df['rating'].describe())

Total ratings: 1518
Unique users who rated something: 200
Unique movies that received ratings: 50

Rating statistics:
count    1518.000000
mean        2.981234
std         1.147521
min         1.000000
25%         2.000000
50%         3.000000
75%         4.000000
max         5.000000

---

Measuring Sparsity

Sparsity is the fraction of possible ratings that are missing. A 200-user × 50-movie matrix could have 10,000 ratings. It has roughly 1,500 — meaning 85% of cells are empty.

total_possible = n_users * n_movies
total_observed = len(ratings_df)
sparsity = 1 - (total_observed / total_possible)

print(f"Total possible ratings: {total_possible}")
print(f"Observed ratings:       {total_observed}")
print(f"Sparsity:               {sparsity:.1%}")

Total possible ratings: 10000
Observed ratings:       1518
Sparsity:               84.8%

Warning

85% sparsity sounds extreme, but it is typical. Netflix's dataset (circa 2006) was over 98% sparse. Sparsity causes serious problems for similarity-based methods — two users may share very few rated movies in common, making their similarity score unreliable.

---

Rating Distribution

import matplotlib.pyplot as plt

ratings_df['rating'].hist(bins=20, edgecolor='black', figsize=(8, 4))
plt.title("Distribution of Ratings")
plt.xlabel("Rating (1–5)")
plt.ylabel("Count")
plt.tight_layout()
plt.show()

print(ratings_df['rating'].value_counts(bins=5).sort_index())

Ratings cluster around 2–4, with a slight right skew. Extreme ratings (1.0, 5.0) are less common because the latent-factor model produces smooth affinity scores before adding noise.

---

Movies Per User

How many movies does a typical user rate?

ratings_per_user = ratings_df.groupby('user_id')['rating'].count()

print(ratings_per_user.describe())

ratings_per_user.hist(bins=20, edgecolor='black', figsize=(8, 4))
plt.title("Ratings Per User")
plt.xlabel("Number of movies rated")
plt.ylabel("Number of users")
plt.tight_layout()
plt.show()

count    200.000000
mean       7.590000
std        2.845000
min        1.000000
25%        5.000000
50%        7.000000
75%       10.000000
max       17.000000

Most users have rated between 5 and 10 movies. A small number of users are power raters. This distribution matters: similarity calculations are more reliable for active users.

---

Ratings Per Movie

ratings_per_movie = ratings_df.groupby('movie_id')['rating'].count()
ratings_per_movie = ratings_per_movie.reset_index()
ratings_per_movie.columns = ['movie_id', 'rating_count']
ratings_per_movie = ratings_per_movie.merge(movies_df[['movie_id', 'title']], on='movie_id')
ratings_per_movie = ratings_per_movie.sort_values('rating_count', ascending=False)

print("Most-rated movies:")
print(ratings_per_movie.head(10)[['title', 'rating_count']])

ratings_per_movie['rating_count'].hist(bins=20, edgecolor='black', figsize=(8, 4))
plt.title("Ratings Per Movie (Long-Tail Distribution)")
plt.xlabel("Number of ratings received")
plt.ylabel("Number of movies")
plt.tight_layout()
plt.show()

Info

The long-tail distribution: a few popular movies get many ratings, most movies get very few. This is the pattern in every real ratings dataset. Item-based collaborative filtering is sensitive to this — popular items dominate recommendations unless you correct for it.

---

Building the User-Item Matrix

Convert the long-format ratings to a wide matrix: rows are users, columns are movies, values are ratings (NaN where unrated).

ratings_matrix_df = ratings_df.pivot_table(
    index='user_id',
    columns='movie_id',
    values='rating'
)

print(f"Matrix shape: {ratings_matrix_df.shape}")
print(f"Non-NaN cells: {ratings_matrix_df.notna().sum().sum()}")
print(ratings_matrix_df.iloc[:5, :8])

Matrix shape: (200, 50)
Non-NaN cells: 1518

movie_id    1    2    3    4    5    6    7    8
user_id
1          NaN  NaN  3.2  NaN  NaN  NaN  4.1  NaN
2          4.7  NaN  NaN  NaN  NaN  NaN  NaN  NaN
3          NaN  2.1  NaN  3.8  NaN  NaN  NaN  NaN
...

This matrix is what collaborative filtering operates on. The goal: fill in the NaN cells intelligently.

---

The Cold Start Problem

Warning

Cold start is the fundamental challenge every recommender faces.

• New user cold start: a brand-new user has rated nothing. There is no signal to base personalised recommendations on. Fallback: show the popularity baseline.
• New item cold start: a brand-new movie has received no ratings. Collaborative filtering cannot recommend it at all. Fallback: use content-based filtering on the movie's genre/metadata.

In this project, content-based filtering partially solves the new-item problem because it uses genre features, not ratings history.

---

Checkpoint

Before moving to EDA, confirm you have:

• [ ] Both DataFrames created and their shapes match
• [ ] Sparsity computed and understood
• [ ] ratings_matrix_df built with pivot_table

---

README | Next: EDA →