Exploratory Data Analysis

Before building any model, you need to understand your data. For recommender systems, EDA reveals how ratings are distributed, which movies dominate the data, how active users are, and how sparse the signal is. The patterns you find here will directly shape your modelling choices.

---

Learning Objectives

• Identify rating distribution skew and what it means for model training
• Distinguish most-rated movies from highest-rated movies and understand popularity bias
• Quantify and visualise matrix sparsity
• Detect genre preference correlations across users

---

Setup — Regenerate the Dataset

Run this block once at the top of your notebook. All subsequent sections assume movies_df and ratings_df are in scope.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

rng = np.random.default_rng(42)

n_users = 200
n_movies = 50

movies = {
    "movie_id": range(1, n_movies + 1),
    "title": [
        "The Matrix", "Inception", "Interstellar", "The Dark Knight", "Pulp Fiction",
        "Forrest Gump", "The Shawshank Redemption", "Schindler's List", "Goodfellas", "Fight Club",
        "The Silence of the Lambs", "Se7en", "The Usual Suspects", "Memento", "American Beauty",
        "Gladiator", "Braveheart", "Saving Private Ryan", "Apocalypse Now", "Full Metal Jacket",
        "Toy Story", "Finding Nemo", "The Lion King", "WALL-E", "Up",
        "Home Alone", "The Princess Bride", "Groundhog Day", "Ferris Bueller's Day Off", "Clueless",
        "The Terminator", "Alien", "Predator", "RoboCop", "Total Recall",
        "Jurassic Park", "Indiana Jones", "Star Wars", "The Lord of the Rings", "Harry Potter",
        "Titanic", "The Notebook", "Pretty Woman", "Sleepless in Seattle", "When Harry Met Sally",
        "Parasite", "Crouching Tiger", "Cinema Paradiso", "Amélie", "Pan's Labyrinth"
    ],
    "genre": [
        "Sci-Fi", "Sci-Fi", "Sci-Fi", "Action", "Crime",
        "Drama", "Drama", "Drama", "Crime", "Drama",
        "Thriller", "Thriller", "Thriller", "Thriller", "Drama",
        "Action", "Action", "War", "War", "War",
        "Animation", "Animation", "Animation", "Animation", "Animation",
        "Comedy", "Comedy", "Comedy", "Comedy", "Comedy",
        "Action", "Sci-Fi", "Action", "Action", "Sci-Fi",
        "Adventure", "Adventure", "Sci-Fi", "Fantasy", "Fantasy",
        "Romance", "Romance", "Romance", "Romance", "Romance",
        "Foreign", "Foreign", "Foreign", "Foreign", "Foreign"
    ]
}
movies_df = pd.DataFrame(movies)

user_factors = rng.normal(0, 1, (n_users, 5))
movie_factors = rng.normal(0, 1, (n_movies, 5))
base_ratings = user_factors @ movie_factors.T
base_ratings = (
    (base_ratings - base_ratings.min())
    / (base_ratings.max() - base_ratings.min())
    * 4 + 1
)

mask = rng.random((n_users, n_movies)) < 0.15
ratings_matrix = np.where(
    mask,
    np.clip(base_ratings + rng.normal(0, 0.3, base_ratings.shape), 1, 5).round(1),
    0
)

rows = []
for user_id in range(1, n_users + 1):
    for movie_id in range(1, n_movies + 1):
        if ratings_matrix[user_id - 1, movie_id - 1] > 0:
            rows.append({
                "user_id": user_id,
                "movie_id": movie_id,
                "rating": ratings_matrix[user_id - 1, movie_id - 1],
            })

ratings_df = pd.DataFrame(rows)

print(f"movies_df: {movies_df.shape}")
print(f"ratings_df: {ratings_df.shape}")

movies_df: (50, 3)
ratings_df: (1518, 3)

---

Rating Distribution

What values do users actually give? Are ratings clustered around a particular score?

fig, axes = plt.subplots(1, 2, figsize=(12, 4))

# Histogram of raw ratings
axes[0].hist(ratings_df['rating'], bins=20, edgecolor='black', color='steelblue')
axes[0].set_title("Distribution of Ratings")
axes[0].set_xlabel("Rating (1–5)")
axes[0].set_ylabel("Count")

# Rounded to nearest integer for a cleaner view
rounded = ratings_df['rating'].round().astype(int)
rounded.value_counts().sort_index().plot(kind='bar', ax=axes[1],
                                          color='steelblue', edgecolor='black')
axes[1].set_title("Rating Counts by Integer Star")
axes[1].set_xlabel("Stars")
axes[1].set_ylabel("Count")
axes[1].tick_params(axis='x', rotation=0)

plt.tight_layout()
plt.show()

print(ratings_df['rating'].describe())

count    1518.000000
mean        2.981234
std         1.147521
min         1.000000
25%         2.000000
50%         3.000000
75%         4.000000
max         5.000000

Info

The distribution is roughly symmetric around 3. Real-world platforms like Amazon show a strong J-curve (lots of 5s and 1s, few 2s and 3s) because people only write reviews when they love or hate something. The latent-factor structure here produces more moderate ratings, which is actually closer to logged implicit feedback.

---

Most-Rated vs Highest-Rated Movies

These two lists are rarely the same. A movie can be hugely popular but mediocre, or a hidden gem with a tiny number of ratings but a near-perfect score.

# Most-rated (popularity)
ratings_per_movie = ratings_df.groupby('movie_id').agg(
    rating_count=('rating', 'count'),
    avg_rating=('rating', 'mean')
).reset_index()
ratings_per_movie = ratings_per_movie.merge(movies_df[['movie_id', 'title', 'genre']], on='movie_id')

most_rated = ratings_per_movie.sort_values('rating_count', ascending=False).head(10)
highest_rated = ratings_per_movie[ratings_per_movie['rating_count'] >= 5].sort_values(
    'avg_rating', ascending=False
).head(10)

print("=== TOP 10 BY RATING COUNT (most popular) ===")
print(most_rated[['title', 'genre', 'rating_count', 'avg_rating']].to_string(index=False))

print("\n=== TOP 10 BY AVERAGE RATING (best rated, min 5 reviews) ===")
print(highest_rated[['title', 'genre', 'rating_count', 'avg_rating']].to_string(index=False))

Warning

Popularity bias: if you recommend the most-rated movies to everyone, you create a feedback loop. Popular movies get more exposure → more ratings → they stay popular. Lesser-known films never get a chance. Every production recommender must explicitly counteract this with diversity constraints or exploration strategies.

---

Genre Distribution

How many movies exist per genre in this dataset?

genre_counts = movies_df['genre'].value_counts()

genre_counts.plot(kind='bar', figsize=(10, 4), color='teal', edgecolor='black')
plt.title("Number of Movies per Genre")
plt.xlabel("Genre")
plt.ylabel("Movie Count")
plt.xticks(rotation=45, ha='right')
plt.tight_layout()
plt.show()

print(genre_counts)

Sci-Fi       8
Action       6
Drama        5
Animation    5
Comedy       5
Romance      5
Foreign      5
Thriller     4
War          3
Adventure    2
Fantasy      2

Sci-Fi and Action are the largest categories. Adventure and Fantasy are small — content-based filtering will have fewer candidates to recommend within those genres.

---

User Activity — How Many Movies Does Each User Rate?

ratings_per_user = ratings_df.groupby('user_id')['rating'].count()

fig, axes = plt.subplots(1, 2, figsize=(12, 4))

axes[0].hist(ratings_per_user, bins=20, edgecolor='black', color='coral')
axes[0].set_title("Ratings Per User")
axes[0].set_xlabel("Number of movies rated")
axes[0].set_ylabel("Number of users")

# Cumulative distribution
sorted_counts = ratings_per_user.sort_values(ascending=False)
axes[1].plot(range(len(sorted_counts)), sorted_counts.values, color='coral')
axes[1].set_title("User Activity (sorted)")
axes[1].set_xlabel("User rank (most active first)")
axes[1].set_ylabel("Ratings given")
axes[1].axhline(ratings_per_user.mean(), color='black', linestyle='--', label=f'Mean: {ratings_per_user.mean():.1f}')
axes[1].legend()

plt.tight_layout()
plt.show()

print(ratings_per_user.describe())
print(f"\nUsers with fewer than 5 ratings: {(ratings_per_user < 5).sum()}")
print(f"Users with more than 12 ratings: {(ratings_per_user > 12).sum()}")

count    200.000000
mean       7.590000
std        2.845000
min        1.000000
25%        5.000000
50%        7.000000
75%       10.000000
max       17.000000

Users with fewer than 5 ratings: 32
Users with more than 12 ratings: 18

Info

The heavy tail of power users is common in real datasets. Active users provide more signal for collaborative filtering, but they also have more influence over the similarity matrix than casual users. This is worth correcting for in production systems.

---

Movie Popularity — How Many Ratings Does Each Movie Receive?

ratings_per_movie_hist = ratings_df.groupby('movie_id')['rating'].count()

ratings_per_movie_hist.hist(bins=15, edgecolor='black', figsize=(8, 4), color='mediumpurple')
plt.title("Ratings Per Movie (Long-Tail Distribution)")
plt.xlabel("Number of ratings received")
plt.ylabel("Number of movies")
plt.tight_layout()
plt.show()

print(f"Movies with fewer than 5 ratings: {(ratings_per_movie_hist < 5).sum()}")
print(f"Movies with more than 15 ratings: {(ratings_per_movie_hist > 15).sum()}")
print(f"\nTop 5 most-rated:")
top5 = ratings_per_movie_hist.sort_values(ascending=False).head(5)
top5_with_titles = top5.reset_index().merge(movies_df[['movie_id', 'title']], on='movie_id')
print(top5_with_titles[['title', 'rating']].to_string(index=False))

The long-tail shape here mirrors real streaming platforms. A handful of blockbusters accumulate most of the ratings; the majority of the catalogue is rarely seen. Item-based collaborative filtering performs poorly on the long-tail.

---

Rating by Genre — Which Genres Get the Best Reviews?

ratings_with_genre = ratings_df.merge(movies_df[['movie_id', 'genre']], on='movie_id')

# Boxplot of rating by genre
genre_order = (
    ratings_with_genre.groupby('genre')['rating']
    .median()
    .sort_values(ascending=False)
    .index.tolist()
)

fig, ax = plt.subplots(figsize=(12, 5))
ratings_with_genre.boxplot(
    column='rating', by='genre',
    order=genre_order, ax=ax,
    boxprops=dict(color='steelblue'),
    medianprops=dict(color='red', linewidth=2)
)
plt.title("Rating Distribution by Genre")
plt.suptitle("")
plt.xlabel("Genre")
plt.ylabel("Rating")
plt.xticks(rotation=45, ha='right')
plt.tight_layout()
plt.show()

print("\nMean rating per genre:")
print(ratings_with_genre.groupby('genre')['rating'].agg(['mean', 'count', 'std']).round(2).sort_values('mean', ascending=False))

Tip

Wide boxes (high variance) in a genre mean users disagree sharply about those movies. Narrow boxes mean users tend to agree. High variance genres are interesting for personalisation — there is real signal to exploit.

---

Sparsity — Visualising the Ratings Matrix

# Compute sparsity
total_possible = n_users * n_movies
total_observed = len(ratings_df)
sparsity = 1 - (total_observed / total_possible)
print(f"Sparsity: {sparsity:.1%}  ({total_observed} ratings out of {total_possible} possible)")

# Build the full matrix (NaN for unrated)
ratings_matrix_df = ratings_df.pivot_table(
    index='user_id', columns='movie_id', values='rating'
)

# Heatmap — presence/absence only (1 if rated, 0 if not)
presence = ratings_matrix_df.notna().astype(int)

fig, ax = plt.subplots(figsize=(14, 6))
sns.heatmap(
    presence,
    cmap='Blues',
    cbar=False,
    xticklabels=False,
    yticklabels=False,
    ax=ax
)
ax.set_title(f"Ratings Matrix — Observed Cells (blue = rated, white = missing)\nSparsity: {sparsity:.1%}")
ax.set_xlabel("Movie ID (50 movies)")
ax.set_ylabel("User ID (200 users)")
plt.tight_layout()
plt.show()

Sparsity: 84.8%  (1518 ratings out of 10000 possible)

The heatmap will be mostly white with scattered blue dots. This visual makes the challenge concrete: your model must infer preferences for 85% of the matrix from the 15% that is observed.

---

Genre Correlation — Do Sci-Fi Fans Also Like Action?

Compute each user's average rating per genre, then correlate genres across users to find which taste profiles co-occur.

# Average rating each user gives to each genre
ratings_with_genre = ratings_df.merge(movies_df[['movie_id', 'genre']], on='movie_id')
user_genre_ratings = ratings_with_genre.groupby(['user_id', 'genre'])['rating'].mean().unstack()

print(f"User-genre matrix shape: {user_genre_ratings.shape}")
print(f"Coverage (non-NaN entries): {user_genre_ratings.notna().sum().sum()} / {user_genre_ratings.size}")

# Correlation between genres (across users)
genre_corr = user_genre_ratings.corr()

fig, ax = plt.subplots(figsize=(10, 8))
sns.heatmap(
    genre_corr,
    annot=True, fmt='.2f',
    cmap='RdBu_r', center=0,
    vmin=-1, vmax=1,
    ax=ax
)
ax.set_title("Genre Preference Correlations\n(positive = fans of one tend to like the other)")
plt.tight_layout()
plt.show()

print("\nHighest positive genre correlations:")
corr_pairs = (
    genre_corr.where(np.triu(np.ones(genre_corr.shape), k=1).astype(bool))
    .stack()
    .sort_values(ascending=False)
)
print(corr_pairs.head(5))

Info

Genre correlations are computed only from users who rated movies in both genres. Users who only rated one genre contribute NaN for the other, so coverage is partial. This is fine for EDA but would need careful handling in a feature engineering pipeline.

---

Key Findings

Success

What EDA revealed:

1. Ratings cluster around 2–4 stars. Extreme ratings are rare. This symmetric distribution means mean-centering per user will be meaningful in feature engineering.

2. Most-rated ≠ best-rated. Popularity and quality are different signals. A baseline recommender that uses only rating count will push blockbusters to every user regardless of taste.

3. 85% sparsity — the fundamental challenge. Most user-movie pairs have no observed signal. Collaborative filtering must generalise from the sparse observations.

4. Heavy-tail user activity. A small number of power users dominate the rating count. Low-activity users (< 5 ratings) are the hardest to personalise for — this is the cold start problem in practice.

5. Long-tail movie popularity. A few movies are rated by many users; most movies are rated by very few. Item similarity on rarely-rated movies will be noisy.

6. Genre correlations exist. Sci-Fi and Action tend to be liked by similar users; Romance and War tend to be liked by different users. Content-based filtering can exploit this indirectly through genre features.

---

← Dataset Guide | Next: Feature Engineering →