Evaluation and Interpretation

Building a recommender is the easy part. Knowing whether it works is harder. Evaluation surfaces which approach actually serves users — and reveals the gaps between offline metrics and what would happen in a real product.

---

Learning Objectives

• Explain why offline evaluation for recommenders is different from standard ML evaluation
• Implement Hit Rate, Precision@K, and Recall@K for each approach
• Run a qualitative check on recommendation outputs for specific users
• Compare approaches on a single results table
• Articulate what offline metrics cannot measure

---

Setup

Run the full setup block from model-building.md. This file assumes train_df, test_df, movies_df, the recommendation functions, and all similarity matrices are already in scope.

# Quick verification — confirm all objects exist
assert 'train_df' in dir() or 'train_df' in globals(), "Run the model-building setup first"
print(f"train_df: {len(train_df)} ratings")
print(f"test_df:  {len(test_df)} ratings, {len(test_df)} held-out items")
print(f"Users with held-out items: {test_df['user_id'].nunique()}")

---

The Evaluation Challenge

Standard classification accuracy asks: "was the prediction correct?" For a recommender, there is no single correct answer. A user who rated Inception might also love Interstellar — both are valid recommendations. You never observe the counterfactual: what the user would have done if shown a different recommendation.

Offline evaluation uses the held-out item as a proxy for ground truth. The assumption: if a user went out of their way to rate a movie, they engaged with it enough to count as a "relevant" item. The question becomes: does your recommendation list contain that held-out item?

Warning

Offline metrics systematically underestimate recommender quality. A model might recommend a movie the user would love but has never rated — the metric penalises this as a miss even though it was a good recommendation. This is why online A/B testing is the only definitive evaluation.

---

Setup — Generate Top-10 Recommendations for Each User

Produce recommendation lists from all three approaches for every user in the test set.

def generate_all_recommendations(test_df, top_n=10):
    """
    For each user in test_df, generate top-n recommendations
    from popularity, content-based, and user-CF approaches.
    Returns a dict: user_id → {'popular': [...], 'content': [...], 'user_cf': [...]}
    """
    test_users = test_df['user_id'].unique()
    results = {}

    # Popularity is the same for everyone — compute once
    popular_list = (
        train_df.groupby('movie_id')['rating']
        .count()
        .sort_values(ascending=False)
        .head(top_n)
        .index.tolist()
    )

    for uid in test_users:
        # Content-based
        cb = get_content_recommendations(user_id=uid, n=top_n)
        cb_ids = (
            cb.merge(movies_df[['movie_id', 'title']], on='title')['movie_id'].tolist()
            if not cb.empty else []
        )

        # User-CF
        cf = get_user_cf_recommendations(user_id=uid, n=top_n)
        cf_ids = (
            cf.merge(movies_df[['movie_id', 'title']], on='title')['movie_id'].tolist()
            if not cf.empty else []
        )

        results[uid] = {
            'popular': popular_list,
            'content': cb_ids,
            'user_cf': cf_ids
        }

    return results


# Build recommendation lists (top-10 per user)
all_recs = generate_all_recommendations(test_df, top_n=10)

print(f"Generated recommendations for {len(all_recs)} users")
print(f"\nSample — user 1 recommendations:")
print(f"  Popular:  {all_recs[1]['popular'][:5]}...")
print(f"  Content:  {all_recs[1]['content'][:5]}...")
print(f"  User-CF:  {all_recs[1]['user_cf'][:5]}...")

---

Hit Rate

Hit Rate answers the simplest useful question: did at least one of the top-N recommendations match the held-out item?

def compute_hit_rate(test_df, all_recs, approach, top_k=10):
    """
    For each user, check if the held-out movie_id appears
    in the top-k recommendations for the given approach.
    Hit rate = fraction of users for whom it does.
    """
    hits = 0
    total = 0
    for _, row in test_df.iterrows():
        uid = row['user_id']
        held_out = row['movie_id']
        if uid not in all_recs:
            continue
        rec_list = all_recs[uid][approach][:top_k]
        if held_out in rec_list:
            hits += 1
        total += 1
    return hits / total if total > 0 else 0.0


hr_popular  = compute_hit_rate(test_df, all_recs, 'popular',  top_k=10)
hr_content  = compute_hit_rate(test_df, all_recs, 'content',  top_k=10)
hr_user_cf  = compute_hit_rate(test_df, all_recs, 'user_cf',  top_k=10)

print("=== Hit Rate @ 10 ===")
print(f"  Popularity baseline: {hr_popular:.3f}  ({hr_popular:.1%})")
print(f"  Content-based:       {hr_content:.3f}  ({hr_content:.1%})")
print(f"  User-CF:             {hr_user_cf:.3f}  ({hr_user_cf:.1%})")

Info

Hit Rate is the most lenient metric — it only asks "did you include the right answer anywhere in your list?" A Hit Rate of 0.10 means you caught the held-out item for 10% of users. At this dataset's sparsity level (~15% fill rate), even a hit rate of 5–15% indicates useful signal. Random recommendations would score roughly 10/50 = 20% for a top-10 list, which is why popularity often performs deceptively well.

---

Precision@K and Recall@K

With only one held-out item per user, Precision@K and Recall@K carry a specific interpretation.

• Precision@K = (number of relevant items in top-K list) / K. With one held-out item, this is either 1/K (item was in the list) or 0 (it was not).
• Recall@K = (number of relevant items in top-K list) / (total relevant items). With one relevant item, Recall@K equals Hit Rate@K.

def compute_precision_at_k(test_df, all_recs, approach, k):
    """Fraction of top-k recommendations that are the held-out item (averaged over users)."""
    precisions = []
    for _, row in test_df.iterrows():
        uid = row['user_id']
        held_out = row['movie_id']
        if uid not in all_recs:
            continue
        rec_list = all_recs[uid][approach][:k]
        precision = 1.0 / k if held_out in rec_list else 0.0
        precisions.append(precision)
    return np.mean(precisions) if precisions else 0.0


def compute_recall_at_k(test_df, all_recs, approach, k):
    """With one relevant item per user, Recall@K == Hit Rate@K."""
    return compute_hit_rate(test_df, all_recs, approach, top_k=k)


# Compute for K=5 and K=10
results_table = {}
for approach in ['popular', 'content', 'user_cf']:
    results_table[approach] = {
        'Hit Rate@10': compute_hit_rate(test_df, all_recs, approach, top_k=10),
        'P@5':         compute_precision_at_k(test_df, all_recs, approach, k=5),
        'P@10':        compute_precision_at_k(test_df, all_recs, approach, k=10),
        'R@5':         compute_recall_at_k(test_df, all_recs, approach, k=5),
        'R@10':        compute_recall_at_k(test_df, all_recs, approach, k=10),
    }

results_df = pd.DataFrame(results_table).T
results_df.index = ['Popularity Baseline', 'Content-Based', 'User-CF']
print("\n=== Evaluation Results ===")
print(results_df.round(4).to_string())

=== Evaluation Results ===
                     Hit Rate@10    P@5   P@10    R@5   R@10
Popularity Baseline       0.172  0.023  0.017  0.115  0.172
Content-Based             0.091  0.012  0.009  0.061  0.091
User-CF                   0.132  0.018  0.013  0.091  0.132

Warning

The popularity baseline often outperforms personalised approaches in offline evaluation. This is not because it is a better recommender — it is because it always recommends the most-seen movies, and the held-out item is more likely to be a popular movie (since popular movies get more ratings in the first place). Offline metrics have a popularity bias baked in.

---

Qualitative Evaluation — Do Recommendations Make Sense?

Metrics tell you the score; qualitative inspection tells you why. Pick three users and compare their history to the recommendations from each method.

def show_user_analysis(user_id, all_recs, test_df, train_df, movies_df, top_n=5):
    """Print a user's rating history and recommendations from all three approaches."""
    print(f"\n{'='*60}")
    print(f"USER {user_id}")
    print(f"{'='*60}")

    history = (
        train_df[train_df['user_id'] == user_id]
        .merge(movies_df[['movie_id', 'title', 'genre']], on='movie_id')
        .sort_values('rating', ascending=False)
    )
    print(f"\nRated {len(history)} movies (training set):")
    print(history[['title', 'genre', 'rating']].to_string(index=False))

    held_out = test_df[test_df['user_id'] == user_id]
    if not held_out.empty:
        held_movie = held_out.merge(movies_df[['movie_id', 'title', 'genre']], on='movie_id')
        print(f"\nHeld-out (test) item:")
        print(held_movie[['title', 'genre', 'rating']].to_string(index=False))

    if user_id in all_recs:
        for approach_key, label in [('popular', 'Popularity'), ('content', 'Content-Based'), ('user_cf', 'User-CF')]:
            rec_ids = all_recs[user_id][approach_key][:top_n]
            rec_titles = movies_df[movies_df['movie_id'].isin(rec_ids)][['movie_id', 'title', 'genre']]
            # Preserve ranking order
            rec_titles = rec_titles.set_index('movie_id').loc[rec_ids].reset_index()
            print(f"\n{label} recommendations:")
            print(rec_titles[['title', 'genre']].to_string(index=False))


for uid in [1, 5, 42]:
    show_user_analysis(uid, all_recs, test_df, train_df, movies_df)

Look at the output and ask: given what user 1 has rated, do the user-CF recommendations make intuitive sense? Does content-based recommend movies in genres the user has shown no interest in? Does the popularity baseline ignore obvious preferences?

---

Comparison Table

import matplotlib.pyplot as plt

metrics = ['Hit Rate@10', 'P@5', 'P@10']
approaches = ['Popularity Baseline', 'Content-Based', 'User-CF']
colors = ['#2196F3', '#4CAF50', '#FF9800']

fig, axes = plt.subplots(1, 3, figsize=(13, 4))
for i, metric in enumerate(metrics):
    values = results_df[metric].values
    bars = axes[i].bar(approaches, values, color=colors, edgecolor='black', width=0.5)
    axes[i].set_title(metric)
    axes[i].set_ylim(0, max(values) * 1.3)
    axes[i].set_xticklabels(approaches, rotation=20, ha='right')
    for bar, val in zip(bars, values):
        axes[i].text(bar.get_x() + bar.get_width() / 2, bar.get_height() + 0.001,
                     f'{val:.3f}', ha='center', va='bottom', fontsize=9)

plt.suptitle("Recommender Evaluation — Offline Metrics")
plt.tight_layout()
plt.show()

print("\nFull results table:")
print(results_df.round(4).to_string())

---

Popularity Bias

The popularity baseline recommends the same small set of blockbusters to everyone. What fraction of recommendation slots across all users go to the top-5 most popular movies?

# Count how often each movie appears in user-CF recommendations across all users
cf_rec_counts = {}
for uid, recs in all_recs.items():
    for movie_id in recs['user_cf'][:10]:
        cf_rec_counts[movie_id] = cf_rec_counts.get(movie_id, 0) + 1

cf_rec_series = (
    pd.Series(cf_rec_counts)
    .sort_values(ascending=False)
    .reset_index()
)
cf_rec_series.columns = ['movie_id', 'appearance_count']
cf_rec_series = cf_rec_series.merge(movies_df[['movie_id', 'title', 'genre']], on='movie_id')

print("Top 10 most recommended movies (User-CF, across all users):")
print(cf_rec_series.head(10)[['title', 'genre', 'appearance_count']].to_string(index=False))

# What fraction of total recommendation slots do the top-5 occupy?
total_slots = sum(cf_rec_counts.values())
top5_slots = cf_rec_series.head(5)['appearance_count'].sum()
print(f"\nTop-5 movies account for {top5_slots / total_slots:.1%} of all recommendation slots")

Warning

If 5 movies (10% of the catalogue) appear in 40%+ of recommendation slots, you have a popularity bias problem. Users who like niche genres see the same blockbusters everyone else sees. In production, recommenders use diversity constraints (maximum appearances per item) and re-ranking to ensure catalogue coverage.

---

What Offline Metrics Cannot Capture

What matters in production	Why offline metrics miss it
Position in the list	We treat all top-10 recommendations equally; in reality, position 1 gets 10× more clicks than position 10
Novelty	A recommendation of The Dark Knight to someone who obviously knows it exists is low-value, even if it is "relevant"
Serendipity	A surprising recommendation that delights the user is invisible in offline data
Click-through rate	Whether users actually clicked on a recommendation is not in the ratings data
Session context	What mood is the user in right now? Offline evaluation ignores temporal context
Business constraints	Studios pay for promotion slots; new releases need visibility — offline metrics ignore all of this

Success

The only way to truly evaluate a recommender is to deploy it and measure real user behaviour through A/B testing: randomly assign users to treatment (new recommender) vs control (current system), then measure engagement, retention, and conversion over days or weeks.

---

What Would You Do Next

Given the results from this offline evaluation, here are the highest-leverage next steps:

Better matrix factorisation — NMF Non-negative Matrix Factorisation (NMF) enforces non-negativity on latent factors, which makes them more interpretable ("factor 3 corresponds to action-thriller fans"). Replace TruncatedSVD with sklearn.decomposition.NMF.

from sklearn.decomposition import NMF

nmf = NMF(n_components=15, random_state=42, max_iter=300)
user_latent_nmf = nmf.fit_transform(ratings_matrix.fillna(0).values)
movie_latent_nmf = nmf.components_.T
ratings_reconstructed_nmf = user_latent_nmf @ movie_latent_nmf.T
print(f"NMF reconstruction error: {nmf.reconstruction_err_:.2f}")

Implicit feedback Real systems often do not have explicit ratings — they have clicks, plays, purchases, and dwell time. The implicit Python library implements ALS (Alternating Least Squares) for implicit feedback datasets.

Two-tower neural model Encode users and items as dense embeddings learned by a neural network. The user tower and item tower each produce a vector; similarity is computed by dot product. This scales to billions of items with approximate nearest-neighbour search (FAISS, ScaNN).

Online A/B test Deploy both content-based and user-CF as treatment arms. Measure 7-day retention and average session length. Shut down the arm that underperforms after reaching statistical significance (typically 2–4 weeks).

---

← Model Building | Next: Interview Questions →