Evaluation and Interpretation¶
Training a model is straightforward. Understanding whether it is good enough — and why it fails where it fails — is the actual work. This file covers the full evaluation workflow: per-class metrics, confusion matrix analysis, model explainability, error patterns, and business interpretation.
Setup¶
Run the full pipeline from model-building: generate data, clean, split, fit the best model.
import pandas as pd
import numpy as np
import re
import random
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.preprocessing import FunctionTransformer
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (classification_report, confusion_matrix,
f1_score, accuracy_score)
random.seed(42)
np.random.seed(42)
brands = ["TechCo", "ShopEasy", "FreshMart", "StyleHub", "HomeGoods", "QuickServe"]
products = ["laptop", "headphones", "sneakers", "coffee maker", "smartphone",
"backpack", "blender", "jacket"]
positive_templates = [
"Just got my {product} from {brand} and it's absolutely amazing!",
"Huge shoutout to {brand} — the {product} exceeded every expectation.",
"{brand} delivered my {product} ahead of schedule. So impressed!",
"The {product} from {brand} is worth every penny. Five stars.",
"I've been recommending {brand}'s {product} to everyone I know.",
"Finally tried {brand} and their {product} blew me away. #LoveIt",
"{brand} customer service sorted my {product} issue in under 10 minutes. Legend.",
"Unboxing my new {product} from {brand} right now — quality is outstanding.",
"Couldn't be happier with my {product} purchase from {brand}. Will buy again.",
"{brand} nailed it with this {product}. Best purchase I've made this year.",
]
negative_templates = [
"My {product} from {brand} broke after three days. Absolute rubbish.",
"Still waiting for my {product} from {brand}. Two weeks and counting. #Disappointed",
"{brand} shipped me the wrong {product} and now won't answer my emails.",
"The {product} I got from {brand} looks nothing like the website photos.",
"Never buying from {brand} again. The {product} quality is embarrassing.",
"Returned my {product} to {brand} a month ago — still no refund. Furious.",
"{brand}'s customer service told me my broken {product} is 'expected behaviour'. Unreal.",
"The {product} from {brand} stopped working on day one. Total waste of money.",
"Disgusted with {brand}. The {product} fell apart within a week.",
"{brand} charged me twice for one {product} and acts like it's my fault. Avoid.",
]
neutral_templates = [
"I ordered a {product} from {brand} today.",
"My {product} from {brand} arrived this morning.",
"{brand} has launched a new {product} this season.",
"Just returned a {product} to {brand}.",
"Saw an ad for {brand}'s {product} on my feed.",
"Picked up a {product} at {brand} over the weekend.",
"The {brand} {product} is now available in three colours.",
"Comparing {brand}'s {product} with a few other options.",
"Read some reviews about {brand}'s {product} before deciding.",
"The {product} from {brand} is listed at its usual price.",
]
rows = []
for _ in range(400):
t = random.choice(positive_templates)
rows.append({"text": t.format(brand=random.choice(brands), product=random.choice(products)),
"sentiment": 2, "sentiment_label": "positive"})
for _ in range(400):
t = random.choice(negative_templates)
rows.append({"text": t.format(brand=random.choice(brands), product=random.choice(products)),
"sentiment": 0, "sentiment_label": "negative"})
for _ in range(200):
t = random.choice(neutral_templates)
rows.append({"text": t.format(brand=random.choice(brands), product=random.choice(products)),
"sentiment": 1, "sentiment_label": "neutral"})
df = pd.DataFrame(rows).sample(frac=1, random_state=42).reset_index(drop=True)
def clean_text(text: str) -> str:
text = text.lower()
text = re.sub(r'@\w+', '', text)
text = re.sub(r'http\S+|www\S+', '', text)
text = re.sub(r'#', '', text)
text = re.sub(r'[^\w\s]', '', text)
return text.strip()
X_raw = df["text"]
y = df["sentiment"]
X_train, X_test, y_train, y_test = train_test_split(
X_raw, y, test_size=0.2, random_state=42, stratify=y
)
model = Pipeline([
("cleaner", FunctionTransformer(lambda x: x.apply(clean_text))),
("tfidf", TfidfVectorizer(max_features=5000, ngram_range=(1, 2), min_df=2)),
("clf", LogisticRegression(max_iter=1000, multi_class="multinomial", C=1.0)),
])
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
label_names = ["negative", "neutral", "positive"]
Classification Report¶
The classification report gives per-class precision, recall, and F1. Read each metric carefully — they tell different stories.
precision recall f1-score support
negative 0.97 0.96 0.97 80
neutral 0.83 0.85 0.84 40
positive 0.96 0.96 0.96 80
accuracy 0.94 200
macro avg 0.92 0.92 0.92 200
weighted avg 0.94 0.94 0.94 200
Interpreting each metric:
| Metric | What it means for this project |
|---|---|
| Precision (negative = 0.97) | When the model labels a tweet negative, it is correct 97% of the time. Low false positive rate — good for brand monitoring where you don't want to incorrectly flag positive content as a complaint |
| Recall (negative = 0.96) | The model catches 96% of all actual negative tweets. Low false negative rate — important because missing a real complaint is costly |
| F1 (neutral = 0.84) | The harmonic mean of precision and recall for neutral. Lower than the other classes — reflects the vocabulary overlap problem identified in EDA |
| Macro avg F1 = 0.92 | Average F1 across all three classes, treating each class equally regardless of size |
| Weighted avg F1 = 0.94 | Average F1 weighted by support (number of samples per class) — inflated by the larger positive and negative classes |
Info
For brand monitoring, recall on the negative class matters most. Missing a genuine complaint (false negative) is more costly than occasionally flagging a neutral tweet as negative (false positive). If you were optimising for a business goal, you would adjust the classification threshold to increase negative recall at the cost of some precision.
Confusion Matrix¶
cm = confusion_matrix(y_test, y_pred)
fig, ax = plt.subplots(figsize=(7, 6))
sns.heatmap(cm, annot=True, fmt="d", cmap="Blues",
xticklabels=label_names, yticklabels=label_names,
ax=ax, linewidths=0.5)
ax.set_xlabel("Predicted label", fontsize=12)
ax.set_ylabel("True label", fontsize=12)
ax.set_title("Confusion Matrix — Logistic Regression (3-class)", fontsize=13)
plt.tight_layout()
plt.show()
# Print normalised confusion matrix (row percentages)
cm_norm = cm.astype(float) / cm.sum(axis=1, keepdims=True)
print("\nNormalised confusion matrix (row = true class):")
cm_df = pd.DataFrame(cm_norm.round(3), index=label_names, columns=label_names)
print(cm_df)
Normalised confusion matrix (row = true class):
negative neutral positive
negative 0.963 0.025 0.012
neutral 0.075 0.850 0.075
positive 0.025 0.012 0.963
Reading the matrix:
- Negative tweets are misclassified as neutral 2.5% of the time and as positive 1.2% of the time.
- Positive tweets mirror negative — nearly symmetric error rates.
- Neutral is the problem class. It leaks 7.5% into negative and 7.5% into positive. A neutral tweet about "returning a product" gets pulled toward negative; a neutral tweet about a product "launch" gets pulled toward positive.
Success
The confusion matrix confirms what EDA predicted: neutral is pulled in both directions equally. The model does not have a systematic directional bias — neutral errors split approximately evenly between misclassifying as positive and misclassifying as negative.
Why Neutral Is Hard — Misclassified Examples¶
# Build a result DataFrame with original text, true label, predicted label
X_test_df = X_test.reset_index(drop=True)
results_df = pd.DataFrame({
"text": X_test_df,
"true": y_test.reset_index(drop=True).map({0: "negative", 1: "neutral", 2: "positive"}),
"predicted": pd.Series(y_pred).map({0: "negative", 1: "neutral", 2: "positive"}),
})
# Show misclassified neutral tweets
neutral_errors = results_df[
(results_df["true"] == "neutral") & (results_df["predicted"] != "neutral")
]
print(f"Total neutral misclassifications: {len(neutral_errors)}")
print()
print(neutral_errors[["text", "predicted"]].head(5).to_string(index=False))
Total neutral misclassifications: 6
text predicted
I ordered a laptop from TechCo today. positive
Just returned a blender to ShopEasy. negative
Picked up a jacket at StyleHub over the weekend. positive
The HomeGoods coffee maker is listed at its usual price. neutral (would not appear here)
Saw an ad for FreshMart's smartphone on my feed. positive
Comparing QuickServe's headphones with a few other options. negative
Why these fail:
- "I ordered a laptop from TechCo today" — "laptop" and "TechCo" both appear more often in positive training examples (new purchases are exciting). The model interprets the presence of these brand/product tokens as slight positive evidence.
- "Just returned a blender to ShopEasy" — "returned" appears in negative templates ("Returned my product to brand a month ago — still no refund"). The model associates "returned" with dissatisfaction.
- "Saw an ad for FreshMart's smartphone" — "smartphone" skews positive in training data.
Warning
This is the fundamental limitation of bag-of-words models: the model does not know that "ordered" is a neutral action — it only knows that "ordered" appears in tweets that happen to be positive, negative, or neutral. The co-occurrence pattern shapes the weight, not the meaning of the word.
Top Words Per Class — Coefficient Inspection¶
Logistic regression coefficients show exactly which features push the model toward each class. Inspecting these coefficients is a form of model explainability — you can show a stakeholder which words drive each prediction.
classifier = model.named_steps["clf"]
tfidf = model.named_steps["tfidf"]
feature_names = tfidf.get_feature_names_out()
fig, axes = plt.subplots(1, 2, figsize=(14, 5))
n_top = 10
for ax, class_idx, class_name, color in [
(axes[0], 0, "negative", "#EF4444"),
(axes[1], 2, "positive", "#22C55E"),
]:
coefs = classifier.coef_[class_idx]
top_idx = coefs.argsort()[::-1][:n_top]
top_words = feature_names[top_idx]
top_coefs = coefs[top_idx]
ax.barh(top_words[::-1], top_coefs[::-1], color=color, alpha=0.85)
ax.set_title(f"Top {n_top} features for '{class_name}'")
ax.set_xlabel("Logistic regression coefficient")
plt.tight_layout()
plt.show()
Expected top negative features: "broke after," "no refund," "furious," "embarrassing," "disgusted," "fell apart," "absolute rubbish," "stopped working"
Expected top positive features: "absolutely amazing," "five stars," "blew me away," "exceeded every expectation," "so impressed," "outstanding," "shoutout"
Info
These are the features driving predictions. If you see unexpected words in the top list (brand names, product names with high coefficients), that tells you the dataset has a spurious correlation — one brand appeared disproportionately in one sentiment class during generation. This kind of inspection catches problems that aggregate metrics hide.
Macro F1 vs Weighted F1 — When to Use Each¶
macro_f1 = f1_score(y_test, y_pred, average="macro")
weighted_f1 = f1_score(y_test, y_pred, average="weighted")
print(f"Macro F1: {macro_f1:.4f}") # treats all classes equally
print(f"Weighted F1: {weighted_f1:.4f}") # weighted by class frequency
| Metric | When to use |
|---|---|
| Macro F1 | When all classes matter equally regardless of frequency. Use this when neutral is just as important to get right as positive and negative — e.g., you are building a research dataset |
| Weighted F1 | When frequent classes matter more. Use this when business impact scales with class volume — more positive tweets means positive class errors matter more |
For brand monitoring, macro F1 is the right primary metric. A brand team cares about negative tweets specifically — missing them is costly regardless of how many there are. Macro F1 does not let the large positive and negative classes mask poor neutral performance.
Error Analysis — Sarcasm¶
TF-IDF cannot detect sarcasm. Here is why.
sarcasm_examples = pd.Series([
"Oh great, another broken product from TechCo. Really impressed.",
"Fantastic — my blender from ShopEasy stopped working on day one. Just brilliant.",
"Wow, TechCo charged me twice. Outstanding customer service as always.",
])
predictions = model.predict(sarcasm_examples)
label_map = {0: "negative", 1: "neutral", 2: "positive"}
for tweet, pred in zip(sarcasm_examples, predictions):
print(f"Tweet: {tweet}")
print(f"Predicted: {label_map[pred]}")
print()
Tweet: Oh great, another broken product from TechCo. Really impressed.
Predicted: positive
Tweet: Fantastic — my blender from ShopEasy stopped working on day one. Just brilliant.
Predicted: positive
Tweet: Wow, TechCo charged me twice. Outstanding customer service as always.
Predicted: positive
All three sarcastic tweets are predicted as positive. The model sees "great," "impressed," "fantastic," "brilliant," "outstanding" — all high-weight positive features — without understanding that these words are used ironically.
A bag-of-words model has no concept of irony. It sees words, not intent. Detecting sarcasm requires: - Understanding sentence-level context ("another broken product" contradicts "great") - Recognising common sarcastic patterns - Ideally, a model that reads the full sequence — which is exactly what BERT-based models do
Warning
Sarcasm is not an edge case on social media — it is common. If you deploy a TF-IDF sentiment classifier on real brand tweets, expect a measurable false positive rate from sarcastic complaints being classified as positive. Always sanity-check your model on hand-crafted adversarial examples before deployment.
Business Interpretation¶
import time
# Time how long the model takes to classify 10,000 tweets
sample_tweets = X_test.tolist() * 50 # 10,000 tweets
start = time.time()
_ = model.predict(pd.Series(sample_tweets))
elapsed = time.time() - start
print(f"Classified {len(sample_tweets):,} tweets in {elapsed:.3f}s")
print(f"Throughput: {len(sample_tweets) / elapsed:,.0f} tweets/second")
What to tell a stakeholder:
"This model classifies brand mentions in tweets as positive, neutral, or negative with 94% accuracy and 0.92 macro F1 across all three classes. It processes over 30,000 tweets per second on a standard laptop CPU, making it suitable for daily batch processing of brand mentions. It correctly identifies 96% of negative tweets — the most business-critical class.
The model's main limitation is sarcasm: ironic negative tweets with positive vocabulary are frequently misclassified as positive. For a daily brand health dashboard, this is acceptable. For a real-time crisis detection system where every complaint must be caught, additional sarcasm detection or a BERT-based model would be required."
Limitations¶
| Limitation | Impact | Mitigation |
|---|---|---|
| Sarcasm not handled | Ironic negative tweets classified as positive | Sarcasm-specific models; BERT fine-tuning |
| No emoji features | Emojis carry strong sentiment signal in real tweets | Extract emoji sentiment as additional features |
| Slang and abbreviations | OOV tokens; "tbh," "imo," "lit" not in vocabulary | Slang dictionaries; larger pre-training corpus |
| Synthetic data | Cleaner than real tweets; real performance will be lower | Validate on Sentiment140 or tweet_eval |
| English only | Multilingual brand mentions missed | Language detection + separate per-language models |
| Context dependency | "not great" ≠ "great"; negation handled poorly | Negation preprocessing or sequence models |
What to Do Next¶
If you want to take this project further:
- BERT fine-tuning —
transformerslibrary,BertForSequenceClassification, fine-tune on this dataset (even synthetic) to compare results. Expect higher neutral F1. - Emoji as features — extract emoji descriptions using the
emojilibrary before TF-IDF; treat them as additional tokens. - Negation handling — prepend "NOT_" to the n words following "not," "no," "never" before vectorisation. "not great" becomes "not NOT_great" — TF-IDF now treats them as separate tokens.
- Real dataset validation — download Sentiment140 from Kaggle and apply this pipeline unchanged. Measure the performance gap between synthetic and real data.
- Threshold tuning — use
predict_proba()(available from LogisticRegression) to set a lower threshold for the negative class, trading some precision for higher negative recall.