Exploratory Data Analysis¶
Before building a model, understand what the text actually looks like. EDA on text data answers three questions: Are the classes distinguishable by vocabulary? Do structural features like length differ across classes? Which words carry the most signal?
Setup — Generate the Dataset¶
This file is self-contained. Run this block first to create df, then proceed through each section.
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import re
from collections import Counter
import random
random.seed(42)
np.random.seed(42)
brands = ["TechCo", "ShopEasy", "FreshMart", "StyleHub", "HomeGoods", "QuickServe"]
products = ["laptop", "headphones", "sneakers", "coffee maker", "smartphone",
"backpack", "blender", "jacket"]
positive_templates = [
"Just got my {product} from {brand} and it's absolutely amazing!",
"Huge shoutout to {brand} — the {product} exceeded every expectation.",
"{brand} delivered my {product} ahead of schedule. So impressed!",
"The {product} from {brand} is worth every penny. Five stars.",
"I've been recommending {brand}'s {product} to everyone I know.",
"Finally tried {brand} and their {product} blew me away. #LoveIt",
"{brand} customer service sorted my {product} issue in under 10 minutes. Legend.",
"Unboxing my new {product} from {brand} right now — quality is outstanding.",
"Couldn't be happier with my {product} purchase from {brand}. Will buy again.",
"{brand} nailed it with this {product}. Best purchase I've made this year.",
]
negative_templates = [
"My {product} from {brand} broke after three days. Absolute rubbish.",
"Still waiting for my {product} from {brand}. Two weeks and counting. #Disappointed",
"{brand} shipped me the wrong {product} and now won't answer my emails.",
"The {product} I got from {brand} looks nothing like the website photos.",
"Never buying from {brand} again. The {product} quality is embarrassing.",
"Returned my {product} to {brand} a month ago — still no refund. Furious.",
"{brand}'s customer service told me my broken {product} is 'expected behaviour'. Unreal.",
"The {product} from {brand} stopped working on day one. Total waste of money.",
"Disgusted with {brand}. The {product} fell apart within a week.",
"{brand} charged me twice for one {product} and acts like it's my fault. Avoid.",
]
neutral_templates = [
"I ordered a {product} from {brand} today.",
"My {product} from {brand} arrived this morning.",
"{brand} has launched a new {product} this season.",
"Just returned a {product} to {brand}.",
"Saw an ad for {brand}'s {product} on my feed.",
"Picked up a {product} at {brand} over the weekend.",
"The {brand} {product} is now available in three colours.",
"Comparing {brand}'s {product} with a few other options.",
"Read some reviews about {brand}'s {product} before deciding.",
"The {product} from {brand} is listed at its usual price.",
]
rows = []
for _ in range(400):
t = random.choice(positive_templates)
rows.append({"text": t.format(brand=random.choice(brands), product=random.choice(products)),
"sentiment": 2, "sentiment_label": "positive"})
for _ in range(400):
t = random.choice(negative_templates)
rows.append({"text": t.format(brand=random.choice(brands), product=random.choice(products)),
"sentiment": 0, "sentiment_label": "negative"})
for _ in range(200):
t = random.choice(neutral_templates)
rows.append({"text": t.format(brand=random.choice(brands), product=random.choice(products)),
"sentiment": 1, "sentiment_label": "neutral"})
df = pd.DataFrame(rows).sample(frac=1, random_state=42).reset_index(drop=True)
print(df.shape) # (1000, 3)
Class Distribution¶
counts = df["sentiment_label"].value_counts()
total = len(df)
fig, ax = plt.subplots(figsize=(7, 4))
colors = {"negative": "#EF4444", "neutral": "#94A3B8", "positive": "#22C55E"}
bars = ax.bar(counts.index, counts.values,
color=[colors[c] for c in counts.index])
for bar, count in zip(bars, counts.values):
pct = count / total * 100
ax.text(bar.get_x() + bar.get_width() / 2, bar.get_height() + 5,
f"{count} ({pct:.0f}%)", ha="center", va="bottom", fontsize=11)
ax.set_title("Class Distribution", fontsize=13)
ax.set_ylabel("Count")
ax.set_ylim(0, 480)
plt.tight_layout()
plt.show()
Info
Positive and negative are balanced at 400 each. Neutral is the minority at 200. A dummy classifier that always predicts "positive" scores 40% accuracy — the floor you must beat.
Text Length Analysis¶
Do longer tweets signal stronger sentiment? Add structural features to the DataFrame, then visualise distributions by class.
# Add structural columns
df["char_length"] = df["text"].str.len()
df["word_count"] = df["text"].str.split().str.len()
print(df.groupby("sentiment_label")[["char_length", "word_count"]].describe().round(1))
char_length word_count
count mean std count mean std
sentiment_label
negative 400.0 76.4 14.3 400.0 13.2 2.5
neutral 200.0 47.1 7.8 200.0 8.8 1.4
positive 400.0 69.8 13.1 400.0 12.2 2.3
fig, axes = plt.subplots(1, 2, figsize=(12, 4))
class_colors = {"positive": "#22C55E", "negative": "#EF4444", "neutral": "#94A3B8"}
for label, grp in df.groupby("sentiment_label"):
axes[0].hist(grp["char_length"], alpha=0.6, bins=20,
label=label, color=class_colors[label])
axes[1].hist(grp["word_count"], alpha=0.6, bins=15,
label=label, color=class_colors[label])
axes[0].set_title("Character length by sentiment")
axes[0].set_xlabel("Characters")
axes[0].legend()
axes[1].set_title("Word count by sentiment")
axes[1].set_xlabel("Words")
axes[1].legend()
plt.tight_layout()
plt.show()
Success
Key finding: Negative tweets are on average ~29 characters longer than neutral tweets and ~7 characters longer than positive tweets. Complaints require more words — people explain what went wrong. Neutral tweets are short factual statements. Word count alone is a weak but real signal.
Most Common Words Per Class¶
Simple frequency analysis after lowercasing and splitting on whitespace. This is not a substitute for TF-IDF but it shows you what each class is talking about.
def top_words(texts, n=15):
"""Count word frequencies after lowercase. Returns list of (word, count) tuples."""
all_words = []
for text in texts:
all_words.extend(text.lower().split())
return Counter(all_words).most_common(n)
# Get top words per class
for label in ["positive", "negative", "neutral"]:
subset = df[df["sentiment_label"] == label]["text"]
words, counts = zip(*top_words(subset, n=15))
print(f"\n--- {label.upper()} ---")
for w, c in zip(words, counts):
print(f" {w:<20} {c}")
--- POSITIVE ---
from 272
the 201
my 198
techco 88
...
amazing 47
outstanding 39
impressed 35
--- NEGATIVE ---
from 291
my 245
the 238
broke 62
refund 58
furious 51
--- NEUTRAL ---
from 142
a 120
product 98
brand 94
today 55
fig, axes = plt.subplots(1, 3, figsize=(16, 5))
labels = ["positive", "negative", "neutral"]
colors = ["#22C55E", "#EF4444", "#94A3B8"]
for ax, label, color in zip(axes, labels, colors):
subset = df[df["sentiment_label"] == label]["text"]
word_counts = top_words(subset, n=15)
words, counts = zip(*word_counts)
ax.barh(words[::-1], counts[::-1], color=color, alpha=0.85)
ax.set_title(f"Top 15 words — {label}")
ax.set_xlabel("Frequency")
plt.tight_layout()
plt.show()
Warning
Stop words like "from," "my," and "the" dominate raw frequency counts. TF-IDF downweights these automatically because they appear across all classes. For raw counts, you would normally filter stop words — but here the goal is exploration, not feature engineering.
Brand Mentions Per Sentiment¶
Which brands attract the most negative attention in this dataset?
brand_sentiment = []
for _, row in df.iterrows():
for brand in brands:
if brand in row["text"]:
brand_sentiment.append({"brand": brand, "sentiment_label": row["sentiment_label"]})
brand_df = pd.DataFrame(brand_sentiment)
pivot = brand_df.pivot_table(index="brand", columns="sentiment_label",
aggfunc="size", fill_value=0)
pivot["total"] = pivot.sum(axis=1)
print(pivot.sort_values("total", ascending=False))
# Stacked bar chart
pivot_pct = pivot[["negative", "neutral", "positive"]].div(pivot["total"], axis=0)
pivot_pct.sort_values("negative", ascending=False).plot(
kind="bar", stacked=True, figsize=(9, 5),
color=["#EF4444", "#94A3B8", "#22C55E"],
title="Sentiment proportion by brand"
)
plt.ylabel("Proportion")
plt.xticks(rotation=30)
plt.tight_layout()
plt.show()
Info
Because templates are filled randomly, each brand should have roughly equal sentiment distribution (about 40% positive, 40% negative, 20% neutral). In a real dataset, brands differ significantly — this is exactly the signal you would present to a stakeholder.
Bigrams Per Class¶
A single word "broke" is informative. The bigram "broke after" is more informative — it confirms the item failed rather than "broke a record." Extract top bigrams per class using simple consecutive word pairs.
def top_bigrams(texts, n=10):
"""Return top n bigrams from a series of text strings."""
all_bigrams = []
for text in texts:
words = text.lower().split()
all_bigrams.extend(zip(words, words[1:]))
return Counter(all_bigrams).most_common(n)
for label in ["positive", "negative", "neutral"]:
subset = df[df["sentiment_label"] == label]["text"]
print(f"\n--- {label.upper()} — Top 10 bigrams ---")
for bigram, count in top_bigrams(subset, n=10):
print(f" {' '.join(bigram):<30} {count}")
--- POSITIVE — Top 10 bigrams ---
from techco 21
from shopeasy 18
the laptop 17
every expectation 15
absolutely amazing 14
so impressed 13
five stars 12
...
--- NEGATIVE — Top 10 bigrams ---
my laptop 22
broke after 19
from techco 17
two weeks 16
still waiting 15
no refund 14
fell apart 13
...
--- NEUTRAL — Top 10 bigrams ---
from techco 14
a laptop 12
from brand 10
this season 9
usual price 8
...
Success
Bigrams like "absolutely amazing," "broke after," and "still waiting" are strong class-specific signals. When you set ngram_range=(1,2) in TfidfVectorizer, these bigrams become features — they often outperform unigrams alone on short text.
Vocabulary Overlap — What Separates Positive from Negative?¶
def class_vocab(texts):
words = set()
for text in texts:
words.update(text.lower().split())
return words
pos_vocab = class_vocab(df[df["sentiment_label"] == "positive"]["text"])
neg_vocab = class_vocab(df[df["sentiment_label"] == "negative"]["text"])
neutral_vocab = class_vocab(df[df["sentiment_label"] == "neutral"]["text"])
# Words exclusive to one class
only_positive = pos_vocab - neg_vocab - neutral_vocab
only_negative = neg_vocab - pos_vocab - neutral_vocab
print("Words found ONLY in positive tweets:")
print(sorted(only_positive)[:20])
print("\nWords found ONLY in negative tweets:")
print(sorted(only_negative)[:20])
shared = pos_vocab & neg_vocab
print(f"\nWords shared by positive AND negative: {len(shared)}")
print(f"Overlap rate: {len(shared) / len(pos_vocab | neg_vocab):.1%}")
Words found ONLY in positive tweets:
['#LoveIt', 'amazing!', 'blew', 'happier', 'impressed!', 'legend.',
'nailed', 'outstanding.', 'outperformed', 'shoutout']
Words found ONLY in negative tweets:
['#Disappointed', 'Absolute', 'Avoid.', 'Disgusted', 'Furious.',
'Unreal.', 'apart', 'behaviour'.', 'charged', 'embarrassing.']
Words shared by positive AND negative: 87
Overlap rate: 34%
Success
Key finding: 34% of the combined vocabulary is shared between positive and negative tweets. The separating signal lives in a specific subset of words — sentiment-charged adjectives and verbs. This is exactly what TF-IDF captures: high-weight features for words that are frequent in one class and rare in others.
EDA Summary¶
| Finding | Implication for modeling |
|---|---|
| Negative tweets are ~7 words longer on average | Text length is a weak but real feature; consider adding word_count as a numeric feature |
| Strong class-exclusive vocabulary exists | TF-IDF with sufficient max_features should separate positive and negative well |
| Neutral tweets share vocabulary with both classes | Expect lower recall for neutral; plan for error analysis |
| Bigrams like "broke after" and "absolutely amazing" are highly discriminative | Use ngram_range=(1,2) in TfidfVectorizer |
| All six brands appear at similar rates across classes | Brand name alone is not a sentiment predictor in this dataset |
Info
EDA did not change the data — it changed your understanding of it. You now know which classes will be easy to separate (positive vs. negative) and which will be hard (neutral vs. either). That knowledge shapes your evaluation strategy: track neutral recall specifically.