Dataset Guide

Understanding your data before modeling is not optional — it is the job. This guide walks through how the synthetic tweet dataset is constructed, what the class distribution looks like, and why tweet text is genuinely hard to classify.

---

Dataset Generation Code

The dataset is generated entirely in Python — no API key, no download, no sign-up. Run this once and you have 1,000 rows ready to work with.

import pandas as pd
import numpy as np
import random

random.seed(42)
np.random.seed(42)

# --- Brand and product vocabulary ---
brands = ["TechCo", "ShopEasy", "FreshMart", "StyleHub", "HomeGoods", "QuickServe"]
products = ["laptop", "headphones", "sneakers", "coffee maker", "smartphone",
            "backpack", "blender", "jacket"]

# --- Sentence templates ---
positive_templates = [
    "Just got my {product} from {brand} and it's absolutely amazing!",
    "Huge shoutout to {brand} — the {product} exceeded every expectation.",
    "{brand} delivered my {product} ahead of schedule. So impressed!",
    "The {product} from {brand} is worth every penny. Five stars.",
    "I've been recommending {brand}'s {product} to everyone I know.",
    "Finally tried {brand} and their {product} blew me away. #LoveIt",
    "{brand} customer service sorted my {product} issue in under 10 minutes. Legend.",
    "Unboxing my new {product} from {brand} right now — quality is outstanding.",
    "Couldn't be happier with my {product} purchase from {brand}. Will buy again.",
    "{brand} nailed it with this {product}. Best purchase I've made this year.",
]

negative_templates = [
    "My {product} from {brand} broke after three days. Absolute rubbish.",
    "Still waiting for my {product} from {brand}. Two weeks and counting. #Disappointed",
    "{brand} shipped me the wrong {product} and now won't answer my emails.",
    "The {product} I got from {brand} looks nothing like the website photos.",
    "Never buying from {brand} again. The {product} quality is embarrassing.",
    "Returned my {product} to {brand} a month ago — still no refund. Furious.",
    "{brand}'s customer service told me my broken {product} is 'expected behaviour'. Unreal.",
    "The {product} from {brand} stopped working on day one. Total waste of money.",
    "Disgusted with {brand}. The {product} fell apart within a week.",
    "{brand} charged me twice for one {product} and acts like it's my fault. Avoid.",
]

neutral_templates = [
    "I ordered a {product} from {brand} today.",
    "My {product} from {brand} arrived this morning.",
    "{brand} has launched a new {product} this season.",
    "Just returned a {product} to {brand}.",
    "Saw an ad for {brand}'s {product} on my feed.",
    "Picked up a {product} at {brand} over the weekend.",
    "The {brand} {product} is now available in three colours.",
    "Comparing {brand}'s {product} with a few other options.",
    "Read some reviews about {brand}'s {product} before deciding.",
    "The {product} from {brand} is listed at its usual price.",
]

# --- Generate rows ---
rows = []

for _ in range(400):   # positive
    template = random.choice(positive_templates)
    text = template.format(brand=random.choice(brands), product=random.choice(products))
    rows.append({"text": text, "sentiment": 2, "sentiment_label": "positive"})

for _ in range(400):   # negative
    template = random.choice(negative_templates)
    text = template.format(brand=random.choice(brands), product=random.choice(products))
    rows.append({"text": text, "sentiment": 0, "sentiment_label": "negative"})

for _ in range(200):   # neutral
    template = random.choice(neutral_templates)
    text = template.format(brand=random.choice(brands), product=random.choice(products))
    rows.append({"text": text, "sentiment": 1, "sentiment_label": "neutral"})

# --- Shuffle and build DataFrame ---
df = pd.DataFrame(rows).sample(frac=1, random_state=42).reset_index(drop=True)

print(df.shape)          # (1000, 3)
print(df.head())
print(df["sentiment_label"].value_counts())

(1000, 3)
                                                text  sentiment sentiment_label
0  Never buying from FreshMart again. The headph...          0        negative
1  The laptop from HomeGoods is listed at its us...          1         neutral
2  Huge shoutout to ShopEasy — the jacket exceed...          2        positive
...

sentiment_label
positive    400
negative    400
neutral     200
dtype: int64

Tip

Save the DataFrame to CSV after generating it so you don't have to regenerate it every time: df.to_csv("tweets.csv", index=False). Load with df = pd.read_csv("tweets.csv") in subsequent files.

---

Class Distribution

import matplotlib.pyplot as plt

counts = df["sentiment_label"].value_counts()
total = len(df)

fig, ax = plt.subplots(figsize=(7, 4))
bars = ax.bar(counts.index, counts.values,
              color=["#EF4444", "#94A3B8", "#22C55E"])

for bar, count in zip(bars, counts.values):
    pct = count / total * 100
    ax.text(bar.get_x() + bar.get_width() / 2, bar.get_height() + 5,
            f"{count}\n({pct:.0f}%)", ha="center", va="bottom", fontsize=11)

ax.set_title("Class Distribution — Synthetic Tweet Dataset", fontsize=13)
ax.set_ylabel("Number of tweets")
ax.set_ylim(0, 480)
ax.set_xlabel("Sentiment class")
plt.tight_layout()
plt.show()

Class	Count	Percentage
Positive	400	40%
Negative	400	40%
Neutral	200	20%

Info

Positive and negative are balanced at 40% each. Neutral is the minority class at 20%. A classifier that always predicted "positive" would score 40% accuracy — that is the baseline you need to beat.

---

Sample Tweets Per Class

Positive (sentiment = 2)

#	Tweet
1	"Just got my headphones from TechCo and it's absolutely amazing!"
2	"Huge shoutout to ShopEasy — the jacket exceeded every expectation."
3	"{brand} delivered my coffee maker ahead of schedule. So impressed!"

Negative (sentiment = 0)

#	Tweet
1	"Never buying from FreshMart again. The headphones quality is embarrassing."
2	"Still waiting for my laptop from StyleHub. Two weeks and counting. #Disappointed"
3	"Returned my blender to HomeGoods a month ago — still no refund. Furious."

Neutral (sentiment = 1)

#	Tweet
1	"I ordered a laptop from TechCo today."
2	"The StyleHub jacket is now available in three colours."
3	"Comparing QuickServe's smartphone with a few other options."

---

Why Three Classes?

Most beginner sentiment tutorials use binary classification: positive vs. negative. Real social media data is not binary.

Neutral is real and ubiquitous. When someone tweets "I ordered a laptop from TechCo," they are not expressing approval or disapproval — they are stating a fact. Treating that as positive or negative adds noise to any brand monitoring system. In real tweet corpora, neutral is often the majority class — the brand appears in conversation without strong sentiment attached.

Three-class classification is also a more useful business tool. A brand team wants to know:

• How much of the conversation is positive? (use for marketing)
• How much is negative? (route to support)
• How much is neutral? (monitor for sentiment shift)

Binary classification collapses neutral into one of the other two, which distorts every downstream metric.

Warning

If you collapse to binary (positive vs. negative) and ignore neutral tweets in evaluation, your model will appear stronger than it actually is. Always report performance across all intended classes.

---

Why Synthetic Data?

Real tweet data requires either the Twitter API (which now requires a paid developer plan) or a Kaggle dataset download. Both introduce friction that gets in the way of learning the NLP workflow.

Synthetic data lets you: - Start coding immediately — no authentication, no rate limits, no terms of service - Control the difficulty — you know exactly what signals the templates contain - Reproduce results exactly — random.seed(42) gives every student the same dataset - Focus on the pipeline — the preprocessing and modeling steps are identical to real data

The tradeoff is that synthetic data is cleaner than reality. The templates have consistent grammar and spelling. Real tweets contain abbreviations, emojis, typos, slang, and multiple languages in the same corpus.

Real dataset pointers for follow-up:

Dataset	Size	Notes
Kaggle Sentiment140	1.6M tweets	Binary (0/4), real Twitter data from 2009
HuggingFace tweet_eval	Varies by task	Covers sentiment, irony, emotion — benchmark quality
Twitter API v2	Live	Requires developer account; free tier is rate-limited

Tip

Once your pipeline works on this synthetic dataset, swap in Sentiment140 by changing the data loading line. The cleaning, vectorisation, and modeling code transfers unchanged.

---

Key Challenges of Tweet Text

Tweet text breaks almost every assumption that classical NLP was built on.

Challenge	Example	Effect on classification
Short length	"Meh."	Very sparse TF-IDF vectors; most features are zero
Informal language	"omg this is lit 🔥"	Out-of-vocabulary tokens; standard tokenisers struggle
@mentions	"@TechCo your product sucks"	@TechCo is not a meaningful feature — needs removal
URLs	"Check this out https://t.co/xyz"	URLs carry no sentiment signal — needs removal
Hashtags	"#Disappointed with #ShopEasy"	The # is punctuation; Disappointed is the signal
Sarcasm	"Oh great, another broken product"	Positive vocabulary, negative meaning — hard for any bag-of-words model
Abbreviations	"tbh idk if this is worth it"	Standard vocabulary maps won't find these
Repeated characters	"sooooo good!!!"	Normalisation needed; "sooooo" ≠ "so" in raw form

Warning

This dataset does not include emojis, abbreviations, or multilingual text. When you apply this pipeline to real tweet data, expect performance to drop and plan for additional preprocessing steps to address these cases.

---

What Makes Neutral Hard to Classify

Neutral tweets are structurally similar to both positive and negative tweets. Compare:

• Positive: "My coffee maker from TechCo is outstanding."
• Neutral: "I ordered a coffee maker from TechCo today."
• Negative: "My coffee maker from TechCo broke after three days."

All three contain the words "coffee maker" and "TechCo." The positive tweet signals sentiment through "outstanding." The negative tweet signals through "broke." The neutral tweet has no polarity word — it shares vocabulary with both classes without expressing either.

A TF-IDF model learns weights for individual words. Words like "outstanding" and "broke" are good signal. But neutral tweets mostly contain neutral function words ("ordered," "arrived," "today") that also appear in the context of positive and negative tweets.

The result: neutral tweets get pulled toward whichever class the surrounding vocabulary most resembles. A tweet like "I ordered a laptop from TechCo today" might score slightly positive because "laptop" and "TechCo" appear more often in positive training examples.

Info

This is a fundamental limitation of bag-of-words approaches. Context-aware models like BERT read the full sentence and can distinguish "I ordered a product" (neutral) from "I love this product" (positive) because they understand word order and context, not just word presence.

---

← README | Next: EDA →