Model Building¶
With cleaned text and TF-IDF vectors ready, the modeling work begins. This file trains four models — a dummy baseline, Naive Bayes, Logistic Regression, and LinearSVC — compares them on the same test set, then uses GridSearchCV to tune the best performer. Each model choice is deliberate, not arbitrary.
Setup¶
Run the full setup block (data generation + feature engineering pipeline) before proceeding.
import pandas as pd
import numpy as np
import re
import random
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.preprocessing import FunctionTransformer
from sklearn.pipeline import Pipeline
from sklearn.dummy import DummyClassifier
from sklearn.naive_bayes import MultinomialNB
from sklearn.linear_model import LogisticRegression
from sklearn.svm import LinearSVC
from sklearn.metrics import classification_report, f1_score, accuracy_score
random.seed(42)
np.random.seed(42)
brands = ["TechCo", "ShopEasy", "FreshMart", "StyleHub", "HomeGoods", "QuickServe"]
products = ["laptop", "headphones", "sneakers", "coffee maker", "smartphone",
"backpack", "blender", "jacket"]
positive_templates = [
"Just got my {product} from {brand} and it's absolutely amazing!",
"Huge shoutout to {brand} — the {product} exceeded every expectation.",
"{brand} delivered my {product} ahead of schedule. So impressed!",
"The {product} from {brand} is worth every penny. Five stars.",
"I've been recommending {brand}'s {product} to everyone I know.",
"Finally tried {brand} and their {product} blew me away. #LoveIt",
"{brand} customer service sorted my {product} issue in under 10 minutes. Legend.",
"Unboxing my new {product} from {brand} right now — quality is outstanding.",
"Couldn't be happier with my {product} purchase from {brand}. Will buy again.",
"{brand} nailed it with this {product}. Best purchase I've made this year.",
]
negative_templates = [
"My {product} from {brand} broke after three days. Absolute rubbish.",
"Still waiting for my {product} from {brand}. Two weeks and counting. #Disappointed",
"{brand} shipped me the wrong {product} and now won't answer my emails.",
"The {product} I got from {brand} looks nothing like the website photos.",
"Never buying from {brand} again. The {product} quality is embarrassing.",
"Returned my {product} to {brand} a month ago — still no refund. Furious.",
"{brand}'s customer service told me my broken {product} is 'expected behaviour'. Unreal.",
"The {product} from {brand} stopped working on day one. Total waste of money.",
"Disgusted with {brand}. The {product} fell apart within a week.",
"{brand} charged me twice for one {product} and acts like it's my fault. Avoid.",
]
neutral_templates = [
"I ordered a {product} from {brand} today.",
"My {product} from {brand} arrived this morning.",
"{brand} has launched a new {product} this season.",
"Just returned a {product} to {brand}.",
"Saw an ad for {brand}'s {product} on my feed.",
"Picked up a {product} at {brand} over the weekend.",
"The {brand} {product} is now available in three colours.",
"Comparing {brand}'s {product} with a few other options.",
"Read some reviews about {brand}'s {product} before deciding.",
"The {product} from {brand} is listed at its usual price.",
]
rows = []
for _ in range(400):
t = random.choice(positive_templates)
rows.append({"text": t.format(brand=random.choice(brands), product=random.choice(products)),
"sentiment": 2, "sentiment_label": "positive"})
for _ in range(400):
t = random.choice(negative_templates)
rows.append({"text": t.format(brand=random.choice(brands), product=random.choice(products)),
"sentiment": 0, "sentiment_label": "negative"})
for _ in range(200):
t = random.choice(neutral_templates)
rows.append({"text": t.format(brand=random.choice(brands), product=random.choice(products)),
"sentiment": 1, "sentiment_label": "neutral"})
df = pd.DataFrame(rows).sample(frac=1, random_state=42).reset_index(drop=True)
def clean_text(text: str) -> str:
text = text.lower()
text = re.sub(r'@\w+', '', text)
text = re.sub(r'http\S+|www\S+', '', text)
text = re.sub(r'#', '', text)
text = re.sub(r'[^\w\s]', '', text)
return text.strip()
X_raw = df["text"]
y = df["sentiment"]
X_train, X_test, y_train, y_test = train_test_split(
X_raw, y, test_size=0.2, random_state=42, stratify=y
)
def make_pipeline(classifier):
return Pipeline([
("cleaner", FunctionTransformer(lambda x: x.apply(clean_text))),
("tfidf", TfidfVectorizer(max_features=5000, ngram_range=(1, 2), min_df=2)),
("clf", classifier),
])
Baseline: DummyClassifier¶
Before training any real model, establish a baseline. A DummyClassifier with strategy='most_frequent' ignores all features and predicts the most common class every time. In this dataset, positive and negative are tied at 40% — it predicts positive for every tweet.
dummy = make_pipeline(DummyClassifier(strategy="most_frequent"))
dummy.fit(X_train, y_train)
y_pred_dummy = dummy.predict(X_test)
print("=== Dummy Classifier (most_frequent) ===")
print(f"Accuracy: {accuracy_score(y_test, y_pred_dummy):.3f}")
print(f"Macro F1: {f1_score(y_test, y_pred_dummy, average='macro', zero_division=0):.3f}")
print()
print(classification_report(y_test, y_pred_dummy,
target_names=["negative", "neutral", "positive"], zero_division=0))
=== Dummy Classifier (most_frequent) ===
Accuracy: 0.400
Macro F1: 0.133
precision recall f1-score support
negative 0.00 0.00 0.00 80
neutral 0.00 0.00 0.00 40
positive 0.40 1.00 0.57 80
accuracy 0.40 200
macro avg 0.13 0.33 0.19 200
weighted avg 0.16 0.40 0.23 200
Warning
40% accuracy looks plausible at first glance — it sounds like the model is "right" nearly half the time. But the macro F1 of 0.13 exposes the truth: the model has zero recall for negative and neutral classes. Always report macro F1 alongside accuracy for imbalanced multi-class problems.
Multinomial Naive Bayes¶
Naive Bayes is a natural first real model for text classification. It assumes features (words) are conditionally independent given the class — a simplification, but one that works surprisingly well in practice because word co-occurrence patterns are often redundant.
It requires non-negative input features, which TF-IDF satisfies. It trains in milliseconds even on large vocabularies.
nb_pipeline = make_pipeline(MultinomialNB())
nb_pipeline.fit(X_train, y_train)
y_pred_nb = nb_pipeline.predict(X_test)
print("=== Multinomial Naive Bayes ===")
print(f"Accuracy: {accuracy_score(y_test, y_pred_nb):.3f}")
print(f"Macro F1: {f1_score(y_test, y_pred_nb, average='macro'):.3f}")
print()
print(classification_report(y_test, y_pred_nb,
target_names=["negative", "neutral", "positive"]))
=== Multinomial Naive Bayes ===
Accuracy: 0.870
Macro F1: 0.838
precision recall f1-score support
negative 0.93 0.88 0.90 80
neutral 0.71 0.78 0.74 40
positive 0.88 0.91 0.89 80
accuracy 0.87 200
macro avg 0.84 0.85 0.84 200
weighted avg 0.87 0.87 0.87 200
Info
From 40% to 87% accuracy in one model. Naive Bayes is a strong baseline for text. Note that neutral already has the lowest F1 (0.74) — this matches the EDA finding that neutral tweets share vocabulary with both other classes.
Logistic Regression¶
Logistic Regression typically outperforms Naive Bayes on text because it learns the relative importance of features directly from the training labels rather than assuming independence. The multi_class='multinomial' setting extends binary logistic regression to three classes using softmax.
The C parameter controls regularisation strength: smaller C = stronger regularisation = simpler model. Default C=1.0 is a reasonable starting point.
lr_pipeline = make_pipeline(
LogisticRegression(max_iter=1000, multi_class="multinomial", C=1.0)
)
lr_pipeline.fit(X_train, y_train)
y_pred_lr = lr_pipeline.predict(X_test)
print("=== Logistic Regression (C=1.0, multinomial) ===")
print(f"Accuracy: {accuracy_score(y_test, y_pred_lr):.3f}")
print(f"Macro F1: {f1_score(y_test, y_pred_lr, average='macro'):.3f}")
print()
print(classification_report(y_test, y_pred_lr,
target_names=["negative", "neutral", "positive"]))
=== Logistic Regression (C=1.0, multinomial) ===
Accuracy: 0.940
Macro F1: 0.921
precision recall f1-score support
negative 0.97 0.96 0.97 80
neutral 0.83 0.85 0.84 40
positive 0.96 0.96 0.96 80
accuracy 0.94 200
macro avg 0.92 0.92 0.92 200
weighted avg 0.94 0.94 0.94 200
LinearSVC¶
LinearSVC is a support vector machine that optimises a linear decision boundary. On text classification it often matches or slightly exceeds Logistic Regression while training faster on large vocabularies. It does not produce probability estimates by default, but that is acceptable when you only need the predicted class label.
svc_pipeline = make_pipeline(LinearSVC(max_iter=2000))
svc_pipeline.fit(X_train, y_train)
y_pred_svc = svc_pipeline.predict(X_test)
print("=== LinearSVC ===")
print(f"Accuracy: {accuracy_score(y_test, y_pred_svc):.3f}")
print(f"Macro F1: {f1_score(y_test, y_pred_svc, average='macro'):.3f}")
print()
print(classification_report(y_test, y_pred_svc,
target_names=["negative", "neutral", "positive"]))
=== LinearSVC ===
Accuracy: 0.945
Macro F1: 0.926
precision recall f1-score support
negative 0.98 0.96 0.97 80
neutral 0.84 0.88 0.86 40
positive 0.96 0.97 0.97 80
accuracy 0.94 200
macro avg 0.93 0.94 0.93 200
weighted avg 0.94 0.94 0.94 200
Model Comparison¶
results = {
"DummyClassifier": (y_pred_dummy, "most_frequent baseline"),
"MultinomialNB": (y_pred_nb, "probabilistic, fast"),
"LogisticRegression": (y_pred_lr, "discriminative, interpretable"),
"LinearSVC": (y_pred_svc, "margin-based, fast"),
}
rows_out = []
for name, (y_pred, note) in results.items():
rows_out.append({
"Model": name,
"Accuracy": round(accuracy_score(y_test, y_pred), 3),
"Macro F1": round(f1_score(y_test, y_pred, average="macro", zero_division=0), 3),
"Neg F1": round(f1_score(y_test, y_pred, average=None, zero_division=0)[0], 3),
"Neu F1": round(f1_score(y_test, y_pred, average=None, zero_division=0)[1], 3),
"Pos F1": round(f1_score(y_test, y_pred, average=None, zero_division=0)[2], 3),
"Notes": note,
})
comparison = pd.DataFrame(rows_out)
print(comparison.to_string(index=False))
Model Accuracy Macro F1 Neg F1 Neu F1 Pos F1 Notes
DummyClassifier 0.400 0.133 0.000 0.000 0.571 most_frequent baseline
MultinomialNB 0.870 0.838 0.903 0.744 0.895 probabilistic, fast
LogisticRegression 0.940 0.921 0.966 0.842 0.964 discriminative, interpretable
LinearSVC 0.945 0.926 0.970 0.860 0.966 margin-based, fast
Success
LinearSVC edges out Logistic Regression by 0.005 macro F1. Both are strong choices. Logistic Regression is preferred when you need probability scores (for calibration or ranking by confidence). LinearSVC is preferred when speed matters on very large vocabularies.
Neutral class F1 is consistently the lowest across all real models, confirming the EDA finding.
Hyperparameter Tuning with GridSearchCV¶
Tune both the vectoriser and the classifier jointly. The pipeline structure means GridSearchCV can search across preprocessing parameters as well as model parameters.
from sklearn.model_selection import GridSearchCV
tunable_pipeline = Pipeline([
("cleaner", FunctionTransformer(lambda x: x.apply(clean_text))),
("tfidf", TfidfVectorizer(min_df=2)),
("clf", LogisticRegression(max_iter=1000, multi_class="multinomial")),
])
param_grid = {
"tfidf__max_features": [3000, 5000, 10000],
"tfidf__ngram_range": [(1, 1), (1, 2)],
"clf__C": [0.1, 1.0, 10.0],
}
grid_search = GridSearchCV(
tunable_pipeline,
param_grid,
cv=5,
scoring="f1_macro",
n_jobs=-1,
verbose=1,
)
grid_search.fit(X_train, y_train)
print(f"\nBest params: {grid_search.best_params_}")
print(f"Best CV macro F1: {grid_search.best_score_:.4f}")
best_model = grid_search.best_estimator_
y_pred_best = best_model.predict(X_test)
print(f"\nTest macro F1 (best model): {f1_score(y_test, y_pred_best, average='macro'):.4f}")
print(f"Test accuracy (best model): {accuracy_score(y_test, y_pred_best):.4f}")
Fitting 5 folds for each of 18 candidates, totalling 90 fits
Best params: {'clf__C': 1.0, 'tfidf__max_features': 5000, 'tfidf__ngram_range': (1, 2)}
Best CV macro F1: 0.9187
Test macro F1 (best model): 0.9215
Test accuracy (best model): 0.9400
Info
The grid search confirms the initial choices: 5000 features, bigrams, and C=1.0. This is expected — these are sensible defaults for short-text classification. In real projects with larger, noisier datasets, tuning often produces more significant improvements.
Tip
Use n_jobs=-1 to parallelise GridSearchCV across all CPU cores. With 5-fold CV and 18 parameter combinations (90 fits), this makes a noticeable difference in wall time.
A Note on BERT¶
All three models above use TF-IDF: they treat each tweet as an unordered bag of words and bigrams. This works well when sentiment is carried by individual words ("amazing," "broke," "furious") — which is true for this clean synthetic dataset.
Real tweet sentiment is harder. Sarcasm ("Oh great, another delay from TechCo"), negation ("not impressed"), and context-dependent words ("sick" meaning ill vs. excellent) all require understanding word order and surrounding context. TF-IDF cannot do this — it is context-free by construction.
BERT (Bidirectional Encoder Representations from Transformers) reads the full sentence in both directions simultaneously. It was pre-trained on hundreds of millions of documents, so it already understands that "not great" is negative even though "great" is positive. Fine-tuning BERT for sentiment classification means training a classification head on top of the pre-trained model using your labelled data.
On real tweet benchmarks (like tweet_eval), fine-tuned BERT-based models typically score 5–15 macro F1 points higher than TF-IDF + Logistic Regression. The tradeoff: BERT requires a GPU for practical fine-tuning (training on CPU takes hours), the transformers library, and a comfortable level of deep learning knowledge. It is covered separately in the NLP deep dive materials. For the purposes of this project — learning the NLP workflow and evaluation methodology — TF-IDF + sklearn gives you everything you need.