Exercises: NLP Basics¶
These exercises follow the full pipeline from raw text to deployed classifier. Work through them in order — each exercise builds on the previous one. By the end you will have built, evaluated, and interpreted a sentiment classifier from scratch.
Dataset: A realistic set of product reviews provided inline. All exercises use it.
Warm-Up Exercises¶
These exercises check your understanding of individual concepts before combining them.
Warm-Up 1: Preprocessing Pipeline¶
Write a function clean_text(text: str) -> str that:
- Lowercases the text
- Removes punctuation and digits (keep only letters and spaces)
- Strips leading/trailing whitespace
- Collapses multiple spaces into one
Test it on these inputs:
tests = [
"This product is AMAZING!!! 10/10 would buy again.",
" Terrible quality. 1-star. Do NOT recommend. ",
"Check https://shop.com — best deals #sale #tech",
]
Expected behaviour: URLs remain as broken tokens (that is fine for this exercise), punctuation is gone, everything is lowercase.
Show answer
import re
def clean_text(text: str) -> str:
text = text.lower()
text = re.sub(r"[^a-z\s]", "", text)
text = re.sub(r"\s+", " ", text).strip()
return text
tests = [
"This product is AMAZING!!! 10/10 would buy again.",
" Terrible quality. 1-star. Do NOT recommend. ",
"Check https://shop.com — best deals #sale #tech",
]
for t in tests:
print(repr(clean_text(t)))
# Output:
# 'this product is amazing would buy again'
# 'terrible quality star do not recommend'
# 'check httpsshopcom best deals sale tech'
Warm-Up 2: Stopword Audit¶
Run the code below to inspect NLTK's English stopword list. Answer the questions after:
import nltk
nltk.download("stopwords", quiet=True)
from nltk.corpus import stopwords
stop_words = set(stopwords.words("english"))
print(f"Total stopwords: {len(stop_words)}")
# Check for negation words
negation_words = ["not", "no", "never", "neither", "nor", "nothing",
"nobody", "nowhere", "cannot"]
for word in negation_words:
status = "IN" if word in stop_words else "NOT IN"
print(f" '{word}' is {status} the stopword list")
Questions to answer (write your answers before checking):
- How many of the negation words are in the NLTK stopword list?
- Why does this matter specifically for sentiment classification?
- What would you do differently if your task were topic classification (sports vs politics) rather than sentiment?
Show answer
Total stopwords: 179
'not' is IN the stopword list
'no' is IN the stopword list
'never' is IN the stopword list
'neither' is IN the stopword list
'nor' is IN the stopword list
'nothing' is IN the stopword list
'nobody' is IN the stopword list
'nowhere' is IN the stopword list
'cannot' is NOT IN the stopword list (it appears as "can" + "not" after splitting)
Q1: Most negation words (not, no, never, neither, nor, nothing, nobody, nowhere) ARE in the NLTK stopword list.
Q2: For sentiment classification, removing "not" changes the meaning entirely. "not good" becomes "good" after stopword removal — the model learns the wrong label. This is a silent failure: the code runs without errors but the model learns corrupted data.
Q3: For topic classification, negation words carry almost no topic signal. Whether an article says "the team did not win" or "the team won" — it is still a sports article. For topic tasks, removing all stopwords including negations is usually fine.
Warm-Up 3: Stemming vs Lemmatisation Comparison¶
Apply both stemming (PorterStemmer) and lemmatisation (WordNetLemmatizer) to the words below. Note where they agree and where they differ.
words = ["running", "ran", "runner", "studies", "studying", "studied",
"better", "worst", "happily", "happiness", "unhappy"]
For each word, output: original | stemmed | lemmatised
Show answer
import nltk
nltk.download("wordnet", quiet=True)
nltk.download("omw-1.4", quiet=True)
from nltk.stem import PorterStemmer, WordNetLemmatizer
stemmer = PorterStemmer()
lemmatizer = WordNetLemmatizer()
words = ["running", "ran", "runner", "studies", "studying", "studied",
"better", "worst", "happily", "happiness", "unhappy"]
print(f"{'Original':15s} {'Stemmed':15s} {'Lemmatised (v)':15s} {'Lemmatised (a)':15s}")
print("-" * 65)
for word in words:
stemmed = stemmer.stem(word)
lemma_v = lemmatizer.lemmatize(word, pos="v")
lemma_a = lemmatizer.lemmatize(word, pos="a")
print(f"{word:15s} {stemmed:15s} {lemma_v:15s} {lemma_a:15s}")
# Output:
# Original Stemmed Lemmatised (v) Lemmatised (a)
# -----------------------------------------------------------------
# running run run running
# ran ran run ran <- stemmer misses this
# runner runner runner runner
# studies studi study studies
# studying studi study studying
# studied studi study studied
# better better better good <- lemmatiser handles comparative
# worst worst worst bad <- lemmatiser handles superlative
# happily happili happily happily
# happiness happi happiness happiness
# unhappy unhappi unhappy unhappy
Key observations:
- Stemmer correctly groups "running/studying/studies" but produces non-words ("studi", "happili")
- Lemmatiser handles irregular forms ("ran" → "run" with pos="v", "better" → "good" with pos="a")
- Lemmatiser requires the correct POS tag to work well — wrong POS gives poor results
Main Exercises¶
Main Exercise 1: N-gram Comparison Experiment¶
Use the product reviews dataset from 04-sentiment-classification.md. Train three separate TF-IDF + Logistic Regression classifiers with different n-gram ranges:
- Model A: unigrams only (1, 1)
- Model B: bigrams only (2, 2)
- Model C: unigrams + bigrams (1, 2)
For each model, report the F1 score on the test set and list the top 5 positive and top 5 negative features.
Which n-gram range performs best? Why do you think that is?
Show answer
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import f1_score
import numpy as np
positive_reviews = [
"This product exceeded all my expectations. Highly recommend.",
"Excellent quality, fast shipping, and great customer service.",
"I've been using this for three months and it still works perfectly.",
"Solid build quality. Worth every penny.",
"Amazing product. Bought two more for my family.",
"Works exactly as described. Very happy with my purchase.",
"Surprisingly good for the price. Will definitely buy again.",
"Five stars. Does exactly what it says on the tin.",
"Best purchase I've made all year. Cannot recommend enough.",
"Outstanding quality and arrived ahead of schedule.",
"Great product, very easy to use, works perfectly.",
"Very satisfied. The quality is much better than expected.",
"Brilliant. Exactly what I needed. Fast delivery too.",
"Good value for money. Would recommend to anyone.",
"Lovely product, well packaged, no issues at all.",
]
negative_reviews = [
"Broke after two weeks. Complete waste of money.",
"Not what was advertised. Very disappointed with the quality.",
"Terrible product. Stopped working after three days.",
"Would not recommend. Poor quality and slow delivery.",
"This is not worth the price. Extremely disappointed.",
"Arrived damaged and customer service was unhelpful.",
"Do not buy this. It broke the first time I used it.",
"Cheap material, poor construction. A total disappointment.",
"Took six weeks to arrive and was completely useless.",
"One star. This is the worst product I have ever bought.",
"Nothing worked out of the box. Had to return immediately.",
"Packaging was fine but the product itself is garbage.",
"Misleading description. This is not what I ordered.",
"Very poor quality. Fell apart within a week.",
"Avoid this seller. Product is a cheap knockoff.",
]
df = pd.DataFrame({
"review": positive_reviews + negative_reviews,
"sentiment": [1] * 15 + [0] * 15,
})
X_train, X_test, y_train, y_test = train_test_split(
df["review"], df["sentiment"],
test_size=0.25, random_state=42, stratify=df["sentiment"]
)
configs = {
"Unigrams (1,1)": (1, 1),
"Bigrams (2,2)": (2, 2),
"Uni+Bi (1,2)": (1, 2),
}
for name, ngram_range in configs.items():
pipe = Pipeline([
("tfidf", TfidfVectorizer(ngram_range=ngram_range, max_features=5000,
sublinear_tf=True, stop_words=None)),
("model", LogisticRegression(C=1.0, max_iter=1000, random_state=42)),
])
pipe.fit(X_train, y_train)
y_pred = pipe.predict(X_test)
f1 = f1_score(y_test, y_pred, average="macro")
feature_names = pipe.named_steps["tfidf"].get_feature_names_out()
coefs = pipe.named_steps["model"].coef_[0]
sorted_idx = np.argsort(coefs)
print(f"\n{name} | F1={f1:.3f}")
print(f" Top 5 negative: {list(feature_names[sorted_idx[:5]])}")
print(f" Top 5 positive: {list(feature_names[sorted_idx[-5:][::-1]])}")
# Expected pattern:
# - Unigrams alone: decent F1 (~0.80-0.87)
# - Bigrams alone: lower F1 on small dataset (too sparse, few bigrams seen in training)
# - Unigrams + Bigrams: best F1 — captures both word and phrase signals
The unigrams + bigrams model typically wins because bigrams like "highly recommend", "not worth", and "broke after" carry strong sentiment that single words cannot capture. Pure bigrams underperform on small datasets because most bigrams only appear once or twice — not enough signal to learn from.
Main Exercise 2: Build a Full Sentiment Classifier with Evaluation¶
Using the same product reviews dataset, build a production-patterned classifier that:
- Splits data with stratification
- Uses
Pipeline(no leakage) - Reports precision, recall, F1 for each class
- Prints the confusion matrix (labelled)
- Predicts sentiment on three new unseen reviews you write yourself
- Prints the top 10 most positive and top 10 most negative words
Show answer
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, confusion_matrix
# Dataset (same as above — use both lists from the previous exercise)
df = pd.DataFrame({
"review": positive_reviews + negative_reviews,
"sentiment": [1] * 15 + [0] * 15,
})
X_train, X_test, y_train, y_test = train_test_split(
df["review"], df["sentiment"],
test_size=0.25, random_state=42, stratify=df["sentiment"]
)
pipe = Pipeline([
("tfidf", TfidfVectorizer(
ngram_range=(1, 2),
max_features=5000,
sublinear_tf=True,
)),
("model", LogisticRegression(C=1.0, max_iter=1000, random_state=42)),
])
pipe.fit(X_train, y_train)
y_pred = pipe.predict(X_test)
# --- Evaluation ---
print(classification_report(y_test, y_pred, target_names=["Negative", "Positive"]))
cm = confusion_matrix(y_test, y_pred)
print("Confusion Matrix:")
print(f" Predicted Neg Predicted Pos")
print(f"Actual Negative {cm[0,0]:5d} {cm[0,1]:5d}")
print(f"Actual Positive {cm[1,0]:5d} {cm[1,1]:5d}")
# --- New predictions ---
new_reviews = [
"This exceeded every expectation. Genuinely impressed.",
"Complete junk. Broke on first use. Avoid at all costs.",
"It is okay. Not great, not terrible. Does the job.",
]
preds = pipe.predict(new_reviews)
probs = pipe.predict_proba(new_reviews)
print("\nNew Review Predictions:")
for review, pred, prob in zip(new_reviews, preds, probs):
label = "Positive" if pred == 1 else "Negative"
conf = max(prob)
print(f" [{label} | {conf:.0%}] {review[:60]}...")
# --- Top words ---
features = pipe.named_steps["tfidf"].get_feature_names_out()
coefs = pipe.named_steps["model"].coef_[0]
idx = np.argsort(coefs)
print("\nTop 10 Negative Words/Phrases:")
for i in idx[:10]:
print(f" {features[i]:35s} {coefs[i]:+.3f}")
print("\nTop 10 Positive Words/Phrases:")
for i in idx[-10:][::-1]:
print(f" {features[i]:35s} {coefs[i]:+.3f}")
Main Exercise 3: Threshold Tuning¶
A product marketplace wants to automatically flag reviews as "potentially negative" for human review before publication. They have decided that: - Missing a negative review (false negative) costs 5x more than incorrectly flagging a positive review (false positive) - They want to minimise total cost
Using the trained classifier from Exercise 2: 1. Get the predicted probabilities for the test set 2. Evaluate precision and recall for the negative class at thresholds: 0.2, 0.3, 0.4, 0.5, 0.6 3. Calculate total cost at each threshold (assume 1 false negative = 5 units, 1 false positive = 1 unit) 4. Which threshold minimises total cost?
Show answer
from sklearn.metrics import confusion_matrix
import numpy as np
# Get probabilities for the NEGATIVE class (class 0)
y_scores_negative = pipe.predict_proba(X_test)[:, 0]
# A "positive" flag means "this review is negative" — our positive class for this task
# We are predicting: is this review negative? Yes/No
# y_test_neg: 1 if the actual review is negative, 0 if positive
y_test_array = np.array(y_test)
y_test_neg = (y_test_array == 0).astype(int) # 1 = negative review
FN_COST = 5 # cost of missing a negative review
FP_COST = 1 # cost of flagging a positive review
print(f"{'Threshold':>10} {'Precision':>10} {'Recall':>10} {'FP':>5} {'FN':>5} {'Cost':>8}")
print("-" * 55)
for threshold in [0.2, 0.3, 0.4, 0.5, 0.6]:
y_pred_neg = (y_scores_negative >= threshold).astype(int)
cm = confusion_matrix(y_test_neg, y_pred_neg)
tn, fp, fn, tp = cm.ravel() if cm.size == 4 else (0, 0, 0, cm[0,0])
precision = tp / (tp + fp) if (tp + fp) > 0 else 0
recall = tp / (tp + fn) if (tp + fn) > 0 else 0
cost = fn * FN_COST + fp * FP_COST
print(f"{threshold:>10.1f} {precision:>10.3f} {recall:>10.3f} {fp:>5d} {fn:>5d} {cost:>8d}")
# Expected output (approximate):
# Threshold Precision Recall FP FN Cost
# -------------------------------------------------------
# 0.2 0.571 1.000 3 0 3
# 0.3 0.714 1.000 2 0 2
# 0.4 0.875 0.875 1 1 6
# 0.5 0.875 0.875 1 1 6
# 0.6 1.000 0.750 0 2 10
# Threshold 0.3 minimises total cost: catches all negatives (recall=1.0)
# with only 2 false positives, giving total cost = 2.
# At threshold 0.4, one missed negative (FN=1) costs 5 — worse than flagging 3 extra.
The key insight: when false negatives are more costly than false positives, lower your threshold. The "right" threshold is not 0.5 — it depends on the cost asymmetry of your specific problem. Always define this with stakeholders before training.
Stretch Exercises¶
Stretch 1: Cross-Validation for Text Classifiers¶
The train/test split on 30 examples is unreliable — results vary significantly by random seed. Use cross_val_score with 5-fold cross-validation to get a more stable F1 estimate for both the Logistic Regression and Naive Bayes classifiers.
Which model is more consistent across folds? What does high variance across folds tell you about the dataset?
Show answer
from sklearn.model_selection import cross_val_score, StratifiedKFold
from sklearn.naive_bayes import MultinomialNB
import numpy as np
X_all = df["review"]
y_all = df["sentiment"]
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
lr_pipe = Pipeline([
("tfidf", TfidfVectorizer(ngram_range=(1, 2), max_features=5000, sublinear_tf=True)),
("model", LogisticRegression(C=1.0, max_iter=1000, random_state=42)),
])
nb_pipe = Pipeline([
("tfidf", TfidfVectorizer(ngram_range=(1, 2), max_features=5000, sublinear_tf=False)),
("model", MultinomialNB(alpha=1.0)),
])
for name, pipe in [("Logistic Regression", lr_pipe), ("Naive Bayes", nb_pipe)]:
scores = cross_val_score(pipe, X_all, y_all, cv=cv, scoring="f1_macro")
print(f"{name}:")
print(f" Fold scores: {scores.round(3)}")
print(f" Mean F1: {scores.mean():.3f} ± {scores.std():.3f}")
# High variance across folds (e.g., ±0.15) indicates the dataset is too small
# for reliable evaluation. The model's apparent performance depends heavily
# on which examples land in the test fold. This is why 30 examples is a toy —
# real classifiers need at least 500–1000 labelled examples per class.
Stretch 2: Feature Importance with a Heatmap¶
For the TF-IDF + Logistic Regression model trained on the full dataset (no split), extract the top 15 positive and top 15 negative features and visualise them as a horizontal bar chart using matplotlib. Colour positive features green and negative features red.
Show answer
import matplotlib.pyplot as plt
import numpy as np
pipe_full = Pipeline([
("tfidf", TfidfVectorizer(ngram_range=(1, 2), max_features=5000, sublinear_tf=True)),
("model", LogisticRegression(C=1.0, max_iter=1000, random_state=42)),
])
pipe_full.fit(df["review"], df["sentiment"])
features = pipe_full.named_steps["tfidf"].get_feature_names_out()
coefs = pipe_full.named_steps["model"].coef_[0]
idx = np.argsort(coefs)
# Top 15 negative and positive
top_neg = [(features[i], coefs[i]) for i in idx[:15]]
top_pos = [(features[i], coefs[i]) for i in idx[-15:][::-1]]
all_features = top_neg + top_pos[::-1] # negative at bottom, positive at top
feature_labels = [f[0] for f in all_features]
feature_coefs = [f[1] for f in all_features]
colors = ["#ef4444" if c < 0 else "#22c55e" for c in feature_coefs]
fig, ax = plt.subplots(figsize=(10, 9))
bars = ax.barh(range(len(feature_labels)), feature_coefs, color=colors, edgecolor="none")
ax.set_yticks(range(len(feature_labels)))
ax.set_yticklabels(feature_labels, fontsize=9)
ax.axvline(0, color="#6b7280", linewidth=0.8)
ax.set_xlabel("Logistic Regression Coefficient")
ax.set_title("Top Features: Sentiment Classifier\n(green = positive, red = negative)")
plt.tight_layout()
plt.savefig("sentiment_features.png", dpi=150, bbox_inches="tight")
plt.show()
print("Saved to sentiment_features.png")
Stretch 3: HuggingFace vs TF-IDF Comparison¶
If you have transformers and torch installed, compare the HuggingFace distilbert-base-uncased-finetuned-sst-2-english pipeline against your TF-IDF + Logistic Regression classifier on the test set. Report F1 for both.
Write a short analysis (4–6 sentences) addressing: - Which model performs better on this dataset? - Why might the transformer struggle or excel on this small dataset? - What would you need to change to make the comparison fair on a larger dataset?
Show answer
# Note: requires: pip install transformers torch
# First run downloads ~250MB of model weights
from transformers import pipeline as hf_pipeline
from sklearn.metrics import f1_score
# HuggingFace sentiment pipeline
hf_sentiment = hf_pipeline(
"sentiment-analysis",
model="distilbert-base-uncased-finetuned-sst-2-english",
truncation=True,
)
# Predict on test set
hf_results = hf_sentiment(list(X_test))
hf_preds = [1 if r["label"] == "POSITIVE" else 0 for r in hf_results]
hf_f1 = f1_score(y_test, hf_preds, average="macro")
lr_f1 = f1_score(y_test, y_pred, average="macro") # from Exercise 2
print(f"TF-IDF + Logistic Regression F1: {lr_f1:.3f}")
print(f"DistilBERT (pre-trained) F1: {hf_f1:.3f}")
# Analysis:
# On this 30-example dataset, TF-IDF + LR often matches or beats DistilBERT.
# Why? DistilBERT was fine-tuned on SST-2 (movie reviews), not product reviews.
# The domain shift (movie sentiment → product sentiment) introduces error.
# The TF-IDF model was trained directly on product review vocabulary, giving it
# a domain advantage. To make the comparison fair on a larger dataset, you would
# fine-tune DistilBERT on your product review data for 3–5 epochs. That would
# leverage both the pre-trained language knowledge AND your domain vocabulary.
Interview Questions¶
Q1: What is the difference between stemming and lemmatisation? When would you choose each?
Show answer
Stemming chops word endings using heuristic rules (PorterStemmer) — fast but produces non-words ("studies" → "studi"). Lemmatisation uses a vocabulary and morphological analysis to return the actual dictionary base form ("studies" → "study"). Choose stemming when speed matters and the task is tolerant of imprecise normalisation (large-scale search indexing). Choose lemmatisation when accuracy matters, the vocabulary is smaller, and you need the output to be human-readable (named entity systems, information extraction). In practice, for TF-IDF-based classifiers, the performance difference is often negligible — run a quick experiment before adding the complexity.
Q2: Why does removing stopwords hurt sentiment classification?
Show answer
Because several stopwords are also negation words. NLTK's English stopword list includes "not", "no", "never", "neither", "nor", "nothing", "nobody". Removing "not" from "not good" produces "good" — the model trains on a sample that says positive things but has a negative label. This is a silent data corruption: the code runs without errors, but the model learns from reversed signal. The fix is to subtract negation words from your stopword list before applying it to sentiment data.
Q3: Explain TF-IDF in plain language. Why does it outperform raw word counts?
Show answer
TF-IDF scores each word in each document by multiplying two quantities: how often the word appears in this document (Term Frequency), and how rare the word is across all documents (Inverse Document Frequency). Words that appear frequently in one document but rarely elsewhere get high scores — they are distinctive. Words that appear everywhere ("the", "is", "and") get near-zero IDF scores and are effectively ignored. Raw counts fail because they reward words that appear often everywhere, not words that are meaningfully distinctive. TF-IDF solves both the frequency bias and the document-length normalisation problem that raw counts suffer from.
Q4: When should you use a transformer instead of TF-IDF + Logistic Regression?
Show answer
Use a transformer when: (1) you have more than ~50,000 labelled examples and your TF-IDF baseline has plateaued below the required performance, (2) your task requires semantic understanding that bag-of-words cannot capture (paraphrase detection, semantic search, complex QA), (3) you need to handle nuanced language like sarcasm, double negation, or coreference resolution, or (4) you need a generation or sequence-to-sequence output (translation, summarisation). Stick with TF-IDF + LR when: your dataset is small (< 5k examples), latency and compute are constrained, you need to interpret model predictions to stakeholders, or your baseline already meets the performance requirement. A transformer is not always better — it is better when the complexity it brings is justified by the problem.
Q5: What is data leakage in the context of text vectorisation, and how does a sklearn Pipeline prevent it?
Show answer
Data leakage in text vectorisation occurs when the TF-IDF vectoriser is fitted on the full dataset (train + test combined) before the train/test split. During fitting, the vectoriser computes IDF weights for every word, including words that only appear in the test set. Those IDF weights influence the feature matrix that the model trains on, so the model indirectly benefits from information about the test data — inflating evaluation metrics. A sklearn Pipeline prevents this by design: pipe.fit(X_train, y_train) calls fit_transform() on all intermediate steps (the vectoriser) using only X_train. When pipe.predict(X_test) is called, the vectoriser uses transform() — applying the vocabulary and IDF weights it learned from training data only, never updating them with test data.