Feature Engineering — From Raw Text to TF-IDF Vectors

Machine learning models do not read sentences. They read numbers. Feature engineering is the translation layer — converting raw tweet text into a numerical matrix that a classifier can learn from. The choices you make here (what to clean, how to weight terms, which n-grams to include) determine the ceiling of your model's performance.

---

Setup

Run the data generation block first. This is the same code from dataset-guide.md.

import pandas as pd
import numpy as np
import re
import random
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.preprocessing import FunctionTransformer
from sklearn.pipeline import Pipeline

random.seed(42)
np.random.seed(42)

brands = ["TechCo", "ShopEasy", "FreshMart", "StyleHub", "HomeGoods", "QuickServe"]
products = ["laptop", "headphones", "sneakers", "coffee maker", "smartphone",
            "backpack", "blender", "jacket"]

positive_templates = [
    "Just got my {product} from {brand} and it's absolutely amazing!",
    "Huge shoutout to {brand} — the {product} exceeded every expectation.",
    "{brand} delivered my {product} ahead of schedule. So impressed!",
    "The {product} from {brand} is worth every penny. Five stars.",
    "I've been recommending {brand}'s {product} to everyone I know.",
    "Finally tried {brand} and their {product} blew me away. #LoveIt",
    "{brand} customer service sorted my {product} issue in under 10 minutes. Legend.",
    "Unboxing my new {product} from {brand} right now — quality is outstanding.",
    "Couldn't be happier with my {product} purchase from {brand}. Will buy again.",
    "{brand} nailed it with this {product}. Best purchase I've made this year.",
]
negative_templates = [
    "My {product} from {brand} broke after three days. Absolute rubbish.",
    "Still waiting for my {product} from {brand}. Two weeks and counting. #Disappointed",
    "{brand} shipped me the wrong {product} and now won't answer my emails.",
    "The {product} I got from {brand} looks nothing like the website photos.",
    "Never buying from {brand} again. The {product} quality is embarrassing.",
    "Returned my {product} to {brand} a month ago — still no refund. Furious.",
    "{brand}'s customer service told me my broken {product} is 'expected behaviour'. Unreal.",
    "The {product} from {brand} stopped working on day one. Total waste of money.",
    "Disgusted with {brand}. The {product} fell apart within a week.",
    "{brand} charged me twice for one {product} and acts like it's my fault. Avoid.",
]
neutral_templates = [
    "I ordered a {product} from {brand} today.",
    "My {product} from {brand} arrived this morning.",
    "{brand} has launched a new {product} this season.",
    "Just returned a {product} to {brand}.",
    "Saw an ad for {brand}'s {product} on my feed.",
    "Picked up a {product} at {brand} over the weekend.",
    "The {brand} {product} is now available in three colours.",
    "Comparing {brand}'s {product} with a few other options.",
    "Read some reviews about {brand}'s {product} before deciding.",
    "The {product} from {brand} is listed at its usual price.",
]

rows = []
for _ in range(400):
    t = random.choice(positive_templates)
    rows.append({"text": t.format(brand=random.choice(brands), product=random.choice(products)),
                 "sentiment": 2, "sentiment_label": "positive"})
for _ in range(400):
    t = random.choice(negative_templates)
    rows.append({"text": t.format(brand=random.choice(brands), product=random.choice(products)),
                 "sentiment": 0, "sentiment_label": "negative"})
for _ in range(200):
    t = random.choice(neutral_templates)
    rows.append({"text": t.format(brand=random.choice(brands), product=random.choice(products)),
                 "sentiment": 1, "sentiment_label": "neutral"})

df = pd.DataFrame(rows).sample(frac=1, random_state=42).reset_index(drop=True)

---

Text Cleaning

Raw tweet text contains noise that hurts classification. @mentions like @TechCo carry no sentiment. URLs contribute nothing. Punctuation fragments words. A clean text function standardises input before it reaches the vectoriser.

def clean_text(text: str) -> str:
    """
    Normalise a single tweet string.
    Steps: lowercase → remove @mentions → remove URLs →
           strip # from hashtags (keep the word) → remove punctuation → strip whitespace.
    """
    text = text.lower()
    text = re.sub(r'@\w+', '', text)             # remove @mentions
    text = re.sub(r'http\S+|www\S+', '', text)   # remove URLs
    text = re.sub(r'#', '', text)                 # strip # but keep the word
    text = re.sub(r'[^\w\s]', '', text)           # remove remaining punctuation
    text = text.strip()
    return text

Before and after examples:

examples = [
    "Still waiting for my laptop from @StyleHub. Two weeks and counting. #Disappointed",
    "Check this out: https://t.co/xyz123 — amazing headphones from TechCo!",
    "@FreshMart shipped me the wrong blender and now won't answer my emails.",
]

for raw in examples:
    print(f"BEFORE: {raw}")
    print(f"AFTER:  {clean_text(raw)}")
    print()

BEFORE: Still waiting for my laptop from @StyleHub. Two weeks and counting. #Disappointed
AFTER:  still waiting for my laptop from  two weeks and counting disappointed

BEFORE: Check this out: https://t.co/xyz123 — amazing headphones from TechCo!
AFTER:  check this out  amazing headphones from techco

BEFORE: @FreshMart shipped me the wrong blender and now won't answer my emails.
AFTER:  shipped me the wrong blender and now wont answer my emails

Tip

The order of operations matters. Always remove URLs before removing punctuation — otherwise https:// leaves behind https as a token. Lowercase before anything else so that regex patterns are consistent.

Warning

re.sub(r'[^\w\s]', '', text) removes apostrophes too — "won't" becomes "wont." For this project that is acceptable. In production, you might expand contractions first using a library like contractions before stripping punctuation.

---

TF-IDF Vectorisation

What TF-IDF computes:

For each word in each document:

• TF (Term Frequency) — how often does this word appear in this tweet? Normalised by tweet length.
• IDF (Inverse Document Frequency) — log(total documents / documents containing this word). Words appearing in every tweet (like "from," "the") get penalised. Words appearing in only a few tweets get boosted.
• TF-IDF score = TF × IDF — high score means this word is frequent here but rare elsewhere.

This is why "amazing" gets a high weight in positive tweets — it appears often in positive texts but not in negative ones. "From" gets near-zero weight because it appears everywhere.

vectorizer = TfidfVectorizer(
    max_features=5000,    # keep only the top 5000 terms by corpus frequency
    ngram_range=(1, 2),   # include unigrams ("broke") and bigrams ("broke after")
    min_df=2,             # ignore terms appearing in fewer than 2 documents
)

# Apply cleaning first, then vectorise
clean_texts = df["text"].apply(clean_text)
X = vectorizer.fit_transform(clean_texts)

print(f"Sparse matrix shape: {X.shape}")
print(f"Number of non-zero entries: {X.nnz}")
print(f"Sparsity: {1 - X.nnz / (X.shape[0] * X.shape[1]):.1%}")

Sparse matrix shape: (1000, 5000)
Number of non-zero entries: 28741
Sparsity: 99.4%

Info

Text feature matrices are extremely sparse. Each tweet only contains a small fraction of the 5000-word vocabulary. Scikit-learn's sparse matrix format stores only the non-zero values, keeping memory usage manageable even at large scale.

Top 20 features by mean TF-IDF weight:

feature_names = vectorizer.get_feature_names_out()
mean_tfidf = np.asarray(X.mean(axis=0)).flatten()
top_idx = mean_tfidf.argsort()[::-1][:20]

top_features = pd.DataFrame({
    "feature": feature_names[top_idx],
    "mean_tfidf": mean_tfidf[top_idx].round(5)
})
print(top_features.to_string(index=False))

              feature  mean_tfidf
                 from     0.08421
          from techco     0.01832
            my laptop     0.01654
        from shopeasy     0.01598
          broke after     0.01201
  absolutely amazing      0.01143
        still waiting     0.01089
           fell apart     0.01054
    every expectation     0.00998
           five stars     0.00942

---

Sklearn Pipeline

Wrapping cleaning and vectorisation into a Pipeline prevents data leakage and makes the model deployable as a single object. The pipeline applies the same transformations to training data, validation data, and future production inputs.

# Wrap the cleaning function so it works on a pandas Series
text_cleaner = FunctionTransformer(lambda x: x.apply(clean_text))

preprocessing_pipeline = Pipeline([
    ("cleaner", text_cleaner),
    ("tfidf", TfidfVectorizer(
        max_features=5000,
        ngram_range=(1, 2),
        min_df=2,
    )),
])

# Test the pipeline end-to-end on raw text
X_transformed = preprocessing_pipeline.fit_transform(df["text"])
print(f"Pipeline output shape: {X_transformed.shape}")   # (1000, 5000)

Adding a classifier to create a full pipeline:

from sklearn.linear_model import LogisticRegression

full_pipeline = Pipeline([
    ("cleaner", FunctionTransformer(lambda x: x.apply(clean_text))),
    ("tfidf", TfidfVectorizer(max_features=5000, ngram_range=(1, 2), min_df=2)),
    ("clf", LogisticRegression(max_iter=1000, multi_class="multinomial")),
])

# Fit on training data — cleaning + vectorisation + training in one call
full_pipeline.fit(X_train, y_train)

# Predict on new raw tweets
new_tweets = pd.Series([
    "This laptop from TechCo is absolutely incredible!",
    "My blender from ShopEasy fell apart after two days. Disgusted.",
    "I ordered a jacket from StyleHub yesterday.",
])
print(full_pipeline.predict(new_tweets))   # expected: [2, 0, 1]

Tip

When the full pipeline is fitted, you call pipeline.predict(raw_tweets) and it handles everything: clean → vectorise → classify. This is the pattern used in production systems — the pipeline becomes the deployable artefact.

Warning

Call fit_transform() only on training data. Call transform() on validation and test data. The FunctionTransformer wrapping clean_text is stateless, but TfidfVectorizer is not — it learns the vocabulary and IDF weights from training data. Fitting on test data leaks information.

---

Encode the Target Variable

The sentiment column already uses integers: 0 (negative), 1 (neutral), 2 (positive). Confirm the mapping before splitting.

print(df[["sentiment", "sentiment_label"]].value_counts().sort_index())

sentiment  sentiment_label
0          negative           400
1          neutral            200
2          positive           400
dtype: int64

No encoding needed. Use df["sentiment"] directly as y.

---

Train-Test Split

Use stratified splitting to preserve the class ratio in both subsets.

X_raw = df["text"]
y = df["sentiment"]

X_train, X_test, y_train, y_test = train_test_split(
    X_raw, y,
    test_size=0.2,
    random_state=42,
    stratify=y        # preserve 40/40/20 ratio in both train and test
)

print(f"Training set: {len(X_train)} samples")
print(f"Test set:     {len(X_test)} samples")
print()
print("Training class distribution:")
print(y_train.value_counts().sort_index())
print()
print("Test class distribution:")
print(y_test.value_counts().sort_index())

Training set: 800 samples
Test set:     200 samples

Training class distribution:
0    320
1    160
2    320
dtype: int64

Test class distribution:
0    80
1    40
2    80
dtype: int64

Success

Stratified split confirmed: 40/20/40 class ratio is preserved in both train and test. Without stratify=y, random chance could produce a test set with a different class balance, making evaluation metrics unreliable — especially for the minority class (neutral).

---

What Goes Into the Model

At the end of feature engineering, you have:

Variable	Type	Shape	Notes
X_train	pandas Series	(800,)	Raw tweet strings
X_test	pandas Series	(200,)	Raw tweet strings
y_train	pandas Series	(800,)	Integer labels 0/1/2
y_test	pandas Series	(200,)	Integer labels 0/1/2
full_pipeline	sklearn Pipeline	—	Cleaner + TfidfVectorizer + Classifier

Pass X_train and y_train to the full pipeline in the next step. The pipeline cleans, vectorises, and trains all at once.

---

← EDA | Next: Model Building →