Transformers Overview

TF-IDF is powerful, but it has a fundamental limitation: it treats words as independent tokens with no memory of their position or context. "Not good" and "good" differ by one word, yet TF-IDF treats them almost identically. Transformers solve this — they read entire sequences and build representations where every word is informed by every other word in the sentence. Understanding when that extra power is worth the cost is the skill this note builds.

Learning Objectives

• Articulate the core limitation of Bag of Words that transformers solve
• Develop intuition for word embeddings and why "similar words are close in vector space"
• Understand the self-attention mechanism at an intuition level (without the maths)
• Use HuggingFace transformers.pipeline() for inference in five lines of code
• Make an informed decision between classical NLP and transformers for a given scenario

---

The Limitation of Bag of Words

BoW and TF-IDF have three connected problems:

1. No word order

sentences = [
    "The dog bit the man",
    "The man bit the dog",
]

# To a CountVectorizer, these are identical — same words, same counts
from sklearn.feature_extraction.text import CountVectorizer
cv = CountVectorizer()
X = cv.fit_transform(sentences)
print(X.toarray())
# Output:
# [[1 1 1 1]   <- same
#  [1 1 1 1]]  <- same
# They are the same vector. But one is news; the other is a horror story.

2. No context

"Bank" means something different in "the river bank" versus "the bank approved my loan". TF-IDF assigns the same vector to both. A reader knows which is which from context — but a word-count model has no mechanism for this.

3. No semantic similarity

To TF-IDF, "good", "great", "excellent", and "superb" are completely unrelated tokens. They get separate columns in the matrix with no connection. A model must see each word in sufficient training examples to learn their shared meaning — which requires large amounts of labelled data.

---

Word Embeddings: Similar Words, Similar Vectors

Before transformers, the field moved from count vectors to dense embeddings — Word2Vec (2013), GloVe (2014). The core insight: train a neural network to predict a word from its neighbours. The hidden layer weights, after training, encode meaning. Similar words end up with similar weights.

# Conceptual illustration — not runnable without gensim installed
# pip install gensim

from gensim.downloader import load

# Download a small pre-trained model (~66MB)
# model = load("glove-wiki-gigaword-50")

# With a loaded model you can do:
# model.most_similar("king")
# → [('queen', 0.75), ('prince', 0.70), ('monarch', 0.68), ...]

# The famous arithmetic:
# model["king"] - model["man"] + model["woman"] ≈ model["queen"]
# This works because directions in embedding space encode semantic relationships.

# Similarity is measured as cosine similarity between vectors:
# cos(king, queen) ≈ 0.75  (similar)
# cos(king, pizza) ≈ 0.02  (unrelated)

Word embeddings encode semantic meaning, but they are still static — the word "bank" always has the same vector regardless of whether it appears next to "river" or "loan". Transformers introduced contextual embeddings: the representation of a word changes based on the words around it.

---

The Transformer Architecture: Intuition First

The transformer's key innovation is self-attention: a mechanism that lets each word look at every other word in the sentence and decide how much attention to pay to each.

Self-Attention in Plain Language

Imagine reading the sentence: "The trophy did not fit in the suitcase because it was too big."

When you process "it", you automatically look back at "trophy" and "suitcase" and reason that "it" refers to "trophy" because "big" fits a trophy better than a suitcase. That backward look — across the entire sentence — is what attention does.

In a transformer, when encoding the word "it", the model computes attention weights over every other word:

"The"     → low attention
"trophy"  → high attention (0.72)
"did"     → low attention
"not"     → medium attention
"fit"     → medium attention
"in"      → low attention
"the"     → low attention
"suitcase"→ medium attention (0.21)
"because" → low attention
"it"      → self-attention (0.04)
"was"     → low attention
"too"     → low attention
"big"     → medium attention (0.18)

The final representation of "it" is a weighted average of all words' representations, with high weight on "trophy". This is how the model resolves the reference.

BERT: Bidirectional Transformers

BERT (Bidirectional Encoder Representations from Transformers, Google 2018) is trained on two self-supervised tasks:

1. Masked Language Modelling: Randomly mask 15% of tokens, predict the masked words. Forces the model to understand context from both left and right.
2. Next Sentence Prediction: Given two sentences, predict whether the second follows the first. Teaches document-level coherence.

After pretraining on 3.3 billion words of text, BERT's representations are so rich that fine-tuning on a small labelled dataset for a downstream task (sentiment, NER, QA) achieves state-of-the-art results with a fraction of the data a model trained from scratch would need.

Info

BERT's paper is titled "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" (Devlin et al., 2018). The foundational architecture paper is "Attention Is All You Need" (Vaswani et al., 2017) — both are worth reading once you are comfortable with the basics. Attention Is All You Need

---

Using Transformers in Practice: HuggingFace

The HuggingFace transformers library makes running pre-trained models as simple as calling a function. The pipeline() abstraction handles tokenisation, model inference, and post-processing.

# pip install transformers torch
# First run downloads the model weights (~250MB for small models)

from transformers import pipeline

# Sentiment analysis using a pre-trained distilBERT model
sentiment_analyser = pipeline(
    task="sentiment-analysis",
    model="distilbert-base-uncased-finetuned-sst-2-english",
)

reviews = [
    "This product is absolutely fantastic. I love it.",
    "Terrible experience. Would not buy again.",
    "Not bad for the price, but could be better.",
    "I didn't not enjoy this — it was actually pretty good.",  # double negation
]

results = sentiment_analyser(reviews)

for review, result in zip(reviews, results):
    print(f"[{result['label']:8s} | {result['score']:.0%}] {review}")

# Output:
# [POSITIVE | 100%] This product is absolutely fantastic. I love it.
# [NEGATIVE |  99%] Terrible experience. Would not buy again.
# [NEGATIVE |  87%] Not bad for the price, but could be better.
# [POSITIVE |  76%] I didn't not enjoy this — it was actually pretty good.
# Note: The transformer correctly handles double negation — TF-IDF would likely fail here.

Other tasks with the same interface

from transformers import pipeline

# Named Entity Recognition
ner = pipeline("ner", grouped_entities=True)
text = "Apple CEO Tim Cook announced a new product in Cupertino last Tuesday."
entities = ner(text)
for entity in entities:
    print(f"{entity['word']:20s} → {entity['entity_group']} ({entity['score']:.0%})")
# Output:
# Apple                → ORG  (99%)
# Tim Cook             → PER  (99%)
# Cupertino            → LOC  (98%)
# last Tuesday         → DATE (82%)

# Zero-shot classification (no fine-tuning required)
classifier = pipeline("zero-shot-classification")
text = "The new laptop has exceptional battery life and a crisp display."
labels = ["electronics review", "food review", "travel review", "movie review"]
result = classifier(text, candidate_labels=labels)
print(result["labels"][0], f"{result['scores'][0]:.0%}")
# Output: electronics review 98%

# Text summarisation
summariser = pipeline("summarization", max_length=50, min_length=20)
long_text = """
Natural language processing is a subfield of linguistics, computer science, and artificial
intelligence concerned with the interactions between computers and human language, in particular
how to program computers to process and analyze large amounts of natural language data.
The goal is a computer capable of understanding the contents of documents, including the
contextual nuances of the language within them.
"""
summary = summariser(long_text)[0]["summary_text"]
print(summary)

---

When to Use Transformers vs Classical NLP

This decision comes up constantly in data science projects. Here is a principled framework:

Factor	Use TF-IDF + Classical ML	Use Transformers
Dataset size	< 50k labelled examples	> 50k, or with pre-trained weights
Task complexity	Single-label classification	Multi-label, NER, QA, generation
Compute budget	CPU inference, tight latency SLA	GPU available, latency flexible
Interpretability	Must explain individual predictions	Black box acceptable
Deployment	REST API, edge devices	Cloud inference, batch jobs
Development time	Hours to days	Days to weeks
Handles negation / sarcasm	Poorly	Well
Semantic similarity	No	Yes

The practitioner's decision tree

Is this a text classification problem?
│
├── Yes: How many labelled examples do you have?
│   │
│   ├── < 5,000 → TF-IDF + Logistic Regression / Naive Bayes
│   │   (More data points than model parameters. Transformers will overfit.)
│   │
│   ├── 5,000–50,000 → TF-IDF baseline first. If F1 < 80%, try fine-tuned BERT.
│   │
│   └── > 50,000 → Fine-tuned BERT or domain-specific model likely worth it.
│
└── Is this NER, QA, translation, or generation?
    └── Transformers — there is no classical equivalent for these tasks.

Tip

Always start with the TF-IDF baseline. It takes an hour to build and gives you a performance floor. If TF-IDF + Logistic Regression reaches 88% F1 and the business requirement is 85%, you are done — and you have saved weeks of engineering work on a transformer pipeline.

Warning

A fine-tuned BERT model trained on 200 labelled examples will likely underperform TF-IDF + Logistic Regression trained on the same 200 examples. Transformers have hundreds of millions of parameters and need either large fine-tuning datasets or careful few-shot prompting to reach their potential. "Transformer" does not mean "always better".

---

The Landscape: Major Models and Their Roles

Model	Organisation	Best for
BERT	Google	Classification, NER, extractive QA
DistilBERT	HuggingFace	Smaller/faster BERT, 97% of BERT's performance
RoBERTa	Meta	Classification, more robust than BERT
GPT-2 / GPT-3 / GPT-4	OpenAI	Text generation, zero-shot tasks
T5	Google	Any text-to-text task (translation, summarisation, QA)
sentence-transformers	HuggingFace	Semantic similarity, search, clustering

Success

For most text classification tasks in data science projects, the progression is: (1) TF-IDF + LR baseline, (2) fine-tuned DistilBERT if baseline is insufficient, (3) full BERT or RoBERTa if compute is available and the performance gap justifies it. Very rarely do you need to train from scratch — the pre-trained models are general enough that fine-tuning on your domain data almost always works.

---

---

What's Next

You've covered the attention mechanism and how it captures long-range word dependencies, BERT's masked language modelling pre-training and its bidirectional context, the HuggingFace pipeline API for zero-shot classification and sentiment, sentence-transformers for semantic similarity, and the decision framework for when to use TF-IDF versus transformers. Next up: 06-exercises — where you'll build a complete NLP pipeline from raw text to deployed classifier, applying preprocessing, TF-IDF, Logistic Regression, and transformer embeddings to real datasets under exam-style constraints.

Optional Deep Dive

Read the original "Attention Is All You Need" paper by Vaswani et al. (2017, available at arxiv.org/abs/1706.03762) — it is the foundational transformer paper and is surprisingly accessible. Section 3 (the model architecture) and Section 3.2.1 (scaled dot-product attention) give you the mathematical grounding for understanding why attention replaced recurrent networks for sequence modelling.

← Sentiment Classification | Next: Exercises →