Classical NLP¶
Before transformers, NLP was built on sparse representations and hand-crafted linguistic features. These techniques still appear in production systems where speed, interpretability, or data scarcity makes neural models impractical. Interviewers test whether you understand the foundations that modern methods were built to improve upon.
Q1: What is Bag of Words, and what information does it lose?¶
Show answer
Bag of Words (BoW) represents a document as a vector of word counts, with one dimension per vocabulary term. The order of words is discarded — only which words appear, and how many times, is recorded.
from sklearn.feature_extraction.text import CountVectorizer
corpus = [
"the dog bit the man",
"the man bit the dog"
]
vectorizer = CountVectorizer()
X = vectorizer.fit_transform(corpus)
print(vectorizer.get_feature_names_out())
# ['bit', 'dog', 'man', 'the']
print(X.toarray())
# [[1 1 1 2]
# [1 1 1 2]]
# Both sentences produce identical vectors
What BoW loses: - Word order — "dog bit man" and "man bit dog" are identical vectors - Syntax — no subject/object structure is preserved - Semantics — "car" and "automobile" are completely unrelated in BoW space - Negation — "not good" and "good" differ by one stop word that is often removed
Despite these limitations, BoW with good preprocessing performs surprisingly well for topic classification and spam detection where content words are the dominant signal.
Q2: What is TF-IDF? Explain the formula and the intuition behind it.¶
Show answer
TF-IDF (Term Frequency-Inverse Document Frequency) weights each word by how frequent it is in the current document relative to how common it is across all documents.
Formula:
TF(t, d) = count(t in d) / total words in d
IDF(t) = log(N / df(t)) where N = total documents, df(t) = documents containing t
TF-IDF(t, d) = TF(t, d) × IDF(t)
Intuition: - A word that appears frequently in one document but rarely across the corpus is highly distinctive for that document → high TF-IDF - A word that appears frequently everywhere (like "the") gets a low IDF, which suppresses its weight regardless of its term frequency → low TF-IDF - This is a statistical approximation of "what makes this document unique"
from sklearn.feature_extraction.text import TfidfVectorizer
import pandas as pd
corpus = [
"machine learning is great",
"deep learning uses neural networks",
"machine learning and deep learning"
]
vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(corpus)
df = pd.DataFrame(X.toarray(), columns=vectorizer.get_feature_names_out())
print(df.round(3))
# "learning" appears in all 3 docs → lower IDF → lower weight than "neural"
Why TF-IDF beats raw counts: "the" appears 1000 times in a document but occurs in every document — raw count makes it look important. IDF kills its weight. "photosynthesis" appears 5 times but only in biology documents — IDF boosts its weight.
Q3: What are n-grams, and when do higher-order n-grams help?¶
Show answer
An n-gram is a contiguous sequence of N tokens. In classical NLP, including bigrams and trigrams in the feature space lets BoW and TF-IDF capture local context.
- Unigrams: ["machine", "learning", "is", "powerful"]
- Bigrams: ["machine learning", "learning is", "is powerful"]
- Trigrams: ["machine learning is", "learning is powerful"]
When higher-order n-grams help: - Phrases with fixed meaning: "New York", "not good", "machine learning" - Negation: "not happy" vs "happy" — a bigram captures the negation that unigrams miss - Domain-specific jargon: medical, legal, and technical texts have meaningful multi-word terms
Tradeoffs: - Vocabulary grows combinatorially: from 10K unigrams to potentially 100K+ with bigrams - Sparsity increases — most bigrams appear in very few documents - In practice, (1,2) range (unigrams + bigrams) is the sweet spot for most classification tasks
from sklearn.feature_extraction.text import TfidfVectorizer
corpus = ["not a good product", "a very good product"]
v = TfidfVectorizer(ngram_range=(1, 2))
v.fit(corpus)
print([f for f in v.get_feature_names_out() if 'good' in f])
# ['good', 'good product', 'not good']
# "not good" is now a separable feature
Q4: Why is cosine similarity used for text vectors instead of Euclidean distance?¶
Show answer
Text vectors from BoW or TF-IDF are high-dimensional and sparse. Euclidean distance has two problems in this setting:
- Length sensitivity: a long document and a short document about the same topic will have very different vector magnitudes. A long document with many word repetitions will appear farther from a short document on the same topic than from an unrelated document of similar length.
- Curse of dimensionality: in very high-dimensional spaces, Euclidean distances between random points converge — everything appears "equally far" from everything else.
Cosine similarity measures the angle between two vectors, not the distance between their endpoints. It is length-invariant — normalising by vector magnitude removes the document length effect.
cosine_similarity(A, B) = (A · B) / (||A|| × ||B||)
Range: [-1, 1], but for TF-IDF vectors (non-negative): [0, 1]
from sklearn.metrics.pairwise import cosine_similarity
from sklearn.feature_extraction.text import TfidfVectorizer
docs = [
"machine learning is fascinating",
"machine learning machine learning machine learning", # same topic, just longer
"cooking recipes for dinner"
]
tfidf = TfidfVectorizer()
X = tfidf.fit_transform(docs)
sims = cosine_similarity(X[0], X[1:])
print(sims)
# Doc 1 vs Doc 2 (same topic, different length): high similarity
# Doc 1 vs Doc 3 (different topic): low similarity
Q5: Why does Naive Bayes work well for text classification despite the independence assumption?¶
Show answer
Naive Bayes classifies text by computing:
The "naive" assumption is that words are conditionally independent given the class — knowing "machine" appears tells us nothing new about whether "learning" appears, given we already know the document is about ML. This is obviously false.
Why it works anyway: - For classification (not probability estimation), you only need the ranking of class probabilities to be correct — even badly calibrated probabilities can produce the right argmax - Word presence patterns tend to cluster by topic even with false independence - The model is extremely fast to train and works well with small datasets
Multinomial vs Bernoulli Naive Bayes: - Multinomial NB: uses word counts or TF-IDF weights — appropriate when word frequency matters (spam detection, topic classification) - Bernoulli NB: uses binary word presence/absence — appropriate for short texts (tweet classification)
from sklearn.naive_bayes import MultinomialNB
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.pipeline import Pipeline
pipeline = Pipeline([
('tfidf', TfidfVectorizer()),
('clf', MultinomialNB())
])
train_texts = ["buy cheap pills now", "meeting at 3pm tomorrow", "win a free prize"]
train_labels = ["spam", "ham", "spam"]
pipeline.fit(train_texts, train_labels)
print(pipeline.predict(["free money for you"])) # ['spam']
Q6: What is part-of-speech (POS) tagging, and when is it used?¶
Show answer
POS tagging assigns each token its grammatical role in the sentence: noun, verb, adjective, adverb, preposition, etc.
"The quick brown fox jumps"
→ [("The", DET), ("quick", ADJ), ("brown", ADJ), ("fox", NOUN), ("jumps", VERB)]
When it is used: - Lemmatisation: the correct lemma depends on POS. "Saw" → "see" (verb) or "saw" (noun). Without POS, lemmatisers guess. - Named Entity Recognition: POS features (is it a proper noun?) are strong NER signals - Information extraction: extracting subject-verb-object triples requires knowing which words fill which roles - Text generation: grammar checking and controlled generation require syntactic awareness
import nltk
nltk.download('averaged_perceptron_tagger', quiet=True)
nltk.download('punkt', quiet=True)
from nltk import pos_tag, word_tokenize
text = "The quick brown fox jumps over the lazy dog"
tokens = word_tokenize(text)
tagged = pos_tag(tokens)
print(tagged[:4])
# [('The', 'DT'), ('quick', 'JJ'), ('brown', 'JJ'), ('fox', 'NN')]
POS tagging is less prominent in modern transformer-based pipelines because models like BERT implicitly learn syntactic structure during pre-training.
Q7: What is Named Entity Recognition (NER), and how is it evaluated?¶
Show answer
Named Entity Recognition identifies and classifies spans of text as named entities: persons, organisations, locations, dates, monetary values, etc.
Example:
How it works (classical): - Sequence labelling: each token gets a label (see BIO tagging in Q8 of applications file) - Features: POS tags, capitalisation, word shape, surrounding words, gazetteer lookups - Models: Conditional Random Fields (CRF) were the standard before neural models
How NER is evaluated: - Entity-level F1 (not token-level) — a prediction counts as correct only if the entire entity span and type are correct - Partial matches (getting the span but wrong type, or vice versa) count as errors - Evaluate separately per entity type (PER, ORG, LOC) — models vary in quality across types
Q8: What is dependency parsing, and what can you extract from it?¶
Show answer
Dependency parsing identifies grammatical relationships between words — specifically, which word is the head (governor) and which is the dependent (modifier).
Example:
"The dog chased the cat"
→ chased → nsubj → dog (dog is the nominal subject of chased)
→ chased → dobj → cat (cat is the direct object of chased)
What you can extract: - Subject-verb-object triples: for open information extraction ("who did what to whom") - Adjective-noun pairs: for aspect-based sentiment ("the battery life is terrible") - Negation scope: which words are negated by "not" or "never"
import spacy
nlp = spacy.load("en_core_web_sm")
doc = nlp("The dog chased the cat quickly.")
for token in doc:
if token.dep_ in ('nsubj', 'dobj', 'ROOT'):
print(f"{token.text:10} {token.dep_:10} → head: {token.head.text}")
# dog nsubj → head: chased
# chased ROOT → head: chased
# cat dobj → head: chased
Dependency parsing is computationally expensive. Use it when you specifically need structured relation extraction, not for general text classification.
Q9: What are sentiment lexicons like VADER and TextBlob? When are they good enough?¶
Show answer
Sentiment lexicons map words and phrases to sentiment scores without training a model. They are rule-based systems that apply hand-crafted heuristics.
VADER (Valence Aware Dictionary and sEntiment Reasoner): - Designed for social media text - Handles capitalisation ("GREAT" > "great"), punctuation ("great!!!" > "great"), and common slang - Returns positive, negative, neutral, and compound scores - No training required
TextBlob: - Pattern-based, trained on movie reviews - Returns polarity (−1 to 1) and subjectivity (0 to 1) - Less accurate than VADER on social media text
from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer
analyzer = SentimentIntensityAnalyzer()
texts = [
"The product is AMAZING!!!",
"Not bad, but could be better",
"Absolutely terrible. I want a refund."
]
for text in texts:
score = analyzer.polarity_scores(text)
print(f"{score['compound']:+.3f} {text}")
# +0.738 The product is AMAZING!!!
# +0.431 Not bad, but could be better
# -0.734 Absolutely terrible. I want a refund.
When lexicons are good enough: - Prototyping and baselines - No labelled training data available - Short, informal social media text (VADER's strength) - When inference speed or compute cost prohibits ML models
When you need ML: - Domain-specific sentiment (financial text, medical reviews) where lexicon coverage is poor - Fine-grained sentiment (5-star ratings, aspect-level) - When the lexicon's compound score consistently misclassifies your domain
Q10: What are the core limitations of classical NLP, and why did neural approaches surpass them?¶
Show answer
Classical NLP methods have well-understood, fundamental limitations:
Sparsity: BoW and TF-IDF produce sparse vectors of size equal to vocabulary (often 50K–500K). Most entries are zero. Sparse vectors are memory-inefficient and make similarity calculations unreliable.
No semantic similarity: "Car" and "automobile" are completely unrelated in BoW space. There is no way to encode that synonyms should be similar.
No context: "Bank" in "river bank" and "bank" in "bank account" produce the same vector. Classical methods have one representation per word, context-free.
Manual feature engineering: POS tags, dependency features, gazetteers — every feature required domain knowledge and manual design. This does not scale to new domains.
Fixed vocabulary: New words (COVID, blockchain, GPT) are OOV by default.
Why neural methods won: - Dense, low-dimensional embeddings capture semantic similarity (Word2Vec showed "king − man + woman ≈ queen") - Recurrent and attention-based models capture word order and long-range dependencies - Contextual embeddings (BERT) give different representations to "bank" depending on surrounding context - Pre-training on massive text corpora transfers rich linguistic knowledge to downstream tasks, reducing the need for labelled data
That said, classical methods remain useful for baselines, interpretability requirements, and resource-constrained environments.
Q11: How would you build a text classifier from scratch using classical methods?¶
Show answer
A production-ready classical text classification pipeline:
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.datasets import fetch_20newsgroups
# Load data
categories = ['sci.space', 'talk.politics.misc', 'rec.sport.baseball']
data = fetch_20newsgroups(subset='train', categories=categories, remove=('headers', 'footers'))
pipeline = Pipeline([
('tfidf', TfidfVectorizer(
ngram_range=(1, 2),
min_df=3,
max_df=0.9,
sublinear_tf=True # apply log(1+tf) to dampen high-frequency effects
)),
('clf', LogisticRegression(C=5.0, max_iter=1000, multi_class='multinomial'))
])
scores = cross_val_score(pipeline, data.data, data.target, cv=5, scoring='f1_macro')
print(f"F1 (macro): {scores.mean():.3f} ± {scores.std():.3f}")
Key decisions:
- sublinear_tf=True: dampens the effect of very frequent terms within a document — usually improves performance
- ngram_range=(1,2): adds bigrams for phrase-level features
- Logistic Regression over Naive Bayes: better when features are correlated (which they always are in text)
- Use Pipeline to prevent data leakage — the TF-IDF vocabulary is fit only on training folds
Q12: What is the cosine similarity threshold for "similar" documents, and how do you set it?¶
Show answer
There is no universal threshold — it depends on the document type, preprocessing, and use case. Setting it is an empirical process, not a mathematical one.
General guidelines: - Short texts (tweets, titles): cosine > 0.8 often indicates near-duplicate content - Long documents (articles, papers): cosine > 0.3–0.5 may indicate topic similarity - After aggressive stemming/stop word removal, scores tend to be higher overall
How to set it in practice: 1. Take a random sample of document pairs 2. Label them manually as "similar" or "not similar" 3. Compute cosine similarity for each pair 4. Find the threshold that maximises F1 on your labelled pairs
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
import numpy as np
docs = [
"Python is a programming language",
"Python programming language tutorial",
"The history of ancient Rome"
]
tfidf = TfidfVectorizer()
X = tfidf.fit_transform(docs)
sim_matrix = cosine_similarity(X)
print(np.round(sim_matrix, 3))
# Docs 0 and 1 will show high similarity; Doc 2 will be near 0 for both
The common mistake is picking 0.5 or 0.7 arbitrarily. Always calibrate against human judgements on your specific data.