Text Preprocessing¶
Raw text is the messiest input a model can receive — inconsistent casing, HTML tags, emojis, encoding errors, and zero structure. How you clean and represent text often matters more than which model you choose. These questions test whether you understand the tradeoffs, not just the mechanics.
Q1: Why does text preprocessing matter? Can't modern models handle raw text?¶
Show answer
Modern transformer models (BERT, GPT) do handle more raw text than classical methods — they include their own tokenisers and are pre-trained on noisy web data. But preprocessing still matters for three reasons:
- Classical models (TF-IDF, Naive Bayes, Bag of Words) are extremely sensitive to surface-level variation. "Run", "run", and "RUNNING" are three different tokens without normalisation.
- Noise reduction improves signal-to-noise ratio — URLs, boilerplate, HTML tags, and repeated whitespace consume vocabulary slots and dilute the signal.
- Domain-specific cleaning is always necessary — medical text has different conventions than tweets. No pre-trained tokeniser knows your domain's abbreviations or formatting quirks.
The common mistake candidates make is treating preprocessing as a fixed recipe. The right preprocessing depends on the model, the domain, and the task.
Q2: What is tokenisation, and what are the three main levels?¶
Show answer
Tokenisation is the process of splitting a text string into discrete units (tokens) that the model will process individually.
Word-level tokenisation: Split on whitespace and punctuation. Simple and interpretable. - Vocabulary size: tens of thousands - Problem: "running" and "run" are separate tokens with no shared representation - Problem: OOV (out-of-vocabulary) words get unknown tokens
Character-level tokenisation: Each character is a token. - Vocabulary size: ~100–300 characters - Handles any word, no OOV problem - Problem: sequences are very long; the model must learn word structure from scratch
Subword tokenisation (BPE, WordPiece, SentencePiece): Common words stay whole; rare words split into meaningful subunits. - "unhappiness" → ["un", "##happiness"] in WordPiece - Vocabulary size: ~30,000–50,000 - No OOV problem — any word can be expressed from subwords - This is why subword tokenisation dominates in modern NLP
# Word-level with NLTK
import nltk
nltk.download('punkt', quiet=True)
from nltk.tokenize import word_tokenize
text = "She's running quickly."
print(word_tokenize(text))
# ["She", "'s", "running", "quickly", "."]
# Character-level (built-in)
print(list(text))
# ['S', 'h', 'e', "'", 's', ' ', 'r', 'u', 'n', 'n', 'i', 'n', 'g', ' ', 'q', 'u', 'i', 'c', 'k', 'l', 'y', '.']
Q3: When does lowercasing help, and when does it hurt?¶
Show answer
When it helps: - Bag of Words and TF-IDF models — "Apple" and "apple" should be the same token for most tasks - Reduces vocabulary size, which speeds up training and reduces sparsity - Sentiment analysis, topic modelling, document classification — casing carries little signal
When it hurts:
- Named Entity Recognition (NER) — "Apple" (company) vs "apple" (fruit). Capitalisation is a strong signal that a word is a proper noun.
- Acronyms — "US" (United States) vs "us" (pronoun) are disambiguated by case
- Code or technical text — True vs true have different meanings
The rule: lowercase by default for classical models, preserve case for sequence labelling tasks like NER and when using pre-trained contextual models (BERT was pre-trained with casing).
Q4: What is stop word removal, and when should you skip it?¶
Show answer
Stop words are high-frequency function words (the, a, is, in, of) that carry little semantic meaning on their own.
When stop word removal helps: - Bag of Words and TF-IDF models — stop words dominate term frequencies and drown out meaningful content words - Topic modelling (LDA) — topics should be about content words, not "the" and "is" - Search indexes where storage and speed matter
When stop word removal hurts: - Semantic similarity and sentence embeddings — removing stop words changes sentence meaning. "This is not good" → "good" completely inverts the sentiment. - Sentiment analysis — "not bad" → "bad" is a disaster - Question answering — "who", "what", "when" are stop words that define the question type - Transformer models — they are trained on full text and rely on all tokens
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
import nltk
nltk.download('stopwords', quiet=True)
nltk.download('punkt', quiet=True)
stop_words = set(stopwords.words('english'))
text = "This is not a good product at all"
tokens = word_tokenize(text.lower())
filtered = [w for w in tokens if w not in stop_words]
print(filtered)
# ['good', 'product'] — "not" is lost, sentiment is reversed
Q5: What is the difference between stemming and lemmatisation? When do you use each?¶
Show answer
Both reduce words to a base form, but they work differently.
Stemming applies heuristic rules to chop word endings. Fast but crude. - "running" → "run", "studies" → "studi" (not a real word), "caring" → "car" (wrong) - Uses: Porter Stemmer, Snowball Stemmer - Results are not real words — cannot be interpreted directly
Lemmatisation uses vocabulary and morphological analysis to return the actual dictionary base form (lemma). - "running" → "run", "studies" → "study", "better" → "good" - Requires knowledge of the word's part of speech - Slower but produces valid words
When to use stemming: speed matters, downstream task is retrieval or BoW classification, exact word form is irrelevant.
When to use lemmatisation: you need interpretable tokens, the task involves understanding meaning (e.g. chatbots, QA), or you are building features for a downstream model where word identity matters.
from nltk.stem import PorterStemmer
from nltk.stem import WordNetLemmatizer
import nltk
nltk.download('wordnet', quiet=True)
stemmer = PorterStemmer()
lemmatizer = WordNetLemmatizer()
words = ["running", "studies", "caring", "better"]
for word in words:
print(f"{word:12} stem={stemmer.stem(word):10} lemma={lemmatizer.lemmatize(word, pos='v')}")
# running stem=run lemma=run
# studies stem=studi lemma=study
# caring stem=car lemma=care
# better stem=better lemma=better (needs pos='a' for adjective)
Q6: How do you handle punctuation and special characters?¶
Show answer
There is no single correct answer — it depends on the task.
When to remove punctuation: - BoW / TF-IDF models — punctuation adds noise with no semantic value - Simple text classification
When to keep punctuation: - Sentiment analysis — "Great!" vs "Great." carry different intensities. "I love it... not." changes meaning entirely. - Sentence boundary detection — periods and question marks define sentence boundaries - Code or technical text — parentheses, brackets, and operators are meaningful
Special characters to handle explicitly:
- HTML entities (&, <) — decode before any other processing
- Emojis — either strip or replace with text descriptions (😊 → "happy face") using libraries like emoji
- Currency symbols ($, €) — may be meaningful in financial text
import re
text = "Check out https://example.com! Price: $99.99 <b>Buy now</b> 😊"
# Remove HTML tags
text = re.sub(r'<[^>]+>', '', text)
# Remove URLs
text = re.sub(r'http\S+|www\S+', '', text)
# Remove non-alphabetic characters (keep spaces)
text_clean = re.sub(r'[^a-zA-Z\s]', '', text)
print(text_clean.strip())
# "Check out Price Buy now"
Q7: How do regular expressions help in text cleaning? Give practical examples.¶
Show answer
Regular expressions (regex) are the workhorse of text cleaning. They let you find and replace patterns in text without writing brittle string-matching loops.
Common patterns you should know:
import re
text = "Contact: user@email.com or call +1-800-555-0100 on 2024-01-15"
# Remove email addresses
no_email = re.sub(r'\b[\w.+-]+@[\w-]+\.\w+\b', '', text)
# Remove phone numbers
no_phone = re.sub(r'\+?[\d\-\(\)\s]{7,15}', '', text)
# Remove dates (simple YYYY-MM-DD)
no_date = re.sub(r'\b\d{4}-\d{2}-\d{2}\b', '', text)
# Remove extra whitespace (always run this last)
text_clean = re.sub(r'\s+', ' ', no_email).strip()
# Extract all numbers from text
numbers = re.findall(r'\d+\.?\d*', text)
print(numbers) # ['1', '800', '555', '0100', '2024', '01', '15']
The common mistake is writing overly greedy patterns (.+ instead of [^>]+) that match more than intended. Always test regex on a sample before applying to the full corpus.
Q8: How should you handle numbers, dates, and URLs in text?¶
Show answer
Numbers:
- For BoW/TF-IDF: replace all numbers with a single placeholder token like <NUM>. This prevents "2020", "2021", "2019" from being treated as separate high-frequency features.
- For tasks where the numeric value matters (financial text): keep the numbers and potentially bin or normalise them.
Dates:
- For general classification: replace with <DATE> token
- For temporal tasks: parse into a structured format and extract features (day of week, month, year) separately
URLs:
- Almost always remove or replace with <URL>. URLs are highly unique strings that fragment vocabulary without adding meaning.
- Exception: sometimes the domain (youtube.com vs wikipedia.org) is meaningful — extract the domain before removing.
import re
def normalize_text(text):
text = re.sub(r'http\S+|www\S+', '<URL>', text)
text = re.sub(r'\b\d{1,2}[/-]\d{1,2}[/-]\d{2,4}\b', '<DATE>', text)
text = re.sub(r'\b\d{4}-\d{2}-\d{2}\b', '<DATE>', text)
text = re.sub(r'\b\d+\.?\d*\b', '<NUM>', text)
return text
sample = "Sales grew 25% in 2023-12-01. See https://report.com"
print(normalize_text(sample))
# "Sales grew <NUM>% in <DATE>. See <URL>"
Q9: What is a text preprocessing pipeline, and what is the correct order of steps?¶
Show answer
A preprocessing pipeline transforms raw text into model-ready input in a defined sequence. Order matters because later steps depend on earlier ones.
Typical pipeline for classical NLP: 1. Decode encoding (handle UTF-8, fix mojibake) 2. HTML/XML stripping 3. URL and email removal 4. Expand contractions ("don't" → "do not") 5. Lowercase 6. Remove special characters / punctuation 7. Tokenise 8. Remove stop words 9. Stem or lemmatise 10. Remove very short tokens (length < 2) 11. Vectorise (BoW / TF-IDF / embeddings)
Why order matters: lowercasing before punctuation removal ensures you don't create empty tokens. Tokenising before stop word removal lets you match whole words, not substrings.
import re
from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords
from nltk.stem import WordNetLemmatizer
lemmatizer = WordNetLemmatizer()
stop_words = set(stopwords.words('english'))
def preprocess(text):
text = text.lower()
text = re.sub(r'http\S+', '', text)
text = re.sub(r'[^a-z\s]', '', text)
tokens = word_tokenize(text)
tokens = [lemmatizer.lemmatize(t) for t in tokens if t not in stop_words and len(t) > 1]
return tokens
print(preprocess("She's running 3 miles! Check https://example.com"))
# ['running', 'mile', 'check']
Q10: What is the out-of-vocabulary (OOV) problem, and how do different approaches handle it?¶
Show answer
The OOV problem occurs when the model encounters a word at inference time that was not in its training vocabulary.
In classical BoW/TF-IDF models: - OOV words are silently ignored — the document vector has no entry for them - This is a silent failure: a technical document full of OOV domain terms gets an empty vector
In Word2Vec / GloVe:
- No embedding exists for OOV words
- Common workaround: map OOV words to a mean of their character n-gram embeddings, or use the <UNK> token embedding
In FastText: - OOV words are represented via character n-gram embeddings — even an unseen word gets a meaningful vector from its subword components - "unhappiness" unseen at training → reconstructed from "un", "happy", "ness" n-grams
In BPE/subword tokenisers (BERT, GPT): - No OOV problem by design — any word decomposes into known subword pieces - The worst case is character-level decomposition, which always succeeds
# Demonstrating OOV in sklearn's CountVectorizer
from sklearn.feature_extraction.text import CountVectorizer
train = ["the cat sat on the mat", "dogs are great pets"]
vectorizer = CountVectorizer()
vectorizer.fit(train)
test = ["the cat chased the cryptocurrency"] # "cryptocurrency" is OOV
vec = vectorizer.transform(test)
print(vec.toarray())
# "cryptocurrency" contributes nothing — silently dropped
Q11: How do you handle encoding issues in real-world text data?¶
Show answer
In production, text comes from many sources — web scrapes, PDFs, user inputs, legacy databases — each with different encoding assumptions.
Common problems:
- Mojibake: garbled characters from encoding mismatch (e.g. UTF-8 bytes interpreted as Latin-1) — "café" becomes "café"
- Mixed encodings: one file contains both UTF-8 and Latin-1 encoded strings
- BOM (Byte Order Mark): some Windows tools prepend a BOM character () that appears as garbage at the start of text
How to handle:
# Always open files with explicit encoding
with open('data.txt', 'r', encoding='utf-8', errors='replace') as f:
text = f.read()
# Detect encoding with chardet when unknown
import chardet
with open('mystery_file.txt', 'rb') as f:
raw = f.read()
detected = chardet.detect(raw)
text = raw.decode(detected['encoding'], errors='replace')
# Strip BOM if present
text = text.lstrip('')
# Normalize unicode (e.g. different forms of the same character)
import unicodedata
text = unicodedata.normalize('NFC', text)
Always use errors='replace' or errors='ignore' rather than letting encoding errors crash your pipeline on production data.
Q12: What is vocabulary size, and why does it matter?¶
Show answer
Vocabulary size is the total number of unique tokens in your corpus after preprocessing. It directly determines the dimensionality of your feature space in classical NLP.
Why it matters: - In BoW/TF-IDF: each document becomes a vector of length equal to vocabulary size. A vocabulary of 100,000 tokens means 100,000-dimensional sparse vectors — high memory cost, high noise. - Rare words (those appearing in only 1–2 documents) add dimensions but carry no generalisable signal — they overfit. - Very common words (stop words) add dimensions with no discriminative power.
How to control vocabulary size:
- min_df: ignore terms appearing in fewer than N documents (removes rare/noisy terms)
- max_df: ignore terms appearing in more than X% of documents (removes near-stop-words)
- max_features: keep only the top N most frequent terms
from sklearn.feature_extraction.text import TfidfVectorizer
vectorizer = TfidfVectorizer(
min_df=5, # ignore terms in fewer than 5 documents
max_df=0.9, # ignore terms in more than 90% of documents
max_features=10000
)
# This keeps the vocabulary focused on informative, generalisable terms
The common mistake is using default settings and ending up with a 500,000-token vocabulary full of typos, rare proper nouns, and numeric artifacts.
Q13: What is the difference between types and tokens, and why does it matter for NLP?¶
Show answer
- Token: every individual word occurrence in a corpus. "the cat sat on the mat" has 6 tokens.
- Type: every unique word form. "the cat sat on the mat" has 5 types ("the" appears twice).
Why it matters: - Vocabulary size = number of types, not tokens - Type-token ratio (TTR = types / tokens) measures lexical diversity. Short texts have high TTR; long texts tend to have lower TTR as words repeat. - Zipf's Law: in any natural language corpus, the frequency of any word is inversely proportional to its rank. A tiny number of types account for a huge proportion of tokens. The top 100 words in English cover ~50% of all token occurrences — which is why stop word removal can reduce token count drastically while barely touching type count.
Understanding this distribution explains why min_df cuts so much noise: the long tail of the frequency distribution is vast but contributes little information per type.
Q14: How do you preprocess text differently for a sentiment analysis task vs a named entity recognition task?¶
Show answer
The preprocessing must align with what signals the task depends on.
Sentiment analysis:
- Lowercase — "Great" and "great" carry the same sentiment
- Preserve negations — "not good" must stay as two tokens, not be stop-word-filtered to "good"
- Preserve exclamation marks and ellipses — they carry emotional intensity
- Replace URLs with <URL> (they carry no sentiment)
- Expand contractions ("can't" → "can not") to help with negation detection
Named Entity Recognition: - Do NOT lowercase — capitalisation is the strongest signal for proper nouns - Preserve punctuation — sentence boundaries matter for sequence labelling - Preserve numbers and dates — they are often part of entities - Use sentence-level tokenisation before word tokenisation — NER operates at sentence level - Do NOT remove stop words — NER is a sequence task; every word needs a label
This is one of the most common mistakes in interview case studies: candidates apply a fixed preprocessing recipe without considering whether the preprocessing destroys the signals the task requires.
Q15: What are n-grams in the context of tokenisation, and when do you include them in preprocessing?¶
Show answer
An n-gram is a contiguous sequence of N tokens from a text. - Unigrams: individual words - Bigrams: two-word sequences ("New York", "ice cream", "not good") - Trigrams: three-word sequences ("New York City")
N-grams capture local context that unigrams miss. "New York" as a bigram is a named entity; "New" and "York" as unigrams are just common words.
When to include bigrams/trigrams: - When phrase-level meaning matters (sentiment phrases like "not good", "very fast") - When named entities are important ("San Francisco", "machine learning") - Text classification tasks where topic-specific phrases are discriminative features
Tradeoff: including bigrams and trigrams increases vocabulary size dramatically (potentially by an order of magnitude) and increases sparsity.
from sklearn.feature_extraction.text import TfidfVectorizer
corpus = ["I do not like this movie", "I really love this film"]
# Unigrams only
v1 = TfidfVectorizer(ngram_range=(1, 1))
# Unigrams and bigrams
v2 = TfidfVectorizer(ngram_range=(1, 2))
print("Unigram vocab size:", len(v1.fit(corpus).vocabulary_))
print("Uni+Bigram vocab size:", len(v2.fit(corpus).vocabulary_))
# Unigram vocab size: 8
# Uni+Bigram vocab size: 19