NLP Overview

Computers do not understand text. They understand numbers. Every NLP pipeline's first job is to convert text into numbers without losing too much meaning — and that translation is harder than it looks, because human language is ambiguous, context-dependent, and relentlessly creative.

By the end of this note you will understand why text is difficult for machines, what kinds of problems NLP solves, and how the whole pipeline fits together before you write a single line of preprocessing code.

Learning Objectives

• Articulate why language understanding is genuinely hard (not just "messy data")
• Name the major NLP tasks and recognise them in real products
• Describe the text-to-numbers pipeline at a high level
• Distinguish between NLP for data science (applied) and NLP research

---

Why Language is Hard for Machines

Ambiguity is the default, not the exception

Read this sentence: "I saw the man with the telescope."

Did you use the telescope to see the man, or did you see a man who was holding a telescope? A human resolves this instantly from context. A computer, working character by character, has no default resolution.

Three types of ambiguity appear constantly in real text:

Type	Example	Why it's hard
Lexical	"Bank" (river / financial)	Same word, different meaning
Syntactic	"Flying planes can be dangerous"	Same parse, two structures
Referential	"The trophy didn't fit in the suitcase because it was too big"	What does "it" refer to?

Context changes everything

Sentiment is the clearest example. "This phone's battery lasts forever" is positive in a product review and sarcastic after the phone dies in two hours. The words are identical; the meaning is inverted. Negation compounds this: "I didn't not enjoy it" requires understanding double negation, which most naive pipelines get wrong.

Warning

A model trained on positive/negative reviews will assign positive sentiment to the sarcastic review "Oh great, another bug update" because "great" appears in positive training examples. Handling sarcasm well requires context that basic bag-of-words models cannot capture.

Scale and sparsity

A typical English vocabulary is 170,000 words. A single sentence uses 10–20 of them. Any numeric representation of that sentence will be almost entirely zeros. Managing this sparsity efficiently is one of the core engineering challenges in classical NLP.

---

What NLP Is Actually Used For

These are the tasks you will encounter in industry data science roles:

Text Classification

Assign a label to a document. Sentiment analysis (positive/negative/neutral), spam detection (spam/ham), topic classification (sports/politics/tech), intent detection in chatbots.

Why it matters: Automates decisions that used to require human reading. A support ticket classifier that routes tickets to the right team before a human reads them saves thousands of hours per month at scale.

Named Entity Recognition (NER)

Extract structured information from unstructured text — people, organisations, locations, dates, monetary values.

"Apple acquired Beats for $3 billion in 2014" → Apple (ORG), Beats (ORG), $3 billion (MONEY), 2014 (DATE)

Sequence-to-Sequence Tasks

Machine translation, text summarisation, question answering. The input is a sequence; the output is a different sequence. These tasks almost always use transformers today.

Information Retrieval / Semantic Search

Find documents relevant to a query. Classic search uses keyword matching (TF-IDF under the hood). Semantic search uses embeddings so that "cheap accommodation" matches "affordable hotel" even when they share no words.

---

The Text-to-Numbers Pipeline

Every NLP system — from a spam filter to GPT — follows this basic flow:

Raw text → Preprocessing → Tokenisation → Vectorisation → Model → Prediction

Here is what each step actually does:

Step	What happens	Example
Preprocessing	Remove noise, normalise case, strip punctuation	"GREAT!!!" → "great"
Tokenisation	Split text into units (tokens)	"not good" → ["not", "good"]
Vectorisation	Convert tokens to numbers	["not", "good"] → [0, 0, 1, 0, 1, ...]
Modelling	Learn patterns between numbers and labels	Logistic Regression, BERT
Prediction	Apply the learned function to new text	"not good" → negative

# The entire pipeline in concept
raw_text = "This product is not as good as I expected!!!"

# Step 1: Preprocess
import re
clean = re.sub(r"[^a-zA-Z\s]", "", raw_text).lower().strip()
# "this product is not as good as i expected"

# Step 2: Tokenise
tokens = clean.split()
# ['this', 'product', 'is', 'not', 'as', 'good', 'as', 'i', 'expected']

# Steps 3–5 happen inside sklearn pipelines — covered in the next three notes

Info

Tokenisation is more nuanced than .split(). "don't" splits to ["don't"] or ["do", "n't"] depending on the tokeniser. Subword tokenisers used by BERT split "tokenisation" into ["token", "##isation"]. The choice of tokeniser affects everything downstream.

---

NLP in Data Science vs NLP Research

The distinction matters for how you approach problems:

Dimension	Data Science (applied)	NLP Research
Goal	Solve a specific business problem	Advance the state of the art
Typical input	1k–100k labelled examples	Millions of examples or self-supervised
Success metric	F1, business impact	BLEU, ROUGE, benchmark leaderboards
Tool of choice	TF-IDF + LR, fine-tuned BERT	Custom architectures, new pretraining
Deployment reality	Must run in production, low latency	Research environment, batch inference

Success

As a data scientist, your job is to solve the business problem with the simplest tool that is good enough. A TF-IDF + Logistic Regression pipeline trained on 5,000 labelled support tickets will often reach 85–90% accuracy and be deployable in an afternoon. A fine-tuned BERT model might reach 93% and take two weeks to build and productionise. Whether that 8% gap justifies the cost is a business question, not a technical one.

---

NLP Tasks at a Glance

# You do not need to run this — it is a mental map
nlp_tasks = {
    "Text Classification": ["sentiment analysis", "spam detection", "topic labelling"],
    "Named Entity Recognition": ["person/org/location extraction", "date/money tagging"],
    "Text Generation": ["autocomplete", "summarisation", "translation"],
    "Information Retrieval": ["search", "question answering", "document similarity"],
    "Sequence Labelling": ["part-of-speech tagging", "chunking"],
}

for task, examples in nlp_tasks.items():
    print(f"{task}: {', '.join(examples)}")

# Output:
# Text Classification: sentiment analysis, spam detection, topic labelling
# Named Entity Recognition: person/org/location extraction, date/money tagging
# Text Generation: autocomplete, summarisation, translation
# Information Retrieval: search, question answering, document similarity
# Sequence Labelling: part-of-speech tagging, chunking

Tip

When scoping an NLP project, always start with the question: "Is this a classification problem, an extraction problem, or a generation problem?" The answer determines your toolkit and your labelling strategy.

---

---

What's Next

You've covered the NLP task taxonomy (classification, extraction, generation), the text-to-features pipeline from raw text through tokens to numbers, the difference between applied NLP for data science and NLP research, and the practitioner's decision framework for when a simple pipeline is sufficient. Next up: 02-text-preprocessing — where you'll implement the complete preprocessing pipeline — lowercasing, URL removal, tokenisation, stop word filtering with negation preservation, stemming, and lemmatisation — and wrap it into a sklearn-compatible transformer.

Optional Deep Dive

Read "Speech and Language Processing" by Jurafsky and Martin (free 3rd edition draft at web.stanford.edu/~jurafsky/slp3/) Chapter 2 — it covers the formal linguistic definitions of tokens, types, and n-grams, and explains the Unicode normalisation and punctuation handling decisions that matter when your data includes multilingual or informal text.

← Agenda | Next: Text Preprocessing →