Language Models¶
Language models are the engine behind modern NLP. Whether you are evaluating a text classifier, building a chatbot, or fine-tuning BERT, you are working with or on top of a language model. Interviews at companies using NLP in production will probe your understanding of how these models work, how to evaluate them, and where they fail.
Q1: What is a language model, and what probability does it model?¶
Show answer
A language model assigns a probability to a sequence of words. Formally, it models:
Each word's probability is conditioned on all preceding words. The model answers: given everything said so far, what is the probability of the next word?
Why this matters: - A model that assigns high probability to fluent, coherent sequences has learned something meaningful about language structure, grammar, and facts about the world - Language modelling is the pre-training objective for GPT-family models (predict next token) and a core component of BERT (predict masked tokens) - Language model probability is used for speech recognition (prefer "I recognise speech" over "I wreck a nice beach"), spell correction, and text generation
Two families: - Autoregressive (causal) LMs: predict each token from all previous tokens (GPT, GPT-2, GPT-4). Natural for generation — you sample one token at a time. - Masked LMs: predict masked tokens from both left and right context (BERT). Better for understanding tasks, not generation.
Q2: How do n-gram language models work, and what is their fundamental problem?¶
Show answer
An n-gram language model approximates the full conditional probability by using only the last n−1 words as context (the Markov assumption).
Bigram: P(w_t | w_1...w_{t-1}) ≈ P(w_t | w_{t-1})
Trigram: P(w_t | w_1...w_{t-1}) ≈ P(w_t | w_{t-2}, w_{t-1})
MLE estimation:
The sparsity problem: Many valid n-gram sequences never appear in the training corpus. Their MLE probability is 0. Multiplying probabilities across a sentence means any sentence containing one unseen n-gram gets probability 0 — regardless of how fluent it is.
from collections import Counter, defaultdict
import numpy as np
corpus = ["the cat sat on the mat", "the cat ate the rat"]
tokens = [sent.split() for sent in corpus]
bigrams = [(tokens[s][i], tokens[s][i+1]) for s in range(len(tokens))
for i in range(len(tokens[s])-1)]
bigram_counts = Counter(bigrams)
unigram_counts = Counter(w for sent in tokens for w in sent)
def bigram_prob(w1, w2):
return bigram_counts[(w1, w2)] / unigram_counts[w1]
print(bigram_prob("the", "cat")) # 0.5 — seen
print(bigram_prob("cat", "dog")) # 0.0 — unseen → zero probability
Q3: What is Laplace smoothing, and what other smoothing techniques exist?¶
Show answer
Smoothing redistributes some probability mass from seen n-grams to unseen ones — preventing zero probabilities.
Laplace (add-1) smoothing: Add 1 to every n-gram count before normalising.
def bigram_prob_laplace(w1, w2, vocab_size):
return (bigram_counts[(w1, w2)] + 1) / (unigram_counts[w1] + vocab_size)
vocab_size = len(unigram_counts)
print(bigram_prob_laplace("the", "cat", vocab_size)) # nonzero
print(bigram_prob_laplace("cat", "dog", vocab_size)) # nonzero (small)
Better smoothing techniques: - Kneser-Ney smoothing: subtracts a fixed discount from counts; distributes saved mass based on how many unique contexts a word appears in (not just frequency). It is the empirical gold standard for n-gram LMs. - Interpolation: blend unigram, bigram, and trigram probabilities with learned weights — fall back to lower-order n-grams when higher-order counts are sparse - Stupid backoff (used at Google scale): not a true probability distribution, but computationally cheap — works well in practice
Laplace smoothing over-smooths heavily. Kneser-Ney is what you should name when asked for a production-quality smoothing approach.
Q4: What is perplexity, and how does it measure language model quality?¶
Show answer
Perplexity (PPL) is the standard intrinsic evaluation metric for language models. It measures how "surprised" the model is by a held-out test set.
Perplexity = exp(H)
where H = -1/N × Σ log P(w_t | context)
(H is the cross-entropy per token on the test set)
Intuition: A perplexity of K means the model is as uncertain as if it had to choose uniformly among K options at every step. Lower perplexity = better model = fewer words are surprising.
- A random model over a 10,000-word vocabulary: PPL ≈ 10,000
- Trigram LM on English: PPL ≈ 60–100
- BERT/GPT-2 on English: PPL ≈ 10–30
import numpy as np
def perplexity(probabilities):
log_probs = np.log(probabilities)
avg_log_prob = np.mean(log_probs)
return np.exp(-avg_log_prob)
probs_good_model = [0.6, 0.7, 0.5, 0.8, 0.65] # high confidence
probs_bad_model = [0.1, 0.05, 0.2, 0.08, 0.12] # low confidence
print(f"Good model PPL: {perplexity(probs_good_model):.2f}")
print(f"Bad model PPL: {perplexity(probs_bad_model):.2f}")
Limitation: perplexity is an intrinsic metric — a model can have low perplexity and still perform poorly on downstream tasks. Always combine with task-specific evaluation.
Q5: Why do neural language models outperform n-gram models?¶
Show answer
N-gram models have two hard limits: context window size and sparsity.
Fixed, short context: Trigrams look back 2 words. "The patient who came from the remote mountain village was admitted to the ___" — the subject ("patient") is 11 words back, well outside a trigram's reach.
Sparsity: Even with smoothing, n-gram models struggle with valid word sequences that rarely appear together in training data.
Neural LMs solve these problems: - Continuous representations: words are embedded into a dense vector space. Similar words have similar embeddings, so the model generalises to unseen combinations. - Variable-length context: RNN-based LMs maintain a hidden state that propagates information arbitrarily far back (in theory). Transformers attend to all previous tokens simultaneously. - Parameter sharing: the same embedding and weights apply across all positions — no explosion of parameters with context length.
The tradeoff: neural LMs require far more compute and data than n-gram models. For simple keyword-based applications or resource-constrained deployments, an n-gram model can be the right choice.
Q6: What does BERT learn during pre-training, and what are its two objectives?¶
Show answer
BERT (Devlin et al., 2018) is pre-trained on two unsupervised objectives using unlabelled text.
Masked Language Modelling (MLM): - 15% of tokens are randomly selected - 80% of selected tokens are replaced with [MASK]; 10% replaced with a random word; 10% kept unchanged - The model must predict the original token at each masked position - This forces bidirectional context — the model sees both left and right context when predicting each masked word
Next Sentence Prediction (NSP): - The model receives two sentences and predicts whether sentence B actually follows sentence A in the original text (50% of pairs are genuine, 50% random) - Designed to teach sentence-level relationships for tasks like question answering and natural language inference - Later research (RoBERTa) found NSP provides little benefit and BERT performs better without it
What BERT learns: - Syntax: attention heads specialise in syntactic relationships (subject-verb agreement, coreference) - Semantics: token representations capture word sense in context (polysemy) - Facts: some world knowledge is encoded in parameters (Eiffel Tower is in Paris)
from transformers import BertForMaskedLM, BertTokenizer
import torch
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForMaskedLM.from_pretrained('bert-base-uncased')
text = "The [MASK] is the largest planet in the solar system."
inputs = tokenizer(text, return_tensors='pt')
mask_idx = (inputs['input_ids'] == tokenizer.mask_token_id).nonzero(as_tuple=True)[1]
with torch.no_grad():
logits = model(**inputs).logits
top_token = tokenizer.decode(logits[0, mask_idx].argmax(dim=-1))
print(top_token) # "jupiter"
Q7: How does GPT differ from BERT in architecture and objective?¶
Show answer
BERT and GPT both use the Transformer architecture, but they use different variants and objectives.
BERT: - Uses the Transformer encoder (bidirectional attention) - Sees the full sentence including future tokens — cannot generate text token by token - Trained with MLM (predict masked tokens from both directions) - Optimised for understanding tasks: classification, NER, QA
GPT (and GPT-2, GPT-3, GPT-4): - Uses the Transformer decoder (causal/unidirectional attention) - Each token can only attend to previous tokens — future tokens are masked - Trained with causal language modelling: predict the next token given all previous tokens - Naturally suited for text generation: generate one token at a time, autoregressively
BERT: [CLS] The cat [MASK] on [MASK] mat [SEP]
predicts: "sat" and "the" using both left and right context
GPT: The → cat → sat → on → the → mat
each prediction uses only left context
When to use which: - BERT-family models for classification, NER, question answering — tasks that require understanding the full input - GPT-family models for text generation, summarisation, dialogue — tasks that require producing output sequentially
Q8: What is the pre-training / fine-tuning paradigm, and why is it powerful?¶
Show answer
The two-stage paradigm that dominates modern NLP:
Stage 1: Pre-training Train a large model on a massive unlabelled corpus using a self-supervised objective (MLM for BERT, causal LM for GPT). The model learns general language representations — grammar, syntax, semantics, world knowledge. This is expensive (thousands of GPU-hours) but done once.
Stage 2: Fine-tuning Take the pre-trained model and continue training on a small labelled dataset for a specific task. Only a small number of new parameters are added (a classification head). The model adapts its general representations to the target domain.
Why it is powerful: - A task that previously required 100,000 labelled examples can now be solved with 1,000, because the model already knows language - The pre-trained model acts as a very strong initialisation for gradient descent — fine-tuning converges fast - One base model can be fine-tuned for dozens of tasks
from transformers import BertForSequenceClassification, BertTokenizer
from transformers import Trainer, TrainingArguments
model = BertForSequenceClassification.from_pretrained(
'bert-base-uncased',
num_labels=2 # binary classification task
)
# All BERT weights initialised from pre-training
# Only the classification head is randomly initialised
# Fine-tuning updates all weights but the pre-trained weights are a strong starting point
Q9: What is the difference between zero-shot, few-shot, and fine-tuning?¶
Show answer
These describe how much task-specific data a model uses.
Zero-shot: The model is given a task description in the prompt but no labelled examples. The model must generalise from its pre-training alone. - "Classify this review as positive or negative: 'The product broke after two days.'" - Works surprisingly well for GPT-3/GPT-4 on many tasks; often poor for specialised domains
Few-shot: The model is given a small number of labelled examples in the prompt (typically 3–10), then asked to handle a new input. - Demonstrated in the GPT-3 paper — performance improves significantly with even 1–5 examples - No weight updates — examples are in the context window
Fine-tuning: - Update the model weights on hundreds to thousands of labelled examples - Produces the best performance when labelled data is available - More expensive, produces a task-specific model
When to use each: - Zero-shot / few-shot: prototype quickly, no labelled data, using a powerful model via API - Fine-tuning: production deployment, consistent quality requirements, domain-specific language
# Few-shot prompting (no code — it's a prompt design question)
prompt = """
Classify movie reviews as positive or negative.
Review: "A masterpiece of storytelling." → positive
Review: "Completely boring and predictable." → negative
Review: "The acting was decent but the plot was confusing." → """
# The model completes: "negative" or "mixed"
Q10: What is Byte Pair Encoding (BPE), and why do modern LLMs use it for tokenisation?¶
Show answer
BPE is a subword tokenisation algorithm that builds a vocabulary of variable-length subword units through iterative merging of frequent character pairs.
How BPE works: 1. Start with a character-level vocabulary 2. Count all adjacent symbol pairs in the training corpus 3. Merge the most frequent pair into a new single token 4. Repeat until vocabulary size reaches the target (e.g. 50,000)
Example:
- Start: l o w e r and l o w e s t (space-separated characters)
- Most frequent pair: l o → merge to lo
- Next: lo w → low
- Continue: low e → lowe, then lowe r → lower
Why LLMs use it: - No OOV problem — any input string decomposes into known subword pieces - Vocabulary is compact — common words are single tokens; rare words split into subwords - Efficient trade-off between sequence length (shorter than character-level) and vocabulary coverage - Language-agnostic — works for any language without separate dictionaries
# GPT-2/GPT-4 use BPE via tiktoken
import tiktoken
enc = tiktoken.encoding_for_model("gpt-4")
tokens = enc.encode("unhappiness in the cryptocurrency market")
print(tokens)
print([enc.decode([t]) for t in tokens])
# ["un", "happiness", " in", " the", " cryptocurrency", " market"]
# (exact splits depend on the BPE vocabulary)
Q11: What is hallucination in LLMs, and what causes it?¶
Show answer
Hallucination is when a language model generates plausible-sounding but factually incorrect or entirely fabricated information — stated confidently without any signal that the information is uncertain.
Examples: - Citing academic papers that don't exist - Stating incorrect dates, statistics, or quotes - Fabricating legal cases, medical facts, or company details
Root causes: - Language models optimise for fluency, not factuality. The training objective is to predict the next token — a fluent but false statement scores just as well as a fluent and true one. - Knowledge stored in weights decays and is static. The model's "knowledge" is frozen at training time. Events after the cutoff date are unknown. - Exposure to misinformation in training data. Web-scraped corpora contain false claims. - Attention mechanism can fail to retrieve relevant context. With long documents, the model may not attend properly to the relevant passage and "fill in" from parameters instead.
Mitigation strategies: - Retrieval-Augmented Generation (RAG): retrieve relevant documents at inference time and condition generation on them — grounds the model in verifiable source material - Fine-tuning on fact-checked data: reduces hallucination rates in specific domains - Self-consistency: sample multiple outputs and take the majority — inconsistent answers flag uncertainty - Calibrated uncertainty: train models to say "I don't know" when confidence is low
Q12: What are temperature and top-p sampling? How do they affect text generation?¶
Show answer
When an LLM generates text, it produces a probability distribution over the vocabulary for the next token. Sampling parameters control how a token is selected from that distribution.
Temperature (τ): Scales the logits before the softmax. High temperature → flatter distribution → more randomness. Low temperature → sharper distribution → more deterministic.
- τ = 0.0: greedy decoding — always pick the highest probability token
- τ = 1.0: sample from the raw distribution
- τ = 1.5+: very random, often incoherent
- τ < 0.5: conservative and repetitive
Top-p (nucleus) sampling: Rather than sampling over the full vocabulary, truncate the distribution to the smallest set of tokens whose cumulative probability exceeds p. Then sample from this nucleus.
p = 0.9: at each step, take the top tokens that together account for 90% of probability mass- This dynamically adjusts the number of candidates — confident steps use fewer tokens, uncertain steps use more
import numpy as np
def temperature_sampling(logits, temperature=1.0):
scaled = logits / temperature
probs = np.exp(scaled) / np.exp(scaled).sum()
return np.random.choice(len(probs), p=probs)
def top_p_sampling(logits, p=0.9):
probs = np.exp(logits) / np.exp(logits).sum()
sorted_idx = np.argsort(probs)[::-1]
cumulative = np.cumsum(probs[sorted_idx])
cutoff = np.searchsorted(cumulative, p) + 1
nucleus = sorted_idx[:cutoff]
nucleus_probs = probs[nucleus] / probs[nucleus].sum()
return np.random.choice(nucleus, p=nucleus_probs)
Practical settings: - Creative writing: temperature 0.8–1.2, top-p 0.9 - Code generation or factual QA: temperature 0.0–0.3 (near-deterministic) - The two parameters can be combined: apply top-p first, then temperature
Q13: What is the difference between encoder-only, decoder-only, and encoder-decoder transformer architectures?¶
Show answer
The original Transformer (Vaswani et al., 2017) had both an encoder and decoder. Later models specialised into encoder-only or decoder-only variants.
Encoder-only (BERT, RoBERTa, DistilBERT): - Bidirectional attention — each token attends to all others - Cannot generate text (no causal mask) - Produces rich contextual representations of the input - Best for: classification, NER, semantic similarity, extractive QA
Decoder-only (GPT-2, GPT-3, GPT-4, LLaMA): - Causal (unidirectional) attention — each token attends only to previous tokens - Natural for autoregressive generation - Best for: text generation, dialogue, code generation, few-shot learning
Encoder-decoder (T5, BART, mT5): - Encoder reads the input with bidirectional attention - Decoder generates output conditioned on encoder representations, with causal attention - Best for: sequence-to-sequence tasks — summarisation, translation, abstractive QA
Rule of thumb for interviews: - Understanding task → encoder-only (BERT-family) - Generation task → decoder-only (GPT-family) - Transformation task (one sequence → different sequence) → encoder-decoder (T5/BART)