Word Embeddings¶
Word embeddings are the bridge between raw text and the numerical representations that neural models operate on. They encode semantic relationships — words with similar meanings end up close in vector space. Understanding their mechanics, limitations, and evolution from static to contextual embeddings is expected knowledge at any mid-to-senior NLP role.
Q1: Why does one-hot encoding fail for words?¶
Show answer
One-hot encoding represents each word as a vector of length equal to vocabulary size, with a 1 in the word's position and 0 everywhere else.
Problem 1: Dimensionality A vocabulary of 50,000 words produces 50,000-dimensional vectors. Matrix operations with these vectors are memory-intensive and slow.
Problem 2: No semantic similarity Every pair of one-hot vectors is orthogonal — the dot product between any two different words is exactly 0. "Car" and "automobile" are just as unrelated as "car" and "democracy". The representation encodes no information about meaning.
import numpy as np
vocab = ["king", "queen", "man", "woman", "car"]
word_to_idx = {word: i for i, word in enumerate(vocab)}
def one_hot(word):
vec = np.zeros(len(vocab))
vec[word_to_idx[word]] = 1
return vec
king = one_hot("king")
queen = one_hot("queen")
car = one_hot("car")
print("king · queen =", np.dot(king, queen)) # 0.0
print("king · car =", np.dot(king, car)) # 0.0
# Completely indistinguishable in terms of relatedness
Word embeddings solve both problems: dense vectors (typically 100–300 dimensions) where semantically similar words have high cosine similarity.
Q2: What is Word2Vec, and how do the two training objectives differ?¶
Show answer
Word2Vec (Mikolov et al., 2013) learns dense word vectors by training a shallow neural network to predict words from context. The key insight: words that appear in similar contexts have similar meanings.
CBOW (Continuous Bag of Words): - Input: surrounding context words - Task: predict the centre word - Example: ["The", "cat", ___, "on", "the"] → predict "sat" - Faster to train, slightly better on frequent words
Skip-gram: - Input: the centre word - Task: predict each surrounding context word - Example: "sat" → predict ["The", "cat", "on", "the"] - Slower to train, better for rare words and larger datasets
How vectors are learned: The network is trained with negative sampling — for each positive (word, context) pair, it generates K random negative pairs. The weights of the input layer after training become the word embeddings.
from gensim.models import Word2Vec
from nltk.tokenize import word_tokenize
corpus = [
"the king rules the kingdom",
"the queen rules the kingdom",
"the man is a king",
"the woman is a queen"
]
tokenised = [word_tokenize(sent) for sent in corpus]
model = Word2Vec(
sentences=tokenised,
vector_size=50,
window=3,
min_count=1,
sg=1, # 1=Skip-gram, 0=CBOW
epochs=100
)
print(model.wv.most_similar("king"))
print(model.wv.similarity("king", "queen"))
Q3: What semantic relationships do word vectors capture?¶
Show answer
Word2Vec vectors encode semantic and syntactic regularities as linear relationships in vector space.
Semantic analogies: The famous example: king − man + woman ≈ queen The vector arithmetic works because "king" and "queen" differ by roughly the same offset as "man" and "woman" — the embedding has learned a "gender" direction.
Other regularities captured: - Capitals: France − Paris ≈ Germany − Berlin - Verb tenses: walk − walked ≈ run − ran - Plural: dog − dogs ≈ cat − cats - Company/product: Microsoft − Windows ≈ Google − Android
# With a well-trained model on large corpus
result = model.wv.most_similar(positive=["king", "woman"], negative=["man"], topn=3)
print(result)
# [("queen", 0.87), ...]
# Nearest neighbours reflect semantic clusters
# model.wv.most_similar("paris", topn=5)
# → london, berlin, madrid, rome, amsterdam
Important caveat: these analogies work best on large corpora (billions of words). On small corpora, the vectors may not encode meaningful relationships. Never demo analogy tasks on small custom-trained models in an interview.
Q4: What is the polysemy problem, and how does Word2Vec fail to solve it?¶
Show answer
Polysemy means a single word has multiple distinct meanings depending on context.
Examples: - "bank" → financial institution OR river bank - "bat" → flying mammal OR sports equipment - "fine" → satisfactory OR penalty OR thin
Word2Vec's failure: Word2Vec assigns one fixed vector per word token regardless of context. The vector for "bank" is a weighted average of all the contexts in which "bank" appeared — a compromise that represents neither meaning well.
"bank" vector ≈ average of:
- financial contexts: interest, loan, deposit, money
- geographic contexts: river, shore, stream, slope
→ The resulting vector sits between both clusters, close to neither
Why this matters in practice: A sentence encoder using Word2Vec for "I went to the river bank" and "I went to the investment bank" produces similar representations — a downstream model cannot distinguish the intent. This is why contextual embeddings (ELMo, BERT) were developed: they produce a different vector for the same word depending on its sentence context.
Q5: How does GloVe differ from Word2Vec?¶
Show answer
GloVe (Global Vectors for Word Representation, Pennington et al., 2014) learns embeddings from a global word co-occurrence matrix rather than local context windows.
Word2Vec approach: - Processes the corpus as a stream of local (word, context) pairs - Optimises predictions one window at a time - Essentially implicit matrix factorisation — never explicitly sees full co-occurrence statistics
GloVe approach: - First builds a global co-occurrence matrix: how often does word i appear near word j across the entire corpus? - Then factorises this matrix to produce embeddings - The objective directly encodes: if words co-occur frequently, their vectors should be close
GloVe objective: dot(v_i, v_j) ≈ log P(i | j) ≈ log(X_ij)
where X_ij = co-occurrence count of words i and j
Practical differences: - GloVe tends to perform slightly better on word analogy benchmarks - Word2Vec trains faster on streaming data (online learning) - Pre-trained GloVe vectors (from Common Crawl, Wikipedia) are widely available as drop-in embeddings - Both share the same polysemy limitation — one vector per word type
In practice, the performance difference between GloVe and Word2Vec on downstream tasks is small. The choice of pre-training corpus matters much more than the choice of algorithm.
Q6: What is FastText, and how does it handle out-of-vocabulary words?¶
Show answer
FastText (Bojanowski et al., 2017) extends Word2Vec by representing each word as the sum of its character n-gram vectors.
Example: the word "eating" is represented as:
- Character n-grams of length 3–6: <ea, eat, ati, tin, ing, ng>, <eat, eati, ...
- The word vector is the sum of all its n-gram vectors
Why this solves OOV: A word not seen during training (e.g. "unhappiness") can still be embedded because its character n-grams ("un", "happ", "iness") were likely seen in other words. The embedding is constructed from these known subword pieces.
Other advantages: - Better representations for morphologically rich languages (German, Finnish, Turkish) - Better on rare words — even infrequent words get stable embeddings via their subword components - Useful for social media text with abbreviations, slang, and misspellings
from gensim.models import FastText
corpus = [["the", "cat", "runs", "fast"], ["dogs", "run", "faster"]]
model = FastText(sentences=corpus, vector_size=50, window=3, min_count=1, epochs=50)
# OOV word: "running" was not in training corpus
# FastText can still produce a vector via character n-grams
vec = model.wv["running"] # Returns a vector, not a KeyError
print(model.wv.similarity("run", "running")) # Should be high
Q7: What are contextual embeddings, and how do ELMo and BERT solve the polysemy problem?¶
Show answer
Contextual embeddings produce a different vector for the same word depending on the sentence it appears in. "Bank" in a financial context gets a different embedding than "bank" in a geographic context.
ELMo (Embeddings from Language Models): - Uses a bidirectional LSTM language model trained on large text - Word representation = weighted combination of all LSTM layer outputs - Different layers capture different levels of abstraction (lower = syntax, higher = semantics) - The word's representation changes based on full sentence context
BERT (Bidirectional Encoder Representations from Transformers): - Uses the Transformer attention mechanism — every word attends to every other word in both directions simultaneously - Pre-trained on Masked Language Modelling: randomly mask 15% of tokens, predict them from context - The final hidden state for each token is a contextual embedding - "bank" in "river bank" and "bank account" produce measurably different vectors
from transformers import BertTokenizer, BertModel
import torch
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertModel.from_pretrained('bert-base-uncased')
def get_word_embedding(sentence, target_word):
inputs = tokenizer(sentence, return_tensors='pt')
with torch.no_grad():
outputs = model(**inputs)
tokens = tokenizer.convert_ids_to_tokens(inputs['input_ids'][0])
idx = tokens.index(target_word)
return outputs.last_hidden_state[0, idx, :]
emb_finance = get_word_embedding("I deposited money at the bank", "bank")
emb_river = get_word_embedding("We sat by the river bank", "bank")
cos = torch.nn.functional.cosine_similarity
print(cos(emb_finance.unsqueeze(0), emb_river.unsqueeze(0)).item())
# Noticeably below 1.0 — different representations for the same token
Q8: How do you measure similarity between word embeddings?¶
Show answer
Cosine similarity is the standard measure. It computes the cosine of the angle between two vectors — length-invariant and bounded in [−1, 1].
For word embeddings: - 1.0: identical direction (synonyms, paraphrases) - 0.5–0.9: semantically related words - ~0.0: no relationship - Negative: rare for word embeddings; indicates semantic opposition
Why not Euclidean distance: Higher-frequency words tend to have larger magnitude vectors. Euclidean distance mixes magnitude (frequency proxy) with direction (semantic meaning). Cosine similarity isolates the directional component.
import numpy as np
def cosine_similarity(v1, v2):
return np.dot(v1, v2) / (np.linalg.norm(v1) * np.linalg.norm(v2))
# Standard evaluation benchmarks for embedding quality:
# - WordSim-353: human similarity judgements for 353 word pairs
# - SimLex-999: stricter similarity (not relatedness) for 999 pairs
# - Correlate embedding cosine similarities with human ratings using Spearman ρ
Q9: How do you use pre-trained word embeddings as features for a downstream task?¶
Show answer
Pre-trained embeddings transfer linguistic knowledge from large corpora to tasks where you have limited labelled data.
Document-level classification (e.g. sentiment analysis): Average the word vectors of all tokens in the document to get a fixed-size document representation.
import numpy as np
def document_vector(text, wv, vector_size):
tokens = text.lower().split()
vectors = [wv[t] for t in tokens if t in wv]
if not vectors:
return np.zeros(vector_size)
return np.mean(vectors, axis=0)
# Use as sklearn-compatible features
from sklearn.linear_model import LogisticRegression
# X_train = np.array([document_vector(t, model.wv, 100) for t in train_texts])
# clf = LogisticRegression(max_iter=500).fit(X_train, train_labels)
Frozen vs fine-tuned embeddings: - Frozen: pre-trained vectors as fixed input features — good when training data is small - Fine-tuned: initialise with pre-trained vectors and allow them to update during training — better when you have enough data and the domain differs from the pre-training corpus
When to use sentence transformers instead:
For document-level tasks, sentence-transformers (SBERT) produces sentence-level embeddings optimised for semantic similarity — they outperform averaged word vectors significantly and should be the default when compute allows.
Q10: What kinds of biases exist in word embeddings, and how do they arise?¶
Show answer
Word embeddings trained on human-generated text absorb the biases present in that text — reflecting historical societal biases, not objective semantic relationships.
Documented biases: - Gender bias: "doctor" is closer to "he" than "she"; "nurse" is closer to "she" than "he" - Racial bias: names associated with different ethnic groups produce different similarity patterns to positive/negative words - Occupational bias: certain professions cluster closer to male names, matching historical labour market patterns
Why they arise: - Training corpora (Wikipedia, news, books) reflect historical and cultural stereotypes - Distributional semantics learns what humans write, including their prejudices - Biased text → biased co-occurrence statistics → biased embedding geometry
How to measure bias: - WEAT (Word Embedding Association Test): measures effect size of association between target and attribute word sets
Mitigation approaches: - Hard debiasing: identify the "gender direction" in embedding space and project it out from neutral word vectors - Counterfactual data augmentation: create gender-swapped versions of training sentences - Adversarial debiasing: train an adversary to predict sensitive attributes; punish the encoder for making this easy
# Measuring gender bias conceptually
# male_words = ["man", "he", "his", "male", "father"]
# female_words = ["woman", "she", "her", "female", "mother"]
# target = "doctor"
# bias = mean([similarity(target, m) for m in male_words])
# - mean([similarity(target, f) for f in female_words])
# positive bias → word is closer to male words
Biases in embeddings propagate to downstream models — a resume-screening model trained on biased embeddings will inherit gender bias in its predictions, often without the model developer noticing.
Q11: When would you choose FastText over Word2Vec or GloVe?¶
Show answer
Choose FastText when:
OOV words are expected at inference time. Social media, user-generated content, and domain-specific text regularly contain novel words, misspellings, neologisms, and abbreviations. FastText handles all of them via character n-grams.
Your domain has rich morphology. Languages like German, Turkish, or Finnish have extremely productive word formation. FastText's subword approach handles these naturally.
You have a small training corpus. FastText's character n-gram sharing means even rare words benefit from the statistics of their component substrings — effectively giving rare words more training signal than Word2Vec would.
Choose Word2Vec or GloVe when: - You are working with clean English text with consistent spelling - You have a large corpus and stable vocabulary - You need faster training (no n-gram overhead in Word2Vec) - You want widely available pre-trained vectors (GloVe Common Crawl, Google News Word2Vec)
In modern NLP, all three static embedding methods are increasingly superseded by contextual embeddings — but FastText remains the best static embedding for resource-constrained production deployments.
Q12: What is the difference between a word embedding and a sentence embedding?¶
Show answer
Word embeddings represent individual tokens as dense vectors. Static embeddings (Word2Vec, GloVe, FastText) give the same vector regardless of context; contextual embeddings (ELMo, BERT) vary by sentence context.
Sentence embeddings represent an entire sentence or document as a single fixed-size vector, designed so that semantically similar sentences have similar vectors.
Methods for sentence embeddings: - Averaging word vectors: fast but loses word order — mediocre for semantic similarity - BERT [CLS] token: the CLS token output is used as a sentence representation, but vanilla BERT was not fine-tuned for this task and performs poorly on similarity benchmarks - Sentence-BERT (SBERT): fine-tunes BERT with a siamese network on sentence pairs labelled for semantic similarity — the go-to choice for production semantic search - Universal Sentence Encoder (USE): Google's pre-trained sentence encoder
from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity
model = SentenceTransformer('all-MiniLM-L6-v2')
sentences = [
"The cat sat on the mat",
"A feline rested on a rug", # paraphrase of sentence 0
"The stock market crashed today" # unrelated
]
embeddings = model.encode(sentences)
sims = cosine_similarity(embeddings)
print(sims.round(3))
# Sentences 0 and 1 show high similarity; sentence 2 shows low similarity with both