Glossary¶
Key terms used throughout this course. Each entry has a brief definition and a link to where the concept is taught in depth. Use this as a quick-reference when a term appears in notes or exercises and you need a fast reminder before reading further.
A¶
Accuracy The fraction of predictions a classifier gets correct: (TP + TN) / total. Misleading on imbalanced datasets — a model predicting the majority class always achieves high accuracy while catching nothing. → Classification Metrics
Activation function A non-linear function applied to a neuron's output — ReLU, sigmoid, tanh, softmax. Without non-linearity, stacking layers is equivalent to a single linear transformation. → Neural Network Intuition
Adam An adaptive optimiser that combines momentum and RMSProp to compute per-parameter learning rates. The default choice for training most neural networks. → Training Neural Networks
A/B testing A controlled experiment that randomly assigns users to a control group (A) and a treatment group (B) to measure the causal effect of a change. Requires proper sample size calculation and a single pre-specified primary metric. → A/B Testing
AUC (Area Under the ROC Curve) A threshold-independent summary of classifier performance. AUC = 1.0 is perfect; AUC = 0.5 is a random classifier. Interpreted as the probability the model ranks a random positive higher than a random negative. → Classification Metrics
B¶
Backpropagation The algorithm for computing gradients in a neural network by applying the chain rule backwards through the computation graph. Gradients are used by the optimiser to update weights. → Training Neural Networks
Bagging (Bootstrap Aggregating) An ensemble method that trains multiple models in parallel on random bootstrap samples of the training data, then averages their predictions. Reduces variance. Random Forest uses bagging. → Tree-Based Regression
Batch normalisation A technique that normalises the inputs to each layer across the mini-batch during training. Stabilises training, allows higher learning rates, and acts as a mild regulariser. → Training Neural Networks
Bias (statistical) The systematic error of a model — the gap between the expected prediction and the true value. A high-bias model is too simple and underfits. Distinct from algorithmic bias (unfairness). → ML Fundamentals
Bias-variance tradeoff Total error = Bias² + Variance + irreducible noise. Reducing bias (more complex model) tends to increase variance, and vice versa. The goal is the sweet spot that minimises total error on unseen data. → ML Fundamentals
Boosting An ensemble method that trains models sequentially, where each model focuses on the errors of the previous one. Reduces bias. Examples: AdaBoost, Gradient Boosting, XGBoost, LightGBM. → Trees, Forests, and Boosting
Bootstrap A resampling technique that draws samples with replacement from a dataset to estimate statistics or train ensemble members. Sampling with replacement means some observations appear multiple times, some not at all.
C¶
Categorical encoding Converting categorical variables into numeric representations a model can use. Methods include one-hot encoding, ordinal encoding, target encoding, and label encoding. Choice depends on model type and cardinality. → Categorical Features
Chi-squared test A statistical test for independence between two categorical variables. The test statistic measures how far observed frequencies are from expected frequencies under independence. → Hypothesis Testing
Class imbalance When one class in a classification problem occurs much more frequently than another (e.g., fraud: 0.1%, legitimate: 99.9%). Requires special handling: resampling, class weights, or threshold adjustment. → Classification Metrics
Classification A supervised learning task where the model predicts a discrete category (class). Binary classification has two classes; multi-class has three or more. → Classification Overview
Clustering An unsupervised learning task that groups data points so that points within a group are more similar to each other than to points in other groups. No labels are used. → Clustering Overview
Confusion matrix A table showing true positives, false positives, true negatives, and false negatives for a classifier. The foundation for computing precision, recall, F1, and accuracy. → Classification Metrics
Confidence interval A range of values that contains the true population parameter with a specified probability (e.g., 95%). Wider intervals indicate more uncertainty. Does not mean there is a 95% chance the true value is in any particular interval. → Confidence Interval
Correlation A measure of the linear relationship between two variables, ranging from -1 (perfect negative) to +1 (perfect positive). Zero correlation does not imply independence. Correlation does not imply causation. → Correlation
Cross-validation A model evaluation technique that trains and validates on K different data splits, averaging the scores. More reliable than a single train-test split because every sample is used for both training and validation. → Cross-Validation
Curse of dimensionality As the number of features grows, the volume of feature space grows exponentially, making data increasingly sparse. Distance-based algorithms degrade; models need exponentially more data. → ML Fundamentals
D¶
Data augmentation Artificially expanding a training dataset by creating modified versions of existing samples — flipping images, adding noise, paraphrasing text. Reduces overfitting when labelled data is scarce. → Overfitting and Regularization
Data leakage When information from outside the training window is used to build a model — either target leakage (a feature derived from the target) or train-test contamination (preprocessing fit on the full dataset). Produces unrealistically high validation scores. → Train-Test Split and Leakage
DBSCAN Density-Based Spatial Clustering of Applications with Noise. Groups points that are closely packed, marks low-density points as outliers. Does not require specifying the number of clusters in advance. → DBSCAN
Decision tree A model that makes predictions by learning a series of binary splits on feature values. Interpretable but prone to overfitting without pruning or depth limits. → Tree-Based Regression
Dimensionality reduction Transforming data from a high-dimensional space to a lower-dimensional one while preserving as much structure as possible. PCA is the most common linear method; t-SNE and UMAP are non-linear alternatives.
Dropout A regularisation technique for neural networks that randomly zeroes a fraction of neuron outputs during each training step, preventing co-adaptation and reducing overfitting. Turned off at inference time. → Overfitting and Regularization
E¶
Early stopping A regularisation technique that halts training when validation loss stops improving, preventing the model from memorising training noise. Requires a held-out validation set monitored during training. → Overfitting and Regularization
ElasticNet A regularisation method combining L1 (Lasso) and L2 (Ridge) penalties. Useful when features are correlated — Lasso alone would arbitrarily select one; ElasticNet can keep both. → Ridge, Lasso, and ElasticNet
Epoch One complete pass through the entire training dataset during neural network training. A model typically trains for multiple epochs; the right number is found via early stopping or cross-validation.
Ensemble method A model that combines the predictions of multiple base models. Ensembles almost always outperform individual models by reducing variance (bagging) or bias (boosting). → Trees, Forests, and Boosting
F¶
F1 score The harmonic mean of precision and recall: 2 × (Precision × Recall) / (Precision + Recall). Useful when both false positives and false negatives matter and the dataset is imbalanced. → Classification Metrics
Feature engineering The process of creating, transforming, or selecting input variables to improve model performance. Often contributes more to model quality than algorithm choice. → Feature Engineering Overview
Feature importance A measure of how much each feature contributes to a model's predictions. Tree-based models provide impurity-based importance; permutation importance is a more reliable alternative. → Feature Engineering
Feature scaling Transforming numeric features to a common scale. Standardisation (zero mean, unit variance) and min-max normalisation are the most common. Required for distance-based models (KNN, SVM, K-means) and gradient descent. → Numeric Features
Fine-tuning Adapting a pre-trained model to a new, specific task by continuing training on task-specific data, typically with a lower learning rate. Most of the original weights are retained. → Transformers Overview
G¶
Gradient descent An iterative optimisation algorithm that updates model parameters in the direction of the negative gradient of the loss function. Variants include batch GD, stochastic GD (SGD), and mini-batch GD. → Training Neural Networks
Gradient boosting A boosting method that fits each new model to the residual errors (negative gradients of the loss) of the ensemble so far. XGBoost, LightGBM, and CatBoost are efficient implementations. → Trees, Forests, and Boosting
GridSearchCV An exhaustive hyperparameter search that evaluates every combination of specified parameter values using cross-validation. Use RandomizedSearchCV when the parameter space is large. → Model Selection and Tuning
H¶
Hierarchical clustering A clustering approach that builds a tree of clusters (dendrogram) by iteratively merging (agglomerative) or splitting (divisive) groups. The number of clusters is chosen by cutting the dendrogram. → Hierarchical Clustering
Hyperparameter A parameter set before training that controls the learning process — learning rate, max depth, number of estimators. Distinct from model parameters, which are learned from data. Tuned via grid search, random search, or Bayesian optimisation. → Model Selection and Tuning
Hypothesis testing A statistical procedure to decide whether data provides enough evidence to reject a default assumption (the null hypothesis). Produces a p-value; rejected if p < α. → Hypothesis Testing
I¶
Imputation Filling in missing values with estimated values — mean, median, mode, or model-predicted values. The choice depends on the missingness mechanism (MCAR, MAR, MNAR). → Data Cleaning
Information gain The reduction in entropy (or impurity) achieved by splitting on a feature. Used by decision tree algorithms (ID3, C4.5) to choose the best split at each node.
K¶
K-fold cross-validation Cross-validation with K non-overlapping folds. Train on K-1 folds, validate on the remaining fold, rotate K times. K=5 or K=10 are standard choices. → Cross-Validation
K-means clustering A centroid-based clustering algorithm that assigns points to the nearest centroid and updates centroids iteratively. Requires K to be specified in advance. Sensitive to outliers and initial centroid placement. → K-Means
KNN (K-Nearest Neighbours) A non-parametric algorithm that predicts by averaging (regression) or majority-voting (classification) the K nearest training points. No explicit training phase — all computation happens at prediction time. → KNN and Naive Bayes
L¶
L1 regularisation (Lasso) Adds the sum of absolute weights as a penalty to the loss function. Drives some weights to exactly zero, performing implicit feature selection. → Ridge, Lasso, and ElasticNet
L2 regularisation (Ridge) Adds the sum of squared weights as a penalty. Shrinks all weights toward zero without zeroing them. Handles multicollinearity well. → Ridge, Lasso, and ElasticNet
Learning rate A hyperparameter controlling how much model weights change per gradient step. Too high → diverges; too low → converges slowly. Learning rate scheduling (decay, warmup) is used to adapt it during training.
Lemmatisation Reducing a word to its canonical dictionary form (lemma). "Running" → "run", "better" → "good". More accurate than stemming but slower. → Text Preprocessing
Linear regression A supervised learning algorithm that models the relationship between features and a continuous target as a linear function. Optimised by minimising mean squared error. → Linear Regression
Logistic regression A classification algorithm that models the probability of a binary outcome using the sigmoid function. Despite the name, it is a classifier, not a regressor. → Logistic Regression
Loss function A function that measures the discrepancy between the model's predictions and the true labels. Training minimises the loss. Common examples: MSE for regression, binary cross-entropy for classification.
M¶
MAE (Mean Absolute Error) Average absolute difference between predictions and true values. Robust to outliers. Interpretable in the same units as the target. → Regression Metrics
MSE (Mean Squared Error) Average squared difference between predictions and true values. Penalises large errors heavily. Differentiable, making it convenient for gradient-based optimisation. → Regression Metrics
Multicollinearity When two or more features are highly correlated. In linear models, causes unstable coefficient estimates with inflated standard errors. Detected via Variance Inflation Factor (VIF). Addressed by Ridge regression or removing one correlated feature. → Regression Interview Prep
N¶
Naive Bayes A probabilistic classifier based on Bayes' theorem that assumes features are conditionally independent given the class. "Naive" because the independence assumption is rarely true. Works well for text classification despite this. → KNN and Naive Bayes
Normalisation Scaling features to a fixed range, typically [0, 1]. Also used loosely to mean standardisation. Use min-max normalisation when you know the bounds; use standardisation otherwise. → Numeric Features
Null hypothesis (H₀) The default assumption in hypothesis testing — no effect, no difference. Rejected when data is sufficiently inconsistent with it (p < α). Failing to reject H₀ is not the same as proving it true. → Hypothesis Testing
O¶
One-hot encoding Representing a categorical variable with K categories as K binary columns (0 or 1), one per category. Can cause the dummy variable trap — drop one column when using with linear models. → Categorical Features
Outlier A data point significantly different from the rest of the distribution. May be a measurement error, a data entry mistake, or a genuinely extreme but valid value. Must be investigated before being removed. → Outlier Detection
Overfitting When a model learns the training data too well — including its noise — and fails to generalise to new data. Detected by a large gap between training and validation scores. → Train-Test Split and Leakage
P¶
P-value The probability of observing data as extreme as the observed result, assuming the null hypothesis is true. A small p-value is evidence against H₀. It is not the probability that H₀ is true. → P-value
Pipeline (scikit-learn) A scikit-learn object that chains preprocessing steps and a model into a single estimator. Ensures transformations are fit only on training data, preventing data leakage. → Pipelines and Leakage
Precision Of all instances the model predicted as positive, the fraction that were actually positive: TP / (TP + FP). Optimise precision when false positives are costly (spam filters, legal review). → Classification Metrics
Principal Component Analysis (PCA) A linear dimensionality reduction technique that projects data onto the directions of maximum variance (principal components). Useful for visualisation, noise reduction, and addressing the curse of dimensionality.
R¶
R² (coefficient of determination) The proportion of variance in the target explained by the model. R² = 1 means perfect fit; R² = 0 means the model does no better than predicting the mean. Can be negative for very poor models. → Regression Metrics
Random Forest An ensemble of decision trees trained on random bootstrap samples of data, with each split restricted to a random subset of features. Reduces variance through averaging. Robust and rarely needs much tuning. → Trees, Forests, and Boosting
Recall (Sensitivity) Of all actual positives, the fraction the model correctly identified: TP / (TP + FN). Optimise recall when false negatives are costly (cancer screening, fraud detection). → Classification Metrics
Regression A supervised learning task where the model predicts a continuous numeric value. → Regression Overview
Regularisation Adding a penalty to the loss function that discourages overly complex models. Reduces variance at the cost of a small increase in bias. L1 and L2 are the most common forms. → Ridge, Lasso, and ElasticNet
Residual The difference between the actual value and the model's predicted value: residual = y − ŷ. Residual analysis is a key tool for diagnosing regression model problems. → Regression Metrics
RMSE (Root Mean Squared Error) Square root of MSE. Expressed in the same units as the target. More sensitive to large errors than MAE due to squaring. → Regression Metrics
ROC curve A plot of True Positive Rate (recall) vs False Positive Rate across all classification thresholds. A perfect classifier hugs the top-left corner. Summarised by AUC. → Classification Metrics
S¶
Scikit-learn The standard Python machine learning library. Provides a consistent fit / transform / predict API across all models and preprocessing steps. → Scikit-learn Workflow
Sigmoid function A function that squashes any real number to (0, 1): σ(x) = 1 / (1 + e^−x). Used as the output activation in binary classification to produce a probability. Suffers from vanishing gradients for deep networks — replaced by ReLU in hidden layers.
Silhouette score A clustering evaluation metric ranging from -1 to +1 that measures how similar a point is to its own cluster versus other clusters. Higher is better. → Evaluation and Scaling
Softmax A function that converts a vector of raw scores (logits) into a probability distribution that sums to 1. Used as the output activation for multi-class classification.
Standardisation (Z-score normalisation) Scaling a feature to have zero mean and unit variance: z = (x − μ) / σ. Most common default for numeric feature scaling. → Numeric Features
Stemming Reducing a word to its root by stripping affixes — often produces non-words ("running" → "run", "studies" → "studi"). Faster but less accurate than lemmatisation. → Text Preprocessing
Supervised learning A machine learning paradigm where models are trained on labelled data (input-output pairs). The model learns to map inputs to outputs. Includes regression and classification. → Supervised and Unsupervised Learning
T¶
Target encoding Encoding a categorical variable by replacing each category with the mean of the target variable for that category. Powerful for high-cardinality features but leaks target information — must be computed within each cross-validation fold. → Categorical Features
TF-IDF (Term Frequency–Inverse Document Frequency) A numeric representation of text that weights a word by how often it appears in a document (TF) divided by how common it is across all documents (IDF). Rare but frequent words get high scores. → BoW and TF-IDF
Tokenisation Splitting raw text into discrete units (tokens) — typically words or subwords. The first step in any NLP pipeline. Modern models use subword tokenisation (BPE) to handle rare and unknown words. → Text Preprocessing
Train-test split Dividing a dataset into a training set (model learns from this) and a test set (used once to report final performance). The test set must never be touched during model development. → Train-Test Split and Leakage
Transfer learning Using a model pre-trained on a large dataset as the starting point for a new, related task. The early layers capture general features; later layers are fine-tuned for the specific task. Common in deep learning and NLP. → Transformers Overview
Transformer A neural network architecture based entirely on attention mechanisms (no recurrence). The foundation for BERT, GPT, and most modern NLP models. Enables parallelism and long-range dependency learning. → Transformers Overview
Type I error (False Positive) Rejecting the null hypothesis when it is actually true. The probability of a Type I error is α (the significance level). → Hypothesis Testing
Type II error (False Negative) Failing to reject the null hypothesis when it is actually false — missing a real effect. The probability is β; power = 1 − β. → Hypothesis Testing
U¶
Underfitting When a model is too simple to capture the true pattern in the data — high training error and high validation error. Addressed by increasing model complexity, adding features, or reducing regularisation. → What is Machine Learning
Unsupervised learning A machine learning paradigm where models learn structure from unlabelled data. Clustering, dimensionality reduction, and anomaly detection are unsupervised tasks. → Supervised and Unsupervised Learning
V¶
Validation set A held-out portion of data used to tune hyperparameters and compare models during development. Distinct from the test set, which is used only once for final evaluation. → Train-Test Split and Leakage
Variance (statistical) A measure of how spread out a distribution is: the average squared deviation from the mean. High variance in a model means it is sensitive to the specific training data — it will change a lot if the training set changes. → Variance and Standard Deviation
Vectorisation In NumPy/Pandas: applying operations to entire arrays without Python loops, using compiled C code. Much faster than element-wise loops. → Vectorization
In NLP: converting text into numeric vectors that a model can process (BoW, TF-IDF, word embeddings). → BoW and TF-IDF
W¶
Word embeddings Dense, low-dimensional vector representations of words that capture semantic meaning. Similar words have similar vectors. Word2Vec, GloVe, and FastText produce static embeddings; BERT produces contextual embeddings. → Sentiment Classification
Word2Vec A neural word embedding model trained to predict surrounding words (Skip-gram) or predict a word from its context (CBOW). Captures semantic relationships: king − man + woman ≈ queen.
X¶
XGBoost An efficient, regularised gradient boosting implementation that is a go-to for tabular data competitions and industry. Key hyperparameters: n_estimators, learning_rate, max_depth, subsample, colsample_bytree. → Trees, Forests, and Boosting
How to use this glossary
When you hit an unfamiliar term in the notes, look it up here for a quick definition, then follow the link to read the full explanation in context. Reading the definition alone rarely builds the intuition — the example and code in the linked note do.