What Is Machine Learning?¶

Every software system before ML was built around explicit rules: an engineer studied a problem, wrote logic, and the computer executed it. ML inverts that contract. Instead of writing the rules, you show the system thousands of examples and let it figure the rules out. That shift sounds subtle. Its consequences are enormous.

Understanding exactly what that shift means — and where it breaks down — is the foundation everything else in Week 2 builds on.

Learning Objectives¶

Explain what ML is in terms of function approximation, not buzzwords
Identify the three paradigms (supervised, unsupervised, reinforcement) at a high level
Describe the full ML project loop from problem definition to deployment
Know when ML is the right tool and, equally important, when it is not
Map a new business problem to the core ML vocabulary: dataset, feature, target, model, evaluation

The Shift from Rules to Patterns¶

How traditional software works¶

A traditional program is a hand-coded decision tree. Consider email spam filtering circa 2000:

def is_spam(email: str) -> bool:
    keywords = ["free money", "click here", "you have won"]
    for keyword in keywords:
        if keyword in email.lower():
            return True
    if email.count("!") > 5:
        return True
    return False

This works until spammers learn your rules and write around them. The engineer then has to update the rules. This is an arms race the rules always lose.

How ML works¶

Instead of writing rules, you collect labeled examples and let an algorithm discover the pattern:

# Traditional programming
# rules + data → output

# Machine learning
# data + desired output → learned rules (model)

With ML, a spam filter sees 100,000 (email, label) pairs where label is spam or not_spam. The algorithm finds statistical patterns that distinguish them — far more patterns than any human would think to encode, and patterns that adapt as you feed it new data.

Info

The formal definition: machine learning is the field of giving computers the ability to learn from data without being explicitly programmed for every case. (Mitchell, 1997: "A computer program is said to learn from experience E with respect to some task T and some performance measure P, if its performance on T, as measured by P, improves with experience E.")

The Mental Model: ML as Function Approximation¶

The cleanest way to think about supervised ML is as function approximation.

You have a true underlying function f that maps inputs to outputs. You do not know f — if you did, you would just write it down. But you have samples from f: pairs of (X, y) where y = f(X) (plus noise).

An ML model learns an approximation f̂ such that f̂(X) ≈ y on the data you have seen, and hopefully generalises — meaning f̂(X_new) ≈ y_new on data it has never seen.

True world:     y = f(X) + noise
What you have:  N examples of (X, y)
What ML does:   learns f̂ such that f̂(X) ≈ y
Goal:           f̂(X_new) ≈ y_new   ← this is generalisation

Everything in ML — regularisation, train/test splits, cross-validation, dropout — exists to improve generalisation. Keep that word in your head throughout Week 2.

Core Vocabulary¶

Term	Definition	Example
Dataset	A table of examples	10,000 rows of customer records
Sample / row	One observation	A single customer
Feature	An input variable (column)	Age, plan type, monthly spend
Target / label	The value to predict	Churn: yes or no
Model	A learned function `f̂`	A trained logistic regression
Training	Fitting `f̂` to data	Running `model.fit(X_train, y_train)`
Prediction	Applying `f̂` to new data	`model.predict(X_new)`
Evaluation	Measuring how well `f̂` generalises	Accuracy on held-out test set

import pandas as pd
from sklearn.datasets import load_breast_cancer

# Load a real dataset to see vocabulary in action
data = load_breast_cancer(as_frame=True)

# Dataset: a DataFrame
df = data.frame
print(df.shape)         # Output: (569, 31)

# Features: columns the model uses as input
X = data.data
print(X.columns.tolist()[:5])
# Output: ['mean radius', 'mean texture', 'mean perimeter', 'mean area', 'mean smoothness']

# Target: what we predict
y = data.target
print(y.value_counts().to_dict())   # Output: {1: 357, 0: 212}
# 1 = malignant, 0 = benign

The Three Paradigms¶

Supervised learning¶

You have (X, y) pairs. The y is known during training. You learn a mapping from X to y.

Classification: y is a category (spam/not spam, disease/healthy, A/B/C)
Regression: y is a number (house price, temperature, number of days to churn)

Proportion of real ML work: ~80%.

Unsupervised learning¶

You have X only — no y. You look for structure in the data itself.

Clustering: group similar samples (customer segments)
Dimensionality reduction: compress many features into fewer (PCA, UMAP)
Density estimation: model the distribution of the data

Proportion of real ML work: ~15%.

Reinforcement learning¶

An agent takes actions in an environment and receives rewards. It learns a policy that maximises cumulative reward over time.

AlphaGo, robotic control, recommendation systems as policies
Not covered in this bootcamp, but important to know the distinction

Info

There are also semi-supervised learning (a small amount of labeled data plus a large amount of unlabeled) and self-supervised learning (the label is derived from the data itself — e.g., predicting the next word in a sentence). These underpin modern language models. Worth knowing the terms even if we do not build them here.

Realistic Problem Mapping¶

For each problem, before you touch any code, ask three questions:

What decision or prediction does a human currently make by hand?
What data exists that correlates with the correct answer?
What does "correct" mean — how would you measure a good model?

Business Problem	Features	Target	Type
Predict house price	bedrooms, area, postcode, age	price (£)	Regression
Predict loan default	income, credit score, debt	default: yes/no	Classification
Segment customers	spend, frequency, tenure	cluster id	Clustering
Flag fraudulent transactions	amount, location, time, merchant	fraud: yes/no	Classification
Recommend products	purchase history, browsing	product to show	Supervised / RL
Compress images for storage	pixel values	compressed representation	Unsupervised

# Identifying features and target from a raw DataFrame
import pandas as pd
import numpy as np

# Synthetic customer churn dataset
np.random.seed(42)
n = 500

churn_df = pd.DataFrame({
    "tenure_months":   np.random.randint(1, 72, n),
    "monthly_spend":   np.random.uniform(20, 200, n).round(2),
    "num_complaints":  np.random.poisson(0.5, n),
    "plan_type":       np.random.choice(["basic", "standard", "premium"], n),
    "customer_id":     [f"C{i:04d}" for i in range(n)],  # identifier — NOT a feature
    "churn":           np.random.choice([0, 1], n, p=[0.75, 0.25]),
})

# Features: what the model learns from
feature_cols = ["tenure_months", "monthly_spend", "num_complaints", "plan_type"]
X = churn_df[feature_cols]

# Target: what we predict
y = churn_df["churn"]

print("Features shape:", X.shape)   # Output: (500, 4)
print("Target distribution:\n", y.value_counts())
# Output:
# 0    375
# 1    125

Warning

customer_id is an identifier, not a feature. Including it leaks identity into the model. If IDs are sequential, the model can learn a spurious pattern between ID range and outcome. Always drop identifier columns before training.

The ML Project Loop¶

This is the workflow you will run — with variations — on every project in this bootcamp.

1. Define the problem
   └─ What decision does the model support?
   └─ What is success? (metric, threshold, business impact)

2. Collect and inspect data
   └─ Where does data come from? Is it reliable?
   └─ How much do you have? Is it representative?

3. Explore and clean data  ← Week 1 was here
   └─ Missing values, outliers, distributions
   └─ Correlations, class imbalance

4. Split into train / validation / test  ← This session
   └─ Separation of training from evaluation is non-negotiable

5. Preprocess and engineer features
   └─ Encode categoricals, scale numerics, create new features
   └─ All preprocessing fitted on train only

6. Train model
   └─ Choose algorithm, fit on training data

7. Evaluate on validation set
   └─ Tune hyperparameters, compare models

8. Final evaluation on test set  ← Done once, at the end
   └─ Honest estimate of real-world performance

9. Interpret results and communicate
   └─ What drives predictions? Where does the model fail?

10. Deploy and monitor
    └─ Performance degrades over time as the world changes

Success

Steps 4 and 5 are where most production ML failures originate. A model with a clean split, no leakage, and proper preprocessing running on a simple algorithm almost always beats a sophisticated model built carelessly. Foundations first.

When ML Is the Right Tool¶

ML makes sense when:

The relationship between inputs and output is too complex to encode as rules
You have sufficient labeled data (or can collect it)
The pattern is stable enough that historical data predicts future data
The cost of a mistake is manageable given the model's error rate

When ML is overkill — use a simpler tool instead¶

Situation	Better approach
You have fewer than 100 examples	Hard-coded rules or expert system
The logic can be stated in < 20 if/else rules	Write the rules
You need a legally auditable decision process	Rules or linear model with documented thresholds
You need zero false positives	Rules with whitelists, not probabilistic models
The relationship changes too fast for historical data	Adaptive rules, human-in-the-loop

Warning

The most expensive ML mistake is spending 3 months building a model for a problem that a two-column lookup table would solve. Before reaching for sklearn, ask: "could a smart intern with a spreadsheet solve this?" If yes, let them.

Why Data Quality Beats Algorithm Choice¶

This is worth saying plainly before you spend any time comparing algorithms:

# A clean dataset with a simple model usually wins.
# A dirty dataset with a sophisticated model usually loses.

# Rough ranking of what matters:
# 1. Problem framing (correct target? right success metric?)
# 2. Data quality and volume
# 3. Feature engineering
# 4. Model selection and hyperparameter tuning

Studies consistently show that in most structured data problems (tabular data, business analytics), model choice accounts for less variation in outcomes than feature engineering and data quality. Random forests and gradient boosting dominate tabular leaderboards not because they are magic but because they handle messy real-world data robustly.

Quote

"More data beats better algorithms. Better data beats more data." — Peter Norvig (paraphrased)

Interview Questions¶

Q1: What is machine learning? Explain it to someone non-technical.

Show answer

Machine learning is a way of teaching a computer to make decisions or predictions by showing it many examples, rather than programming every rule by hand. A spam filter built with ML does not have a list of banned phrases — it has seen millions of emails labeled "spam" or "not spam" and learned the patterns that distinguish them.

Q2: What is the difference between a feature and a target?

Show answer

A feature is an input column — information available at prediction time that the model uses to make its decision. The target is the column the model is trained to predict. In a churn model, features include tenure and monthly spend; the target is whether the customer churned. The target is what you know in training data but do not know for new customers yet.

Q3: When would you not use machine learning?

Show answer

When the logic can be expressed as a small set of explicit rules. When you have too little data for a model to generalise. When the decision requires full legal auditability that a black-box model cannot provide. When the cost of a wrong prediction is catastrophic and the model cannot achieve a high enough precision. When historical data does not predict future outcomes because the environment has fundamentally changed.

Q4: What does "generalisation" mean in ML, and why does it matter?

Show answer

Generalisation means the model performs well on data it was not trained on — data from the real world that arrives after the model is deployed. A model that memorises its training data scores perfectly on training but fails on new inputs. Good generalisation is the entire goal. Every technique in ML — regularisation, dropout, cross-validation, early stopping — exists to improve generalisation.

What's Next¶

You've covered the definition of machine learning, the supervised/unsupervised/reinforcement taxonomy, the ML project loop from problem definition to deployment, when ML is the right tool versus simpler approaches, and why data quality and feature engineering matter more than algorithm choice. Next up: 02-supervised-unsupervised — where you'll go deeper into the distinction between classification and regression, explore clustering and dimensionality reduction, and learn a decision framework for choosing the right learning paradigm for any new problem.

Optional Deep Dive

Read "The Hundred-Page Machine Learning Book" by Andriy Burkov (free PDF available at themlbook.com), Chapter 1 and 2 — it gives a rigorous but concise mathematical grounding for the intuitions introduced here, covering feature vectors, learning algorithms, and the bias-variance tradeoff in about 20 pages.

Agenda | Next: Supervised vs Unsupervised