Skip to content

Neural Network Intuition

Every breakthrough in computer vision, language translation, and protein folding over the last decade runs on the same core idea: stack simple mathematical functions on top of each other, train them end-to-end on data, and the composition learns to represent extraordinarily complex patterns. Understanding that idea deeply — not just knowing the API — is what separates practitioners who can debug neural networks from those who are stuck copying tutorials.

Learning Objectives

  • Describe what a single neuron computes, with numbers
  • Explain why activation functions are non-negotiable
  • Trace a forward pass through a two-layer network by hand
  • Articulate the universal approximation theorem in one sentence
  • Make an honest judgment about when deep learning is worth the complexity

The Neuron: A Linear Function With a Twist

A single neuron takes a vector of inputs, multiplies each input by a learned weight, sums the results, adds a learned bias, and then passes the total through an activation function.

Mathematically:

output = activation(w1*x1 + w2*x2 + ... + wn*xn + b)

That weighted sum is just a dot product. The activation is what makes it interesting.

A Forward Pass With Real Numbers

Suppose a neuron receives two inputs: x1 = 0.5 and x2 = -1.2. Its learned weights are w1 = 0.8 and w2 = 0.3, and its bias is b = 0.1.

import numpy as np

x = np.array([0.5, -1.2])
w = np.array([0.8, 0.3])
b = 0.1

# Step 1: weighted sum
z = np.dot(w, x) + b
print(f"z = {z:.4f}")
# Output: z = 0.1400

# Step 2: apply ReLU activation
output = np.maximum(0, z)
print(f"output = {output:.4f}")
# Output: output = 0.1400

Now replace x1 = 0.5 with x1 = -2.0:

x = np.array([-2.0, -1.2])
z = np.dot(w, x) + b
print(f"z = {z:.4f}")
# Output: z = -1.2600

output = np.maximum(0, z)
print(f"output = {output:.4f}")
# Output: output = 0.0000

ReLU killed the signal. That is intentional — the neuron learned "I am not relevant for this input pattern." Dead neurons act as gates.

Why Activation Functions Are Non-Negotiable

Without an activation function, a stack of Dense layers is mathematically equivalent to a single linear layer, no matter how many layers you add.

# Two linear layers with no activation
# Layer 1: output = W1 @ x + b1
# Layer 2: output = W2 @ (W1 @ x + b1) + b2
#                 = (W2 @ W1) @ x + (W2 @ b1 + b2)
#                 = W_combined @ x + b_combined
# — still just a linear transformation

A nonlinear activation breaks this collapse. After ReLU, the composition of layers cannot be reduced to a single linear equation. The network gains the capacity to represent curved decision boundaries, not just straight lines.

Info

The four activations you need to know:

  • ReLU (max(0, z)) — default for hidden layers. Fast, sparse, works well in practice.
  • Sigmoid (1 / (1 + e^-z)) — squashes output to (0, 1). Use in the final layer for binary classification only.
  • Softmax — squashes a vector to probabilities that sum to 1. Use in the final layer for multiclass classification.
  • tanh — squashes to (-1, 1). Occasionally useful in recurrent networks. Otherwise, prefer ReLU.
import numpy as np
import matplotlib.pyplot as plt

z = np.linspace(-4, 4, 200)

relu    = np.maximum(0, z)
sigmoid = 1 / (1 + np.exp(-z))
tanh    = np.tanh(z)

fig, axes = plt.subplots(1, 3, figsize=(12, 4))
for ax, y, name in zip(axes, [relu, sigmoid, tanh], ["ReLU", "Sigmoid", "tanh"]):
    ax.plot(z, y, linewidth=2)
    ax.axhline(0, color="gray", linewidth=0.5)
    ax.axvline(0, color="gray", linewidth=0.5)
    ax.set_title(name)
    ax.set_xlabel("z")

plt.tight_layout()
plt.savefig("activations.png", dpi=120)
plt.show()

Layers: Organizing Neurons in Parallel

A layer is a collection of neurons that all receive the same input but have their own independent weights. If a layer has 32 neurons, it produces 32 outputs — one per neuron. The next layer receives all 32 as its inputs.

import numpy as np

# Simulate a dense layer with 3 inputs → 4 neurons (no activation for clarity)
np.random.seed(42)
X = np.array([[0.5, -1.2, 0.3]])  # shape: (1, 3)
W = np.random.randn(3, 4)          # shape: (3, 4) — 4 neurons, each with 3 weights
b = np.zeros((1, 4))               # shape: (1, 4)

layer_output = X @ W + b
print("Layer output shape:", layer_output.shape)
# Output: Layer output shape: (1, 4)

# Apply ReLU
activated = np.maximum(0, layer_output)
print("After ReLU:", np.round(activated, 3))

The key insight: the weight matrix W has shape (input_size, num_neurons). Matrix multiplication handles all the neurons simultaneously in one operation. This is why GPUs — which are optimized for matrix multiplication — accelerate deep learning so dramatically.

Why Depth Matters

A shallow network (one hidden layer) can theoretically approximate any continuous function given enough neurons. But "enough neurons" might be an absurdly large number. Depth is more parameter-efficient.

Think of it this way: a deep network composes simple functions hierarchically.

  • Layer 1 detects edges in an image
  • Layer 2 combines edges into corners and textures
  • Layer 3 combines textures into parts (eyes, wheels, windows)
  • Layer 4 combines parts into objects

Each layer builds on the previous one's abstractions. A shallow network would have to represent all of these directly in a single layer — requiring exponentially more neurons to achieve the same representational power.

# Illustration: composing functions
def layer_1(x):
    """Detects simple patterns"""
    return np.maximum(0, 2 * x - 1)

def layer_2(x):
    """Combines patterns from layer 1"""
    return np.maximum(0, -1.5 * x + 0.8)

def deep_network(x):
    return layer_2(layer_1(x))

# The composed function is more complex than either layer alone
x = np.linspace(0, 2, 100)
plt.plot(x, deep_network(x), label="Deep composition")
plt.plot(x, layer_1(x), label="Layer 1 alone", linestyle="--")
plt.legend()
plt.title("Function composition creates richer representations")
plt.show()

Success

The universal approximation theorem in one sentence: a neural network with at least one hidden layer and a nonlinear activation can approximate any continuous function to arbitrary precision, given enough neurons. Depth makes this practical; width alone makes it possible but inefficient.

A Full Forward Pass Through a Two-Layer Network

Here is a complete forward pass with shapes annotated at every step.

import numpy as np

np.random.seed(0)

# Input: 5 samples, 3 features each
X = np.random.randn(5, 3)  # shape: (5, 3)

# Layer 1: 3 inputs → 4 neurons
W1 = np.random.randn(3, 4)  # shape: (3, 4)
b1 = np.zeros(4)             # shape: (4,)

z1 = X @ W1 + b1             # shape: (5, 4)
a1 = np.maximum(0, z1)       # ReLU — shape: (5, 4)

print("After layer 1:", a1.shape)  # (5, 4)

# Layer 2: 4 inputs → 1 output (binary classification)
W2 = np.random.randn(4, 1)  # shape: (4, 1)
b2 = np.zeros(1)             # shape: (1,)

z2 = a1 @ W2 + b2            # shape: (5, 1)
a2 = 1 / (1 + np.exp(-z2))  # Sigmoid — shape: (5, 1)

print("After layer 2 (predictions):", np.round(a2, 3))
# Output: probabilities between 0 and 1 for each of the 5 samples

At this point the network has made predictions — but they are based on random weights, so they are meaningless. Training is the process of adjusting W1, b1, W2, b2 so the predictions match the actual labels.

When Deep Learning Wins and When It Doesn't

This is the judgment call that separates practitioners from tutorial followers.

Deep learning earns its complexity on:

  • Images — CNNs exploit spatial structure that tabular methods cannot
  • Text and sequences — transformers and RNNs model variable-length sequential dependencies
  • Audio — temporal patterns over thousands of timesteps
  • Raw unstructured signals — where feature engineering is impractical at scale
  • Large datasets — neural networks scale with data in ways that tree models do not

Gradient boosting almost always wins on:

  • Structured tabular data under 100k rows
  • Problems with strong categorical features
  • Any setting where inference speed or interpretability matters
  • Any setting where you cannot afford to tune hyperparameters carefully

Warning

The most common mistake beginners make is reaching for a neural network first because it "sounds more powerful." On a 5,000-row dataset with 20 engineered features, XGBoost will outperform a neural network 9 times out of 10 and train in seconds. Ask whether the data is unstructured and large before choosing deep learning.

Tip

A good rule of thumb: if the data could fit in a spreadsheet and make sense to a human analyst, start with gradient boosting. If the data is pixels, tokens, or waveforms, start with deep learning.

Interview Questions

Q1: What does a single neuron compute?

Show answer

A neuron computes a weighted sum of its inputs plus a bias term, then applies a nonlinear activation function to the result: output = activation(W·x + b). The weights and bias are learned during training.

Q2: Why would stacking linear layers without activations be pointless?

Show answer

Any composition of linear functions is itself a linear function. Without nonlinear activations, a 10-layer network is mathematically equivalent to a single linear layer. Activation functions break this collapse and give the network the capacity to represent curved, nonlinear decision boundaries.

Q3: What is the universal approximation theorem?

Show answer

A neural network with at least one hidden layer and a nonlinear activation can approximate any continuous function to arbitrary precision, given sufficient width. It is an existence theorem — it tells you a solution exists but not how to find it or how many neurons you need.

Q4: When would you choose XGBoost over a neural network?

Show answer

For structured tabular data, especially datasets under ~100k rows, XGBoost typically outperforms neural networks with far less tuning. Neural networks shine on unstructured data (images, text, audio) and very large datasets where their data-scaling advantage becomes significant.

What's Next

You've covered neurons as weighted-sum-plus-activation units, the role of activation functions in enabling non-linear decision boundaries, the universal approximation theorem, the forward pass computation from input to output, and the judgment framework for when to choose deep learning over gradient boosting. Next up: 02-training-neural-networks — where you'll learn how backpropagation flows the gradient signal backward through the network, why learning rate is the most important hyperparameter, and how the Adam optimizer manages adaptive per-weight learning rates automatically.

Optional Deep Dive

Read "Neural Networks and Deep Learning" by Michael Nielsen (free at neuralnetworksanddeeplearning.com), Chapter 1 — it walks through the perceptron, the sigmoid neuron, and the gradient descent update rule with detailed interactive visualisations, giving you the mathematical foundation for everything covered in this note.

Back to Agenda | Next: Training Neural Networks