Neural Networks

Neural network questions appear in virtually every ML interview, from phone screens to final rounds. Interviewers use them to probe whether you understand what a model is actually doing — not just which API to call.

---

Q1: What is a neuron and what does it compute?

Show answer

A neuron receives a vector of inputs, computes a weighted sum, adds a bias, and then passes the result through a non-linear activation function.

output = activation(w₁x₁ + w₂x₂ + ... + wₙxₙ + b)
         = activation(wᵀx + b)

The weights control how much each input contributes. The bias shifts the activation threshold. The activation function introduces non-linearity — without it, the entire network collapses to a linear model no matter how many layers you stack.

import numpy as np

def neuron(x, w, b, activation=np.tanh):
    return activation(np.dot(w, x) + b)

x = np.array([1.0, 2.0, 3.0])
w = np.array([0.5, -0.3, 0.8])
b = 0.1

output = neuron(x, w, b)
# output: tanh(0.5*1 + (-0.3)*2 + 0.8*3 + 0.1) = tanh(2.0) ≈ 0.964

A perceptron is the simplest neuron — it uses a step function as activation. Modern networks use smooth differentiable activations (ReLU, sigmoid, tanh) so gradients can flow during training.

---

Q2: Why do neural networks need non-linear activation functions?

Show answer

Without non-linearity, stacking multiple layers is equivalent to a single linear transformation. No matter how deep the network, it can only represent linear functions of the input.

Proof sketch: if layer 1 computes W₁x + b₁ and layer 2 computes W₂(W₁x + b₁) + b₂, the combined operation is (W₂W₁)x + (W₂b₁ + b₂) — still linear.

Non-linear activations allow networks to approximate arbitrarily complex functions. Each layer applies a non-linear transformation that warps the input space, allowing subsequent layers to find patterns that are linearly separable in the transformed space.

Geometric intuition: a deep network is learning a sequence of coordinate transformations that "unfold" the data manifold until the classification boundary becomes a hyperplane.

---

Q3: What are the main activation functions and when do you use each?

Show answer

ReLU (Rectified Linear Unit): f(x) = max(0, x) - Default choice for hidden layers in most modern networks - Computationally cheap, does not saturate for positive inputs, gradient is 1 for positive inputs - Problem: dying ReLU — neurons can get stuck outputting 0 if they receive consistently negative inputs, and the gradient becomes exactly 0, so weights never update

Leaky ReLU: f(x) = x if x > 0 else αx (α ≈ 0.01) - Fixes dying ReLU by allowing a small gradient for negative inputs - Works well in practice; α is a hyperparameter

Sigmoid: f(x) = 1 / (1 + e⁻ˣ) → outputs in (0, 1) - Use at the output layer for binary classification to produce a probability - Saturates at both ends — gradients vanish for large |x|, making it a poor choice for hidden layers in deep networks

Tanh: f(x) = (eˣ - e⁻ˣ) / (eˣ + e⁻ˣ) → outputs in (-1, 1) - Zero-centred (unlike sigmoid), which helps gradients flow better - Still saturates; generally better than sigmoid for hidden layers but worse than ReLU

Softmax: converts a vector of logits to probabilities that sum to 1 - Used at the output layer for multi-class classification

import numpy as np

def relu(x):
    return np.maximum(0, x)

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

def softmax(x):
    exp_x = np.exp(x - np.max(x))  # subtract max for numerical stability
    return exp_x / exp_x.sum()

logits = np.array([2.0, 1.0, 0.5])
probs = softmax(logits)
# array([0.626, 0.230, 0.144]) — sums to 1.0

---

Q4: What is the dying ReLU problem?

Show answer

A ReLU neuron "dies" when its pre-activation output (the weighted sum before the activation) becomes and stays negative. In this state:

• The neuron outputs exactly 0
• The gradient of ReLU for negative inputs is exactly 0
• No gradient flows back through this neuron → its weights never update
• The neuron is permanently inactive for all inputs — it is dead

Common causes: - A large learning rate that overshoots and pushes weights into a regime where most inputs map to negative pre-activations - Poor weight initialisation - No bias or a very negative bias

Fixes: - Use Leaky ReLU or ELU (Exponential Linear Unit) instead of ReLU - Use careful weight initialisation (He initialisation for ReLU networks) - Use a smaller learning rate - Add batch normalisation before activations (keeps pre-activations near zero)

import tensorflow as tf

# Leaky ReLU in Keras
model = tf.keras.Sequential([
    tf.keras.layers.Dense(128),
    tf.keras.layers.LeakyReLU(alpha=0.01),
    tf.keras.layers.Dense(64),
    tf.keras.layers.LeakyReLU(alpha=0.01),
    tf.keras.layers.Dense(10, activation='softmax')
])

---

Q5: What is the Universal Approximation Theorem and what does it actually guarantee?

Show answer

The Universal Approximation Theorem states that a feedforward neural network with at least one hidden layer containing a sufficient number of neurons with a non-linear activation function can approximate any continuous function on a compact subset of Rⁿ to arbitrary precision.

What it guarantees: - A sufficiently wide network can represent (in principle) any function - The theorem establishes that the network family is expressive enough

What it does NOT guarantee: - That training will find those weights — the theorem says the right weights exist, not that gradient descent will find them - Anything about the number of neurons needed — it may be astronomically large - Anything about generalisation — a network can perfectly fit training data and fail on new data - That a specific architecture will work for your specific problem

Practical implication: the theorem motivates using neural networks, but architecture design, initialisation, regularisation, and optimisation remain critical engineering challenges. Expressivity is necessary but not sufficient.

---

Q6: How does a feedforward pass work?

Show answer

A feedforward (forward) pass propagates input data through each layer of the network in sequence to produce a prediction.

For a network with layers L₁, L₂, ..., Lₖ: 1. Input x enters the first layer 2. Each layer computes a = activation(Wx + b) where W and b are that layer's learned parameters 3. The output of one layer is the input to the next 4. The final layer produces the prediction (logits or probabilities)

import numpy as np

def relu(x):
    return np.maximum(0, x)

def softmax(x):
    exp_x = np.exp(x - np.max(x))
    return exp_x / exp_x.sum()

# Two-layer network: input(4) -> hidden(8) -> output(3)
np.random.seed(42)
W1 = np.random.randn(8, 4) * 0.1
b1 = np.zeros(8)
W2 = np.random.randn(3, 8) * 0.1
b2 = np.zeros(3)

x = np.array([1.0, 0.5, -0.3, 0.8])

# Forward pass
z1 = W1 @ x + b1       # pre-activation, layer 1
a1 = relu(z1)           # post-activation, layer 1
z2 = W2 @ a1 + b2       # pre-activation, layer 2
probs = softmax(z2)     # output probabilities

print(f"Predicted class: {probs.argmax()}")
print(f"Probabilities: {probs.round(3)}")

During training, the forward pass computes both the prediction and stores intermediate activations — these are needed during the backward pass to compute gradients.

---

Q7: What loss functions are used for regression vs classification?

Show answer

Regression: - Mean Squared Error (MSE): L = (1/n) Σ (yᵢ - ŷᵢ)² — penalises large errors quadratically; differentiable everywhere; sensitive to outliers. Default choice for regression. - Mean Absolute Error (MAE): L = (1/n) Σ |yᵢ - ŷᵢ| — robust to outliers; gradient is constant (can oscillate near minimum) - Huber Loss: behaves like MSE for small errors, MAE for large errors — combines robustness with smooth optimisation

Binary Classification: - Binary Cross-Entropy: L = -[y·log(ŷ) + (1-y)·log(1-ŷ)] — used with sigmoid output. Measures the divergence between predicted probability distribution and true labels. More informative gradient than MSE for classification.

Multi-class Classification: - Categorical Cross-Entropy: L = -Σᵢ yᵢ·log(ŷᵢ) — used with softmax output. y is one-hot encoded. - Sparse Categorical Cross-Entropy: same as above but y is the integer class label directly — more memory-efficient

import tensorflow as tf

# Regression
model.compile(optimizer='adam', loss='mse')

# Binary classification (sigmoid output)
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

# Multi-class classification (softmax output, one-hot labels)
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

# Multi-class classification (softmax output, integer labels)
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

Why cross-entropy over MSE for classification? MSE applied to probabilities produces a flat loss landscape when predictions are confidently wrong (e.g., predicting 0.001 when true label is 1). Cross-entropy has a much steeper gradient in this region, producing faster and more stable learning.

---

Q8: What does softmax compute and why is it used at the output?

Show answer

Softmax converts a vector of arbitrary real values (logits) into a probability distribution: all values are positive and they sum to 1.

softmax(zᵢ) = exp(zᵢ) / Σⱼ exp(zⱼ)

The exponential function amplifies differences between logits: the largest logit gets a disproportionately large probability. This makes the output distribution "peaky" — confident models assign near-1 probability to one class.

Why it's used: - Output is a valid probability distribution — interpretable as confidence per class - Paired with categorical cross-entropy, gradients are clean and numerically stable - The argmax of the softmax output gives the predicted class

import numpy as np

logits = np.array([3.0, 1.0, 0.2])
exp_logits = np.exp(logits - logits.max())  # subtract max for stability
probs = exp_logits / exp_logits.sum()
# array([0.844, 0.114, 0.042]) — class 0 is predicted with 84.4% confidence

Numerical stability note: computing exp(z) directly can overflow for large z. Subtracting the maximum value before exponentiating (z - max(z)) is mathematically equivalent but prevents overflow.

Temperature scaling: dividing logits by a temperature τ before softmax controls the sharpness of the distribution. Low τ → peakier (more confident). High τ → flatter (more uncertain). Used in knowledge distillation and language model sampling.

---

Q9: What is the vanishing gradient problem and how was it solved?

Show answer

During backpropagation, gradients are computed by repeatedly applying the chain rule. In deep networks with saturating activations (sigmoid, tanh), the gradient at each layer is multiplied by the derivative of the activation function — which is at most 0.25 for sigmoid and at most 1.0 for tanh.

Over many layers, these repeated multiplications cause gradients to shrink exponentially. By the time the gradient reaches the early layers, it is effectively zero — those layers receive no meaningful update signal and stop learning.

Why it's worse in RNNs: gradients must propagate through many time steps, making sequences equivalent to very deep networks in terms of gradient flow.

Solutions: - ReLU activations: gradient is exactly 1 for positive inputs — no shrinkage in the positive regime - Residual connections (ResNets): skip connections provide a direct gradient highway from output to earlier layers - Batch Normalisation: keeps activations in a well-conditioned range, preventing saturation - LSTM / GRU: gating mechanisms in recurrent networks that control gradient flow explicitly - Better initialisation (Xavier / He): starts weights in a range that keeps activations from saturating immediately - Gradient clipping: caps gradient magnitude to prevent explosion (the opposite problem)

import tensorflow as tf

# ResNet-style skip connection
inputs = tf.keras.Input(shape=(128,))
x = tf.keras.layers.Dense(128, activation='relu')(inputs)
x = tf.keras.layers.Dense(128)(x)
x = tf.keras.layers.Add()([x, inputs])  # skip connection
x = tf.keras.layers.Activation('relu')(x)

---

Q10: What is batch size and what does it affect?

Show answer

Batch size is the number of training samples processed in one forward-backward pass before the weights are updated.

• Batch GD (batch size = full dataset): exact gradient; very slow per epoch; requires all data in memory; rarely used in deep learning
• Stochastic GD (batch size = 1): noisy gradient; fast updates; the noise can help escape local minima but makes convergence unstable
• Mini-batch GD (batch size = 32–512): practical compromise; enables parallelism on GPU; gradient is a noisy but useful estimate

What batch size affects:

Effect	Small batch	Large batch
Gradient noise	High (regularising)	Low (more exact)
Memory use	Low	High
GPU utilisation	Underutilised	Well-utilised
Generalisation	Often better	Often worse (sharp minima)
Steps per epoch	More	Fewer

The generalisation finding: large-batch training tends to converge to sharp minima (high curvature, poor generalisation). Small batches converge to flat minima (low curvature, better generalisation) because gradient noise acts as implicit regularisation.

Practical advice: start with batch size 32–128. If training is slow and GPU memory allows, scale up the batch size and scale the learning rate proportionally (linear scaling rule: if batch size doubles, double the learning rate).

---

Q11: What is weight initialisation and why does it matter?

Show answer

Initialising all weights to zero is a common mistake that makes a network fail to learn: every neuron in a layer computes the same output, receives the same gradient, and updates identically — the symmetry is never broken. All neurons remain identical throughout training.

Good initialisation: - Breaks symmetry (weights must differ) - Keeps activations and gradients from exploding or vanishing at the start of training

Xavier / Glorot initialisation (for sigmoid/tanh): Draws weights from a distribution with variance 2 / (fan_in + fan_out). Designed to keep the variance of activations and gradients approximately equal across layers.

He initialisation (for ReLU): Draws weights from a distribution with variance 2 / fan_in. Accounts for the fact that ReLU zeroes out half its inputs, so the effective variance would be halved without the factor of 2.

import tensorflow as tf

# He initialisation (recommended for ReLU networks)
layer = tf.keras.layers.Dense(
    128,
    activation='relu',
    kernel_initializer='he_normal'
)

# Xavier/Glorot (recommended for tanh/sigmoid networks)
layer = tf.keras.layers.Dense(
    128,
    activation='tanh',
    kernel_initializer='glorot_uniform'
)

Rule of thumb: use He initialisation with ReLU and its variants; use Xavier/Glorot with tanh and sigmoid.

---

Q12: What is a neural network doing geometrically?

Show answer

A neural network learns a sequence of non-linear transformations of the input space such that, in the final transformed space, the classification or regression task is easy (ideally linearly separable).

Each layer applies: 1. A linear transformation (the weight matrix W) — rotates, scales, and shears the space 2. A non-linear activation — folds and creases the space

After many such transformations, points that were entangled in the original input space are "unfolded" into a representation where the final linear layer can separate them cleanly.

Concrete example: for 2D XOR (a classic non-linearly separable problem), a single hidden layer with two ReLU neurons can fold the plane so that the four XOR points are now linearly separable. The hidden layer learns to map the input to a new space where a straight line divides the classes.

This geometric view explains: - Why depth matters: more layers enable more complex folding - Why width matters: more neurons give the transformation more degrees of freedom - Why the final layer is always linear: it classifies in the learned representation space

---

Q13: What is the difference between parameters and hyperparameters in a neural network?

Show answer

Parameters are learned from data during training: - Weight matrices W at each layer - Bias vectors b at each layer - Total count = sum of all (inputs × outputs) + biases across all layers

import tensorflow as tf

model = tf.keras.Sequential([
    tf.keras.layers.Dense(64, activation='relu', input_shape=(10,)),
    tf.keras.layers.Dense(32, activation='relu'),
    tf.keras.layers.Dense(1)
])
model.summary()
# Layer 1: 10*64 + 64 = 704 parameters
# Layer 2: 64*32 + 32 = 2,080 parameters
# Layer 3: 32*1 + 1 = 33 parameters
# Total: 2,817 trainable parameters

Hyperparameters are set before training and control the learning process: - Network architecture: number of layers, units per layer, activation functions - Optimiser: learning rate, momentum, weight decay - Training: batch size, number of epochs - Regularisation: dropout rate, L2 penalty strength

Hyperparameter tuning methods: grid search, random search, Bayesian optimisation (e.g. Optuna). In practice, random search over a well-defined range is often more efficient than grid search.

---

Q14: What is the difference between a shallow and a deep network, and why does depth help?

Show answer

A shallow network has one hidden layer. A deep network has multiple hidden layers.

The Universal Approximation Theorem guarantees that a shallow network can approximate any function — but it might need an exponentially large number of neurons to do so. Depth provides a more efficient representation.

Why depth helps: - Early layers learn low-level features (edges in images, character n-grams in text) - Middle layers combine them into mid-level features (shapes, words) - Deep layers combine those into high-level, task-specific features (object categories, sentiment) - This hierarchical composition is aligned with how many real-world signals are structured

The practical evidence: AlexNet (8 layers) beat shallow methods on ImageNet in 2012. ResNet (152 layers) improved further in 2015. Depth — enabled by ReLU activations, batch normalisation, and residual connections — consistently outperforms width for complex tasks.

When shallow works fine: tabular data, small datasets, problems with low intrinsic complexity. A single-layer network or even a linear model may be optimal.

---

Q15: How do you count parameters in a fully connected neural network?

Show answer

For each dense (fully connected) layer:

Parameters = (input_units × output_units) + output_units (bias)

import tensorflow as tf

model = tf.keras.Sequential([
    tf.keras.layers.Dense(128, activation='relu', input_shape=(784,)),
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.Dense(10, activation='softmax')
])

# Layer 1: 784 * 128 + 128 = 100,480 params
# Layer 2: 128 * 64 + 64 = 8,256 params
# Layer 3: 64 * 10 + 10 = 650 params
# Total: 109,386 parameters

model.summary()

Why this matters in interviews: parameter count determines: - Memory footprint of the model - Computational cost of inference - Overfitting risk (more parameters → needs more data and regularisation)

A model with millions of parameters trained on thousands of samples is almost guaranteed to overfit without strong regularisation.

---