← An Example: Face Recognition | Next → Recurrent Networks
Standard Backpropagation minimises:
E(w) = (1/2) × Σ_d Σ_k (t_kd − o_kd)²
This works for many cases. But we may also want to:
Each new objective leads to a different error function E, and hence a different gradient descent update rule.
Gradient descent works for any differentiable error function. Change E → change what the network learns.
Problem: Large weights allow the network to fit noise in training data → overfitting.
Solution: Add a penalty for large weights to the error function:
E_new(w) = (1/2) × Σ_d Σ_k (t_kd − o_kd)² + γ × Σ_ji w_ji²
───────────────────────────────── ──────────────
original squared error weight penalty
The second term grows with weight magnitude. Minimising E_new requires trading off fitting the data against keeping weights small.
Resulting weight update rule:
w_ji ← w_ji × (1 − 2γη) + η × δ_j × x_ji
───────────────── ─────────────────
weight decays here normal gradient step
Every weight is multiplied by (1 − 2γη) before the gradient step — it “decays” toward zero.
| γ value | Effect |
|---|---|
| γ = 0 | Standard Backpropagation (no regularisation) |
| Small γ | Mild preference for smaller weights |
| Large γ | Strong bias toward simple model |
Choosing γ: Use cross-validation. Too small: still overfits. Too large: underfits.
Motivation: In some tasks we know not just what output values should be, but also how the output should vary with input — i.e., desired derivatives.
Example — Translation-invariant character recognition: A digit ‘3’ should be recognised whether it appears slightly to the left or right of centre in the image. The network’s output should be nearly invariant to small translations. This means the derivative of the output with respect to horizontal position should be near zero.
Extended error function:
E = (1/2) Σ_d Σ_k (t_kd − o_kd)²
+ μ × Σ_d Σ_k Σ_j ( desired_deriv_kd_j − actual_deriv_kd_j )²
Applications: Simard et al. (1992) — translation-invariant character recognition; Mitchell & Thrun (1993) — incorporating prior knowledge via derivatives.
Motivation: For binary classification, the network should output probability estimates, not just arbitrary real values. Squared error is not the best objective in this case.
Example: Predicting loan repayment — the target is 0 or 1 (did/didn’t repay), but the true underlying function is a probability like 0.73.
Cross-entropy error function:
E = − Σ_d [ t_d × log(o_d) + (1 − t_d) × log(1 − o_d) ]
where o_d is the network’s probability estimate and t_d is the 0/1 target.
Why cross-entropy is better than squared error for probabilities:
| Property | Squared Error | Cross-Entropy |
|---|---|---|
| Assumes output represents | A value | A probability |
| Gradient when confidently wrong | Small (saturated sigmoid) | Large |
| Maximum likelihood estimator | No | Yes (for Bernoulli targets) |
Cross-entropy gives a stronger learning signal when the network is confidently wrong — it avoids the “vanishing gradient at saturation” problem that squared error has with sigmoid outputs.
Motivation: Some inputs should be treated equivalently — we want the network to enforce this symmetry automatically.
Example — Time-Delay Neural Network (TDNN) for Speech Recognition:
Time windows: t₁ t₂ t₃ ... t₁₂
↓ ↓ ↓ ↓
[Unit] [Unit] [Unit]...[Unit]
↑─────↑─────↑──────────↑
All share identical weights
Implementation:
Why this helps:
Modern relevance: This is the direct theoretical foundation of Convolutional Neural Networks (CNNs) — the standard architecture for image recognition. In CNNs, the same filter weights apply at every spatial position (translation equivariance).
Standard gradient descent uses a fixed step size η. The following methods choose the step size (and sometimes direction) more adaptively.
After choosing a descent direction, find the optimal step size along that line:
α* = argmin E( w − α × ∇E(w) )
α
Each step moves exactly to the minimum along the chosen direction — allowing much larger or smaller steps than a fixed η.
Combines line search with a smarter choice of search direction:
Standard gradient descent: Conjugate gradient:
zig-zags toward minimum more direct path, less backtracking
many small steps n steps for n-dimensional quadratic
Convergence: For a quadratic (bowl-shaped) error surface in n dimensions, conjugate gradient converges in exactly n steps — vs. potentially thousands for fixed-rate gradient descent.
In practice for ANNs:
| Error Function | Use Case | Modern Name |
|---|---|---|
| Squared error | General regression | MSE — Mean Squared Error |
| Squared error + weight penalty | Reduce overfitting | L2 regularisation |
| Squared error + derivative penalty | Enforce invariance | Tangent propagation |
| Cross-entropy | Binary / multiclass classification | Binary cross-entropy |
| Weight sharing | Symmetric inputs (time, space) | Convolutional layers in CNNs |
| Expression | Meaning |
|---|---|
E = (1/2)Σ(t−o)² + γΣw² | Weight decay error function |
w_ji ← w_ji(1−2γη) + ηδ_jx_ji | Weight decay update rule |
E = −Σ[t·log(o) + (1−t)·log(1−o)] | Cross-entropy error |
| Shared weight = mean of per-unit gradients | Weight sharing update |
(1 − 2γη) per iteration.← An Example: Face Recognition | Next → Recurrent Networks