← Remarks on Backpropagation | Next → Advanced Topics: Error Functions
This section walks through applying Backpropagation to a real learning task — recognising which direction a person is facing from a camera image. It illustrates the design decisions every ANN practitioner must make, and what the network actually learns internally.
| Property | Detail |
|---|---|
| Total images | 624 greyscale images |
| Subjects | 20 different people |
| Images per person | ~32 |
| Image resolution | 120 × 128 pixels |
| Pixel depth | Greyscale, 0–255 |
Variations within the dataset:
Task selected: Predict the direction the person is facing — one of four classes:
Other possible targets from the same dataset: identity, gender, expression, wearing sunglasses — all learnable to high accuracy.
Every ANN application requires decisions about representation, architecture, and training. Below are the choices made for this task.
Problem: The full image is 120 × 128 = 15,360 pixels — too many weights for efficient training.
Solution: Downsample to a 30 × 32 coarse image (960 pixels):
Scaling: Pixel values 0–255 are scaled to [0, 1] to match the output range of sigmoid units.
Comparison to ALVINN: ALVINN uses the same idea but samples one random pixel per region instead of averaging — faster computation for real-time driving, at some cost in quality.
1-of-4 (one-hot) encoding for the four face directions:
| Direction | Output vector |
|---|---|
| Looking left | (high, low, low, low) |
| Looking straight | (low, high, low, low) |
| Looking right | (low, low, high, low) |
| Looking up | (low, low, low, high) |
Why 1-of-4, not a single unit with values 0.25/0.50/0.75/1.0?
The sigmoid function approaches 0 and 1 only asymptotically — it never exactly reaches these values with finite weights. If we use target values of 0 and 1:
Solution: Use target values of 0.1 (instead of 0) and 0.9 (instead of 1).
Target for "looking left": (0.9, 0.1, 0.1, 0.1)
Target for "looking right": (0.1, 0.1, 0.9, 0.1)
These values are achievable by a sigmoid unit with finite weights.
Architecture chosen: 960 inputs → 3 hidden units → 4 output units
| Design question | Decision | Rationale |
|---|---|---|
| Number of hidden layers | 1 | One or two sigmoid layers can approximate any function; more layers increase training time |
| Number of hidden units | 3 | Achieves 90% accuracy; sufficient for this task |
| Layer type | Sigmoid + linear output | Standard for classification |
Experiment with 30 hidden units:
General principle: With cross-validation, extra hidden units typically increase overfitting risk rather than generalisation accuracy. The minimum number needed for the task is often best.
| Parameter | Value | Notes |
|---|---|---|
| Learning rate η | 0.3 | Lower values work but slower; too high fails to converge |
| Momentum α | 0.3 | Consistent convergence improvement |
| Weight initialisation | Output units: small random; Input unit weights: 0 | Zero input weights give more interpretable visualisations |
| Training examples | 260 images | Remaining used for validation and test |
| Gradient descent type | Full batch | Entire training set used before each update |
The validation-based procedure from Section 4.6.5:
This three-way split — training / validation / test — ensures the reported accuracy is a genuine measure of generalisation.
| Metric | Value |
|---|---|
| Training images | 260 |
| Test accuracy (3 hidden units) | 90% |
| Baseline (random guess) | 25% |
| Training time (3 hidden) | ~5 minutes (Sun Sparc5) |
| Training time (30 hidden) | ~1 hour |
A 3.6× improvement over random guessing, using only 3 hidden units connecting 960 pixel inputs to 4 output classes.
After 100 training iterations, the hidden unit weights are inspected by plotting the 30×32 weight matrix at the position of each corresponding input pixel.
Observable pattern: The weights concentrate in the region of the image where faces typically appear — the network learns to focus attention on the face and body area, ignoring background.
Example — “Looking right” output unit:
After 1 iteration: Weights are near-random, roughly uniform — no structure visible.
After 100 iterations: Weights organise into face-sensitive patterns. The network has learned which pixel regions carry information about head direction.
This illustrates the key ANN property: internal feature discovery. The designer provided only raw pixels and direction labels; the hidden units independently discovered that facial position and orientation patterns are the relevant features.
Dataset: 624 greyscale images, 20 people, 4 directions
Input: 30×32 pixels, scaled to [0,1] → 960 input units
Hidden layer: 3 sigmoid units
Output layer: 4 units (one per direction)
Target values: 0.9 (target class) / 0.1 (other classes)
Learning rate: η = 0.3
Momentum: α = 0.3
Training set: 260 images
Test accuracy: 90% (vs. 25% chance)
← Remarks on Backpropagation | Next → Advanced Topics: Error Functions