Backpropagation

Unit 4 Navigation

← Introduction to ANN  |  Next → Perceptrons

4.3 When Should You Use Neural Networks?

The Backpropagation algorithm is the most commonly used ANN learning technique. It is appropriate for problems with the following six characteristics:

Characteristic 1: Instances Described by Many Attribute-Value Pairs

The target function is defined over instances described by a vector of predefined features — such as pixel values in an image.

• Input attributes may be highly correlated or independent of one another
• Input values can be any real number (not just binary or integer)

Example: In ALVINN, each instance is described by 960 pixel intensity values.
Counter-example: Symbolic logic problems where inputs are categorical labels are less naturally suited (though ANNs can still be applied).

Unit 4 Navigation

← Perceptrons  |  Next → Remarks on Backpropagation

Why Multilayer Networks?

The perceptron can only represent linear decision surfaces. Real-world problems like vowel recognition or face direction require nonlinear boundaries.

Multilayer networks trained with Backpropagation learn rich nonlinear decision surfaces — curved, irregular boundaries that no single perceptron could produce.

4.5.1 The Sigmoid Unit

The Problem with Perceptrons in Multilayer Networks

We need units that are:

1. Nonlinear — to express complex functions
2. Differentiable everywhere — to apply gradient descent

Perceptrons fail condition 2 (hard threshold is not differentiable at the boundary). Linear units fail condition 1 (stacking linear units still gives a linear function).

Unit 4 Navigation

← Multilayer Networks and Backpropagation  |  Next → An Example: Face Recognition

4.6.1 Convergence and Local Minima

The Local Minima Problem

Unlike the linear unit — which has a single smooth bowl-shaped error surface — multilayer network error surfaces may have many local minima. Backpropagation is therefore only guaranteed to converge to some local minimum, not necessarily the global one.

    E(w)
     │         Local    Global
     │    Local Min     Min
     │    Max  ↓   \   /
     │   / \  / \  /\_/
     │  /   \/   \/
     └──────────────────── w

Despite this, Backpropagation works well in practice. Local minima are less severe than expected for two reasons:

Unit 4 Navigation

← Remarks on Backpropagation  |  Next → Advanced Topics: Error Functions

4.7 Face Recognition with Backpropagation

This section walks through applying Backpropagation to a real learning task — recognising which direction a person is facing from a camera image. It illustrates the design decisions every ANN practitioner must make, and what the network actually learns internally.

4.7.1 The Task

Dataset

Variations within the dataset: