Courses

Unit 4 Navigation

← Unit 4 Overview  |  Next → Appropriate Problems for NN Learning

Overview

Artificial Neural Networks (ANNs) provide a general, practical method for learning real-valued, discrete-valued, and vector-valued functions from examples.

The Backpropagation algorithm — the cornerstone of ANN learning — uses gradient descent to tune network parameters to best fit a training set of input-output pairs.

Why ANNs Matter

ANN learning has been proven effective in:

• Recognising handwritten characters (LeCun et al., 1989)
• Recognising spoken words (Lang et al., 1990)
• Recognising faces (Cottrell, 1990)
• Steering autonomous vehicles (Pomerleau, 1993)

4.1 Biological Motivation

The study of ANNs was inspired by the observation that biological learning systems are built of densely interconnected neurons.

Unit 4 Navigation

← Introduction to ANN  |  Next → Perceptrons

4.3 When Should You Use Neural Networks?

The Backpropagation algorithm is the most commonly used ANN learning technique. It is appropriate for problems with the following six characteristics:

Characteristic 1: Instances Described by Many Attribute-Value Pairs

The target function is defined over instances described by a vector of predefined features — such as pixel values in an image.

• Input attributes may be highly correlated or independent of one another
• Input values can be any real number (not just binary or integer)

Example: In ALVINN, each instance is described by 960 pixel intensity values.
Counter-example: Symbolic logic problems where inputs are categorical labels are less naturally suited (though ANNs can still be applied).

Unit 4 Navigation

← Appropriate Problems for NN Learning  |  Next → Multilayer Networks and Backpropagation

4.4 Perceptrons

What Is a Perceptron?

A perceptron is a single computational unit that takes a vector of real-valued inputs x₁, x₂, …, xₙ, computes their weighted sum, and outputs +1 if the sum exceeds a threshold, -1 otherwise.

Output o(x₁, ..., xₙ):
  = +1   if   w₀ + w₁x₁ + w₂x₂ + ... + wₙxₙ  >  0
  = -1   otherwise

Each wᵢ is a real-valued weight. The quantity −w₀ is the threshold that the weighted sum must exceed.

Unit 4 Navigation

← Perceptrons  |  Next → Remarks on Backpropagation

Why Multilayer Networks?

The perceptron can only represent linear decision surfaces. Real-world problems like vowel recognition or face direction require nonlinear boundaries.

Multilayer networks trained with Backpropagation learn rich nonlinear decision surfaces — curved, irregular boundaries that no single perceptron could produce.

4.5.1 The Sigmoid Unit

The Problem with Perceptrons in Multilayer Networks

We need units that are:

1. Nonlinear — to express complex functions
2. Differentiable everywhere — to apply gradient descent

Perceptrons fail condition 2 (hard threshold is not differentiable at the boundary). Linear units fail condition 1 (stacking linear units still gives a linear function).

Unit 4 Navigation

← Multilayer Networks and Backpropagation  |  Next → An Example: Face Recognition

4.6.1 Convergence and Local Minima

The Local Minima Problem

Unlike the linear unit — which has a single smooth bowl-shaped error surface — multilayer network error surfaces may have many local minima. Backpropagation is therefore only guaranteed to converge to some local minimum, not necessarily the global one.

    E(w)
     │         Local    Global
     │    Local Min     Min
     │    Max  ↓   \   /
     │   / \  / \  /\_/
     │  /   \/   \/
     └──────────────────── w

Despite this, Backpropagation works well in practice. Local minima are less severe than expected for two reasons:

Unit 4 Navigation

← Remarks on Backpropagation  |  Next → Advanced Topics: Error Functions

4.7 Face Recognition with Backpropagation

This section walks through applying Backpropagation to a real learning task — recognising which direction a person is facing from a camera image. It illustrates the design decisions every ANN practitioner must make, and what the network actually learns internally.

4.7.1 The Task

Dataset

Variations within the dataset:

Unit 4 Navigation

← An Example: Face Recognition  |  Next → Recurrent Networks

Why Modify the Error Function?

Standard Backpropagation minimises:

E(w) = (1/2) × Σ_d Σ_k  (t_kd − o_kd)²

This works for many cases. But we may also want to:

• Reduce overfitting by penalising large weights
• Enforce invariance by matching how the output varies with inputs
• Output probabilities rather than arbitrary real values
• Enforce symmetry across equivalent inputs

Each new objective leads to a different error function E, and hence a different gradient descent update rule.

Unit 4 Navigation

← Advanced Topics: Error Functions  |  → Unit 5: Decision Trees & Reinforcement Learning

4.8.3 Recurrent Networks

Why Feedforward Networks Fall Short for Time Series

All networks discussed so far are feedforward — they map a fixed-size input to an output with no memory of previous inputs.

For time series problems, this is a fundamental limitation.

Example: Predict tomorrow’s stock market index y(t+1) from today’s economic indicators x(t).

Feedforward network:   x(t)  →  [Network]  →  y(t+1)

This uses only x(t). But if y(t+1) also depends on x(t−1), x(t−2), etc., the feedforward network has no access to that history.

Unit 4 Topics

Source: Tom M. Mitchell, Machine Learning, Chapter 4
📄 Full Chapter PDF