Unit4 on Chandras Edu | AI, US Stocks, Courses & Jobs India

Introduction to Artificial Neural Networks

Mon, 01 Jun 2026 00:00:00 +0000

← Unit 4 Overview | Next → Appropriate Problems for NN Learning

Overview

Artificial Neural Networks (ANNs) provide a general, practical method for learning real-valued, discrete-valued, and vector-valued functions from examples.

The Backpropagation algorithm — the cornerstone of ANN learning — uses gradient descent to tune network parameters to best fit a training set of input-output pairs.

Why ANNs Matter

ANN learning has been proven effective in:

Recognising handwritten characters (LeCun et al., 1989)
Recognising spoken words (Lang et al., 1990)
Recognising faces (Cottrell, 1990)
Steering autonomous vehicles (Pomerleau, 1993)

4.1 Biological Motivation

The study of ANNs was inspired by the observation that biological learning systems are built of densely interconnected neurons.

Appropriate Problems for Neural Network Learning

Mon, 01 Jun 2026 00:00:00 +0000

← Introduction to ANN | Next → Perceptrons

4.3 When Should You Use Neural Networks?

The Backpropagation algorithm is the most commonly used ANN learning technique. It is appropriate for problems with the following six characteristics:

Characteristic 1: Instances Described by Many Attribute-Value Pairs

The target function is defined over instances described by a vector of predefined features — such as pixel values in an image.

Input attributes may be highly correlated or independent of one another
Input values can be any real number (not just binary or integer)

Example: In ALVINN, each instance is described by 960 pixel intensity values.
Counter-example: Symbolic logic problems where inputs are categorical labels are less naturally suited (though ANNs can still be applied).

Perceptrons

Mon, 01 Jun 2026 00:00:00 +0000

← Appropriate Problems for NN Learning | Next → Multilayer Networks and Backpropagation

4.4 Perceptrons

What Is a Perceptron?

A perceptron is a single computational unit that takes a vector of real-valued inputs x₁, x₂, …, xₙ, computes their weighted sum, and outputs +1 if the sum exceeds a threshold, -1 otherwise.

Output o(x₁, ..., xₙ):
  = +1   if   w₀ + w₁x₁ + w₂x₂ + ... + wₙxₙ  >  0
  = -1   otherwise

Each wᵢ is a real-valued weight. The quantity −w₀ is the threshold that the weighted sum must exceed.

Multilayer Networks and Backpropagation Algorithm

Mon, 01 Jun 2026 00:00:00 +0000

← Perceptrons | Next → Remarks on Backpropagation

Why Multilayer Networks?

The perceptron can only represent linear decision surfaces. Real-world problems like vowel recognition or face direction require nonlinear boundaries.

Multilayer networks trained with Backpropagation learn rich nonlinear decision surfaces — curved, irregular boundaries that no single perceptron could produce.

4.5.1 The Sigmoid Unit

The Problem with Perceptrons in Multilayer Networks

We need units that are:

Nonlinear — to express complex functions
Differentiable everywhere — to apply gradient descent

Perceptrons fail condition 2 (hard threshold is not differentiable at the boundary). Linear units fail condition 1 (stacking linear units still gives a linear function).

Remarks on the Backpropagation Algorithm

Mon, 01 Jun 2026 00:00:00 +0000

← Multilayer Networks and Backpropagation | Next → An Example: Face Recognition

4.6.1 Convergence and Local Minima

The Local Minima Problem

Unlike the linear unit — which has a single smooth bowl-shaped error surface — multilayer network error surfaces may have many local minima. Backpropagation is therefore only guaranteed to converge to some local minimum, not necessarily the global one.

    E(w)
     │         Local    Global
     │    Local Min     Min
     │    Max  ↓   \   /
     │   / \  / \  /\_/
     │  /   \/   \/
     └──────────────────── w

Despite this, Backpropagation works well in practice. Local minima are less severe than expected for two reasons:

An Example: Face Recognition Using Backpropagation

Mon, 01 Jun 2026 00:00:00 +0000

← Remarks on Backpropagation | Next → Advanced Topics: Error Functions

4.7 Face Recognition with Backpropagation

This section walks through applying Backpropagation to a real learning task — recognising which direction a person is facing from a camera image. It illustrates the design decisions every ANN practitioner must make, and what the network actually learns internally.

4.7.1 The Task

Dataset

Property	Detail
Total images	624 greyscale images
Subjects	20 different people
Images per person	~32
Image resolution	120 × 128 pixels
Pixel depth	Greyscale, 0–255

Variations within the dataset:

Advanced Topics of ANN: Alternative Error Functions

Mon, 01 Jun 2026 00:00:00 +0000

← An Example: Face Recognition | Next → Recurrent Networks

Why Modify the Error Function?

Standard Backpropagation minimises:

E(w) = (1/2) × Σ_d Σ_k  (t_kd − o_kd)²

This works for many cases. But we may also want to:

Reduce overfitting by penalising large weights
Enforce invariance by matching how the output varies with inputs
Output probabilities rather than arbitrary real values
Enforce symmetry across equivalent inputs

Each new objective leads to a different error function E, and hence a different gradient descent update rule.

Recurrent Networks

Mon, 01 Jun 2026 00:00:00 +0000

← Advanced Topics: Error Functions | → Unit 5: Decision Trees & Reinforcement Learning

4.8.3 Recurrent Networks

Why Feedforward Networks Fall Short for Time Series

All networks discussed so far are feedforward — they map a fixed-size input to an output with no memory of previous inputs.

For time series problems, this is a fundamental limitation.

Example: Predict tomorrow’s stock market index y(t+1) from today’s economic indicators x(t).

Feedforward network:   x(t)  →  [Network]  →  y(t+1)

This uses only x(t). But if y(t+1) also depends on x(t−1), x(t−2), etc., the feedforward network has no access to that history.

Unit 4: Artificial Neural Networks

Mon, 01 Jun 2026 00:00:00 +0000

Unit 4 Topics

#	Topic	Description
4.1	Introduction to Artificial Neural Networks	Biological motivation, ANN representations, ALVINN example
4.2	Appropriate Problems for NN Learning	When to use ANNs — 6 key problem characteristics
4.3	Perceptrons	Perceptron model, representational power, training rule, delta rule, gradient descent
4.4	Multilayer Networks and Backpropagation Algorithm	Sigmoid units, Backpropagation algorithm, momentum, derivation
4.5	Remarks on Backpropagation Algorithm	Convergence, local minima, representational power, hidden representations, overfitting
4.6	An Example: Face Recognition	Full design walkthrough — input encoding, network structure, learned representations
4.7	Advanced Topics of ANN — Error Functions	Weight decay, derivative errors, cross-entropy, weight sharing, conjugate gradient
4.8	Recurrent Networks	Time series, recurrent architecture, unfolding in time, dynamic network structure

Source: Tom M. Mitchell, Machine Learning, Chapter 4
📄 Full Chapter PDF

Unit4 on Chandras Edu | AI, US Stocks, Courses & Jobs India

Introduction to Artificial Neural Networks

Unit 4 Navigation

Overview

Why ANNs Matter

4.1 Biological Motivation

Appropriate Problems for Neural Network Learning

Unit 4 Navigation

4.3 When Should You Use Neural Networks?

Characteristic 1: Instances Described by Many Attribute-Value Pairs

Perceptrons

Unit 4 Navigation

4.4 Perceptrons

What Is a Perceptron?

Multilayer Networks and Backpropagation Algorithm

Unit 4 Navigation

Why Multilayer Networks?

4.5.1 The Sigmoid Unit

The Problem with Perceptrons in Multilayer Networks

Remarks on the Backpropagation Algorithm

Unit 4 Navigation

4.6.1 Convergence and Local Minima

The Local Minima Problem

An Example: Face Recognition Using Backpropagation

Unit 4 Navigation

4.7 Face Recognition with Backpropagation

4.7.1 The Task

Dataset

Advanced Topics of ANN: Alternative Error Functions

Unit 4 Navigation

Why Modify the Error Function?

Recurrent Networks

Unit 4 Navigation

4.8.3 Recurrent Networks

Why Feedforward Networks Fall Short for Time Series

Unit 4: Artificial Neural Networks

Unit 4 Topics