Gradient-Descent

← Appropriate Problems for NN Learning | Next → Multilayer Networks and Backpropagation

4.4 Perceptrons

What Is a Perceptron?

A perceptron is a single computational unit that takes a vector of real-valued inputs x₁, x₂, …, xₙ, computes their weighted sum, and outputs +1 if the sum exceeds a threshold, -1 otherwise.

Output o(x₁, ..., xₙ):
  = +1   if   w₀ + w₁x₁ + w₂x₂ + ... + wₙxₙ  >  0
  = -1   otherwise

Each wᵢ is a real-valued weight. The quantity −w₀ is the threshold that the weighted sum must exceed.

← Perceptrons | Next → Remarks on Backpropagation

Why Multilayer Networks?

The perceptron can only represent linear decision surfaces. Real-world problems like vowel recognition or face direction require nonlinear boundaries.

Multilayer networks trained with Backpropagation learn rich nonlinear decision surfaces — curved, irregular boundaries that no single perceptron could produce.

4.5.1 The Sigmoid Unit

The Problem with Perceptrons in Multilayer Networks

We need units that are:

Nonlinear — to express complex functions
Differentiable everywhere — to apply gradient descent

Perceptrons fail condition 2 (hard threshold is not differentiable at the boundary). Linear units fail condition 1 (stacking linear units still gives a linear function).