Neural Networks Explained: From Perceptron to Deep Learning

Neural Networks Explained: From Perceptron to Deep Learning

A neural network is a chain of mathematical functions that transforms an input into an output, tuned by training rather than hand-written rules. The brain metaphor that dominates most textbooks is the wrong entry point — function composition is the model that actually makes the math click.

This guide builds that mental model bottom-up, from the simplest possible network to the deep learning systems behind modern AI. No math background beyond high school algebra is assumed, though calculus gets referenced where it matters.

The Simplest Neural Network: A Single Perceptron

A perceptron takes weighted inputs, sums them, and passes the result through an activation function — the smallest possible unit of a neural network, invented in 1957 and still the foundation of every network built since.

It takes inputs, multiplies each by a weight, sums the result, adds a bias, then passes through an activation function:

Inputs:  x₁ = 0.5, x₂ = 0.3, x₃ = 0.8
Weights: w₁ = 0.4, w₂ = -0.2, w₃ = 0.7

Weighted sum: z = (0.5 × 0.4) + (0.3 × -0.2) + (0.8 × 0.7) + bias
             z = 0.20 + (-0.06) + 0.56 + 0.1 = 0.80

Activation: output = sigmoid(0.80) = 1 / (1 + e^-0.80) ≈ 0.69

The output (0.69) can be interpreted as a probability or confidence. If you're classifying emails as spam or not-spam, an output of 0.69 might mean "69% confident this is spam."

What the Weights Represent

The weights are the learned parameters. A positive weight means "this input increases the probability of spam." A negative weight means "this input decreases the probability." The magnitude tells you how strongly.

After training on thousands of emails, a spam classifier's weights encode: "the word 'Nigerian' strongly predicts spam, but the word 'meeting' slightly predicts not-spam."

The limitation of a single perceptron: It can only learn linear decision boundaries — it can separate data that's linearly separable, but not data that requires a curved or complex boundary. The XOR problem (output 1 when inputs differ, 0 when they're the same) famously cannot be solved by a single perceptron.

Solving XOR: Why We Need Layers

The XOR problem requires a curved decision boundary. To make curved boundaries, we stack perceptrons.

Input Layer → Hidden Layer → Output Layer

x₁, x₂ → [neuron_1]  → output
       → [neuron_2]  →

Each neuron in the hidden layer creates its own linear boundary. Together, they combine into a non-linear boundary. With enough hidden neurons, any decision boundary is achievable.

This insight — that layering simple functions creates complex functions — is the core of deep learning.

The Network as a Function Composition

A neural network with 3 layers applies three functions in sequence:

Input x
  ↓
Layer 1: f₁(x) = activation(W₁·x + b₁)
  ↓
Layer 2: f₂(z₁) = activation(W₂·z₁ + b₂)
  ↓
Output: f₃(z₂) = activation(W₃·z₂ + b₃)

Final output = f₃(f₂(f₁(x)))

W are weight matrices, b are bias vectors. The "learning" is finding the W and b values that make the final output match the targets.

Activation Functions: Adding Non-Linearity

If we remove activation functions, all the layers collapse into one linear transformation. Non-linear activation functions are what allow the network to learn non-linear patterns.

Common Activation Functions

Sigmoid:

σ(x) = 1 / (1 + e^(-x))
Range: (0, 1) — useful for output probabilities in binary classification
Problem: Vanishing gradients in deep networks — gradients shrink exponentially
         as they propagate backward through many sigmoid layers

ReLU (Rectified Linear Unit) — the standard:

ReLU(x) = max(0, x)
If x > 0: output = x
If x ≤ 0: output = 0

Advantages:
- Computationally simple (fast)
- No vanishing gradient problem for positive values  
- Empirically works better than sigmoid in hidden layers

GELU (Gaussian Error Linear Unit) — used in Transformers:

GELU(x) ≈ x × Φ(x)  (where Φ is the Gaussian CDF)
Smoother than ReLU; used in BERT, GPT, and most modern Transformers

Softmax — output layer for classification:

softmax(x)ᵢ = e^xᵢ / Σⱼ e^xⱼ

Converts raw scores to probabilities that sum to 1.
Used when you need to predict one of N classes.

Example: [2.0, 1.0, 0.5] → [0.61, 0.23, 0.16]
         "Cat: 61%, Dog: 23%, Bird: 16%"

Forward Pass: How a Network Makes Predictions

The forward pass is the sequence of computations that transforms input into output. In code:

import numpy as np

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

def relu(x):
    return np.maximum(0, x)

class SimpleNeuralNetwork:
    def __init__(self):
        # Initialize weights randomly (small values to start)
        np.random.seed(42)
        self.W1 = np.random.randn(4, 3) * 0.01  # 4 neurons, 3 input features
        self.b1 = np.zeros((4, 1))
        self.W2 = np.random.randn(1, 4) * 0.01  # 1 output neuron
        self.b2 = np.zeros((1, 1))
    
    def forward(self, X):
        # Layer 1: linear transformation + ReLU
        self.Z1 = np.dot(self.W1, X) + self.b1
        self.A1 = relu(self.Z1)
        
        # Layer 2: linear transformation + sigmoid (binary output)
        self.Z2 = np.dot(self.W2, self.A1) + self.b2
        self.A2 = sigmoid(self.Z2)
        
        return self.A2  # Probability between 0 and 1

# Example
network = SimpleNeuralNetwork()
X = np.array([[1.0], [0.5], [0.3]])  # 3 features, 1 example
output = network.forward(X)
print(f"Output probability: {output[0][0]:.4f}")

Training: How the Network Learns

The Loss Function

The loss (or cost) function measures how wrong the network's predictions are. Common choices:

Binary Cross-Entropy (binary classification):

Loss = -[y × log(ŷ) + (1-y) × log(1-ŷ)]

y = true label (0 or 1)
ŷ = predicted probability

If the true label is 1 and we predict 0.9: loss = -log(0.9) ≈ 0.105 (small — good prediction)
If the true label is 1 and we predict 0.1: loss = -log(0.1) ≈ 2.30  (large — bad prediction)

Mean Squared Error (regression):

Loss = (1/n) × Σ(yᵢ - ŷᵢ)²

Average squared difference between true and predicted values.

Gradient Descent

Gradient descent is the optimization algorithm that reduces the loss. The gradient is the direction of steepest increase in loss — by moving opposite to the gradient (steepest decrease), we reduce the loss.

Weight update rule:
w_new = w_old - learning_rate × ∂Loss/∂w

learning_rate (e.g., 0.001): how big a step to take
∂Loss/∂w: the gradient — how much loss changes when we change w

The learning rate is critical. Too large: the loss oscillates or diverges. Too small: training is extremely slow.

Backpropagation

Backpropagation computes the gradient of the loss with respect to every weight in the network using the chain rule. The chain rule allows us to compute how changes in early layer weights affect the final loss:

Chain rule example (simplified):
∂Loss/∂W₁ = ∂Loss/∂A₂ × ∂A₂/∂Z₂ × ∂Z₂/∂A₁ × ∂A₁/∂Z₁ × ∂Z₁/∂W₁

The "backprop" part is that these gradients propagate backward from output to input — we compute them in reverse layer order and use them to update all weights simultaneously.

Deep Learning: Why Depth Matters

"Deep" learning refers to networks with many layers (typically more than 3). Deep networks can learn hierarchical representations:

Image recognition example (Convolutional Neural Network):

Layer 1:  Learns edges and gradients
Layer 2:  Combines edges into shapes (corners, curves)
Layer 3:  Combines shapes into textures (fur, fabric, skin)
Layer 4:  Combines textures into object parts (eyes, wheels, leaves)
Layer 5:  Combines parts into objects (cats, cars, trees)

No one programmed these features — the network discovered them by finding patterns that minimize the classification loss across millions of images.

This hierarchical feature learning is why deep networks outperform traditional ML on complex data:

• Images: each pixel is a raw feature; deep networks learn which pixels matter and how they combine
• Text: each word is a token; deep networks learn semantic meaning and syntax
• Audio: each time-frequency point is raw; deep networks learn phonemes, words, speakers

Key Architectures in 2026

Convolutional Neural Networks (CNNs): For image data. Use convolutional layers that detect local patterns regardless of position. ResNet, EfficientNet, and Vision Transformers are current best-practice architectures.

Recurrent Neural Networks (RNNs) and LSTMs: For sequence data. Maintain a hidden state that carries information through the sequence. Largely superseded by Transformers but still used in some settings.

Transformers: The dominant architecture for NLP and increasingly for vision and other modalities. Uses attention mechanisms to weigh the importance of each input element when processing each output. The foundation of GPT, BERT, Claude, Gemini, and essentially every major AI system.

Graph Neural Networks (GNNs): For graph-structured data (molecular structures, social networks, knowledge graphs). Growing area of research with significant practical applications.

Implementing a Neural Network with PyTorch

import torch
import torch.nn as nn
import torch.optim as optim

class ClassificationNetwork(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super().__init__()
        self.network = nn.Sequential(
            nn.Linear(input_size, hidden_size),
            nn.ReLU(),
            nn.Dropout(0.3),           # Regularization: randomly drop neurons
            nn.Linear(hidden_size, hidden_size),
            nn.ReLU(),
            nn.Dropout(0.3),
            nn.Linear(hidden_size, output_size)
        )
    
    def forward(self, x):
        return self.network(x)

# Initialize
model = ClassificationNetwork(input_size=10, hidden_size=64, output_size=2)
optimizer = optim.Adam(model.parameters(), lr=0.001)
criterion = nn.CrossEntropyLoss()

# Training loop
for epoch in range(100):
    # Forward pass
    outputs = model(X_train)
    loss = criterion(outputs, y_train)
    
    # Backward pass
    optimizer.zero_grad()
    loss.backward()      # Compute gradients
    optimizer.step()     # Update weights
    
    if epoch % 10 == 0:
        print(f"Epoch {epoch}, Loss: {loss.item():.4f}")

Common Failure Modes

Neural Networks Are Function Approximators, Not Brains

Neural networks are function approximators. Their "intelligence" comes not from any single neuron but from the collective effect of millions of weights, tuned through gradient descent across massive datasets.

Understanding the forward pass (how predictions get made) and the backward pass (how weights get updated) is the foundation for every modern architecture — CNNs, RNNs, Transformers, and whatever comes next.

The math required is genuinely accessible: matrix multiplication, the chain rule, basic calculus. The path from a single perceptron to a model like GPT-4 is longer than it looks, but every step connects to the one before it.

For hands-on practice, see our scikit-learn tutorial for traditional ML implementation and our machine learning beginners guide for the full learning path.

Further Reading

• Best Machine Learning Courses in 2025: Ranked After Taking Them All
• Computer Vision Tutorial: Build an Image Classifier from Scratch
• Recommendation Systems Explained: How Netflix and Amazon Know What You Want
• Machine Learning for Beginners: A Honest Guide to Getting Started
• ML Engineer Roadmap 2025: From Beginner to Hired in 12 Months
• Deploying Python Apps to AWS: A Complete Beginner's Guide
• RAG System Tutorial: Build a Production Retrieval-Augmented Generation System
• Deploy AI Model to Production: FastAPI, Docker, and Cloud Deployment Guide