The calculus behind neural networks

A neural network learns by repeating one loop: guess, measure how wrong the guess was, then nudge every weight in the direction that would have made the guess less wrong. That last step — figuring out which nudge to make — is entirely calculus.

This piece derives it from scratch: the chain rule that makes backpropagation possible, the derivative table for common activations, the surprisingly elegant softmax-plus-cross-entropy shortcut, and a full worked example you can trace by hand.

What a network actually computes

Strip away the hype and a neural network is a fairly plain machine: a chain of matrix multiplications and squashing functions that turns an input into a prediction.

Picture the network as a stack of layers, numbered $l = 1, 2, \dots, L$ . Each layer takes what the layer before it produced, mixes it with some weights, and passes the result through a non-linear function. Written out, layer $l$ does exactly two things:

\mathbf{z}^{(l)} = \mathbf{W}^{(l)} \mathbf{a}^{(l-1)} + \mathbf{b}^{(l)}

\mathbf{a}^{(l)} = \sigma\!\left(\mathbf{z}^{(l)}\right)

The first line is a weighted vote: every unit in the layer looks at everything the previous layer produced, weighs each signal by an importance number, and adds a bias. The second line squashes that vote through an activation function $\sigma$ , which is what lets the network represent curved, non-linear relationships rather than only straight lines.

A few pieces of notation worth keeping straight:

$\mathbf{W}^{(l)} \in \mathbb{R}^{n_l \times n_{l-1}}$ — the weight matrix connecting layer $l-1$ to layer $l$
$\mathbf{b}^{(l)} \in \mathbb{R}^{n_l}$ — the bias vector for layer $l$ , one free parameter per unit
$\mathbf{a}^{(l-1)}$ — the previous layer’s output, with $\mathbf{a}^{(0)} = \mathbf{x}$ , the raw input
$\sigma(\cdot)$ — an activation function applied element-wise

Once the signal reaches the last layer $L$ , the network has a prediction $\hat{y} = \mathbf{a}^{(L)}$ . A loss function $\mathcal{L}(\hat{y}, \mathbf{y})$ then scores how far that prediction sits from the true answer $\mathbf{y}$ . Everything from here on is about one question: how should each weight change to make $\mathcal{L}$ smaller?

The chain rule, and why backpropagation works at all

Training means nudging every weight downhill, using gradient descent:

\mathbf{W}^{(l)} \leftarrow \mathbf{W}^{(l)} - \eta \, \frac{\partial \mathcal{L}}{\partial \mathbf{W}^{(l)}}, \qquad \mathbf{b}^{(l)} \leftarrow \mathbf{b}^{(l)} - \eta \, \frac{\partial \mathcal{L}}{\partial \mathbf{b}^{(l)}}

where $\eta$ is the learning rate — how big a step to take. The entire difficulty is computing that gradient, $\partial \mathcal{L}/\partial \mathbf{W}^{(l)}$ , for every single weight in the network, without recomputing the whole forward pass millions of times. That is exactly what the chain rule buys us.

The multivariable chain rule

The loss $\mathcal{L}$ does not touch a weight $w_{ij}^{(l)}$ directly — it only feels it through the chain of vectors that weight influences downstream. So we route the derivative through the intermediate variable $z_k^{(l)}$ :

\frac{\partial \mathcal{L}}{\partial w_{ij}^{(l)}} = \sum_{k=1}^{n_l} \frac{\partial \mathcal{L}}{\partial z_k^{(l)}} \cdot \frac{\partial z_k^{(l)}}{\partial w_{ij}^{(l)}}

Now the simplification that makes this tractable: because $z_k^{(l)} = \sum_m w_{km}^{(l)} a_m^{(l-1)} + b_k^{(l)}$ , the weight $w_{ij}^{(l)}$ only ever appears in the formula for $z_i^{(l)}$ — nowhere else. So every term in that sum vanishes except the one where $k = i$ , leaving:

\frac{\partial z_i^{(l)}}{\partial w_{ij}^{(l)}} = a_j^{(l-1)}

Passing the blame backward: the delta recurrence

To avoid recomputing this chain from scratch for every weight, define a single quantity per unit — its local error, usually written $\delta$ — that captures exactly how sensitive the loss is to that unit’s raw input:

\delta_i^{(l)} \equiv \frac{\partial \mathcal{L}}{\partial z_i^{(l)}}

The trick is that $\delta_i^{(l)}$ at layer $l$ can be written entirely in terms of the deltas one layer ahead, $\delta^{(l+1)}$ . Applying the chain rule once more, backward through the layer boundary:

\delta_i^{(l)} = \sum_{k=1}^{n_{l+1}} \frac{\partial \mathcal{L}}{\partial z_k^{(l+1)}} \cdot \frac{\partial z_k^{(l+1)}}{\partial a_i^{(l)}} \cdot \frac{\partial a_i^{(l)}}{\partial z_i^{(l)}}

Two of those factors have simple closed forms: $\partial z_k^{(l+1)}/\partial a_i^{(l)} = w_{ki}^{(l+1)}$ , and $\partial a_i^{(l)}/\partial z_i^{(l)} = \sigma'(z_i^{(l)})$ . Substituting both in:

\delta_i^{(l)} = \left[ \sum_{k=1}^{n_{l+1}} \delta_k^{(l+1)} \, w_{ki}^{(l+1)} \right] \cdot \sigma'\!\left(z_i^{(l)}\right)

Written in matrix form, with $\odot$ denoting an element-wise (Hadamard) product, this becomes the single equation that is backpropagation:

\boldsymbol{\delta}^{(l)} = \left[ \left(\mathbf{W}^{(l+1)}\right)^{\!\top} \boldsymbol{\delta}^{(l+1)} \right] \odot \sigma'\!\left(\mathbf{z}^{(l)}\right)

Once you have $\delta^{(l)}$ for a layer, the gradients you actually need fall out immediately: $\partial \mathcal{L}/\partial w_{ij}^{(l)} = \delta_i^{(l)} \, a_j^{(l-1)}$ and $\partial \mathcal{L}/\partial b_i^{(l)} = \delta_i^{(l)}$ . One backward sweep, computing $\delta$ layer by layer from the output back to the input, hands you every gradient in the network. That is the entire algorithm.

Activation functions and their derivatives

The recurrence above needs $\sigma'(z)$ at every layer, so it is worth having the common activations and their derivatives on hand.

Function	Formula	Derivative
Sigmoid — squashes to $(0, 1)$	$\sigma(z) = \dfrac{1}{1+e^{-z}}$	$\sigma(z)\bigl(1-\sigma(z)\bigr)$
Tanh — squashes to $(-1, 1)$	$\tanh(z) = \dfrac{e^{z}-e^{-z}}{e^{z}+e^{-z}}$	$1-\tanh^2(z)$
ReLU — the default for hidden layers	$\max(0,z)$	$1$ if $z>0$ , else $0$

Drag $x$ , $w$ , and $b$ below to move the point along the curve. The tangent line’s slope is $\sigma'(z)$ — the exact quantity backpropagation needs at this unit.

Neuron explorer

x1.0w0.8b-0.1

z = wx + b = 0.700a = σ(z) = 0.668σ′(z) = 0.222

The tangent slope is σ′(z) — the quantity backpropagation needs at this unit.

Push $z$ far into either tail with sigmoid or tanh and watch the tangent go nearly flat. Switch to ReLU and the slope only ever takes two values: no in-between flattening at all.

Softmax and cross-entropy: an elegant special case

Classification problems with more than two categories almost always pair two specific ingredients — and the pairing is not a coincidence. It is chosen because the calculus collapses into something remarkably simple.

For a $K$ -class problem, the output layer uses softmax to turn raw scores into a probability distribution that sums to one:

a_i = \mathrm{Softmax}(z_i) = \frac{e^{z_i}}{\sum_{k=1}^{K} e^{z_k}}

and pairs it with categorical cross-entropy loss, which penalizes low confidence in the correct class:

\mathcal{L} = -\sum_{k=1}^{K} y_k \ln(a_k)

where $\mathbf{y}$ is a one-hot vector — all zeros except a single 1 marking the correct class.

Differentiating softmax

Softmax is trickier than most activations because every output $a_i$ depends on every input $z_j$ , not just its own. That means the derivative is a full Jacobian, and it splits into two cases.

When $i = j$ (a unit’s effect on its own output), the quotient rule gives:

\frac{\partial a_i}{\partial z_i} = a_i(1-a_i)

When $i \ne j$ (how one unit’s score drags down another’s probability):

\frac{\partial a_i}{\partial z_j} = -a_i a_j

Both cases fold into one compact expression using the Kronecker delta $\delta_{ij}$ :

\frac{\partial a_i}{\partial z_j} = a_i\left(\delta_{ij} - a_j\right)

The payoff: differentiating the loss

Chain the loss through every softmax output that $z_j$ influences:

\frac{\partial \mathcal{L}}{\partial z_j} = \sum_{i=1}^{K} \frac{\partial \mathcal{L}}{\partial a_i} \cdot \frac{\partial a_i}{\partial z_j}

The loss’s own derivative with respect to a probability is $\partial \mathcal{L}/\partial a_i = -y_i/a_i$ . Substituting and letting the $a_i$ terms cancel eventually yields:

\frac{\partial \mathcal{L}}{\partial z_j} = a_j - y_j \qquad \Longrightarrow \qquad \boldsymbol{\delta}^{(L)} = \mathbf{a}^{(L)} - \mathbf{y}

Adjust the three raw scores, pick which class is actually correct, and watch the probabilities — and the error signal $\boldsymbol{\delta}=\mathbf{a}-\mathbf{y}$ — update instantly.

Softmax playground

z₁2.0z₂0.5z₃-1.0

True class

loss = −ln(a_true) = 0.241a1 = 0.786 · δ1 = -0.214a2 = 0.175 · δ2 = 0.175a3 = 0.039 · δ3 = 0.039

Bars = probabilities a. Lower marks = error signal δ = a − y.

Set the true class to whichever bar is already tallest and the loss drops close to zero. Set it to the shortest bar instead — the network is confidently wrong — and watch the loss spike while $\delta_{\mathrm{true}}$ swings sharply negative.

A fully worked example, by hand

Formulas are easier to trust once you have pushed real numbers through them. Here is the smallest possible network — one input, one hidden unit, one output — worked from start to finish.

Setup

Input $x = 1.0$ , target $y = 0.0$
Weights $w_1 = 0.5$ , $w_2 = 0.8$ ; biases $b_1 = 0.2$ , $b_2 = -0.1$
Activation: sigmoid. Loss: squared error $\mathcal{L} = \tfrac{1}{2}(a_2-y)^2$

1. Forward pass — make a prediction

Hidden pre-activation: $z_1 = (0.5)(1.0) + 0.2 = 0.7000$
Hidden activation: $a_1 = \sigma(0.7) \approx 0.6682$
Output pre-activation: $z_2 = (0.8)(0.6682) - 0.1 = 0.4346$
Prediction: $a_2 = \sigma(0.4346) \approx 0.6070$
Loss $\approx 0.1842$ — the network is fairly wrong, since the target was 0

2. Backward pass — assign blame

Output error: $\delta_2 = (a_2-y)\cdot a_2(1-a_2) = (0.6070)(0.23855) \approx 0.1448$
Gradient for $w_2$ : $\delta_2 \cdot a_1 = (0.1448)(0.6682) \approx 0.0967$
Hidden error: $\delta_1 = \delta_2 \cdot w_2 \cdot a_1(1-a_1) = (0.1448)(0.8)(0.2217) \approx 0.02568$
Gradient for $w_1$ : $\delta_1 \cdot x = (0.02568)(1.0) \approx 0.02568$

Both weights should shrink slightly — that is the direction that reduces the loss. With learning rate $\eta$ , the update would be $w_2 \leftarrow 0.8 - \eta(0.0967)$ and $w_1 \leftarrow 0.5 - \eta(0.02568)$ . Run this loop thousands of times, over thousands of examples, and those small nudges are the entirety of what “training a neural network” means.

Questions that usually come up next

Do I need to compute any of this by hand in practice?
No — frameworks like PyTorch and TensorFlow use automatic differentiation to compute every gradient for you. The value of working through the derivation is that it explains why those tools behave the way they do: why gradients vanish, why certain activation/loss pairings are standard, and what a training curve is actually measuring.

Why is it called “backpropagation” specifically?
Because the local error $\delta$ is computed starting at the output layer and propagated backward, layer by layer, reusing each layer’s result to compute the next. Reusing intermediate results this way is what makes the algorithm efficient.

What happens if the activation function is not differentiable everywhere?
ReLU technically has an undefined derivative at exactly $z=0$ , but in practice frameworks just assign it 0 or 1 there by convention — the probability of landing on that exact point during training is negligible.

Is gradient descent guaranteed to find the best possible weights?
No. The loss surface of a deep network is highly non-convex, so gradient descent generally finds a good minimum rather than provably the best one. In practice, over-parameterized networks tend to have many minima that perform comparably well.