Approximation by Superpositions of a Sigmoidal Function

In 1969 Minsky and Papert published Perceptrons, proving that a single-layer perceptron cannot compute XOR. This result was devastating — it triggered the first "AI winter" and left a lingering doubt: even with hidden layers, do neural networks have fundamental representational blind spots?

By the late 1980s, backpropagation had revived interest by enabling multi-layer training. Networks were solving real problems. But success in practice does not equal proof in theory. The community needed a mathematical answer: can a neural network represent any continuous function, or are there functions forever out of reach?

A sigmoidal function $\sigma$ is any bounded measurable function that approaches 0 as its input goes to $-\infty$ and approaches 1 as its input goes to $+\infty$. The classic example is the logistic sigmoid $\sigma(t) = 1/(1 + e^{-t})$, but the theorem applies to any function with this S-shaped limiting behavior.

A single hidden-layer network takes an input $\mathbf{x} \in \mathbb{R}^n$ and computes:

$\hat{f}(\mathbf{x}) = \sum_{i=1}^{N} \alpha_i \, \sigma(\mathbf{w}_i^\top \mathbf{x} + b_i)$

Each neuron $i$ contributes one S-curve: $\sigma(\mathbf{w}_i^\top \mathbf{x} + b_i)$, scaled by $\alpha_i$. The weight vector $\mathbf{w}_i$ sets the orientation and steepness, the bias $b_i$ shifts it left or right, and $\alpha_i$ controls its amplitude. The full network output is a weighted sum — a superposition — of these S-curves.

Here is the formal claim. Let $\sigma$ be any continuous sigmoidal function. Let $f$ be any continuous function on the unit hypercube $[0,1]^n$. Then for every $\varepsilon > 0$, there exist $N$, weights $\{\mathbf{w}_i\}$, biases $\{b_i\}$, and coefficients $\{\alpha_i\}$ such that:

$\sup_{\mathbf{x} \in [0,1]^n} \left| f(\mathbf{x}) - \sum_{i=1}^{N} \alpha_i \, \sigma(\mathbf{w}_i^\top \mathbf{x} + b_i) \right| < \varepsilon$

In plain words: no matter what continuous function you pick and no matter how tight your error tolerance, there exists a single-layer network that fits within that tolerance everywhere on the domain.

The proof works by contradiction and uses two powerful theorems from functional analysis. Think of it as a courtroom argument:

Step 1 — Assume the opposite. Suppose the set of all possible network outputs $\mathcal{H}$ is not dense in the space of continuous functions $C([0,1]^n)$. This means there is some continuous function that no network output can get close to.

Step 2 — Invoke Hahn-Banach. If $\mathcal{H}$ is not dense, then by the Hahn-Banach separation theorem, there exists a nonzero bounded linear functional $L$ that equals zero on all of $\mathcal{H}$. Think of $L$ as a "detector" that ignores every network output.

Step 3 — Invoke Riesz Representation. By the Riesz theorem, this functional $L$ corresponds to a finite signed measure $\mu$ on $[0,1]^n$: $L(g) = \int g \, d\mu$. So $\mu$ annihilates every sigmoid superposition: $\int \sigma(\mathbf{w}^\top \mathbf{x} + b) \, d\mu = 0$ for all $\mathbf{w}, b$.

Step 4 — Show μ must be zero. Cybenko shows that a measure annihilating all sigmoids must also annihilate all indicator functions of half-spaces, hence all simple functions, hence all bounded measurable functions — so $\mu = 0$. But we said $L \neq 0$. Contradiction. Therefore $\mathcal{H}$ is dense, and the theorem is proved.

A crucial step in the proof relies on a beautiful geometric fact: as the weight magnitude $\|\mathbf{w}\|$ grows large, the sigmoid $\sigma(\mathbf{w}^\top \mathbf{x} + b)$ approaches a step function — it jumps sharply from 0 to 1 across a hyperplane.

This means sigmoids can approximate indicator functions of half-spaces. By combining multiple such "sharp" sigmoids with different orientations and positions, you can carve out any region of the input space. And if you can approximate indicator functions, you can approximate any simple function (a staircase), and simple functions are dense in continuous functions.

The chain of approximation works like this: sigmoids → step functions → indicator functions → simple functions → any continuous function.

The Universal Approximation Theorem is powerful but limited. It is important to understand what it guarantees and what it leaves open:

• No bound on width. The number of neurons $N$ needed could grow exponentially with the input dimension or the desired accuracy. The theorem only says a finite $N$ exists.

• No learning algorithm. Knowing a solution exists does not help you find it. Gradient descent may get stuck in local minima or require unfeasible training time.

• Only continuous functions. The theorem covers $C([0,1]^n)$. Discontinuous functions are not addressed (though later extensions by Hornik handle $L^p$ spaces).

• No depth advantage. The theorem is about width (one wide layer). Later work by Telgarsky (2016) showed that depth gives an exponential efficiency advantage: some functions need exponentially many neurons in one layer but only polynomially many in a deep network.

Cybenko's 1989 proof was a turning point. It told the neural network community: the architecture you already have is theoretically complete. Now invest your energy in training algorithms, data, and depth. That shift of focus — from "can it represent?" to "can it learn efficiently?" — is what led to modern deep learning.