Reducing the Dimensionality of Data with Neural Networks

A 28×28 grayscale image has 784 pixel values — 784 dimensions. But handwritten digits don't fill all possible combinations of 784 numbers. The actual images cluster on a thin, curved surface — a manifold — inside that vast 784-dimensional space. A "2" varies by slant, thickness, and loop size: maybe 5–10 real degrees of freedom, not 784.

Dimensionality reduction finds that small set of numbers. Fewer dimensions mean faster search, better visualization, less noise, and often better classification. The question is: how do you find the right projection?

PCA — Principal Component Analysis — is the workhorse of dimensionality reduction. It finds the directions of greatest variance in the data and projects onto them. But PCA can only find flat directions: hyperplanes. If the manifold curves (and real manifolds always do), PCA has to use extra dimensions just to approximate the curvature, wasting capacity.

An autoencoder is a neural network trained to copy its input to its output — but with a twist: somewhere in the middle there's a bottleneck layer with far fewer neurons than the input. To squeeze 784 pixels through, say, 30 neurons and reconstruct the image on the other side, the network must learn a compressed representation that captures the essence of the data.

The first half — input to bottleneck — is the encoder. The second half — bottleneck to output — is the decoder. After training, throw away the decoder; the encoder is your dimensionality reducer.

A one-hidden-layer autoencoder works, but it's no better than PCA because a single hidden layer with linear activations is PCA. Nonlinear activations help a little, but the real power comes from depth: stacking many layers lets the network compose features hierarchically — edges → parts → objects.

But in 2006, training a 7-layer network from random weight initialization simply didn't work. Gradients vanished through the layers, early layers barely changed, and the network settled into a poor local minimum from which backpropagation couldn't escape. This was the central obstacle that had kept deep learning stuck for two decades.

A Restricted Boltzmann Machine (RBM) is a shallow two-layer network — visible units and hidden units — with no connections within a layer. It learns to model the probability distribution of its inputs using a simple algorithm called Contrastive Divergence (CD): show it data, let the hidden units respond, reconstruct the data from the hidden state, and adjust weights to make the reconstruction match the original.

Hinton's trick: stack RBMs. Train the first RBM on raw pixels. Freeze it. Use its hidden activations as "data" for a second RBM. Freeze. Repeat. Each RBM learns one layer of increasingly abstract features — without ever running backpropagation through the full depth. This greedy, one-layer-at-a-time strategy gives every layer a meaningful starting point before the network is assembled.

Why does this help? Think of it as scouting before the full expedition. Each RBM explores one floor of a building on its own, finding where the doors and hallways are. When you finally connect all floors and walk through the whole building (backpropagation), you already have a rough map — you won't get lost in a dead-end corridor (local minimum).

After pretraining four RBMs (784→1000, 1000→500, 500→250, 250→30), Hinton "unrolls" them into an autoencoder:

Encoder (four pretrained layers): 784 → 1000 → 500 → 250 → 30

Decoder (mirror, using transposed weights): 30 → 250 → 500 → 1000 → 784

Now the full 8-layer network is initialized near a good solution. Standard backpropagation minimizes reconstruction error end-to-end, adjusting all weights together. Because they started close, the network quickly reaches a much better minimum than random initialization ever could.

The paper's flagship architecture for MNIST has nine layers. The encoder side compresses: the input layer takes 784 pixel values (a 28×28 image), Hidden 1 expands to 1000 neurons (over-complete — learns diverse low-level features), Hidden 2 narrows to 500 (combines features into parts), Hidden 3 narrows further to 250 (composes parts into high-level concepts), and the code layer squeezes everything into just 30 dimensions — the bottleneck, the compressed representation.

The decoder side reconstructs: Hidden 4 (250) begins expanding the code, Hidden 5 (500) refines spatial detail, Hidden 6 (1000) restores the full feature set, and the output layer (784) produces the reconstructed image.

The first hidden layer is wider than the input (1000 > 784) — this is intentional. An over-complete first layer can learn many different features; the subsequent narrowing forces selection and abstraction. The true information bottleneck is the 30-unit code layer.

Hinton tested on four datasets: MNIST handwritten digits, Olivetti face images, random curves, and newsgroup documents (bag-of-words). In every experiment, the deep autoencoder with 30-dimensional codes dramatically outperformed PCA with 30 principal components:

• MNIST: reconstruction error was roughly half that of PCA. Digits reconstructed by the autoencoder were sharp and recognizable; PCA's were blurry ghosts.
• Faces: the autoencoder captured expression, lighting, and pose variations that PCA smeared together.
• Documents: in 2D visualization, autoencoder codes cleanly separated document categories that PCA lumped into overlapping clouds.

The improvement was not marginal — it was visually obvious and numerically massive. This was the proof that deep nonlinear dimensionality reduction was worth the complexity.

The choice of bottleneck size is a fundamental trade-off. Too few dimensions and you lose important information — reconstruction becomes blurry. Too many dimensions and the network doesn't learn to compress — it just memorizes. Hinton chose 30 for MNIST as a sweet spot: enough to represent the ~10 digit classes with their style variations, but far below 784.

The experiment below lets you feel this trade-off. As you reduce the bottleneck from 30 to 2, watch how the reconstructed digits degrade — but notice they stay recognizable surprisingly long, because nonlinear compression is remarkably efficient.

Hinton applied the same technique to documents represented as bag-of-words vectors (2000 word counts). The autoencoder architecture for documents was 2000 → 500 → 250 → 125 → 2 — compressing each document to just 2 numbers for visualization.

Using latent space codes for retrieval, the autoencoder outperformed Latent Semantic Analysis (the PCA of text) on precision–recall curves. When projected to 2D, different document categories formed distinct clusters — newsgroups about sports separated cleanly from those about religion or politics. LSA/PCA produced overlapping blobs.

This demonstrated that the autoencoder's nonlinear codes captured semantic structure that linear methods missed: documents about similar topics ended up near each other in code space, even when they shared few surface-level words.

One of the most compelling properties of a well-trained autoencoder is that its latent space is smooth: nearby points in code space produce similar outputs. If you take the code for a "3" and slowly move it toward the code for an "8", the decoded images morph gradually through plausible intermediate digits.

This smoothness arises because the network must reconstruct all training examples through the same 30-dimensional bottleneck. To minimize total error, it arranges codes so that similar inputs map to nearby codes — a manifold in code space that mirrors the manifold of real data. PCA codes are smooth too, but only along flat directions; the autoencoder's manifold can curve to match the data's true geometry.