Representation Learning2006intermediate13 min read

A Fast Learning Algorithm for Deep Belief Nets

خوارزمية تعلُّم سريعة لشبكات الاعتقاد العميق

Hinton, G. E. · Osindero, S. · Teh, Y.-W. — Neural Computation

The problem

Deep directed belief networks — networks with many hidden layers that how data is generated — should in principle be powerful generative models. But them was considered intractable: the "" effect made in densely connected layers exponentially hard, gradients vanished through depth, and the produced poor results because its sleep phase generated fantasies from a bad model. By 2006 the consensus was that deep networks simply couldn't be trained.

The contribution

Three interlocking ideas that made deep generative networks trainable for the first time. First, "" — a theoretical device that cancels explaining-away effects and makes the in each layer factorial, enabling exact greedy learning. Second, a fast greedy that trains the network one layer at a time using , with each new layer provably improving a variational bound on the data log-. Third, a contrastive wake-sleep procedure that fine-tunes the full network after greedy . The result: a three-hidden-layer network that achieved 1.25% error on MNIST — better than the best discriminative methods of the time — while also being a genuine that could synthesize realistic digits.

The impact

This paper broke the two-decade consensus that deep networks were untrainable and launched the modern revolution. The recipe inspired deep autoencoders, stacked denoising autoencoders, and the pre-train → fine-tune paradigm that underlies BERT, GPT, and every modern foundation model. The theoretical insight — that good initialization makes depth trainable — motivated later discoveries like ReLU, batch normalization, and residual connections that eventually made pretraining unnecessary for supervised tasks, but the conceptual breakthrough remains: depth works if you start well.

Imagine building a skyscraper in a city where nobody has built higher than two floors. Every architect who tried just poured concrete from ground to roof and watched it collapse under its own — the foundation couldn't support what came above it.

Hinton's idea: build one floor at a time. Pour the first floor and let it set. Then pour the second floor on a solid first floor. Repeat. Each floor is a Restricted Boltzmann Machine that learns patterns on its own before supporting the next floor. Once all floors stand, send an inspector through the whole building to fine-tune every joint.

The building stands — not because the concrete is different, but because the construction sequence is different. That sequence is greedy layer-wise pretraining.

The problem: why deep networks refused to learn

By 2006 the community faced a paradox. Theory said that deep networks — networks with many hidden layers — should be exponentially more expressive than shallow ones: they can represent complex functions with far fewer parameters. But in practice, every attempt to train a deep network from random initial weights ended in failure.

Two intertwined problems caused this:

Vanishing gradients. multiplies error signals through each layer. With activations, each multiplication shrinks the signal. By the time gradients reach the first layer, they're effectively zero — so early layers never learn meaningful features.
Explaining away. In a directed generative model (a belief net), observing data creates dependencies between hidden causes even when those causes are a priori independent. If two hidden units both explain a visible pattern, activating one "explains away" the need for the other. This makes the posterior over hidden states intractably complex — you can't decompose it into independent factors, so efficient learning seems impossible.

Open in Lab

Toggle between independent priors and observed data to see how explaining away creates hidden dependencies.

The demo wakes as you arrive…

The key insight: complementary priors cancel explaining away

Hinton's breakthrough starts with a theoretical observation. In a logistic belief net with a single , the posterior over hidden units is factorial — each hidden unit is independent given the data. The explaining-away problem only arises when you have multiple hidden layers, because the on the first hidden layer is no longer simple.

The idea: what if you could design the prior so that it perfectly cancels the explaining-away correlations introduced by the likelihood? Such a prior is called a complementary prior. When the prior and likelihood are both in the exponential family, a complementary prior always exists and makes the posterior exactly factorial.

Here's the practical trick: an RBM trained on data with Contrastive Divergence implicitly creates a complementary prior for the layer below it. So training an RBM on the hidden activations of a lower layer is equivalent to adding a complementary prior that cancels explaining away in that layer. Each new RBM you stack fixes the inference problem for the layer beneath it.

The building block: Restricted Boltzmann Machines

A Restricted Boltzmann Machine is a shallow undirected model with two layers: visible units (the data) and hidden units (the learned features). "Restricted" means there are no connections within a layer — every visible unit connects to every hidden unit, but hidden units don't connect to each other, and visible units don't connect to each other.

This restriction makes the RBM tractable in a crucial way: given the visible units, all hidden units are conditionally independent, and vice versa. You can compute the hidden activations from the data in a single parallel step — no iterative inference needed.

The RBM defines an over visible and hidden states. Lower energy means higher probability. Learning adjusts the weights to lower the energy of real data configurations and raise the energy of everything else.

E(\mathbf{v}, \mathbf{h}) = -\sum_i b_i v_i - \sum_j c_j h_j - \sum_{i,j} v_i W_{ij} h_j

Energy function of an RBM — v = visible units, h = hidden units, W = weight matrix, b and c = biases. Lower energy = higher probability. The model learns to assign low energy to real data.

Think of the energy function as a landscape of valleys and hills. Each data point is a ball that should roll into a deep valley. Training carves valleys under real data and fills in valleys elsewhere. The weight $W$ shapes this landscape — it determines which combinations of visible and hidden states are "low energy" (probable) and which are "high energy" (improbable).

Open in Lab

Click 'Train' to watch the energy landscape reshape — valleys deepen under real data patterns.

The demo wakes as you arrive…

Learning rule: Contrastive Divergence

Training an RBM requires computing a that involves summing over all possible configurations — exponentially many. Contrastive Divergence (CD) sidesteps this by approximating the sum with a single reconstruction step:

Positive phase: clamp the data on visible units, compute hidden activations (one parallel step thanks to the bipartite structure).

Negative phase: from those hidden activations, reconstruct the visible units, then recompute the hidden activations from the reconstruction.

Update: adjust weights to make real data statistics larger than reconstruction statistics. The weight update is simply the difference between two outer products — the data's and the reconstruction's.

CD-1 (one step of Gibbs ) is surprisingly effective. It doesn't compute the true gradient, but it points in a direction that consistently lowers the reconstruction error and makes the model assign higher probability to real data.

\Delta W_{ij} = \epsilon \left( \langle v_i h_j \rangle_{\text{data}} - \langle v_i h_j \rangle_{\text{recon}} \right)

Contrastive Divergence weight update — ε = learning rate · ⟨v_i h_j⟩_data = correlation between visible and hidden when data is clamped · ⟨v_i h_j⟩_recon = same correlation after one reconstruction step · Push weights toward the data, away from the model's fantasy.

Open in Lab

Step through one round of Contrastive Divergence: clamp data → compute hidden → reconstruct → update weights.

The demo wakes as you arrive…

The algorithm: greedy layer-wise pretraining

Now the pieces come together. The full algorithm builds a (DBN) from the bottom up:

Step 1: Train an RBM on the raw data (e.g., values). This first RBM learns low-level features — edge detectors, stroke fragments.

Step 2: Freeze the first RBM's weights. Use its hidden activations as "data" for a second RBM. This second RBM learns to model combinations of first-layer features — it discovers parts, curves, corners.

Step 3: Repeat. Each new RBM learns increasingly abstract features from the layer below. The theoretical guarantee says each new layer improves the model's bound on the data log-likelihood.

Step 4 (optional): Fine-tune the full network. Now that every layer has meaningful weights, use a contrastive version of the wake-sleep algorithm to adjust all weights jointly. Because the starting point is good, this converges to a much better solution than random initialization ever could.

The result is a hybrid model: the top two layers are an undirected RBM (associative memory), and the layers below are directed (generative). This hybrid structure is what makes the complementary priors theory work.

Open in Lab

Watch the network grow layer by layer. Each RBM trains on the previous layer's features before the next is added.

The demo wakes as you arrive…

Fine-tuning: the contrastive wake-sleep algorithm

After greedy pretraining, the network has good weights but they're not yet jointly optimized. The contrastive wake-sleep algorithm refines them:

Wake phase: present real data, propagate it bottom-up using recognition (bottom-up) weights to infer hidden states. Use these inferred states to adjust the top-down generative weights — make the generative model better at explaining the real data.

Sleep phase (contrastive version): instead of generating fantasies from the model (which would be bad because the model is still imperfect), start from data, propagate up to the top-level RBM, run a few steps of alternating Gibbs sampling in the top-level RBM, then propagate back down. Adjust the bottom-up recognition weights to better predict the hidden states from these top-down samples.

The key improvement over standard wake-sleep is that the sleep phase fantasies are anchored in real data (through the initial bottom-up pass and Gibbs sampling), so they are more realistic than pure top-down fantasies.

Architecture and MNIST results

The paper tested a DBN with three hidden layers on MNIST handwritten digits. The architecture modeled the joint distribution of digit images and their labels:

Visible layer: 784 pixel units (28×28 image) + 10 label units (one-hot)
Hidden layer 1: 500 units
Hidden layer 2: 500 units
Hidden layer 3 + label: 2000 units (top-level associative memory, as an RBM paired with the label units)

After greedy pretraining and , the network achieved 1.25% error rate on the MNIST test set — better than SVMs (1.4%), deep backprop nets from random init (~2%), and comparable to the best convolutional networks that used hand-designed geometric knowledge.

But the real triumph was that the DBN wasn't just a classifier — it was a generative model. You could clamp a label, sample from the top-level RBM, and propagate down through the directed layers to generate realistic-looking digit images. The network had truly learned the structure of handwritten digits.

Open in Lab

Click any layer to see its role. Toggle 'Generate' to see digits sampled from the model.

The demo wakes as you arrive…

The algorithm in code

Greedy layer-wise pretraining for a Deep Belief Networkpython

Simplified to show the idea — not the real implementation.

import numpy as np

def sigmoid(x):
    return 1 / (1 + np.exp(-np.clip(x, -500, 500)))

def train_rbm(data, n_hidden, lr=0.01, epochs=10, k=1):
    """Train one RBM layer using CD-k (Contrastive Divergence)."""
    n_visible = data.shape[1]
    W = np.random.randn(n_visible, n_hidden) * 0.01
    b_v = np.zeros(n_visible)   # visible biases
    b_h = np.zeros(n_hidden)    # hidden biases

    for epoch in range(epochs):
        for batch in np.array_split(data, len(data) // 64):
            # Positive phase: data → hidden
            h_prob = sigmoid(batch @ W + b_h)
            pos_associations = batch.T @ h_prob

            # Negative phase: k steps of Gibbs sampling
            v_sample = batch
            for _ in range(k):
                h_sample = (sigmoid(v_sample @ W + b_h) > np.random.rand(*h_prob.shape))
                v_sample = sigmoid(h_sample @ W.T + b_v)
            h_recon = sigmoid(v_sample @ W + b_h)
            neg_associations = v_sample.T @ h_recon

            # Weight update: push toward data, away from reconstruction
            W  += lr * (pos_associations - neg_associations) / len(batch)
            b_v += lr * (batch - v_sample).mean(axis=0)
            b_h += lr * (h_prob - h_recon).mean(axis=0)

    return W, b_h, b_v

# === Greedy layer-wise pretraining ===
layer_sizes = [784, 500, 500, 2000]
weights, biases = [], []
current_data = X_train  # shape (N, 784)

for i in range(len(layer_sizes) - 1):
    print(f"Training RBM layer {i+1}: {layer_sizes[i]} → {layer_sizes[i+1]}")
    W, b_h, b_v = train_rbm(current_data, layer_sizes[i+1])
    weights.append(W)
    biases.append(b_h)
    # Hidden activations become next layer's "data"
    current_data = sigmoid(current_data @ W + b_h)

# After pretraining: fine-tune with contrastive wake-sleep
# (or use the pretrained weights to initialize a supervised network)

The theoretical guarantee: each layer helps

One of the paper's most important contributions is a proof that greedy training provably improves the model. Specifically, adding a new RBM layer on top of the existing network can only improve (or maintain) a variational lower bound on the log-likelihood of the training data:

$\log P(\mathbf{v}) \geq \sum_{\mathbf{h}} Q(\mathbf{h}|\mathbf{v}) \log \frac{P(\mathbf{v}, \mathbf{h})}{Q(\mathbf{h}|\mathbf{v})}$

The bound tightens because each new layer provides a better prior for the layer below, replacing a factorial approximation with a richer model. In the limit, if each RBM is trained perfectly, the bound reaches the true log-likelihood.

This was revolutionary because it gave deep learning its first theoretical justification: depth helps, provably.

The generative power: a network that dreams

Unlike a pure classifier, the DBN is a generative model — it learns $P(\text{image}, \text{label})$ , not just $P(\text{label}|\text{image})$ . This means it can generate new data by sampling:

Fix a digit label (say "3")
Run Gibbs sampling in the top-level RBM to explore the "ravine" of 3-like patterns in the energy landscape
Propagate the sample down through the directed generative layers
The output is a new, realistic image of a "3"

Hinton described the manifolds of each digit class as "long ravines" in the free-energy landscape of the top-level RBM. Gibbs sampling walks along these ravines, producing diverse samples within each class. This was a powerful demonstration that the network had truly learned the structure of the data, not just a decision boundary.

Open in Lab

Select a digit class and watch the network 'dream' — generating diverse samples by walking along the energy ravine.

The demo wakes as you arrive…

Why it changed everything

2002
Contrastive Divergence (Hinton)
A fast approximation to the gradient of the log-likelihood for RBMs. Made RBM training practical and became the learning rule that powers the greedy algorithm in this paper.
2006
This paper (Hinton, Osindero, Teh)
Introduced Deep Belief Networks with greedy layer-wise pretraining. Proved that each layer improves a variational bound. Broke the consensus that deep networks are untrainable.
2006
Deep Autoencoders (Hinton & Salakhutdinov)
Applied the same pretraining recipe to autoencoders. Showed deep nonlinear compression dramatically outperforms PCA, extending the impact of the DBN training strategy.
2010
ReLU activation (Nair & Hinton)
ReLU largely solved the vanishing gradient problem for supervised training, reducing the need for unsupervised pretraining but vindicating the bet on depth.
2012
AlexNet — deep CNNs without pretraining
Showed that deep CNNs with ReLU and dropout could be trained from random init on large supervised data. Ended the RBM pretraining era but built on the depth insight.
2013
Knowledge Distillation (Hinton et al.)
Extended the generative insight: a trained deep network's "soft targets" carry dark knowledge that can train a smaller network. Conceptually linked to the DBN's generative capabilities.
2018
BERT & GPT: pre-train → fine-tune at scale
The recipe from 2006 — pretrain unsupervised, fine-tune supervised — reappeared at massive scale with Transformers. The conceptual DNA of this paper runs through every modern foundation model.

CitationHinton, G. E., Osindero, S., Teh, Y.-W.. A Fast Learning Algorithm for Deep Belief Nets. Neural Computation, 2006.

Terms in this paper

Deep Belief Networkشبكة الاحتمالات العميقة
Restricted Boltzmann Machine (RBM)آلة بولتزمان المقيَّدة
Greedy Layer-wise Pretrainingالتدريب المسبق الجشع طبقةً بطبقة
Contrastive Divergence (CD)التباعد التبايني
Complementary Priorsالأوائل المتمِّمة
Explaining Awayالتفسير بالإقصاء
Generative Modelالنموذج التوليدي
Wake-Sleep Algorithmخوارزمية اليقظة والنوم
Fine-Tuningالضبط الدقيق
Unsupervised Learningالتعلّم غير الخاضع للإشراف
Energy Functionدالة الطاقة
Pretrainingالتدريب المسبق