Azimuth

Generative Models1985intermediate12 min read

A Learning Algorithm for Boltzmann Machines

خوارزمية تعلُّم لآلات بولتزمان

Ackley, D. H. · Hinton, G. E. · Sejnowski, T. J. — Cognitive Science

The problem

By the mid-1980s, neural networks could learn simple input–output mappings, but only in single- perceptrons with no hidden units. The had no way to discover useful — features that the programmer didn't hand-design. Multi-layer networks existed in theory, but there was no principled learning algorithm for adjusting weights on connections to hidden units, because there was no direct teaching signal for them. This was the problem: which internal connections deserve credit or blame for the network's overall performance?

The contribution

The Boltzmann machine: a stochastic network whose units flip on/off probabilistically based on an borrowed from statistical physics. The key innovation is a learning rule derived from information theory — adjust each by the difference between two correlations: how often two units are active together when data is clamped (the positive phase) versus when the network runs freely (the negative phase). This simple, local rule provably performs ascent on the of the data, solving the credit assignment problem without . Hidden units learn to discover internal representations automatically.

The impact

The Boltzmann machine was the first with hidden units that had a mathematically grounded learning algorithm. It introduced energy-based modeling to machine learning, inspired Restricted Boltzmann Machines and deep belief networks, and laid conceptual groundwork for modern generative models. Though too slow for practical use, its ideas — stochastic , energy landscapes, the positive/negative phase framework — echo through VAEs, GANs, diffusion models, and the entire pretraining revolution.

Imagine a committee of advisors sitting around a table. Each advisor can only say "agree" or "disagree." Pairs of advisors have relationships — some are allies who want to say the same thing, others are rivals who prefer to differ.

The committee's current mood — the pattern of agrees and disagrees — has a tension score (energy). High tension means many pairs are fighting their natural relationships. Left alone, the committee naturally drifts toward low-tension states where allies agree and rivals disagree.

A Boltzmann machine is this committee. The learning algorithm adjusts the strength of each relationship so that the low-tension patterns the committee naturally settles into match the real data the system needs to model.

The problem: how do hidden units learn without a teacher?

The perceptron can learn an input–output mapping, but only when there is a single layer of adjustable weights connecting inputs directly to outputs. The moment you add hidden units between input and output, you lose the teaching signal: the data tells you what the output should be, but says nothing about what the hidden units should do.

This is the credit assignment problem: when the network makes an error, which of the many internal connections should be strengthened, and which weakened? Without solving this, multi-layer networks were beautiful in theory and useless in practice.

Open in Lab
Click "Make Error" to see which hidden connections should change — and why the answer isn't obvious.
The demo wakes as you arrive…

The core idea: networks as physical systems with energy

The insight came from physics. In a solid like iron, atoms have magnetic spins that point up or down. Neighboring spins interact: aligned spins lower the system's energy, misaligned spins raise it. The whole system naturally settles into low-energy configurations.

Ackley, Hinton, and Sejnowski applied the same idea to a neural network. Each unit () has a binary state — on or off, like a spin pointing up or down. Each connection has a weight. The network's energy is determined by which units are on and how their states agree or disagree with the connection weights. Strong positive weights between co-active units lower the energy; active units connected by negative weights raise it.

This is the key conceptual leap: instead of programming a network, you define an energy landscape. The network's dynamics naturally roll downhill, and learning reshapes the landscape so that the valleys (low-energy states) correspond to the patterns in your data.

E(s)=−∑i<jwij si sj−∑ibi siE(\mathbf{s}) = -\sum_{i<j} w_{ij}\, s_i\, s_j - \sum_i b_i\, s_i
The energy function — the heart of a Boltzmann machine — Each unit i has binary state sᵢ (0 or 1). wᵢⱼ is the weight between units i and j. bᵢ is the bias of unit i. Co-active units connected by positive weights reduce energy; the network prefers these configurations.

Think of the energy function as a landscape of hills and valleys. Each possible combination of unit states is a location on this landscape. The energy value at that location is the altitude. The network "wants" to be in valleys — low-energy states — because those are the configurations it settles into naturally. Learning carves the valleys to match the training data.

Open in Lab
Toggle units on/off and watch the energy change. The network prefers states where co-active units have positive weights.
The demo wakes as you arrive…

Stochastic units and simulated annealing: escaping traps

If units simply switched to whatever state lowered the energy, the network would immediately get stuck in the nearest valley — a that might be far from the best solution. This is exactly the problem with Hopfield networks.

The Boltzmann machine solves this with stochastic updates: each unit flips to "on" with a probability given by the logistic (sigmoid) function of its total input, divided by a T. At high temperature, units flip nearly at random — the network explores wildly. As the temperature drops, it becomes increasingly likely to settle into low-energy states but can still occasionally jump over small barriers.

This process — starting hot and slowly cooling — is , borrowed from metallurgy, where slowly cooling metal lets atoms find their crystalline arrangement instead of freezing into a disordered mess.

P(si=1)=11+e−zi/T,zi=bi+∑jwij sjP(s_i = 1) = \frac{1}{1 + e^{-z_i / T}}, \quad z_i = b_i + \sum_j w_{ij}\, s_j
Stochastic update rule — flipping a unit — Unit i turns on with probability σ(zᵢ/T). At high T, the probability ≈ 0.5 regardless of input (random exploration). At T→0, the unit deterministically goes to the lower-energy state (a Hopfield network).
Open in Lab
Drag the temperature slider and watch units settle. High T = chaos. Low T = frozen pattern.
The demo wakes as you arrive…

Thermal equilibrium: the Boltzmann distribution

After enough stochastic updates at a fixed temperature, the network reaches : it doesn't settle into one state, but visits states with frequencies determined by their energies. Low-energy states are visited exponentially more often than high-energy ones. The precise relationship is the — the same law that governs how molecules distribute themselves across energy levels in physics.

This is what makes the network a : at equilibrium, the visible units produce patterns sampled from a well-defined probability . Learning adjusts the weights so that this distribution matches the distribution of the training data.

P(s)=e−E(s)/T∑ue−E(u)/TP(\mathbf{s}) = \frac{e^{-E(\mathbf{s}) / T}}{\displaystyle\sum_{\mathbf{u}} e^{-E(\mathbf{u}) / T}}
The Boltzmann distribution — probability from energy — The probability of state s is proportional to e^(−E/T). The denominator (partition function) sums over all possible states to normalize. Low-energy states get exponentially higher probability.

The learning rule: two phases, one subtraction

The learning rule is remarkably elegant. For each weight wᵢⱼ, measure two things:

  • Positive phase (clamped): Fix the visible units to a training pattern and let the hidden units reach equilibrium. Record how often units i and j are both active: ⟨sᵢsⱼ⟩_data.

  • Negative phase (free-running): Let the entire network (visible and hidden) run freely until equilibrium. Record the same co-activation statistic: ⟨sᵢsⱼ⟩_model.

Then update: Δwᵢⱼ = ε (⟨sᵢsⱼ⟩_data − ⟨sᵢsⱼ⟩_model).

If two units are active together more often in the data than in the model's fantasies, strengthen their connection. If the model over-predicts their co-activation, weaken it. The learning rule is purely local — each weight only needs information about the two units it connects — yet it provably performs gradient ascent on the log-likelihood of the data.

Δwij=ε(⟨si sj⟩data−⟨si sj⟩model)\Delta w_{ij} = \varepsilon \left( \langle s_i\, s_j \rangle_{\text{data}} - \langle s_i\, s_j \rangle_{\text{model}} \right)
The Boltzmann machine learning rule — ε is the learning rate. The rule increases weights that the data needs but the model under-produces (positive term > negative term) and decreases weights the model over-produces. At convergence: ⟨sᵢsⱼ⟩_data = ⟨sᵢsⱼ⟩_model for all i,j.
Open in Lab
Step through the positive and negative phases. Watch correlations converge as weights update.
The demo wakes as you arrive…

Visible and hidden units: the architecture

A Boltzmann machine has two types of units:

  • Visible units are the interface with the outside world. They are clamped to data patterns during the positive phase and left free during the negative phase.

  • Hidden units are internal detectors. They have no direct contact with the data but learn to represent higher-order regularities — correlations between visible units that can't be captured by direct pairwise weights alone.

Without hidden units, a Boltzmann machine can only model distributions that are expressible through pairwise correlations. Hidden units give it the power to model complex, multi-modal distributions by creating internal representations — exactly what perceptrons couldn't do. The key breakthrough: the same learning rule works regardless of whether a unit is visible or hidden. No separate algorithm is needed for hidden units.

Open in Lab
Explore a 4-visible, 3-hidden Boltzmann machine. Toggle between clamped and free-running modes.
The demo wakes as you arrive…

The encoder problem: proof that hidden units discover representations

To demonstrate the power of hidden units, the paper presented the problem: an N-to-N identity mapping through a . Take N input units and N output units, connected through only log₂(N) hidden units. The network must learn to reproduce any one-hot input on its output — but it can only transmit information through the narrow .

For N=8, the 3 hidden units must discover binary coding on their own. The network was never told about binary — it independently invented a compact encoding (like 101 for input 5) that minimizes . This was dramatic evidence that Boltzmann machines could learn useful internal representations without a teacher — exactly the capability that single-layer perceptrons lacked.

Open in Lab
Watch the 8→3→8 encoder converge. Hidden units independently discover binary coding.
The demo wakes as you arrive…

The idea in code

Boltzmann machine learning algorithm (simplified)python

Simplified to show the idea — not the real implementation.

import numpy as np

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

class BoltzmannMachine:
    def __init__(self, n_visible, n_hidden, lr=0.01):
        n = n_visible + n_hidden
        self.n_vis = n_visible
        self.n_hid = n_hidden
        self.W = np.random.randn(n, n) * 0.01   # symmetric weights
        self.W = (self.W + self.W.T) / 2         # enforce symmetry
        np.fill_diagonal(self.W, 0)               # no self-connections
        self.b = np.zeros(n)
        self.lr = lr

    def _total_input(self, state, i):
        """Net input to unit i from all other active units."""
        return self.b[i] + np.dot(self.W[i], state)

    def _anneal(self, state, free_units, steps=100, T_start=10, T_end=1):
        """Run simulated annealing on free units."""
        for step in range(steps):
            T = T_start * (T_end / T_start) ** (step / steps)
            i = np.random.choice(free_units)
            z = self._total_input(state, i)
            state[i] = 1 if np.random.rand() < sigmoid(z / T) else 0
        return state

    def positive_phase(self, visible_data, n_samples=50):
        """Clamp visible units, let hidden units reach equilibrium."""
        stats = np.zeros_like(self.W)
        state = np.zeros(self.n_vis + self.n_hid)
        hidden_idx = list(range(self.n_vis, self.n_vis + self.n_hid))
        for v in visible_data:
            state[:self.n_vis] = v          # clamp visible
            self._anneal(state, hidden_idx) # free hidden
            stats += np.outer(state, state)
        return stats / len(visible_data)

    def negative_phase(self, n_samples=50):
        """Let all units run freely until equilibrium."""
        stats = np.zeros_like(self.W)
        all_idx = list(range(self.n_vis + self.n_hid))
        for _ in range(n_samples):
            state = np.random.randint(0, 2, self.n_vis + self.n_hid).astype(float)
            self._anneal(state, all_idx, steps=200)
            stats += np.outer(state, state)
        return stats / n_samples

    def learn(self, data, epochs=100):
        """The full learning algorithm: positive − negative."""
        for epoch in range(epochs):
            pos = self.positive_phase(data)
            neg = self.negative_phase()
            self.W += self.lr * (pos - neg)  # the one-line learning rule
            np.fill_diagonal(self.W, 0)
            self.W = (self.W + self.W.T) / 2

The cost: why Boltzmann machines were too slow

The learning rule is mathematically elegant but computationally brutal. Each weight update requires two full equilibrium runs — and reaching equilibrium means running simulated annealing for many iterations. The negative phase is especially expensive: the model must freely explore its entire state space to collect representative statistics.

For a network with just tens of units, training took hours on 1985 hardware. Scaling to hundreds of units was impractical. This computational bottleneck kept Boltzmann machines as a theoretical contribution rather than a practical tool — until the invention of Restricted Boltzmann Machines (RBMs) nearly two decades later.

Legacy and impact

Though Boltzmann machines were rarely used directly, their ideas seeded an entire field. The energy-based perspective, the idea that a network's behavior can be understood as settling into low-energy states, became a foundational lens for understanding neural computation. The two-phase learning framework (data vs. model statistics) reappears in , noise-contrastive estimation, and the objective functions of GANs and diffusion models.

Most directly, the — a Boltzmann machine with no hidden-hidden or visible-visible connections — solved the speed problem by making the positive phase exact and the negative phase fast. Stacking RBMs produced deep belief networks, which reignited in 2006 and led directly to the modern era of foundation models.

  1. 1982

    Hopfield Networks

    Deterministic networks with an energy function. Could store patterns but had no hidden units, no learning for connections, and got stuck in local minima.

  2. 1985

    This paper — Boltzmann Machines

    Added stochastic units, hidden units, and a principled learning rule. Solved credit assignment but too slow for practical use.

  3. 1986

    Backpropagation (Rumelhart, Hinton, Williams)

    An alternative solution to credit assignment — faster than Boltzmann learning but requires differentiable activations and labeled data.

  4. 2002

    Contrastive Divergence (Hinton)

    Approximated the negative phase with just one reconstruction step. Made RBM training practical for the first time.

  5. 2006

    Deep Belief Networks (Hinton et al.)

    Stacked RBMs to pretrain deep networks. Reignited deep learning and proved the Boltzmann machine's conceptual framework could scale.

  6. 2014

    GANs and VAEs

    Modern generative models that inherit the two-phase philosophy: compare data statistics with model statistics, and adjust to close the gap.

CitationAckley, D. H., Hinton, G. E., & Sejnowski, T. J.. A Learning Algorithm for Boltzmann Machines. Cognitive Science, 1985.

Terms in this paper