A network can learn how often pictures occur
Suppose a training set contains many small pictures of stripes. Some have vertical stripes, others horizontal ones. A useful learner should discover that neighbouring pixels follow patterns: it should favour stripe pictures when it runs freely, including plausible ones it has not copied directly. The task is to learn a distribution, meaning the relative frequencies of different possible pictures.
David Ackley, Geoffrey Hinton and Terrence Sejnowski published a learning algorithm for Boltzmann machines in 1985. They combined interconnected units with ideas from statistical mechanics, the physics of many interacting components. The network would keep changing its state, and learning would adjust which arrangements were common.
Allow an occasional uphill move
Start with two-state units whose connections have equal strength in both directions. Their joint arrangement has an energy, and one unit updates at a time. At a low temperature, updates strongly favour states of lower energy; at a higher temperature, a unit can more often choose a locally unfavourable state. In an artificial network, temperature controls randomness. It need not be the temperature of a physical device.
An occasional uphill change can help the network leave a valley. For a small example, imagine two units that strongly favour agreeing. From both off to both on, a one-at-a-time path must temporarily make them disagree. A purely downhill rule can remain stuck in the first arrangement. Random updates can cross the intermediate state.
After sufficient sampling at a fixed positive temperature, the frequency of configurations follows a Boltzmann distribution: lower-energy arrangements are more likely, with the energy differences compared to the temperature. Sampling a range of states is what makes the frequency of patterns available for learning.
Sources: [1]
Let hidden units find a useful code
Visible units carry the data, such as pixels in a picture. Hidden units interact with them but are not prescribed directly by the training examples. They can represent combinations that recur across examples. A hidden unit can help several visible pixels act together even though the model’s connections involve pairs.
The 1985 paper included a small encoder task. Two groups of visible units had to communicate through fewer hidden units. The network had to find an internal code that preserved which input was present. These examples were small enough to inspect, so the authors could examine whether learning had found a useful representation.
Sources: [1]
Compare what the examples request with what the network produces
The learning rule compares two kinds of statistics. In one phase, visible units are held to training examples and the hidden units are allowed to sample. In another phase, the network runs freely. For each connection, compare how often its two ends are on together in these two situations.
If a pair is more often on together with the data present, strengthen its connection. If the free network produces that coincidence too often, weaken it. Learning thus makes examples easier to produce and reduces overproduced alternatives. At equilibrium, this difference gives the direction that improves the model’s match to the data distribution.
Both sets of statistics require adequate sampling. A network trapped in one valley can give misleading estimates, and hidden units can make training difficult. The original general algorithm was computationally expensive. Its contribution was a learning principle grounded in an equilibrium distribution, with explicit conditions for the statistical comparison.
Recall and learning answer different questions
In a Hopfield recall experiment, the connections are already fixed and the question is where a damaged cue settles. In Boltzmann learning, the connections change so the network’s long-run statistics resemble the examples. Random state changes and slower learning changes have different jobs.
The 2024 physics prize recognised Hopfield and Hinton’s foundational contributions to artificial neural networks. Ackley and Sejnowski remain coauthors of the 1985 learning result. The physical connection is especially clear here: equilibrium, fluctuations and interaction strengths became tools for building a model of data. A picture’s likelihood could be changed by reshaping the interactions among simple units.