--- title: "How a Boltzmann machine learns which patterns belong together" description: "Why would a learning network benefit from making random changes?" canonical_url: "https://sandboxphysics.com/stories/thermal-complexity/hinton-boltzmann-learning/" language: "en" page_type: "story" alternate_languages: zh-CN: "https://sandboxphysics.com/zh/stories/thermal-complexity/hinton-boltzmann-learning/" last_modified: "2026-10-24" --- [When a network can remember and learn](https://sandboxphysics.com/stories/thermal-complexity/#networks) Heat · Chance · Collective behaviour # How a Boltzmann machine learns which patterns belong together Why would a learning network benefit from making random changes? Sandbox Physics · 2026-10-24 Nobel contribution: 2024-1 ## A network can learn how often pictures occur Suppose a training set contains many small pictures of stripes. Some have vertical stripes, others horizontal ones. A useful learner should discover that neighbouring pixels follow patterns: it should favour stripe pictures when it runs freely, including plausible ones it has not copied directly. The task is to learn a distribution, meaning the relative frequencies of different possible pictures. David Ackley, Geoffrey Hinton and Terrence Sejnowski published a learning algorithm for Boltzmann machines in 1985. They combined interconnected units with ideas from statistical mechanics, the physics of many interacting components. The network would keep changing its state, and learning would adjust which arrangements were common. Sources: [1] [2] ## Allow an occasional uphill move Start with two-state units whose connections have equal strength in both directions. Their joint arrangement has an energy, and one unit updates at a time. At a low temperature, updates strongly favour states of lower energy; at a higher temperature, a unit can more often choose a locally unfavourable state. In an artificial network, temperature controls randomness. It need not be the temperature of a physical device. An occasional uphill change can help the network leave a valley. For a small example, imagine two units that strongly favour agreeing. From both off to both on, a one-at-a-time path must temporarily make them disagree. A purely downhill rule can remain stuck in the first arrangement. Random updates can cross the intermediate state. After sufficient sampling at a fixed positive temperature, the frequency of configurations follows a Boltzmann distribution: lower-energy arrangements are more likely, with the energy differences compared to the temperature. Sampling a range of states is what makes the frequency of patterns available for learning. Sources: [1] ## Let hidden units find a useful code Visible units carry the data, such as pixels in a picture. Hidden units interact with them but are not prescribed directly by the training examples. They can represent combinations that recur across examples. A hidden unit can help several visible pixels act together even though the model’s connections involve pairs. The 1985 paper included a small encoder task. Two groups of visible units had to communicate through fewer hidden units. The network had to find an internal code that preserved which input was present. These examples were small enough to inspect, so the authors could examine whether learning had found a useful representation. Sources: [1] ## Compare what the examples request with what the network produces The learning rule compares two kinds of statistics. In one phase, visible units are held to training examples and the hidden units are allowed to sample. In another phase, the network runs freely. For each connection, compare how often its two ends are on together in these two situations. If a pair is more often on together with the data present, strengthen its connection. If the free network produces that coincidence too often, weaken it. Learning thus makes examples easier to produce and reduces overproduced alternatives. At equilibrium, this difference gives the direction that improves the model’s match to the data distribution. Both sets of statistics require adequate sampling. A network trapped in one valley can give misleading estimates, and hidden units can make training difficult. The original general algorithm was computationally expensive. Its contribution was a learning principle grounded in an equilibrium distribution, with explicit conditions for the statistical comparison. Sources: [1] [2] ## Recall and learning answer different questions In a Hopfield recall experiment, the connections are already fixed and the question is where a damaged cue settles. In Boltzmann learning, the connections change so the network’s long-run statistics resemble the examples. Random state changes and slower learning changes have different jobs. The 2024 physics prize recognised Hopfield and Hinton’s foundational contributions to artificial neural networks. Ackley and Sejnowski remain coauthors of the 1985 learning result. The physical connection is especially clear here: equilibrium, fluctuations and interaction strengths became tools for building a model of data. A picture’s likelihood could be changed by reshaping the interactions among simple units. Sources: [3] [2] Continue with the physics ## Try the related models ### [Start with the recall problem](https://sandboxphysics.com/labs/hopfield-associative-memory/) Corrupt a cue and watch a fixed network settle before reading how connections can learn. This companion model illustrates Hopfield recall with fixed weights; the Boltzmann training procedure is explained in the article. ## Sources and reading notes 1. [Ackley, Hinton and Sejnowski (1985), A Learning Algorithm for Boltzmann Machines](https://www.cs.toronto.edu/~fritz/absps/cogscibm.pdf) Sections 2–3 explain stochastic updates, equilibrium and the clamped/free statistical comparison; the encoder example shows why hidden units are useful. 2. [Geoffrey Hinton, Publications on Boltzmann machines](https://www.cs.toronto.edu/~hinton/boltzmann.html) The author’s bibliography identifies the 1983 Hinton–Sejnowski contribution, the three-author 1985 learning paper and the 1986 account of learning and relearning. 3. [Royal Swedish Academy of Sciences (2024), Physics prize announcement](https://www.nobelprize.org/prizes/physics/2024/press-release/) Use the official citation for award scope and date. Read the original papers for the mechanisms and coauthors. [Heat, chance and collective behaviour](https://sandboxphysics.com/stories/thermal-complexity/) [How a Hopfield network finds a damaged memory](https://sandboxphysics.com/stories/thermal-complexity/hopfield-energy-memory/)