---
title: "Neural Posterior Calibration and Model Mismatch Interactive Lab | Sandbox Physics"
description: "How can you tell whether a neural network's posterior deserves trust? Test it on many simulated datasets whose true values are known."
canonical_url: "https://sandboxphysics.com/labs/neural-posterior-calibration/"
language: "en"
page_type: "lab"
alternate_languages:
  zh-CN: "https://sandboxphysics.com/zh/labs/neural-posterior-calibration/"
last_modified: "2026-11-14"
---

# Neural Posterior Calibration and Model Mismatch

**How can you tell whether a neural network's posterior deserves trust?** Test it on many simulated datasets whose true values are known. If its 90 percent regions catch the true value about 90 percent of the time, it is calibrated for data like those simulations. Real data can still contain noise or signals the simulator never made, and then even the exact posterior misses, so the data need their own goodness-of-fit check.

Change the noise or add a signal the model lacks to a synthetic ringdown, and watch when a frozen neural posterior and the exact posterior stop covering the truth.

> Interactive simulation: open this page in a web browser to change the parameters and run the model.

Model assumptions

The record is synthetic whitened strain: one damped sinusoid with known amplitude and phase, 128 samples at 4096 Hz, and white Gaussian noise of known level. The prior is uniform over frequency 200 to 300 Hz and damping time 2 to 10 ms. The exact posterior is a two-stage 64 by 64 grid under that model. The network is a five-component mixture density network trained offline on 30.7 million well-specified simulations with noise level 0.15 to 0.6; the browser only evaluates its frozen weights. The overtone, glitch and louder-noise cases are teaching choices, not Kerr overtones or a detector noise model. The live coverage counts repeated noise at one fixed source; the offline charts average over sources drawn from the prior. No real detector data are used.

Cyan: exact grid posterior Amber: frozen neural posterior White: injected source or nominal value

Neither check flags this record.

Where each posterior puts the source

Solid lines: 90% regions. Dashed: 50%. Dots: network draws. Drag the white marker to move the source.

How often the true source falls inside

Repeated noise at this one source. The pale band is the normal spread for an honest 90% region; short dotted marks are a saved 400-record reference when a preset is selected.

Offline calibration over the prior

Offline case: noise inside the training range, no mismatch.

More numbers and the offline rank test

- **Records in the live count**
- **Optimal signal-to-noise ratio**
- **Jensen–Shannon divergence between the posteriors · nats**
- **Exact posterior mean frequency · Hz**
- **Network posterior mean frequency · Hz**
- **Exact posterior mean damping time · ms**
- **Network posterior mean damping time · ms**
- **Noise level actually in the record**

Simulation-based calibration of the network

Rank of the true value among 99 network draws, for 2000 sources from the prior. Filled bars: frequency. Outlined bars: damping time. Grey band: 99% range for uniform ranks.

Training: 30.7 million simulations, 23198 weights, 7 minutes on one CPU. Frozen weights SHA-256 bb213976acf54c96…

## Three ways a posterior can fail

**Background** A damped sinusoid rings in noisy synthetic data. An exact grid calculation and a neural network trained on simulations both estimate its frequency and damping time from each new record.

**Why it matters** Neural posteriors are fast enough to analyse a gravitational-wave event in seconds, so their errors have to be found before anyone relies on them.

- **Focus question**: When does a posterior that looks sharp stop telling the truth?
- **One-sentence intuition**: Calibration on simulations tests the network against the model; only a test against the data checks the model itself.

### Core mathematical model

#### Bayes' theorem for the source parameters

$$
p(\theta\mid d)=\frac{p(d\mid\theta)\,p(\theta)}{p(d)}
$$

The exact posterior follows from the likelihood of the assumed model and the prior.

#### Gaussian white-noise likelihood

$$
\ln p(d\mid\theta)=-\frac{1}{2\sigma^2}\sum_{k}\bigl[d_k-h_k(\theta)\bigr]^2+\text{const}
$$

Every sample has independent noise of standard deviation sigma; the grid evaluates this sum for every candidate source.

#### Coverage of a 90% region

$$
\Pr\bigl[\theta^{\ast}\in R_{0.9}(d)\bigr]\approx0.9
$$

A calibrated posterior's 90% region contains the true source in about nine records out of ten.

### Common difficulties

#### A perfect likelihood does not fix a wrong model

**Typical misconception** If the network matches the exact posterior, the answer must be right.

**Better mental model** Both posteriors use the same assumed model. When the record holds a signal the model lacks, they can agree with each other and both miss the true source.

### Run the experiment

1. 01

   #### Watch the count settle

   Start from Trained conditions and let new records stream for half a minute.

   **What to observe:** Both fractions wander inside the pale band around 0.9. Both posteriors are calibrated here, and the cyan and amber regions overlap in the parameter plane.

2. 02

   #### Leave the training range

   Choose Cleaner than the training data, or drag the noise band edge below 0.15.

   **What to observe:** The exact posterior stays near 0.9, the network falls far below it, and the importance-sampling efficiency drops toward zero. That low efficiency is the warning sign.

3. 03

   #### Hide a signal from both analyses

   Choose A hidden second mode, then A glitch in the record. Read the efficiency and the residual test.

   **What to observe:** Both posteriors miss the source more often than one time in ten. The efficiency stays high because the network still agrees with the assumed model. The residual test catches the glitch but rarely the second mode, which the fit absorbs by shifting frequency and damping time.

Physical conclusion

Check the network against the exact answer, and check the model against the data.

## Related experiments and articles

Trusting posteriors learned from simulations · Reading path

### From a posterior to calibration and model mismatch

Articles on this path that use this experiment, followed by the other experiments they suggest.

1. [Article **How can you tell whether a neural network's posterior deserves trust?**](https://sandboxphysics.com/stories/simulation-based-inference/trusting-a-neural-posterior/)
2. [Article **What a posterior and a 90% credible region promise**](https://sandboxphysics.com/stories/simulation-based-inference/posterior-and-credible-regions/)
3. [Article **Checking posterior calibration with ranks and coverage**](https://sandboxphysics.com/stories/simulation-based-inference/rank-tests-for-posterior-calibration/)
4. [Article **Finding a wrong simulator with importance weights and residuals**](https://sandboxphysics.com/stories/simulation-based-inference/model-mismatch-and-importance-weights/)
5. [Article **Recompute the ringdown posterior calibration**](https://sandboxphysics.com/stories/simulation-based-inference/recompute-the-ringdown-calibration/)
6. [Article **Reading the DINGO evidence: P–P plots, divergences and sample efficiency**](https://sandboxphysics.com/stories/simulation-based-inference/reading-dingo-calibration-evidence/)

Fields & matter · Starter path

## From a field to an electron microscope

Map an electric field and steer a charged particle. Then find the electron's wavelength and use electron beams to see a crystal and a surface.

[Start here → **Make an Electric Field Visible** Move the probe around a dipole and check that the field always points downhill on the potential landscape. Observe · 8 min](https://sandboxphysics.com/labs/electric-field/)

1. [1 Make an Electric Field Visible](https://sandboxphysics.com/labs/electric-field/)
2. [2 A Charge Bending through Fields](https://sandboxphysics.com/labs/charged-particle/)
3. [3 Electron Wavelength & Voltage](https://sandboxphysics.com/labs/electron-wavelength/)
4. [4 Electron Diffraction & Reciprocal Space](https://sandboxphysics.com/labs/electron-diffraction/)
5. [5 TEM Bright Field, Dark Field & SAED](https://sandboxphysics.com/labs/tem-mode-selector/)
6. [6 SEM Secondary-Electron Topography](https://sandboxphysics.com/labs/sem-topography/)

[View the full path →](https://sandboxphysics.com/topics/fields-and-matter/#starter-path)