Bayes' theorem for the source parameters
The exact posterior follows from the likelihood of the assumed model and the prior.
Neural Posterior Calibration and Model Mismatch
How can you tell whether a neural network's posterior deserves trust? Test it on many simulated datasets whose true values are known. If its 90 percent regions catch the true value about 90 percent of the time, it is calibrated for data like those simulations. Real data can still contain noise or signals the simulator never made, and then even the exact posterior misses, so the data need their own goodness-of-fit check.
Change the noise or add a signal the model lacks to a synthetic ringdown, and watch when a frozen neural posterior and the exact posterior stop covering the truth.
Truth inside the exact 90% region
Truth inside the network 90% region
Importance-sampling efficiency
Residual test p-value
Cyan: exact grid posteriorAmber: frozen neural posteriorWhite: injected source or nominal value
Neither check flags this record.
Solid lines: 90% regions. Dashed: 50%. Dots: network draws. Drag the white marker to move the source.
Repeated noise at this one source. The pale band is the normal spread for an honest 90% region; short dotted marks are a saved 400-record reference when a preset is selected.
Offline case: noise inside the training range, no mismatch.
More numbers and the offline rank test
- Records in the live count
- Optimal signal-to-noise ratio
- Jensen–Shannon divergence between the posteriors · nats
- Exact posterior mean frequency · Hz
- Network posterior mean frequency · Hz
- Exact posterior mean damping time · ms
- Network posterior mean damping time · ms
- Noise level actually in the record
Rank of the true value among 99 network draws, for 2000 sources from the prior. Filled bars: frequency. Outlined bars: damping time. Grey band: 99% range for uniform ranks.
Training: 30.7 million simulations, 23198 weights, 7 minutes on one CPU. Frozen weights SHA-256 bb213976acf54c96…
Three ways a posterior can fail
BackgroundA damped sinusoid rings in noisy synthetic data. An exact grid calculation and a neural network trained on simulations both estimate its frequency and damping time from each new record.
Why it mattersNeural posteriors are fast enough to analyse a gravitational-wave event in seconds, so their errors have to be found before anyone relies on them.
- Focus question
- When does a posterior that looks sharp stop telling the truth?
- One-sentence intuition
- Calibration on simulations tests the network against the model; only a test against the data checks the model itself.
Core mathematical model
Gaussian white-noise likelihood
Every sample has independent noise of standard deviation sigma; the grid evaluates this sum for every candidate source.
Coverage of a 90% region
A calibrated posterior's 90% region contains the true source in about nine records out of ten.
Common difficulties
A perfect likelihood does not fix a wrong model
Typical misconceptionIf the network matches the exact posterior, the answer must be right.
Better mental modelBoth posteriors use the same assumed model. When the record holds a signal the model lacks, they can agree with each other and both miss the true source.
Run the experiment
- 01
Watch the count settle
Start from Trained conditions and let new records stream for half a minute.
What to observe: Both fractions wander inside the pale band around 0.9. Both posteriors are calibrated here, and the cyan and amber regions overlap in the parameter plane. - 02
Leave the training range
Choose Cleaner than the training data, or drag the noise band edge below 0.15.
What to observe: The exact posterior stays near 0.9, the network falls far below it, and the importance-sampling efficiency drops toward zero. That low efficiency is the warning sign. - 03
Hide a signal from both analyses
Choose A hidden second mode, then A glitch in the record. Read the efficiency and the residual test.
What to observe: Both posteriors miss the source more often than one time in ten. The efficiency stays high because the network still agrees with the assumed model. The residual test catches the glitch but rarely the second mode, which the fit absorbs by shifting frequency and damping time.
Related experiments and articles
You are hereNeural Posterior Calibration and Model Mismatch
Articles
- Uses this experimentHow can you tell whether a neural network's posterior deserves trust?
- Uses this experimentWhat a posterior and a 90% credible region promise
- Uses this experimentChecking posterior calibration with ranks and coverage
- Uses this experimentFinding a wrong simulator with importance weights and residuals
- Uses this experimentRecompute the ringdown posterior calibration
- Uses this experimentReading the DINGO evidence: P–P plots, divergences and sample efficiency
Trusting posteriors learned from simulations · Reading path
From a posterior to calibration and model mismatch
Articles on this path that use this experiment, followed by the other experiments they suggest.
- ArticleHow can you tell whether a neural network's posterior deserves trust?
- ArticleWhat a posterior and a 90% credible region promise
- ArticleChecking posterior calibration with ranks and coverage
- ArticleFinding a wrong simulator with importance weights and residuals
- ArticleRecompute the ringdown posterior calibration
- ArticleReading the DINGO evidence: P–P plots, divergences and sample efficiency