Checking posterior calibration with ranks and coverage

This chapter follows what a posterior and a 90% credible region promise and uses the same conjugate Gaussian model. New tools are the binomial distribution and the chi-square test, each introduced where it is needed. All numbers come from the saved run described in the method tutorial.

Why the rank of an exact posterior is uniform

Simulation-based calibration, or SBC, repeats four steps $N$ times.

1. Draw a true value $\tilde\theta$ from the prior $p(\theta)$.
2. Simulate data $\tilde y$ from the likelihood $p(y\mid\tilde\theta)$.
3. Draw $L$ samples $\theta_1,\dots,\theta_L$ from the posterior being tested, $q(\theta\mid\tilde y)$.
4. Record the rank $r=\#\{\ell:\theta_\ell<\tilde\theta\}$, an integer from $0$ to $L$.

The pair $(\tilde\theta,\tilde y)$ was drawn from $p(\theta)\,p(y\mid\theta)$. By Bayes' theorem that equals $p(y)\,p(\theta\mid y)$, so given $\tilde y$ the true value is itself a draw from the exact posterior. If $q$ is the exact posterior, $\tilde\theta,\theta_1,\dots,\theta_L$ are $L+1$ independent draws from one continuous distribution. Every ordering of them is equally likely, so the true value is equally likely to sit in each of the $L+1$ positions:

This is the uniformity result at the centre of the method of Talts and colleagues. Talts and colleagues, 2018, Section 4.1 Nothing in the argument needs the posterior to be normal or the parameter to be one number; it is applied to one parameter at a time.

How tall should each bar be?

The saved run uses $N=4000$ simulations and $L=99$ draws, so there are 100 possible ranks. Grouping five ranks per bar gives 20 bars. For an exact posterior each simulation lands in a given bar with probability $1/20$, independently of the others, so the count $c$ in one bar follows a binomial distribution with $4000$ trials and success probability $0.05$. Its mean and standard deviation are

The central 99% of that binomial runs from $165$ to $236$. Talts and colleagues draw this band behind the histogram. A single bar outside it is not alarming by itself: with 20 bars, an exact posterior has a fair chance of one bar straying. A joint check is the chi-square statistic

where $c_b$ is the count in bar $b$. For a flat histogram $X^2$ follows a chi-square distribution with 19 degrees of freedom, and a small tail probability is a warning.

Warnings also arrive by chance. In the saved run the exact posterior itself gave $X^2=36.5$, a tail probability of $0.0093$, with two bars outside the band. The run was kept rather than reseeded. A separate study repeated the whole experiment 200 times: the exact posterior was flagged at the 1% level in 2 of the 200 runs, close to the 1% false-alarm rate the threshold promises. Every wrong posterior below was flagged in all 200.

Reading the shape

The saved run tests four deliberately wrong posteriors against the exact one. Each wrong posterior is normal, so its rank histogram can be predicted without simulation. If $z=(\tilde\theta-\mu)/\sqrt v$ is the true value in units of the exact posterior, then $z$ is standard normal, and the fraction of a wrong posterior lying below the truth is a known function of $z$. A posterior with the right centre but its width multiplied by $k$ gives $u(z)=\Phi(z/k)$, and the rank is binomial with probability $u(z)$.

The predicted coverage for the narrow posterior is a one-line calculation. Its 90% interval reaches $1.645\times0.6\simeq0.987$ exact standard deviations from the centre, and $z$ is standard normal, so the coverage is $2\Phi(0.987)-1\simeq0.676$. The same reasoning with $1.645\times1.5$ gives $0.986$ for the wide one.

Each shape has a direct reading. A posterior that is too narrow often leaves the truth outside all of its draws, which piles ranks into the two end bars: a cup. A posterior that is too wide almost always surrounds the truth, so ranks gather in the middle: a hump. A posterior shifted upward puts most draws above the truth, so low ranks dominate. A posterior that uses a prior narrower than the one used to simulate shrinks too hard toward zero and comes out too narrow as well, so it also gives a cup. Talts and colleagues show the same cup for a regression model fitted with too narrow a prior. Talts and colleagues, 2018, Sections 4.2 and 6.1

Expected coverage at every level

A rank histogram looks at one parameter at a time. Expected coverage, defined by Hermans and colleagues, uses the posterior's own highest-density regions instead, and so can be applied to the joint distribution of several parameters. For each simulated pair, find the smallest credibility level $\alpha$ whose highest-density region contains the true value. Calibrated posteriors give values of $\alpha$ spread uniformly between zero and one, so the fraction of simulations with $\alpha$ below any level equals that level. A curve below the diagonal means overconfident regions. Hermans and colleagues, 2022, Section 2

The frozen neural posterior in the Lab learns a two-parameter posterior for a synthetic ringdown with the mixture-density approach of Papamakarios and Murray: a network outputs a mixture of Gaussians and is trained only on simulated pairs of source and record. Papamakarios and Murray, 2016 Over 2000 fresh simulations drawn from the prior, its 90% joint regions contained the true source in $0.897$ of them, the same fraction as the exact grid posterior. Its rank tests gave tail probabilities of $0.462$ for frequency and $0.800$ for damping time.

What averages over the prior can miss

Every check above is an average over sources drawn from the prior. Three failures can hide inside that average.

The first is a wrong joint distribution with correct one-dimensional pieces. While building the Lab, the network's sampler had an error in the term that sets the correlation between frequency and damping time. The rank tests still passed, with tail probabilities $0.462$ and $0.781$. A direct comparison between the sampler's moments and the density's own moments caught it: they differed by $0.223$, against $0.0038$ after the fix.

The second is a posterior that ignores the data. If $q(\theta\mid y)$ simply returns the prior, the true value and the draws are again independent draws from one distribution, so the ranks are perfectly flat. Rank tests check calibration, not sharpness.

The third is a region that is right on average and wrong in a corner of parameter space. Splitting the 400-simulation checks by noise level shows whether calibration holds across the training range.

With 400 records, the standard error of a coverage near $0.9$ is about $\sqrt{0.9\times0.1/400}=0.015$. All four rows agree with $0.9$ and with each other within two to three standard errors, and the network tracks the exact grid in every row.

Lab tasks

Open the neural posterior Lab and expand the offline rank test under the data row.

1. Predict: how many of the 20 bars for each parameter would you expect outside the 99% band if the network were exact? Then count.
2. Compare: switch to the faint, short ringdown and read the running coverage after the curve settles. Is this a failure of the network, or a property of the exact posterior at a fixed source near the prior edge?
3. Explain: why does the offline curve for the trained conditions lie on the diagonal at every level, while the running curve for one fixed source can sit below $0.9$?

Exercises

1. With $N=1000$ simulations and 20 bars, what are the mean and standard deviation of one bar's count for an exact posterior?
2. A posterior has the right width but is centred too low by half a standard deviation. Which end of the rank histogram fills up?
3. Why is a flat rank histogram not enough to call a posterior useful?