---
title: "Checking posterior calibration with ranks and coverage"
description: "Simulation-based calibration ranks the true value among posterior draws. Flat ranks mean calibrated; humps, cups and slopes name the failure."
canonical_url: "https://sandboxphysics.com/stories/simulation-based-inference/rank-tests-for-posterior-calibration/"
language: "en"
page_type: "story"
alternate_languages:
  zh-CN: "https://sandboxphysics.com/zh/stories/simulation-based-inference/rank-tests-for-posterior-calibration/"
last_modified: "2026-11-12"
---

[Trusting posteriors learned from simulations](https://sandboxphysics.com/stories/simulation-based-inference/)

# Checking posterior calibration with ranks and coverage

Sandbox Physics · 2026-11-12

Draw a true value from the prior, simulate data, draw posterior samples and record how many samples fall below the true value. If the posterior is exact, that count, the rank, is equally likely to take every possible value. A histogram of ranks over thousands of simulations should therefore be flat. A hump in the middle means the posterior is too wide, a cup means it is too narrow, and a slope means it is shifted.

This chapter follows [what a posterior and a 90% credible region promise](https://sandboxphysics.com/stories/simulation-based-inference/posterior-and-credible-regions/) and uses the same conjugate Gaussian model. New tools are the binomial distribution and the chi-square test, each introduced where it is needed. All numbers come from the saved run described in the [method tutorial](https://sandboxphysics.com/stories/simulation-based-inference/recompute-the-ringdown-calibration/).

## Why the rank of an exact posterior is uniform

Simulation-based calibration, or SBC, repeats four steps $N$ times.

1. Draw a true value $\tilde\theta$ from the prior $p(\theta)$.
2. Simulate data $\tilde y$ from the likelihood $p(y\mid\tilde\theta)$.
3. Draw $L$ samples $\theta_1,\dots,\theta_L$ from the posterior being tested, $q(\theta\mid\tilde y)$.
4. Record the rank $r=\#\{\ell:\theta_\ell<\tilde\theta\}$, an integer from $0$ to $L$.

The pair $(\tilde\theta,\tilde y)$ was drawn from $p(\theta)\,p(y\mid\theta)$. By Bayes' theorem that equals $p(y)\,p(\theta\mid y)$, so given $\tilde y$ the true value is itself a draw from the exact posterior. If $q$ is the exact posterior, $\tilde\theta,\theta_1,\dots,\theta_L$ are $L+1$ independent draws from one continuous distribution. Every ordering of them is equally likely, so the true value is equally likely to sit in each of the $L+1$ positions:

$$
\Pr(r=k)=\frac{1}{L+1},\qquad k=0,1,\dots,L .
$$

This is the uniformity result at the centre of the method of Talts and colleagues. [Talts and colleagues, 2018, Section 4.1](https://arxiv.org/abs/1804.06788v2) Nothing in the argument needs the posterior to be normal or the parameter to be one number; it is applied to one parameter at a time.

## How tall should each bar be?

The saved run uses $N=4000$ simulations and $L=99$ draws, so there are 100 possible ranks. Grouping five ranks per bar gives 20 bars. For an exact posterior each simulation lands in a given bar with probability $1/20$, independently of the others, so the count $c$ in one bar follows a binomial distribution with $4000$ trials and success probability $0.05$. Its mean and standard deviation are

$$
\langle c\rangle=4000\times0.05=200,\qquad \sigma_c=\sqrt{4000\times0.05\times0.95}\simeq13.8 .
$$

The central 99% of that binomial runs from $165$ to $236$. Talts and colleagues draw this band behind the histogram. A single bar outside it is not alarming by itself: with 20 bars, an exact posterior has a fair chance of one bar straying. A joint check is the chi-square statistic

$$
X^2=\sum_{b=1}^{20}\frac{(c_b-200)^2}{200},
$$

where $c_b$ is the count in bar $b$. For a flat histogram $X^2$ follows a chi-square distribution with 19 degrees of freedom, and a small tail probability is a warning.

Warnings also arrive by chance. In the saved run the exact posterior itself gave $X^2=36.5$, a tail probability of $0.0093$, with two bars outside the band. The run was kept rather than reseeded. A separate study repeated the whole experiment 200 times: the exact posterior was flagged at the 1% level in 2 of the 200 runs, close to the 1% false-alarm rate the threshold promises. Every wrong posterior below was flagged in all 200.

## Reading the shape

The saved run tests four deliberately wrong posteriors against the exact one. Each wrong posterior is normal, so its rank histogram can be predicted without simulation. If $z=(\tilde\theta-\mu)/\sqrt v$ is the true value in units of the exact posterior, then $z$ is standard normal, and the fraction of a wrong posterior lying below the truth is a known function of $z$. A posterior with the right centre but its width multiplied by $k$ gives $u(z)=\Phi(z/k)$, and the rank is binomial with probability $u(z)$.

**Table 3.1** Wrong posteriors in the conjugate Gaussian test, with $n=4$, $s=1$, $N=4000$ and $L=99$.

| Posterior | Histogram shape | Bars outside the 99% band | 90% central coverage, simulated | 90% central coverage, predicted |
| --- | --- | --- | --- | --- |
| Exact | flat | $2$ | $0.914$ | $0.900$ |
| Shifted up by half a standard deviation | falling slope | $15$ | $0.864$ | $0.858$ |
| Width times $1.5$ | hump | $15$ | $0.988$ | $0.986$ |
| Width times $0.6$ | cup | $16$ | $0.685$ | $0.676$ |
| Prior standard deviation $0.5$ | cup | $15$ | $0.704$ | $0.702$ |

The predicted coverage for the narrow posterior is a one-line calculation. Its 90% interval reaches $1.645\times0.6\simeq0.987$ exact standard deviations from the centre, and $z$ is standard normal, so the coverage is $2\Phi(0.987)-1\simeq0.676$. The same reasoning with $1.645\times1.5$ gives $0.986$ for the wide one.

Each shape has a direct reading. A posterior that is too narrow often leaves the truth outside all of its draws, which piles ranks into the two end bars: a cup. A posterior that is too wide almost always surrounds the truth, so ranks gather in the middle: a hump. A posterior shifted upward puts most draws above the truth, so low ranks dominate. A posterior that uses a prior narrower than the one used to simulate shrinks too hard toward zero and comes out too narrow as well, so it also gives a cup. Talts and colleagues show the same cup for a regression model fitted with too narrow a prior. [Talts and colleagues, 2018, Sections 4.2 and 6.1](https://arxiv.org/abs/1804.06788v2)

## Expected coverage at every level

A rank histogram looks at one parameter at a time. Expected coverage, defined by Hermans and colleagues, uses the posterior's own highest-density regions instead, and so can be applied to the joint distribution of several parameters. For each simulated pair, find the smallest credibility level $\alpha$ whose highest-density region contains the true value. Calibrated posteriors give values of $\alpha$ spread uniformly between zero and one, so the fraction of simulations with $\alpha$ below any level equals that level. A curve below the diagonal means overconfident regions. [Hermans and colleagues, 2022, Section 2](https://arxiv.org/abs/2110.06581v3)

The frozen neural posterior in the Lab learns a two-parameter posterior for a synthetic ringdown with the mixture-density approach of Papamakarios and Murray: a network outputs a mixture of Gaussians and is trained only on simulated pairs of source and record. [Papamakarios and Murray, 2016](https://arxiv.org/abs/1605.06376v4) Over 2000 fresh simulations drawn from the prior, its 90% joint regions contained the true source in $0.897$ of them, the same fraction as the exact grid posterior. Its rank tests gave tail probabilities of $0.462$ for frequency and $0.800$ for damping time.

## What averages over the prior can miss

Every check above is an average over sources drawn from the prior. Three failures can hide inside that average.

The first is a wrong joint distribution with correct one-dimensional pieces. While building the Lab, the network's sampler had an error in the term that sets the correlation between frequency and damping time. The rank tests still passed, with tail probabilities $0.462$ and $0.781$. A direct comparison between the sampler's moments and the density's own moments caught it: they differed by $0.223$, against $0.0038$ after the fix.

The second is a posterior that ignores the data. If $q(\theta\mid y)$ simply returns the prior, the true value and the draws are again independent draws from one distribution, so the ranks are perfectly flat. Rank tests check calibration, not sharpness.

The third is a region that is right on average and wrong in a corner of parameter space. Splitting the 400-simulation checks by noise level shows whether calibration holds across the training range.

**Table 3.2** Network coverage of 90% regions in parts of the training range, 400 synthetic records each.

| Subset | Network | Exact grid |
| --- | --- | --- |
| Noise level $0.15$ to $0.25$ | $0.898$ | $0.903$ |
| Noise level $0.25$ to $0.4$ | $0.915$ | $0.915$ |
| Noise level $0.4$ to $0.6$ | $0.903$ | $0.910$ |
| Damping time $2$ to $3$ ms | $0.940$ | $0.925$ |

With 400 records, the standard error of a coverage near $0.9$ is about $\sqrt{0.9\times0.1/400}=0.015$. All four rows agree with $0.9$ and with each other within two to three standard errors, and the network tracks the exact grid in every row.

## Lab tasks

Open the [neural posterior Lab](https://sandboxphysics.com/labs/neural-posterior-calibration/) and expand the offline rank test under the data row.

1. Predict: how many of the 20 bars for each parameter would you expect outside the 99% band if the network were exact? Then count.
2. Compare: switch to the faint, short ringdown and read the running coverage after the curve settles. Is this a failure of the network, or a property of the exact posterior at a fixed source near the prior edge?
3. Explain: why does the offline curve for the trained conditions lie on the diagonal at every level, while the running curve for one fixed source can sit below $0.9$?

## Exercises

1. With $N=1000$ simulations and 20 bars, what are the mean and standard deviation of one bar's count for an exact posterior?
2. A posterior has the right width but is centred too low by half a standard deviation. Which end of the rank histogram fills up?
3. Why is a flat rank histogram not enough to call a posterior useful?

Answers

1. Each bar is binomial with $1000$ trials and probability $0.05$. The mean is $50$ and the standard deviation is $\sqrt{1000\times0.05\times0.95}\simeq6.9$. Talts and colleagues suggest keeping about 20 simulations per bar or more, so 1000 simulations support at most about 50 bars.
2. Most draws now lie below the truth, so the rank is usually large and the high end fills: a rising slope, the mirror image of the shifted row in Table 3.1.
3. The prior itself passes the test, because averaged over data the exact posterior returns the prior. A useful posterior must also be narrow where the data are informative. That needs a second measure, such as the width of the regions or the average of the log density at the truth.

## Sources

1. [Talts and colleagues, 2018, Section 4.1](https://arxiv.org/abs/1804.06788v2)
2. [Hermans and colleagues, 2022, Section 2](https://arxiv.org/abs/2110.06581v3)
3. [Papamakarios and Murray, 2016](https://arxiv.org/abs/1605.06376v4)

## Keep reading

Trusting posteriors learned from simulations · Reading path

### From a posterior to calibration and model mismatch

1. [Article **How can you tell whether a neural network's posterior deserves trust?**](https://sandboxphysics.com/stories/simulation-based-inference/trusting-a-neural-posterior/)
2. [Article **What a posterior and a 90% credible region promise**](https://sandboxphysics.com/stories/simulation-based-inference/posterior-and-credible-regions/)
3. [You are here **Checking posterior calibration with ranks and coverage**](https://sandboxphysics.com/stories/simulation-based-inference/rank-tests-for-posterior-calibration/)
4. [Article **Finding a wrong simulator with importance weights and residuals**](https://sandboxphysics.com/stories/simulation-based-inference/model-mismatch-and-importance-weights/)
5. [Article **Recompute the ringdown posterior calibration**](https://sandboxphysics.com/stories/simulation-based-inference/recompute-the-ringdown-calibration/)
6. [Article **Reading the DINGO evidence: P–P plots, divergences and sample efficiency**](https://sandboxphysics.com/stories/simulation-based-inference/reading-dingo-calibration-evidence/)