What a posterior and a 90% credible region promise

This chapter assumes probability densities and the normal distribution. The Lab on likelihoods and priors in cosmology builds the prior and likelihood picture with sliders; here we turn it into equations. The aims are to compute a posterior by hand and on a grid, derive the coverage promise, and see where it fails for a fixed source.

Prior, likelihood and posterior for one number

Let the unknown be a single number $\theta$. Before measuring, our knowledge is the prior density $p(\theta)$. A measurement produces data $y$, and the model states how likely each data set is for each value of $\theta$: this is the likelihood $p(y\mid\theta)$, read as a function of $\theta$ with $y$ fixed. Bayes' theorem combines them:

The denominator $p(y)$, called the evidence, does not depend on $\theta$. It only makes the posterior integrate to one.

Take a concrete model. The prior is standard normal, $\theta\sim\mathcal N(0,1)$. We make $n$ independent measurements $y_i=\theta+e_i$, each with Gaussian noise of known standard deviation $s$. Then

The right side is quadratic in $\theta$, so the posterior is normal. Collect the $\theta^2$ and $\theta$ terms and complete the square:

The posterior variance $v$ is smaller than both the prior variance $1$ and the single-measurement variance $s^2$. The mean $\mu$ is the data average pulled toward the prior mean zero; this pull is called shrinkage.

A worked example

Use $n=4$ and $s=1$, with measurements $1.3$, $0.2$, $1.1$ and $0.6$. Their sum is $3.2$. The posterior variance is $v=1/5=0.2$, so the posterior standard deviation is $\sqrt{0.2}\simeq0.447$. The mean is $\mu=0.2\times3.2=0.64$, smaller than the data average $0.8$ because of shrinkage.

For a normal posterior the central 90% interval runs $1.645$ standard deviations either side of the mean, where $1.645$ is the point with 95% of a standard normal below it:

For a symmetric single-peaked posterior this central interval is also the highest-density region: the shortest interval holding 90% of the probability.

The same posterior on a grid

Most posteriors have no closed form. The Lab's exact reference therefore computes the posterior on a grid, and the same steps work here.

1. Lay down cells of width $0.01$ from $-3$ to $3$ and take each cell centre as a candidate $\theta_j$.
2. Evaluate the log prior plus log likelihood at each centre. Subtract the largest value before exponentiating so that no number underflows.
3. Divide by the sum so the cell probabilities $P_j$ add to one.
4. Sort cells from highest to lowest $P_j$ and add them up until the total reaches $0.9$. Those cells form the 90% highest-density region.

Doing this gives a grid mean of $0.640$ and standard deviation $0.447$. The region takes 148 cells holding $0.902$ of the probability and spans $[-0.10,\;1.38]$. The small excess over $0.9$ and the shift at the edges come from whole cells being added at once. In the Lab, the same recipe runs on a two-dimensional grid over frequency and damping time, refined around the peak so the cells stay much smaller than the posterior width.

Why the region catches the truth nine times in ten

Call the region $R(y)$ and suppose it holds posterior probability $0.9$ for every data set. Now imagine many repeated experiments: each draws a true value from the prior, then data from the likelihood. The fraction of experiments in which the truth lands inside the region is

Bayes' theorem lets us rewrite $p(\theta)\,p(y\mid\theta)$ as $p(y)\,p(\theta\mid y)$. The inner integral over $\theta$ then becomes the posterior probability of the region, which is $0.9$ for every $y$:

Two assumptions carried this result. The true values were drawn from the same prior the posterior used, and the data came from the same likelihood. Hermans and colleagues call the fraction on the left the expected coverage and use it as their main test of approximate posteriors. Hermans and colleagues, 2022, Definition 2.1 Talts and colleagues start from the same identity: averaged over the data, the posterior returns the prior. Talts and colleagues, 2018, Section 2

Coverage at one fixed true value

Hold the true value fixed and repeat only the noise. In the example, $\mu-\theta$ is then normal with mean $-0.2\,\theta$ and standard deviation $0.4$, because shrinkage pulls every estimate toward zero by a fifth. The 90% interval has half-width $0.736$, so its coverage at fixed $\theta$ is

where $\Phi$ is the standard normal cumulative distribution.

Near the prior centre the exact interval over-covers; far out it under-covers. Weighting these values by the standard normal prior and integrating gives back exactly $0.900$. An exact posterior is calibrated on average, not at every source.

The Lab shows the same effect in two dimensions. Its faint, short ringdown sits near the edge of the prior box. Over 400 synthetic records at that fixed source, the exact grid posterior's 90% region caught the source in $0.860$ of them, with a 95% interval of $0.823$ to $0.891$ on that fraction.

Lab tasks

Open the neural posterior Lab.

1. Predict, then watch: with the trained conditions, where should both running coverage curves settle after a minute? Explain the size of their early wobble from the number of records.
2. Change one condition: choose the faint, short ringdown. Before the curves settle, predict whether the exact posterior's coverage will sit above or below 0.9, using Table 2.1 as a guide.
3. Compare: drag the white marker in the parameter plane toward the centre of the prior box and then toward a corner. Watch the exact 90% contour change shape near the box edge.

Exercises

1. With the same prior and $s=1$, how many measurements are needed to bring the posterior standard deviation below $0.2$?
2. A colleague reports a 90% interval and says the true value has a 90% chance of being inside for this particular source. Under what condition is that a fair reading?
3. In the grid recipe, why must the cells be much narrower than the posterior width? What happens to the 90% region if they are not?