Finding a wrong simulator with importance weights and residuals

This chapter follows checking posterior calibration with ranks and coverage. It adds expectation values under a change of distribution, the chi-square distribution and a first-order Taylor expansion. The Lab's model is defined first, because every number below comes from it.

The teaching model

The record is synthetic whitened strain: 128 samples $d_k$ at times $t_k=k/(4096\,\mathrm{Hz})$, covering $31.25\,\mathrm{ms}$. The assumed signal is one damped sinusoid with fixed amplitude and phase,

with frequency $f$ uniform on $200$ to $300\,\mathrm{Hz}$ and damping time $\tau$ uniform on $2$ to $10\,\mathrm{ms}$. This is the convention of Berti, Cardoso and Will, whose quality factor $Q=\pi f\tau$ is about $4.7$ at the Lab's default source of $250\,\mathrm{Hz}$ and $6\,\mathrm{ms}$. Berti, Cardoso and Will, 2006, Section II The assumed noise is white and Gaussian with known standard deviation $\sigma$ per sample, so

The amplitude and noise are in the same whitened units. At $\sigma=0.2$ the default signal has an optimal signal-to-noise ratio $\sqrt{\textstyle\sum_k h_k^2}/\sigma\simeq13.0$. The network was trained only on records drawn from this model with $\sigma$ between $0.15$ and $0.6$.

Importance weights compare the network with the exact posterior

Let $q(\theta\mid d)$ be the network's posterior for $\theta=(f,\tau)$ and $\pi(\theta)$ the prior. Any expectation under the exact posterior can be written as an expectation under $q$:

Draw $n$ samples $\theta_i$ from the network, compute each weight $w_i$ and replace the expectations by averages. When $q$ equals the exact posterior, every weight equals the same number $p(d)$. When $q$ misses part of the posterior, a few samples in the missed region carry huge weights. The sample efficiency

measures this: it is one for equal weights and close to $1/n$ when one sample dominates. Dax and colleagues use this quantity for the gravitational-wave network DINGO and treat a low value as a sign that the data lie outside what the network learned. Dax and colleagues, 2023

For many samples, $\epsilon$ tends to $\mathbb E_q[w]^2/\mathbb E_q[w^2]$. A one-dimensional example shows how fast it falls. Let the exact posterior be standard normal and the network return a normal with the right centre and standard deviation $k$. Then

A network $1.5$ times too wide has $\epsilon\to0.83$. One $0.75$ times too narrow has $\epsilon\to0.63$. For $k\le1/\sqrt2$ the integral diverges: the weights have infinite variance, and the efficiency of any finite run is unreliable. A proposal that is too narrow is far more dangerous than one that is too wide. This is one reason DINGO is trained with a loss that favours covering all of the posterior mass.

In the Lab's model, the network has a median efficiency of $0.966$ over 2000 records drawn across the training range. For records with $\sigma$ between $0.06$ and $0.1$, cleaner than anything in training, the median falls to $0.029$. On those records the network's 90% regions caught the source in only $0.38$ of 400 cases, while the exact grid caught it in $0.89$.

A residual test checks the model against the data

Importance weights use the assumed likelihood, so they can only tell whether the network matches the assumed model. To test the model, look at what is left after fitting it. If the assumed model is right, then at the true parameters the scaled residuals $(d_k-h_k)/\sigma$ are 128 independent standard normal numbers. Fitting two parameters removes about two degrees of freedom, so the minimum

approximately follows a chi-square distribution with $126$ degrees of freedom, with mean $126$ and standard deviation $\sqrt{252}\simeq15.9$. The Lab reports the probability of a value at least this large. A value below $0.01$ flags the record.

On well-specified records the test flagged $0.9\%$, close to the $1\%$ it should. It flagged $96.5\%$ of records containing a glitch at $20\,\mathrm{ms}$ and every record whose true noise was $1.5$ times the assumed $\sigma$. In both cases the importance-sampling efficiency stayed well above the $0.1$ warning level, with medians near $0.70$ and $0.96$, because the network still matched the exact posterior of the assumed model. The exact posterior was wrong as well: its 90% regions caught the source in $0.40$ and $0.62$ of the records.

How a small missing signal shifts the answer

Suppose the data contain an extra signal $\delta h$ that the model lacks. Define the noise-weighted inner product $(a\mid b)=\sum_k a_k b_k/\sigma^2$ and the Fisher matrix $\Gamma_{ij}=(\partial_i h\mid\partial_j h)$. Expanding the best fit to first order in $\delta h$, Cutler and Vallisneri obtain the shift

Cutler and Vallisneri, 2007, Section II Both $\Gamma$ and the inner product scale as $1/\sigma^2$, so the shift does not depend on the noise level. The posterior width scales as $\sigma$. A fixed mismatch therefore matters more as the signal gets louder.

The saved grid check tests the formula with a schematic second mode added at the default source. At strength $0.02$ the formula predicts a frequency shift of $0.0888\,\mathrm{Hz}$ and the grid finds $0.0882\,\mathrm{Hz}$; for the damping time it predicts $0.0208\,\mathrm{ms}$ and the grid finds $0.0207\,\mathrm{ms}$. At strength $0.3$, lowering $\sigma$ from $0.6$ to $0.05$ moves the frequency shift from $1.76\,\mathrm{Hz}$ toward a high-signal value near $1.23\,\mathrm{Hz}$, while the shift measured in posterior standard deviations grows from $0.16$ to $1.70$.

The mismatch both checks miss

Put a second mode of strength $0.6$ into the default source at $\sigma=0.2$. The fit absorbs most of it by moving $f$ and $\tau$, so the residual looks almost like noise: the test flagged only $6\%$ of 400 records. The network still matches the exact posterior of the assumed model, so the median efficiency is $0.98$. Yet both posteriors caught the source in only about $0.74$ of the records at nominal 90%.

The rows are the Lab's presets at fixed sources. In the glitch row the efficiency also drops, to a median of $0.29$, because a glitch this strong moves the record away from anything in training; the residual test is what flags it. Talts and colleagues state the boundary in general form: calibration says nothing about whether the model is rich enough to capture the truth. Talts and colleagues, 2018, Section 4 A study of neural methods under deliberately wrong simulators found that none of the mitigations it tried prevented failure in every case. Cannon, Ward and Schmon, 2022 A richer model that includes the second mode, with its own posterior and calibration, is what would expose this case.

Lab tasks

Open the neural posterior Lab.

1. Predict: choose the glitch, then drag the violet handle down toward zero strength. Which readout recovers first, the efficiency or the residual p-value?
2. Change one condition: in the hidden second mode preset, raise the strength from $0.6$ to $1$. Does the residual test start to fire, and does the coverage gap grow?
3. Compare: drag the noise band edge just below $0.15$ and then just above $0.6$. Which side of the training range hurts the network more, and why does the exact posterior not care?

Exercises

1. Twenty importance weights are all equal except one, which is ten times larger. Compute $\epsilon$.
2. Why does the residual test use $126$ degrees of freedom and not $128$?
3. Using the scaling argument above, by what factor does the frequency shift in posterior standard deviations grow when $\sigma$ halves, if the linear formula holds?