---
title: "Finding a wrong simulator with importance weights and residuals"
description: "Importance weights test the network against the assumed likelihood; a residual test checks the likelihood against the data. A hidden second mode can pass both."
canonical_url: "https://sandboxphysics.com/stories/simulation-based-inference/model-mismatch-and-importance-weights/"
language: "en"
page_type: "story"
alternate_languages:
  zh-CN: "https://sandboxphysics.com/zh/stories/simulation-based-inference/model-mismatch-and-importance-weights/"
last_modified: "2026-11-12"
---

[Trusting posteriors learned from simulations](https://sandboxphysics.com/stories/simulation-based-inference/)

# Finding a wrong simulator with importance weights and residuals

Sandbox Physics · 2026-11-12

Calibration on simulations tests a neural posterior against its simulator, not the simulator against the data. Two further checks fill part of that gap. Importance weights compare the network with the exact posterior of the assumed model on the one record you have, so they catch a network that is out of its depth. A residual test asks whether the data minus the best-fit signal looks like the assumed noise, so it catches some ways the model itself is wrong. An unmodelled signal that the fit can absorb by shifting its parameters passes both.

This chapter follows [checking posterior calibration with ranks and coverage](https://sandboxphysics.com/stories/simulation-based-inference/rank-tests-for-posterior-calibration/). It adds expectation values under a change of distribution, the chi-square distribution and a first-order Taylor expansion. The Lab's model is defined first, because every number below comes from it.

## The teaching model

The record is synthetic whitened strain: 128 samples $d_k$ at times $t_k=k/(4096\,\mathrm{Hz})$, covering $31.25\,\mathrm{ms}$. The assumed signal is one damped sinusoid with fixed amplitude and phase,

$$
h_k(f,\tau)=A\,\mathrm e^{-t_k/\tau}\cos(2\pi f t_k+\varphi),\qquad A=1,\;\varphi=0,
$$

with frequency $f$ uniform on $200$ to $300\,\mathrm{Hz}$ and damping time $\tau$ uniform on $2$ to $10\,\mathrm{ms}$. This is the convention of Berti, Cardoso and Will, whose quality factor $Q=\pi f\tau$ is about $4.7$ at the Lab's default source of $250\,\mathrm{Hz}$ and $6\,\mathrm{ms}$. [Berti, Cardoso and Will, 2006, Section II](https://arxiv.org/abs/gr-qc/0512160v2) The assumed noise is white and Gaussian with known standard deviation $\sigma$ per sample, so

$$
\ln p(d\mid f,\tau)=-\frac{1}{2\sigma^2}\sum_{k=0}^{127}\bigl[d_k-h_k(f,\tau)\bigr]^2+\text{const}.
$$

The amplitude and noise are in the same whitened units. At $\sigma=0.2$ the default signal has an optimal signal-to-noise ratio $\sqrt{\textstyle\sum_k h_k^2}/\sigma\simeq13.0$. The network was trained only on records drawn from this model with $\sigma$ between $0.15$ and $0.6$.

## Importance weights compare the network with the exact posterior

Let $q(\theta\mid d)$ be the network's posterior for $\theta=(f,\tau)$ and $\pi(\theta)$ the prior. Any expectation under the exact posterior can be written as an expectation under $q$:

$$
\mathbb E_{p}[g]=\frac{\int g(\theta)\,\pi(\theta)\,p(d\mid\theta)\,\mathrm d\theta}{\int\pi(\theta)\,p(d\mid\theta)\,\mathrm d\theta}
=\frac{\mathbb E_{q}[g\,w]}{\mathbb E_{q}[w]},\qquad w(\theta)=\frac{\pi(\theta)\,p(d\mid\theta)}{q(\theta\mid d)} .
$$

Draw $n$ samples $\theta_i$ from the network, compute each weight $w_i$ and replace the expectations by averages. When $q$ equals the exact posterior, every weight equals the same number $p(d)$. When $q$ misses part of the posterior, a few samples in the missed region carry huge weights. The sample efficiency

$$
\epsilon=\frac{\bigl(\sum_i w_i\bigr)^2}{n\sum_i w_i^2}
$$

measures this: it is one for equal weights and close to $1/n$ when one sample dominates. Dax and colleagues use this quantity for the gravitational-wave network DINGO and treat a low value as a sign that the data lie outside what the network learned. [Dax and colleagues, 2023](https://arxiv.org/abs/2210.05686v2)

For many samples, $\epsilon$ tends to $\mathbb E_q[w]^2/\mathbb E_q[w^2]$. A one-dimensional example shows how fast it falls. Let the exact posterior be standard normal and the network return a normal with the right centre and standard deviation $k$. Then

$$
\frac{\mathbb E_q[w^2]}{\mathbb E_q[w]^2}=\int\frac{p(\theta)^2}{q(\theta)}\,\mathrm d\theta=\frac{k^2}{\sqrt{2k^2-1}} .
$$

A network $1.5$ times too wide has $\epsilon\to0.83$. One $0.75$ times too narrow has $\epsilon\to0.63$. For $k\le1/\sqrt2$ the integral diverges: the weights have infinite variance, and the efficiency of any finite run is unreliable. A proposal that is too narrow is far more dangerous than one that is too wide. This is one reason DINGO is trained with a loss that favours covering all of the posterior mass.

In the Lab's model, the network has a median efficiency of $0.966$ over 2000 records drawn across the training range. For records with $\sigma$ between $0.06$ and $0.1$, cleaner than anything in training, the median falls to $0.029$. On those records the network's 90% regions caught the source in only $0.38$ of 400 cases, while the exact grid caught it in $0.89$.

## A residual test checks the model against the data

Importance weights use the assumed likelihood, so they can only tell whether the network matches the assumed model. To test the model, look at what is left after fitting it. If the assumed model is right, then at the true parameters the scaled residuals $(d_k-h_k)/\sigma$ are 128 independent standard normal numbers. Fitting two parameters removes about two degrees of freedom, so the minimum

$$
\chi^2_{\min}=\min_{f,\tau}\frac{1}{\sigma^2}\sum_{k}\bigl[d_k-h_k(f,\tau)\bigr]^2
$$

approximately follows a chi-square distribution with $126$ degrees of freedom, with mean $126$ and standard deviation $\sqrt{252}\simeq15.9$. The Lab reports the probability of a value at least this large. A value below $0.01$ flags the record.

On well-specified records the test flagged $0.9\%$, close to the $1\%$ it should. It flagged $96.5\%$ of records containing a glitch at $20\,\mathrm{ms}$ and every record whose true noise was $1.5$ times the assumed $\sigma$. In both cases the importance-sampling efficiency stayed well above the $0.1$ warning level, with medians near $0.70$ and $0.96$, because the network still matched the exact posterior of the assumed model. The exact posterior was wrong as well: its 90% regions caught the source in $0.40$ and $0.62$ of the records.

## How a small missing signal shifts the answer

Suppose the data contain an extra signal $\delta h$ that the model lacks. Define the noise-weighted inner product $(a\mid b)=\sum_k a_k b_k/\sigma^2$ and the Fisher matrix $\Gamma_{ij}=(\partial_i h\mid\partial_j h)$. Expanding the best fit to first order in $\delta h$, Cutler and Vallisneri obtain the shift

$$
\delta\theta^{i}=\bigl(\Gamma^{-1}\bigr)^{ij}\,\bigl(\partial_j h\mid\delta h\bigr).
$$

[Cutler and Vallisneri, 2007, Section II](https://arxiv.org/abs/0707.2982v2) Both $\Gamma$ and the inner product scale as $1/\sigma^2$, so the shift does not depend on the noise level. The posterior width scales as $\sigma$. A fixed mismatch therefore matters more as the signal gets louder.

The saved grid check tests the formula with a schematic second mode added at the default source. At strength $0.02$ the formula predicts a frequency shift of $0.0888\,\mathrm{Hz}$ and the grid finds $0.0882\,\mathrm{Hz}$; for the damping time it predicts $0.0208\,\mathrm{ms}$ and the grid finds $0.0207\,\mathrm{ms}$. At strength $0.3$, lowering $\sigma$ from $0.6$ to $0.05$ moves the frequency shift from $1.76\,\mathrm{Hz}$ toward a high-signal value near $1.23\,\mathrm{Hz}$, while the shift measured in posterior standard deviations grows from $0.16$ to $1.70$.

## The mismatch both checks miss

Put a second mode of strength $0.6$ into the default source at $\sigma=0.2$. The fit absorbs most of it by moving $f$ and $\tau$, so the residual looks almost like noise: the test flagged only $6\%$ of 400 records. The network still matches the exact posterior of the assumed model, so the median efficiency is $0.98$. Yet both posteriors caught the source in only about $0.74$ of the records at nominal 90%.

**Table 4.1** Which check sees which failure, 400 synthetic records per row, nominal 90% regions.

| Record | Network coverage | Exact coverage | Median efficiency | Residual flags |
| --- | --- | --- | --- | --- |
| Trained conditions | $0.905$ | $0.900$ | $0.990$ | $0.8\%$ |
| Cleaner than the training data | $0.285$ | $0.893$ | $0.020$ | $1.3\%$ |
| Glitch in the record | $0.373$ | $0.113$ | $0.292$ | $100\%$ |
| Noise louder than assumed | $0.648$ | $0.650$ | $0.980$ | $100\%$ |
| Hidden second mode | $0.735$ | $0.743$ | $0.983$ | $6.3\%$ |

The rows are the Lab's presets at fixed sources. In the glitch row the efficiency also drops, to a median of $0.29$, because a glitch this strong moves the record away from anything in training; the residual test is what flags it. Talts and colleagues state the boundary in general form: calibration says nothing about whether the model is rich enough to capture the truth. [Talts and colleagues, 2018, Section 4](https://arxiv.org/abs/1804.06788v2) A study of neural methods under deliberately wrong simulators found that none of the mitigations it tried prevented failure in every case. [Cannon, Ward and Schmon, 2022](https://arxiv.org/abs/2209.01845v1) A richer model that includes the second mode, with its own posterior and calibration, is what would expose this case.

## Lab tasks

Open the [neural posterior Lab](https://sandboxphysics.com/labs/neural-posterior-calibration/).

1. Predict: choose the glitch, then drag the violet handle down toward zero strength. Which readout recovers first, the efficiency or the residual p-value?
2. Change one condition: in the hidden second mode preset, raise the strength from $0.6$ to $1$. Does the residual test start to fire, and does the coverage gap grow?
3. Compare: drag the noise band edge just below $0.15$ and then just above $0.6$. Which side of the training range hurts the network more, and why does the exact posterior not care?

## Exercises

1. Twenty importance weights are all equal except one, which is ten times larger. Compute $\epsilon$.
2. Why does the residual test use $126$ degrees of freedom and not $128$?
3. Using the scaling argument above, by what factor does the frequency shift in posterior standard deviations grow when $\sigma$ halves, if the linear formula holds?

Answers

1. Write the weights as nineteen of $1$ and one of $10$. Then $\sum w=29$ and $\sum w^2=19+100=119$, so $\epsilon=29^2/(20\times119)\simeq0.353$. A single outlier removes almost two thirds of the useful samples.
2. The minimum is taken over two fitted parameters. For a model that is close to linear in its parameters near the best fit, each fitted parameter removes one degree of freedom from the sum of squared residuals.
3. The shift itself is unchanged and the posterior width halves, so the shift in posterior standard deviations doubles. The grid values at $\sigma=0.3$ and $0.15$, $0.29$ and $0.57$, agree with this.

## Sources

1. [Berti, Cardoso and Will, 2006, Section II](https://arxiv.org/abs/gr-qc/0512160v2)
2. [Dax and colleagues, 2023](https://arxiv.org/abs/2210.05686v2)
3. [Cutler and Vallisneri, 2007, Section II](https://arxiv.org/abs/0707.2982v2)
4. [Talts and colleagues, 2018, Section 4](https://arxiv.org/abs/1804.06788v2)
5. [Cannon, Ward and Schmon, 2022](https://arxiv.org/abs/2209.01845v1)

## Keep reading

Trusting posteriors learned from simulations · Reading path

### From a posterior to calibration and model mismatch

1. [Article **How can you tell whether a neural network's posterior deserves trust?**](https://sandboxphysics.com/stories/simulation-based-inference/trusting-a-neural-posterior/)
2. [Article **What a posterior and a 90% credible region promise**](https://sandboxphysics.com/stories/simulation-based-inference/posterior-and-credible-regions/)
3. [Article **Checking posterior calibration with ranks and coverage**](https://sandboxphysics.com/stories/simulation-based-inference/rank-tests-for-posterior-calibration/)
4. [You are here **Finding a wrong simulator with importance weights and residuals**](https://sandboxphysics.com/stories/simulation-based-inference/model-mismatch-and-importance-weights/)
5. [Article **Recompute the ringdown posterior calibration**](https://sandboxphysics.com/stories/simulation-based-inference/recompute-the-ringdown-calibration/)
6. [Article **Reading the DINGO evidence: P–P plots, divergences and sample efficiency**](https://sandboxphysics.com/stories/simulation-based-inference/reading-dingo-calibration-evidence/)