---
title: "Reading the DINGO evidence: P–P plots, divergences and sample efficiency"
description: "A figure-by-figure guide to the 2021 DINGO letter and the 2023 importance-sampling letter, with what each test shows and how out-of-range data are flagged."
canonical_url: "https://sandboxphysics.com/stories/simulation-based-inference/reading-dingo-calibration-evidence/"
language: "en"
page_type: "story"
alternate_languages:
  zh-CN: "https://sandboxphysics.com/zh/stories/simulation-based-inference/reading-dingo-calibration-evidence/"
last_modified: "2026-11-12"
---

[Trusting posteriors learned from simulations](https://sandboxphysics.com/stories/simulation-based-inference/)

# Reading the DINGO evidence: P–P plots, divergences and sample efficiency

Sandbox Physics · 2026-11-12

Two letters carry most of the case that the DINGO network gives trustworthy gravitational-wave posteriors. The 2021 letter shows calibration on simulated signals in Figure 2 and agreement with a standard sampler on eight real events in Figure 4. The 2023 letter adds importance weights, whose sample efficiency verifies each event and flags data the network was not trained for. None of these tests asks whether the assumed waveform and noise model describe the data. This guide reads the main text of arXiv version 2 of both letters; their supplemental material was not read, and none of their results is reproduced here.

The guide assumes the course chapters on [rank tests](https://sandboxphysics.com/stories/simulation-based-inference/rank-tests-for-posterior-calibration/) and [importance weights](https://sandboxphysics.com/stories/simulation-based-inference/model-mismatch-and-importance-weights/). Every number below comes from the cited paper unless it is marked as a result of this site's toy model.

## What was read and what kind of source it is

**Table 6.1** Sources, versions and reading depth.

| Source | Status | Read for this guide |
| --- | --- | --- |
| Dax and colleagues, arXiv 2106.12594 version 2 | published letter, 2021 | main text, Figures 2 and 4 |
| Dax and colleagues, arXiv 2210.05686 version 2 | published letter, 2023 | main text, Tables I and II |
| Talts and colleagues, arXiv 1804.06788 version 2 | preprint, no journal version listed on arXiv | Sections 1 to 6 |
| Cannon, Ward and Schmon, arXiv 2209.01845 version 1 | preprint, no journal version listed on arXiv | abstract, introduction, Section 2.1, conclusion |
| Cranmer, Brehmer and Louppe, arXiv 1911.01429 version 3 | published review | abstract and introduction |

The arXiv records list Physical Review Letters 127, 241103 for the first letter and 130, 171403 for the second. [Dax and colleagues, 2021](https://arxiv.org/abs/2106.12594v2) [Dax and colleagues, 2023](https://arxiv.org/abs/2210.05686v2) The review by Cranmer and colleagues places this work in the wider field of simulation-based inference, where a simulator can produce data but its likelihood is hard or impossible to write down. [Cranmer, Brehmer and Louppe, 2020](https://arxiv.org/abs/1911.01429v3) DINGO is an unusual member of that field, because a gravitational-wave likelihood can be evaluated. That fact makes the 2023 check possible.

## Figure 2 of the 2021 letter: the P–P plot

The test uses 1000 simulated signals in stationary Gaussian noise, drawn the same way as the training data. For each parameter, the authors take the percentile of the true value within its one-dimensional posterior and plot the cumulative distribution of these percentiles. A calibrated posterior gives uniform percentiles, so each curve should follow the diagonal. Kolmogorov–Smirnov p-values for the separate parameters are combined into one p-value of $0.46$.

This is the rank test of Talts and colleagues drawn as a cumulative curve instead of a histogram. If $u_j$ is the percentile of the truth in simulation $j$, the plotted curve is

$$
\hat F(x)=\frac{1}{N}\sum_{j=1}^{N}\mathbf 1\{u_j\le x\},\qquad N=1000,
$$

and calibration predicts $\hat F(x)\approx x$. [Talts and colleagues, 2018, Section 4.1](https://arxiv.org/abs/1804.06788v2) Read the figure for three things.

1. It tests marginals only. A wrong correlation between two parameters leaves every curve on the diagonal. This site's toy model met exactly that failure: a sampler bug passed one-dimensional rank tests with tail probabilities $0.462$ and $0.781$.
2. It averages over the prior. A posterior that is too wide in one region and too narrow in another can still produce a diagonal curve.
3. Its noise is the training noise. The letter says this directly: real data contain noise that is neither strictly stationary nor Gaussian and therefore lie outside the training distribution.

## Figure 4 of the 2021 letter: divergences on real events

The main result runs DINGO and the LALInference MCMC sampler on the eight binary black hole events of the first catalogue that fit the network's prior, with component masses from $10$ to $80$ solar masses. DINGO draws 50,000 samples with 30 iterations of its time-alignment step in about 20 seconds. Training took about 10 days on one A100 GPU with $5\times10^{6}$ waveforms.

Figure 4 reports the Jensen–Shannon divergence between the two samplers' one-dimensional posteriors. The divergence runs from $0$ to $\ln 2\approx0.69$ nats. The mean over events and parameters is $0.0009$ nats. Two LALInference runs that differ only in their random seed give $0.0007$ nats. A maximum of $0.002$ nats, taken from an earlier study, counts as indistinguishable; two events meet it for every parameter, and the others exceed it for one to three parameters. For scale, the letter states that changing the noise spectrum estimate or the waveform model moves the divergence by about $0.02$ nats.

Both samplers use the same waveform model and the same noise assumptions. Figure 4 shows that the network reproduces the posterior of that model on real data. Whether the model fits the data is outside the figure, and the waveform-model shift of $0.02$ nats is twenty times the network's error.

## The 2023 letter: importance weights as a per-event check

The second letter treats the network output $q(\theta\mid d)$ as a proposal. Each of $n$ draws gets a weight

$$
w_i=\frac{p(d\mid\theta_i)\,p(\theta_i)}{q(\theta_i\mid d)},\qquad
n_{\mathrm{eff}}=\frac{\bigl(\sum_i w_i\bigr)^2}{\sum_i w_i^2},\qquad
\epsilon=\frac{n_{\mathrm{eff}}}{n}.
$$

A perfect network gives equal weights and $\epsilon=1$. The authors argue that the network's training loss, a forward Kullback–Leibler divergence, makes the proposal cover the whole posterior, so that broad tails get downweighted instead of missing regions going unseen.

**Table 6.2** Sample efficiencies printed in the 2023 letter.

| Case | Waveform model | Efficiency |
| --- | --- | --- |
| GW150914 | IMRPhenomXPHM | 28.8% |
| GW151012 | IMRPhenomXPHM | 12.5% |
| Median of 42 events | IMRPhenomXPHM | 10.9% |
| Median of 42 events | SEOBNRv4PHM | 4.4% |
| Median of 37 events, smaller mass prior | IMRPhenomPv2 | 36.8% |
| Signal-free data with a blip glitch | not given in this passage | about 0.001% |
| Adversarial examples | not given in this passage | about 0.01% |

Table I compares Dingo-IS with LALInference and Bilby for the two named events, giving divergences in units of $10^{-3}$ nats and evidence estimates. Table II colours each of the 42 events by efficiency and states that results can be trusted for medium and high values. The cut-offs for those colours are in the supplemental material, which this guide did not read.

The low-efficiency events have stated reasons: chirp masses near the lower prior edge, glitches in the data, strong precession that neither waveform model may capture, and one event that may be a noise fluctuation. On a glitch with no signal at all, efficiency falls to about one usable draw in 100,000. The authors conclude that low efficiency can have several causes but often serves as a useful heuristic for data outside the training distribution.

## What efficiency cannot see

The weights use the assumed likelihood. A low efficiency therefore means the network disagrees with the exact posterior of the assumed model on this record. A high efficiency means they agree, and says nothing about whether the model is right.

In this site's toy model, which is not DINGO, a hidden second mode of strength $0.6$ shows how large that gap can be. Efficiency stays at a median of $0.983$ and the residual test flags $6.3\%$ of records, yet both the network and the exact posterior cover the source in only $0.743$ of records at the nominal 90% level. This is a model result on synthetic ringdowns. It suggests a kind of failure to look for and does not measure one in real data.

Cannon and colleagues ran neural simulation-based methods against deliberately misspecified simulators and found that misspecification can damage performance severely; no mitigation they tested prevented failure in all cases. [Cannon, Ward and Schmon, 2022](https://arxiv.org/abs/2209.01845v1) Their paper is a preprint, and the general problem remains open.

**Table 6.3** Status of each claim used in this guide.

| Claim | Kind | Checked here |
| --- | --- | --- |
| DINGO P–P curves follow the diagonal on 1000 simulations | published result | no, read from Figure 2 |
| Mean divergence from LALInference is $0.0009$ nats | published result | no, read from Figure 4 |
| Low efficiency flags glitches and adversarial inputs | published result | no, read from the main text |
| Marginal rank tests miss a wrong correlation | toy-model result | yes, failure log of the tutorial |
| A mismatch absorbed by the fit keeps efficiency high | toy-model result | yes, Lab preset and validation file |
| Neural methods can fail badly under misspecification | preprint result | no, abstract and conclusion only |
| A general test for simulator misspecification | open problem | no |

The analogous checks on the toy model can be rerun with the [method tutorial](https://sandboxphysics.com/stories/simulation-based-inference/recompute-the-ringdown-calibration/).

## A checklist for the next paper you read

- Which data were the calibration simulations drawn from, and are they the training distribution?
- Is coverage tested only per parameter, or also jointly and by region of parameter space?
- Is the reference posterior computed under the same model? If so, the comparison tests the network, not the model.
- What efficiency or other per-event diagnostic is reported, and what cut-off was chosen before looking at the events?
- Which failure modes were tried on purpose, such as glitches, shifted noise or a wrong waveform family?
- What was not read? Supplements often hold the thresholds that the main text relies on.

Readers who want to try these questions on real detector data can start from the public tutorials of the Gravitational Wave Open Science Center, which walk through common analysis tasks step by step, mostly in Python. [Gravitational Wave Open Science Center tutorials](https://gwosc.org/tutorials/)

## Lab task

Open the [neural posterior Lab](https://sandboxphysics.com/labs/neural-posterior-calibration/).

1. Choose the glitch preset. Read the importance-sampling efficiency next to the residual-test p-value. Which one would a Dingo-IS style check have used, and which one flags the record more clearly here?
2. Choose the hidden second mode. Write down the efficiency, then the coverage. Which line of Table 6.3 does this reproduce?

## Questions

1. In Figure 2, why does a combined p-value of $0.46$ say nothing about posteriors on real events?
2. An event has $\epsilon=0.1\%$ with $n=10^{5}$ proposal draws. How many effective samples remain?
3. Figure 4 compares DINGO with LALInference under one waveform model. What comparison would test the waveform model instead?

Answers

1. The 1000 injections use simulated stationary Gaussian noise drawn like the training data. A good score shows the network learned the training distribution; real noise lies outside it, as the letter itself says.
2. $n_{\mathrm{eff}}=\epsilon n=10^{-3}\times10^{5}=100$. That may still allow rough estimates, but the posterior is carried by a few heavily weighted draws and should be checked.
3. Analyse the same events with a different waveform model and compare posteriors or evidences. The 2023 letter does this with IMRPhenomXPHM and SEOBNRv4PHM. Agreement between two models still leaves open whether either describes the signal fully.

## Sources

1. [Dax and colleagues, 2021](https://arxiv.org/abs/2106.12594v2)
2. [Dax and colleagues, 2023](https://arxiv.org/abs/2210.05686v2)
3. [Cranmer, Brehmer and Louppe, 2020](https://arxiv.org/abs/1911.01429v3)
4. [Talts and colleagues, 2018, Section 4.1](https://arxiv.org/abs/1804.06788v2)
5. [Cannon, Ward and Schmon, 2022](https://arxiv.org/abs/2209.01845v1)
6. [Gravitational Wave Open Science Center tutorials](https://gwosc.org/tutorials/)

## Keep reading

Trusting posteriors learned from simulations · Reading path

### From a posterior to calibration and model mismatch

1. [Article **How can you tell whether a neural network's posterior deserves trust?**](https://sandboxphysics.com/stories/simulation-based-inference/trusting-a-neural-posterior/)
2. [Article **What a posterior and a 90% credible region promise**](https://sandboxphysics.com/stories/simulation-based-inference/posterior-and-credible-regions/)
3. [Article **Checking posterior calibration with ranks and coverage**](https://sandboxphysics.com/stories/simulation-based-inference/rank-tests-for-posterior-calibration/)
4. [Article **Finding a wrong simulator with importance weights and residuals**](https://sandboxphysics.com/stories/simulation-based-inference/model-mismatch-and-importance-weights/)
5. [Article **Recompute the ringdown posterior calibration**](https://sandboxphysics.com/stories/simulation-based-inference/recompute-the-ringdown-calibration/)
6. [You are here **Reading the DINGO evidence: P–P plots, divergences and sample efficiency**](https://sandboxphysics.com/stories/simulation-based-inference/reading-dingo-calibration-evidence/)