Skip to main content
Sandbox Physics
Trusting posteriors learned from simulations

Reading the DINGO evidence: P–P plots, divergences and sample efficiency

Sandbox Physics ·

Two letters carry most of the case that the DINGO network gives trustworthy gravitational-wave posteriors. The 2021 letter shows calibration on simulated signals in Figure 2 and agreement with a standard sampler on eight real events in Figure 4. The 2023 letter adds importance weights, whose sample efficiency verifies each event and flags data the network was not trained for. None of these tests asks whether the assumed waveform and noise model describe the data. This guide reads the main text of arXiv version 2 of both letters; their supplemental material was not read, and none of their results is reproduced here.

The guide assumes the course chapters on rank tests and importance weights. Every number below comes from the cited paper unless it is marked as a result of this site's toy model.

What was read and what kind of source it is

Table 6.1 Sources, versions and reading depth.
SourceStatusRead for this guide
Dax and colleagues, arXiv 2106.12594 version 2published letter, 2021main text, Figures 2 and 4
Dax and colleagues, arXiv 2210.05686 version 2published letter, 2023main text, Tables I and II
Talts and colleagues, arXiv 1804.06788 version 2preprint, no journal version listed on arXivSections 1 to 6
Cannon, Ward and Schmon, arXiv 2209.01845 version 1preprint, no journal version listed on arXivabstract, introduction, Section 2.1, conclusion
Cranmer, Brehmer and Louppe, arXiv 1911.01429 version 3published reviewabstract and introduction

The arXiv records list Physical Review Letters 127, 241103 for the first letter and 130, 171403 for the second. Dax and colleagues, 2021 Dax and colleagues, 2023 The review by Cranmer and colleagues places this work in the wider field of simulation-based inference, where a simulator can produce data but its likelihood is hard or impossible to write down. Cranmer, Brehmer and Louppe, 2020 DINGO is an unusual member of that field, because a gravitational-wave likelihood can be evaluated. That fact makes the 2023 check possible.

Figure 2 of the 2021 letter: the P–P plot

The test uses 1000 simulated signals in stationary Gaussian noise, drawn the same way as the training data. For each parameter, the authors take the percentile of the true value within its one-dimensional posterior and plot the cumulative distribution of these percentiles. A calibrated posterior gives uniform percentiles, so each curve should follow the diagonal. Kolmogorov–Smirnov p-values for the separate parameters are combined into one p-value of 0.460.46.

This is the rank test of Talts and colleagues drawn as a cumulative curve instead of a histogram. If uju_j is the percentile of the truth in simulation jj, the plotted curve is

F^(x)=1N∑j=1N1{uj≤x},N=1000,\hat F(x)=\frac{1}{N}\sum_{j=1}^{N}\mathbf 1\{u_j\le x\},\qquad N=1000,

and calibration predicts F^(x)≈x\hat F(x)\approx x. Talts and colleagues, 2018, Section 4.1 Read the figure for three things.

  1. It tests marginals only. A wrong correlation between two parameters leaves every curve on the diagonal. This site's toy model met exactly that failure: a sampler bug passed one-dimensional rank tests with tail probabilities 0.4620.462 and 0.7810.781.
  2. It averages over the prior. A posterior that is too wide in one region and too narrow in another can still produce a diagonal curve.
  3. Its noise is the training noise. The letter says this directly: real data contain noise that is neither strictly stationary nor Gaussian and therefore lie outside the training distribution.

Figure 4 of the 2021 letter: divergences on real events

The main result runs DINGO and the LALInference MCMC sampler on the eight binary black hole events of the first catalogue that fit the network's prior, with component masses from 1010 to 8080 solar masses. DINGO draws 50,000 samples with 30 iterations of its time-alignment step in about 20 seconds. Training took about 10 days on one A100 GPU with 5×1065\times10^{6} waveforms.

Figure 4 reports the Jensen–Shannon divergence between the two samplers' one-dimensional posteriors. The divergence runs from 00 to ln⁡2≈0.69\ln 2\approx0.69 nats. The mean over events and parameters is 0.00090.0009 nats. Two LALInference runs that differ only in their random seed give 0.00070.0007 nats. A maximum of 0.0020.002 nats, taken from an earlier study, counts as indistinguishable; two events meet it for every parameter, and the others exceed it for one to three parameters. For scale, the letter states that changing the noise spectrum estimate or the waveform model moves the divergence by about 0.020.02 nats.

Both samplers use the same waveform model and the same noise assumptions. Figure 4 shows that the network reproduces the posterior of that model on real data. Whether the model fits the data is outside the figure, and the waveform-model shift of 0.020.02 nats is twenty times the network's error.

The 2023 letter: importance weights as a per-event check

The second letter treats the network output q(θ∣d)q(\theta\mid d) as a proposal. Each of nn draws gets a weight

wi=p(d∣θi) p(θi)q(θi∣d),neff=(∑iwi)2∑iwi2,ϵ=neffn.w_i=\frac{p(d\mid\theta_i)\,p(\theta_i)}{q(\theta_i\mid d)},\qquad n_{\mathrm{eff}}=\frac{\bigl(\sum_i w_i\bigr)^2}{\sum_i w_i^2},\qquad \epsilon=\frac{n_{\mathrm{eff}}}{n}.

A perfect network gives equal weights and ϵ=1\epsilon=1. The authors argue that the network's training loss, a forward Kullback–Leibler divergence, makes the proposal cover the whole posterior, so that broad tails get downweighted instead of missing regions going unseen.

Table 6.2 Sample efficiencies printed in the 2023 letter.
CaseWaveform modelEfficiency
GW150914IMRPhenomXPHM28.8%
GW151012IMRPhenomXPHM12.5%
Median of 42 eventsIMRPhenomXPHM10.9%
Median of 42 eventsSEOBNRv4PHM4.4%
Median of 37 events, smaller mass priorIMRPhenomPv236.8%
Signal-free data with a blip glitchnot given in this passageabout 0.001%
Adversarial examplesnot given in this passageabout 0.01%

Table I compares Dingo-IS with LALInference and Bilby for the two named events, giving divergences in units of 10−310^{-3} nats and evidence estimates. Table II colours each of the 42 events by efficiency and states that results can be trusted for medium and high values. The cut-offs for those colours are in the supplemental material, which this guide did not read.

The low-efficiency events have stated reasons: chirp masses near the lower prior edge, glitches in the data, strong precession that neither waveform model may capture, and one event that may be a noise fluctuation. On a glitch with no signal at all, efficiency falls to about one usable draw in 100,000. The authors conclude that low efficiency can have several causes but often serves as a useful heuristic for data outside the training distribution.

What efficiency cannot see

The weights use the assumed likelihood. A low efficiency therefore means the network disagrees with the exact posterior of the assumed model on this record. A high efficiency means they agree, and says nothing about whether the model is right.

In this site's toy model, which is not DINGO, a hidden second mode of strength 0.60.6 shows how large that gap can be. Efficiency stays at a median of 0.9830.983 and the residual test flags 6.3%6.3\% of records, yet both the network and the exact posterior cover the source in only 0.7430.743 of records at the nominal 90% level. This is a model result on synthetic ringdowns. It suggests a kind of failure to look for and does not measure one in real data.

Cannon and colleagues ran neural simulation-based methods against deliberately misspecified simulators and found that misspecification can damage performance severely; no mitigation they tested prevented failure in all cases. Cannon, Ward and Schmon, 2022 Their paper is a preprint, and the general problem remains open.

Table 6.3 Status of each claim used in this guide.
ClaimKindChecked here
DINGO P–P curves follow the diagonal on 1000 simulationspublished resultno, read from Figure 2
Mean divergence from LALInference is 0.00090.0009 natspublished resultno, read from Figure 4
Low efficiency flags glitches and adversarial inputspublished resultno, read from the main text
Marginal rank tests miss a wrong correlationtoy-model resultyes, failure log of the tutorial
A mismatch absorbed by the fit keeps efficiency hightoy-model resultyes, Lab preset and validation file
Neural methods can fail badly under misspecificationpreprint resultno, abstract and conclusion only
A general test for simulator misspecificationopen problemno

The analogous checks on the toy model can be rerun with the method tutorial.

A checklist for the next paper you read

  • Which data were the calibration simulations drawn from, and are they the training distribution?
  • Is coverage tested only per parameter, or also jointly and by region of parameter space?
  • Is the reference posterior computed under the same model? If so, the comparison tests the network, not the model.
  • What efficiency or other per-event diagnostic is reported, and what cut-off was chosen before looking at the events?
  • Which failure modes were tried on purpose, such as glitches, shifted noise or a wrong waveform family?
  • What was not read? Supplements often hold the thresholds that the main text relies on.

Readers who want to try these questions on real detector data can start from the public tutorials of the Gravitational Wave Open Science Center, which walk through common analysis tasks step by step, mostly in Python. Gravitational Wave Open Science Center tutorials

Lab task

Open the neural posterior Lab.

  1. Choose the glitch preset. Read the importance-sampling efficiency next to the residual-test p-value. Which one would a Dingo-IS style check have used, and which one flags the record more clearly here?
  2. Choose the hidden second mode. Write down the efficiency, then the coverage. Which line of Table 6.3 does this reproduce?

Questions

  1. In Figure 2, why does a combined p-value of 0.460.46 say nothing about posteriors on real events?
  2. An event has ϵ=0.1%\epsilon=0.1\% with n=105n=10^{5} proposal draws. How many effective samples remain?
  3. Figure 4 compares DINGO with LALInference under one waveform model. What comparison would test the waveform model instead?
Answers
  1. The 1000 injections use simulated stationary Gaussian noise drawn like the training data. A good score shows the network learned the training distribution; real noise lies outside it, as the letter itself says.
  2. neff=ϵn=10−3×105=100n_{\mathrm{eff}}=\epsilon n=10^{-3}\times10^{5}=100. That may still allow rough estimates, but the posterior is carried by a few heavily weighted draws and should be checked.
  3. Analyse the same events with a different waveform model and compare posteriors or evidences. The 2023 letter does this with IMRPhenomXPHM and SEOBNRv4PHM. Agreement between two models still leaves open whether either describes the signal fully.

Sources

  1. Dax and colleagues, 2021
  2. Dax and colleagues, 2023
  3. Cranmer, Brehmer and Louppe, 2020
  4. Talts and colleagues, 2018, Section 4.1
  5. Cannon, Ward and Schmon, 2022
  6. Gravitational Wave Open Science Center tutorials

Keep reading

What this page links to directly

You are hereReading the DINGO evidence: P–P plots, divergences and sample efficiency

Experiments

Articles

Trusting posteriors learned from simulations · Reading path

From a posterior to calibration and model mismatch

  1. ArticleHow can you tell whether a neural network's posterior deserves trust?
  2. ArticleWhat a posterior and a 90% credible region promise
  3. ArticleChecking posterior calibration with ranks and coverage
  4. ArticleFinding a wrong simulator with importance weights and residuals
  5. ArticleRecompute the ringdown posterior calibration
  6. You are hereReading the DINGO evidence: P–P plots, divergences and sample efficiency
Something broken, confusing, or missing on this page?Tell us
What kind of feedback?

Sent with this page’s address, its Lab ID, the site language and your device type. Please leave out personal details. Feedback is kept for up to 12 months.Privacy policy