Decide whether an excess needs a new signal

An excess becomes evidence for a signal when the signal model describes the data substantially better than the allowed background models. That comparison must let uncertain background and detector parameters change within their auxiliary constraints. Its reliability is then checked by repeating the entire procedure on simulated experiments, including cases where the background shape is slightly wrong.

Write a probability model for the bins

For an unweighted histogram, take the count $n_i$ in each bin to follow a Poisson distribution. Its mean is $\nu_i=\mu s_i+b_i(\theta)$: $s_i$ is a fixed signal template, $\mu$ its dimensionless strength and $\theta$ a set of nuisance parameters. A nuisance parameter describes an uncertain ingredient, such as calibration or background slope, that is needed for the prediction but is not the main quantity we seek.

An auxiliary measurement supplies a probability factor $C(\theta)$. A Gaussian constraint is appropriate only when it represents that measurement sufficiently well. It does not give an arbitrary licence to bend the background.

A likelihood is this probability expression evaluated as a function of the unknown parameters with the data fixed. Taking its logarithm turns products into sums and makes numerical maximization easier. The best-fit point gives the largest likelihood over the permitted parameter region.

Profile the uncertainty at every signal strength

At each proposed signal strength, find the nuisance values that fit best. The profile likelihood ratio compares that conditional best fit with the unrestricted best fit. A double hat marks the nuisance fit made at fixed signal strength; a single hat marks the overall fit.

If a background slope can imitate part of a signal, it will do so in the conditional fit. The profile then becomes broader than it would be with the slope fixed. A tight result obtained by freezing that uncertain slope has omitted an allowed explanation of the data.

For a discovery test with a nonnegative signal, the statistic $q_0$ is set to zero when the unconstrained best-fit signal would be negative. Under suitable large-sample regularity conditions, the local significance is approximately $\sqrt{q_0}$. Very small counts, weak constraints or parameters at unusual boundaries need calibration beyond that approximation.

A counting example you can check by hand

Take a known background mean of 40 and observe 50 events. The best-fit nonnegative signal yield is ten. Dividing the null Poisson likelihood by the best-fit Poisson likelihood and taking minus twice the logarithm gives

The asymptotic local significance is about 1.52. Under the null model, the exact upper-tail probability is obtained by summing Poisson probabilities from 50 upwards. That exact probability and the asymptotic conversion are different calculations, and they need not agree at low counts. Neither is the probability that the signal hypothesis is true.

If a search chooses the most striking mass after inspecting many masses, its global false-alarm probability is larger than the local one at the chosen mass. Nearby masses are correlated, so simply multiplying by the number of bins is generally insufficient. The repeated search procedure, including the scan, supplies the appropriate calibration.

Coverage asks about intervals over repeated experiments

A nominal 95 percent confidence procedure has 95 percent coverage at a fixed true parameter if 95 percent of intervals from repeated experiments contain that parameter. The true parameter stays fixed during this test; the data and fitted interval change.

Run the complete pipeline many times: generate counts and auxiliary measurements, fit each sample, construct the same interval, then count inclusions. If 950 of 1000 independent intervals contain the fixed truth, the estimated coverage is 0.95 with a binomial standard error of about 0.0069. This sampling error must accompany a small coverage study.

Repeat with a deliberately shifted background or detector response. Undercoverage in that ensemble diagnoses sensitivity to the omitted effect. With more data, statistical errors can shrink around a biased result, so the omitted effect needs an explicit model or uncertainty treatment.

Look for a false signal before claiming a real one

Use the collider-likelihood Lab to fit a sample with the nominal background, then use a preset with a changed background shape. Keep the fitted signal template fixed. Compare the fitted strength, profile width and residual shape. Ask which background freedom is allowed by the auxiliary information, and which discrepancy is absent from the fit model. The default synthetic sample at seed 330057 has 1155 events, fitted strength about 0.896 and a nominal 95 percent profile interval from about 0.596 to 1.22. Its local asymptotic significance is about 6.46 under this particular background model; that is a teaching-sample result, not a measured discovery.

For a transfer task, design a held-out background deformation that is smooth across bins. Fit it first with a fixed background, then with an allowed shape nuisance. Record both the spurious signal and any loss of sensitivity. A flexible background helps only for deformations the model actually contains.

Exercises and explained answers

Exercise 1. With background mean 40, observe 35 events. What is the one-sided discovery statistic for a nonnegative excess signal?

Answer. It is zero. A downward fluctuation supplies no evidence for an added positive signal in this test. The data still constrain signal models through an interval or exclusion procedure.

Exercise 2. An interval covers the input in 90 of 100 pseudoexperiments. How precisely is the coverage estimated?

Answer. The estimate is 0.90 with a binomial standard error of about 0.03. To test a claimed coverage of 0.95, use the binomial distribution under that null: the probability of 90 or fewer inclusions is about 0.0282. That supplies evidence of undercoverage in a one-sided five-percent test, although the estimated coverage still has substantial uncertainty.

Exercise 3. Why should the auxiliary measurements fluctuate in a frequentist coverage test?

Answer. They are part of the random data used by the stated procedure. Holding them fixed defines a conditional test, which can be useful, but answers a different coverage question and must be labelled as such.