Skip to main content
Sandbox Physics
Baryon acoustic oscillations

Counting galaxy pairs against a random catalogue

Sandbox Physics ·

The BAO bump lives in the two-point correlation function ξ(r)\xi(r), the fractional excess of galaxy pairs at separation rr over a random distribution. A survey cannot count random pairs on paper, because its sky coverage and depth are patchy. So it builds a random catalogue with the same footprint and selection, counts data-data, data-random and random-random pairs, and combines them in the Landy-Szalay estimator (DD−2DR+RR)/RR(DD - 2DR + RR)/RR. Errors come from repeating the measurement with one piece of the survey left out at a time. This chapter builds each step and shows what goes wrong when one is skipped.

It follows the sound horizon chapter, which explains where the bump comes from.

What the correlation function measures

Suppose galaxies have mean number density nˉ\bar n. The probability of finding one galaxy in a small volume dV1dV_1 and another in dV2dV_2, a distance rr apart, is

dP=nˉ2 [1+ξ(r)] dV1 dV2.dP = \bar n^2\,\bigl[1 + \xi(r)\bigr]\,dV_1\,dV_2 .

If galaxies were scattered at random, ξ=0\xi = 0. Positive ξ\xi means more pairs than random. In a homogeneous and isotropic universe ξ\xi depends only on the distance rr, not on position or direction. Redshift-space effects break the direction part, and the next chapter uses that.

Integrating over a thin shell of radius rr and thickness Δr\Delta r around every galaxy, the expected number of distinct pairs in a volume VV holding NN galaxies is

Npairs(r)≈N(N−1)2 4πr2 ΔrV [1+ξ(r)].N_{\mathrm{pairs}}(r) \approx \frac{N(N-1)}{2}\,\frac{4\pi r^2\,\Delta r}{V}\,\bigl[1 + \xi(r)\bigr] .

Worked example. The Lab's default catalogue has N=6645N = 6645 galaxies in a box 900 Mpc900\,\mathrm{Mpc} on a side. For the shell from 145145 to 150 Mpc150\,\mathrm{Mpc}, the shell volume is 1.37×106 Mpc31.37\times10^6\,\mathrm{Mpc^3}, a fraction 1.88×10−31.88\times10^{-3} of the box. With 2.21×1072.21\times10^7 distinct pairs, about 41,40041{,}400 fall in the shell if ξ=0\xi = 0, ignoring the box edges. The Poisson scatter is 41,400≈203\sqrt{41{,}400} \approx 203, or 0.5 percent. A bump of a few tenths of a percent in one bin would be lost in that noise, which is why surveys need so many galaxies and combine many bins in a fit.

Why a random catalogue is needed

The formula above assumes a box with uniform coverage. A real survey has edges, holes around bright stars, regions observed more or less completely, and a number density that changes with redshift. All of these change the number of pairs at a given separation without any clustering at all.

The fix is to throw down random points that follow exactly the same footprint, mask and redshift distribution as the galaxies but have no clustering. Their pair counts measure the geometry alone. DESI ships random catalogues with each data release, with a weight column that encodes the same selection effects as the data. In the patch used in the method guide, one DESI random file held 1,043,1891{,}043{,}189 points inside the patch, about 66 times the galaxy count.

Building the estimator

Let DD(r)DD(r), DR(r)DR(r) and RR(r)RR(r) be the pair counts in a bin, each divided by its total number of pairs: ND(ND−1)/2N_D(N_D-1)/2, NDNRN_D N_R and NR(NR−1)/2N_R(N_R-1)/2 for distinct pairs. The simplest estimate compares data pairs with random pairs:

ξ^natural=DDRR−1.\hat\xi_{\mathrm{natural}} = \frac{DD}{RR} - 1 .

Landy and Szalay proposed a combination that is less sensitive to the edges of the survey and has a smaller variance:

ξ^LS=DD−2 DR+RRRR.\hat\xi_{\mathrm{LS}} = \frac{DD - 2\,DR + RR}{RR} .

Eisenstein and colleagues used it for the first BAO detection in 2005, and DESI uses it for its correlation functions.

To see why it works, write the data density near the edge of a survey as nˉ (1+δ)\bar n\,(1+\delta), where δ\delta includes both real clustering and any mismatch between the true mean density and the one implied by the randoms. To first order DD/RR≈1+ξ+2δˉDD/RR \approx 1 + \xi + 2\bar\delta and DR/RR≈1+δˉDR/RR \approx 1 + \bar\delta, where δˉ\bar\delta is the average of δ\delta around the bin. The natural estimator keeps the 2δˉ2\bar\delta term. In the Landy-Szalay combination it cancels: 1+ξ+2δˉ−2(1+δˉ)+1=ξ1 + \xi + 2\bar\delta - 2(1 + \bar\delta) + 1 = \xi.

Each count must be normalised. If they are not, the estimator picks up a constant offset. With raw counts and NR=m NDN_R = m\,N_D randoms, DD:DR:RR≈1:m:m2DD : DR : RR \approx 1 : m : m^2 for ordered pairs, and

ξ^raw≈1−2m+m2m2+ξm2=(m−1)2m2+ξm2.\hat\xi_{\mathrm{raw}} \approx \frac{1 - 2m + m^2}{m^2} + \frac{\xi}{m^2} = \frac{(m-1)^2}{m^2} + \frac{\xi}{m^2} .

Worked example. For m=3m = 3 the offset is 4/9=0.4444/9 = 0.444. At s=95 h−1 Mpcs = 95\,h^{-1}\,\mathrm{Mpc} this adds about 0.444×952≈40000.444 \times 95^2 \approx 4000 to s2ξs^2\xi. The method guide's unnormalised failure case shows an offset of 39453945 there, slightly less because the weighted sums do not stand in an exact ratio of 3.

Pairs can carry weights. DESI multiplies a completeness and systematics weight by an FKP weight, named after Feldman, Kaiser and Peacock, that gives each galaxy the importance that maximises the signal-to-noise at BAO scales. Then each pair counts with wiwjw_i w_j, and the normalisations become (∑w)2−∑w2\bigl(\sum w\bigr)^2 - \sum w^2 for the auto-counts and ∑wD∑wR\sum w_D \sum w_R for the cross-count.

Why plots show s squared times xi

The correlation function falls steeply. In the DESI patch it drops from 1.871.87 at 5 h−1 Mpc5\,h^{-1}\,\mathrm{Mpc} to about 0.0050.005 near 100 h−1 Mpc100\,h^{-1}\,\mathrm{Mpc}. On a linear plot the bump would be invisible. Multiplying by s2s^2 roughly flattens the falling part, so a feature near 100 h−1 Mpc100\,h^{-1}\,\mathrm{Mpc} stands out. Here ss is the separation in redshift-space coordinates, computed with an assumed distance-redshift relation. It equals rr only if that assumption is right.

Errors from the survey itself

Neighbouring bins share galaxies, so their errors are correlated. A standard way to estimate the full covariance from the data alone is the jackknife. Split the survey into NjkN_{\mathrm{jk}} regions of similar size. For each kk, remove region kk and recompute ξ^k\hat\xi_k. Then

Cij=Njk−1Njk∑k=1Njk(ξ^k,i−ξˉi)(ξ^k,j−ξˉj).C_{ij} = \frac{N_{\mathrm{jk}} - 1}{N_{\mathrm{jk}}}\sum_{k=1}^{N_{\mathrm{jk}}}\bigl(\hat\xi_{k,i} - \bar\xi_i\bigr)\bigl(\hat\xi_{k,j} - \bar\xi_j\bigr) .

The factor Njk−1N_{\mathrm{jk}} - 1 makes up for the fact that the leave-one-out samples overlap almost completely.

A fit needs the inverse covariance. An inverse built from a noisy estimate is biased high. For Gaussian errors, Hartlap and colleagues showed that multiplying by (Njk−p−2)/(Njk−1)(N_{\mathrm{jk}} - p - 2)/(N_{\mathrm{jk}} - 1), with pp bins, removes the bias. With 30 regions and 10 bins this factor is 18/29=0.62118/29 = 0.621. When pp approaches NjkN_{\mathrm{jk}}, the estimate becomes useless. The Lab avoids that problem by using only the diagonal of its 27-region jackknife, which ignores the correlations between bins. DESI uses analytic and mock-based covariances instead.

In the DESI patch, neighbouring bins inside the fit range have correlation coefficients up to 0.92, and some distant pairs of bins are anticorrelated, down to −0.58-0.58. With only 30 regions these coefficients are noisy themselves. A fit that treated the bins as independent would misjudge its precision.

Lab tasks

  1. In the default catalogue, find the bin where the transverse wedge peaks. Compare its height with the error bar on a neighbouring bin.
  2. Open the Too faint to see by eye preset. Read the naive detection and compare its error bars with the default.
  3. Open the No sound wave at all preset. Does any bump survive? What does the fitted stretch do?

Exercises

  1. Repeat the worked example for the shell from 9595 to 100 Mpc100\,\mathrm{Mpc}. How many random pairs are expected, and what is the fractional Poisson scatter?
  2. In one bin the normalised counts are DD=1.012DD = 1.012, DR=1.004DR = 1.004 and RR=1.000RR = 1.000. Compute both estimators and say which one is more trustworthy near an edge.
  3. A student uses raw counts with ten times more randoms than galaxies. What constant offset appears in ξ^\hat\xi?
  4. Compute the Hartlap factor for 27 regions and 10 bins, and for 27 regions and 25 bins. What do you conclude?
Answers
  1. The shell volume is 4π×97.52×5=5.97×105 Mpc34\pi \times 97.5^2 \times 5 = 5.97\times10^5\,\mathrm{Mpc^3}, a fraction 8.19×10−48.19\times10^{-4} of the box. Then 2.21×107×8.19×10−4≈18,1002.21\times10^7 \times 8.19\times10^{-4} \approx 18{,}100 pairs, with scatter 18,100≈134\sqrt{18{,}100} \approx 134, or 0.74 percent. Smaller shells hold fewer pairs and are noisier.
  2. Natural: 1.012−1=0.0121.012 - 1 = 0.012. Landy-Szalay: (1.012−2.008+1)/1=0.004(1.012 - 2.008 + 1)/1 = 0.004. The data-random count is 0.40.4 percent high, a sign that the mean density near this bin differs from the random catalogue. Landy-Szalay removes that first-order term, so 0.0040.004 is the better estimate.
  3. With m=10m = 10, the offset is (10−1)2/102=0.81(10-1)^2/10^2 = 0.81.
  4. (27−10−2)/26=0.577(27 - 10 - 2)/26 = 0.577 and (27−25−2)/26=0(27 - 25 - 2)/26 = 0. With 25 bins and 27 regions the inverse covariance carries no usable information, so either use fewer bins, more regions, or a different covariance.

Sources: Eisenstein et al. 2005, the Landy-Szalay estimator in the first BAO detection, DESI 2024 III, section 2 on weights and section 4.1 on two-point codes.

Open the Too faint to see by eye preset and compare its error bars and detection with the default catalogue.

Sources

  1. Eisenstein et al. 2005, the Landy-Szalay estimator in the first BAO detection
  2. DESI 2024 III, section 2 on weights and section 4.1 on two-point codes

Keep reading

What this page links to directly

See this series on the content map →

Baryon acoustic oscillations · Reading path

From a sound wave to galaxy pair counts and DESI distances

  1. ArticleHow can pairs of galaxies measure the universe?
  2. ArticleHow far sound travelled before the drag epoch
  3. You are hereCounting galaxy pairs against a random catalogue
  4. ArticleReading the ruler along and across the line of sight
  5. ArticleMeasure a correlation function from a DESI data patch
  6. ArticleHow to read the DESI BAO results
Something broken, confusing, or missing on this page?Tell us
What kind of feedback?

Sent with this page’s address, its Lab ID, the site language and your device type. Please leave out personal details. Feedback is kept for up to 12 months.Privacy policy