---
title: "Counting galaxy pairs against a random catalogue"
description: "Define the correlation function, build the Landy-Szalay estimator from data and random pairs, and estimate its errors by leaving out pieces of the survey."
canonical_url: "https://sandboxphysics.com/stories/baryon-acoustic-oscillations/counting-galaxy-pairs-with-randoms/"
language: "en"
page_type: "story"
alternate_languages:
  zh-CN: "https://sandboxphysics.com/zh/stories/baryon-acoustic-oscillations/counting-galaxy-pairs-with-randoms/"
last_modified: "2026-11-14"
---

[Baryon acoustic oscillations](https://sandboxphysics.com/stories/baryon-acoustic-oscillations/)

# Counting galaxy pairs against a random catalogue

Sandbox Physics · 2026-11-14

The BAO bump lives in the two-point correlation function $\xi(r)$, the fractional excess of galaxy pairs at separation $r$ over a random distribution. A survey cannot count random pairs on paper, because its sky coverage and depth are patchy. So it builds a random catalogue with the same footprint and selection, counts data-data, data-random and random-random pairs, and combines them in the Landy-Szalay estimator $(DD - 2DR + RR)/RR$. Errors come from repeating the measurement with one piece of the survey left out at a time. This chapter builds each step and shows what goes wrong when one is skipped.

It follows the [sound horizon chapter](https://sandboxphysics.com/stories/baryon-acoustic-oscillations/how-far-sound-travelled-before-the-drag-epoch/), which explains where the bump comes from.

## What the correlation function measures

Suppose galaxies have mean number density $\bar n$. The probability of finding one galaxy in a small volume $dV_1$ and another in $dV_2$, a distance $r$ apart, is

$$
dP = \bar n^2\,\bigl[1 + \xi(r)\bigr]\,dV_1\,dV_2 .
$$

If galaxies were scattered at random, $\xi = 0$. Positive $\xi$ means more pairs than random. In a homogeneous and isotropic universe $\xi$ depends only on the distance $r$, not on position or direction. Redshift-space effects break the direction part, and the [next chapter](https://sandboxphysics.com/stories/baryon-acoustic-oscillations/along-and-across-the-line-of-sight/) uses that.

Integrating over a thin shell of radius $r$ and thickness $\Delta r$ around every galaxy, the expected number of distinct pairs in a volume $V$ holding $N$ galaxies is

$$
N_{\mathrm{pairs}}(r) \approx \frac{N(N-1)}{2}\,\frac{4\pi r^2\,\Delta r}{V}\,\bigl[1 + \xi(r)\bigr] .
$$

Worked example. The Lab's default catalogue has $N = 6645$ galaxies in a box $900\,\mathrm{Mpc}$ on a side. For the shell from $145$ to $150\,\mathrm{Mpc}$, the shell volume is $1.37\times10^6\,\mathrm{Mpc^3}$, a fraction $1.88\times10^{-3}$ of the box. With $2.21\times10^7$ distinct pairs, about $41{,}400$ fall in the shell if $\xi = 0$, ignoring the box edges. The Poisson scatter is $\sqrt{41{,}400} \approx 203$, or 0.5 percent. A bump of a few tenths of a percent in one bin would be lost in that noise, which is why surveys need so many galaxies and combine many bins in a fit.

## Why a random catalogue is needed

The formula above assumes a box with uniform coverage. A real survey has edges, holes around bright stars, regions observed more or less completely, and a number density that changes with redshift. All of these change the number of pairs at a given separation without any clustering at all.

The fix is to throw down random points that follow exactly the same footprint, mask and redshift distribution as the galaxies but have no clustering. Their pair counts measure the geometry alone. DESI ships random catalogues with each data release, with a weight column that encodes the same selection effects as the data. In the patch used in the [method guide](https://sandboxphysics.com/stories/baryon-acoustic-oscillations/measure-a-desi-patch-correlation-function/), one DESI random file held $1{,}043{,}189$ points inside the patch, about 66 times the galaxy count.

## Building the estimator

Let $DD(r)$, $DR(r)$ and $RR(r)$ be the pair counts in a bin, each divided by its total number of pairs: $N_D(N_D-1)/2$, $N_D N_R$ and $N_R(N_R-1)/2$ for distinct pairs. The simplest estimate compares data pairs with random pairs:

$$
\hat\xi_{\mathrm{natural}} = \frac{DD}{RR} - 1 .
$$

Landy and Szalay proposed a combination that is less sensitive to the edges of the survey and has a smaller variance:

$$
\hat\xi_{\mathrm{LS}} = \frac{DD - 2\,DR + RR}{RR} .
$$

Eisenstein and colleagues used it for the first BAO detection in 2005, and DESI uses it for its correlation functions.

To see why it works, write the data density near the edge of a survey as $\bar n\,(1+\delta)$, where $\delta$ includes both real clustering and any mismatch between the true mean density and the one implied by the randoms. To first order $DD/RR \approx 1 + \xi + 2\bar\delta$ and $DR/RR \approx 1 + \bar\delta$, where $\bar\delta$ is the average of $\delta$ around the bin. The natural estimator keeps the $2\bar\delta$ term. In the Landy-Szalay combination it cancels: $1 + \xi + 2\bar\delta - 2(1 + \bar\delta) + 1 = \xi$.

Each count must be normalised. If they are not, the estimator picks up a constant offset. With raw counts and $N_R = m\,N_D$ randoms, $DD : DR : RR \approx 1 : m : m^2$ for ordered pairs, and

$$
\hat\xi_{\mathrm{raw}} \approx \frac{1 - 2m + m^2}{m^2} + \frac{\xi}{m^2} = \frac{(m-1)^2}{m^2} + \frac{\xi}{m^2} .
$$

Worked example. For $m = 3$ the offset is $4/9 = 0.444$. At $s = 95\,h^{-1}\,\mathrm{Mpc}$ this adds about $0.444 \times 95^2 \approx 4000$ to $s^2\xi$. The method guide's unnormalised failure case shows an offset of $3945$ there, slightly less because the weighted sums do not stand in an exact ratio of 3.

Pairs can carry weights. DESI multiplies a completeness and systematics weight by an FKP weight, named after Feldman, Kaiser and Peacock, that gives each galaxy the importance that maximises the signal-to-noise at BAO scales. Then each pair counts with $w_i w_j$, and the normalisations become $\bigl(\sum w\bigr)^2 - \sum w^2$ for the auto-counts and $\sum w_D \sum w_R$ for the cross-count.

## Why plots show s squared times xi

The correlation function falls steeply. In the DESI patch it drops from $1.87$ at $5\,h^{-1}\,\mathrm{Mpc}$ to about $0.005$ near $100\,h^{-1}\,\mathrm{Mpc}$. On a linear plot the bump would be invisible. Multiplying by $s^2$ roughly flattens the falling part, so a feature near $100\,h^{-1}\,\mathrm{Mpc}$ stands out. Here $s$ is the separation in redshift-space coordinates, computed with an assumed distance-redshift relation. It equals $r$ only if that assumption is right.

## Errors from the survey itself

Neighbouring bins share galaxies, so their errors are correlated. A standard way to estimate the full covariance from the data alone is the jackknife. Split the survey into $N_{\mathrm{jk}}$ regions of similar size. For each $k$, remove region $k$ and recompute $\hat\xi_k$. Then

$$
C_{ij} = \frac{N_{\mathrm{jk}} - 1}{N_{\mathrm{jk}}}\sum_{k=1}^{N_{\mathrm{jk}}}\bigl(\hat\xi_{k,i} - \bar\xi_i\bigr)\bigl(\hat\xi_{k,j} - \bar\xi_j\bigr) .
$$

The factor $N_{\mathrm{jk}} - 1$ makes up for the fact that the leave-one-out samples overlap almost completely.

A fit needs the inverse covariance. An inverse built from a noisy estimate is biased high. For Gaussian errors, Hartlap and colleagues showed that multiplying by $(N_{\mathrm{jk}} - p - 2)/(N_{\mathrm{jk}} - 1)$, with $p$ bins, removes the bias. With 30 regions and 10 bins this factor is $18/29 = 0.621$. When $p$ approaches $N_{\mathrm{jk}}$, the estimate becomes useless. The Lab avoids that problem by using only the diagonal of its 27-region jackknife, which ignores the correlations between bins. DESI uses analytic and mock-based covariances instead.

In the DESI patch, neighbouring bins inside the fit range have correlation coefficients up to 0.92, and some distant pairs of bins are anticorrelated, down to $-0.58$. With only 30 regions these coefficients are noisy themselves. A fit that treated the bins as independent would misjudge its precision.

## Lab tasks

1. In the default catalogue, find the bin where the transverse wedge peaks. Compare its height with the error bar on a neighbouring bin.
2. Open the Too faint to see by eye preset. Read the naive detection and compare its error bars with the default.
3. Open the No sound wave at all preset. Does any bump survive? What does the fitted stretch do?

## Exercises

1. Repeat the worked example for the shell from $95$ to $100\,\mathrm{Mpc}$. How many random pairs are expected, and what is the fractional Poisson scatter?
2. In one bin the normalised counts are $DD = 1.012$, $DR = 1.004$ and $RR = 1.000$. Compute both estimators and say which one is more trustworthy near an edge.
3. A student uses raw counts with ten times more randoms than galaxies. What constant offset appears in $\hat\xi$?
4. Compute the Hartlap factor for 27 regions and 10 bins, and for 27 regions and 25 bins. What do you conclude?

Answers

1. The shell volume is $4\pi \times 97.5^2 \times 5 = 5.97\times10^5\,\mathrm{Mpc^3}$, a fraction $8.19\times10^{-4}$ of the box. Then $2.21\times10^7 \times 8.19\times10^{-4} \approx 18{,}100$ pairs, with scatter $\sqrt{18{,}100} \approx 134$, or 0.74 percent. Smaller shells hold fewer pairs and are noisier.
2. Natural: $1.012 - 1 = 0.012$. Landy-Szalay: $(1.012 - 2.008 + 1)/1 = 0.004$. The data-random count is $0.4$ percent high, a sign that the mean density near this bin differs from the random catalogue. Landy-Szalay removes that first-order term, so $0.004$ is the better estimate.
3. With $m = 10$, the offset is $(10-1)^2/10^2 = 0.81$.
4. $(27 - 10 - 2)/26 = 0.577$ and $(27 - 25 - 2)/26 = 0$. With 25 bins and 27 regions the inverse covariance carries no usable information, so either use fewer bins, more regions, or a different covariance.

Sources: [Eisenstein et al. 2005, the Landy-Szalay estimator in the first BAO detection](https://arxiv.org/abs/astro-ph/0501171v1), [DESI 2024 III, section 2 on weights and section 4.1 on two-point codes](https://arxiv.org/abs/2404.03000v1).

[Open the Too faint to see by eye preset and compare its error bars and detection with the default catalogue.](https://sandboxphysics.com/labs/baryon-acoustic-ruler/)

## Sources

1. [Eisenstein et al. 2005, the Landy-Szalay estimator in the first BAO detection](https://arxiv.org/abs/astro-ph/0501171v1)
2. [DESI 2024 III, section 2 on weights and section 4.1 on two-point codes](https://arxiv.org/abs/2404.03000v1)

## Keep reading

Baryon acoustic oscillations · Reading path

### From a sound wave to galaxy pair counts and DESI distances

1. [Article **How can pairs of galaxies measure the universe?**](https://sandboxphysics.com/stories/baryon-acoustic-oscillations/how-galaxy-pairs-measure-the-universe/)
2. [Article **How far sound travelled before the drag epoch**](https://sandboxphysics.com/stories/baryon-acoustic-oscillations/how-far-sound-travelled-before-the-drag-epoch/)
3. [You are here **Counting galaxy pairs against a random catalogue**](https://sandboxphysics.com/stories/baryon-acoustic-oscillations/counting-galaxy-pairs-with-randoms/)
4. [Article **Reading the ruler along and across the line of sight**](https://sandboxphysics.com/stories/baryon-acoustic-oscillations/along-and-across-the-line-of-sight/)
5. [Article **Measure a correlation function from a DESI data patch**](https://sandboxphysics.com/stories/baryon-acoustic-oscillations/measure-a-desi-patch-correlation-function/)
6. [Article **How to read the DESI BAO results**](https://sandboxphysics.com/stories/baryon-acoustic-oscillations/reading-desi-bao-results/)
