Mock data study for next-generation ground-based detectors: The performance loss of matched filtering due to correlated confusion noise

Abstract

The next-generation (3G/XG) ground-based gravitational-wave (GW) detectors such as Einstein Telescope (ET) and Cosmic Explorer (CE) will begin observing in the next decade. Due to the extremely high sensitivity of these detectors, the majority of stellar-mass compact-binary mergers in the entire Universe will be observed. It is also expected that 3G detectors will have significant sensitivity down to 2-7 Hz; the observed duration of binary neutron star signals could increase to several hours or days. The abundance and duration of signals will cause them to overlap in time, which may form a confusion noise that could affect the detection of individual GW sources when using naive matched filtering; matched filtering is only optimal for stationary Gaussian noise. We create mock data for CE and ET using the latest population models informed by the GWTC-3 catalog and investigate the performance loss of matched filtering due to overlapping signals. We find the performance loss mainly comes from a deviation in the noise's measured amplitude spectral density. The redshift reach of CE (ET) can be reduced by 15%-38% (8%-21%) depending on the merger rate estimate. The direct contribution of confusion noise to the total signal-to-noise ratio (SNR) is generally negligible compared to the contribution from instrumental noise. We also find that correlated confusion noise has a negligible effect on the quadrature summation rule of network SNR for ET, but might reduce the network SNR of high detector-frame mass signals for detector networks including CE if no mitigation is applied. For ET, the null stream can mitigate the astrophysical foreground. For CE, we demonstrate that a computationally efficient, straightforward single-detector signal subtraction method suppresses the total noise to almost the instrument noise level; this will allow for near-optimal searches.