Increased maritime traffic is leading to more underwater noise. Despite the use of a standardized protocol (ANSI/ASA S12/64-2009), measuring ship noise is complex, as it is difficult to follow the protocol in real-life conditions. The Marine Acoustic Research Station (MARS) project (www.projet-mars.ca/en) designed a recording station and a measurement protocol that follows closely the international standard. The station operated during summer/fall in 2021, 2022 and 2023 in the Laurentian Channel in the St. Lawrence Estuary (eastern Canada). Four dedicated missions were organized, during which the R/V Coriolis II was measured repeatedly, generating a database of 239 measurements. In this study, we assess the measurement repeatability of the MARS station, comparing simultaneous measurements from identical systems, as well as repeated measurements at varying speeds and distances. We quantify the measurement uncertainties associated to the hydrophone configuration and the measurement protocol. Our results illustrate how array redundancy allows to identify and mitigate hardware-related issues and to reduce the duration of the measurement, a costly parameter. We quantify the repeatability of single measurements within the MARS station, we discuss the implications regarding our ability to validate noise reduction actions from ~3 dB and we propose a measurement protocol to further reduce this limit.