Coherent elastic neutrino-nucleus scattering (CEνNS) provides a unique probe for neutrino properties beyond the Standard Model physics. The reactor neutrino liquid xenon coherent scattering experiment (RELICS), a proposed reactor neutrino program using liquid xenon time projection chamber technology, aims to investigate the CEνNS process of antineutrinos off xenon atomic nuclei. In this work, the design of the experiment is studied and optimized based on Monte Carlo simulations. To achieve a sufficiently low-energy threshold for CEνNS detection, an ionization-only analysis channel is adopted for RELICS. A high emission rate of delayed electrons after a big ionization signal is the major background, leading to an analysis threshold of 120 photoelectrons in the CEνNS search. The second largest background, nuclear recoils induced by cosmic-ray neutrons, is suppressed via a passive water shield. The physics potential of RELICS is explored with a 30 kg·yr exposure at a baseline of 25 m from a reactor core with a 3 GW thermal power. In an energy range of 120–300 photoelectrons, corresponding to an average nuclear recoil from 0.63 to 1.36 keV considering the liquid xenon response and detector-related effect, we expect 4639.7 CEνNS and 1687.8 background events. The sensitivity of RELICS to the weak mixing angle is investigated at a low momentum transfer. Our study shows that RELICS can further improve the constraints on the nonstandard neutrino interaction compared to the current best results. Published by the American Physical Society 2024