Oxygen K-edge X-ray absorption spectroscopy is used routinely to study a range of solid materials. However, liquid samples are studied less frequently at the oxygen K-edge due to the combined challenges of high-vacuum conditions and oxygen contamination of window materials. A modular sample holder design with a twist-seal sample containment system that provides a simple method to encapsulate liquid samples under high-vacuum conditions is presented. This work shows that pure silicon nitride windows have lower oxygen contamination than both diamond- and silicon-rich nitride windows, that the levels of oxygen contamination are related to the age of the windows, and provides a protocol for minimizing the background oxygen contamination. Acid-washed 100 nm-thick silicon nitride windows were found to give good quality oxygen K-edge data on dilute liquid samples.