The Sino-Mongolian border region is a hotspot for international geoscience research and exploration. Mercury (Hg) can be used as a reference element for geochemical exploration and ore prospecting. This is because the change in Hg concentration and its spatial distribution can indirectly reflect the regional mineral distribution and regional environmental mercury pollution. Hg pollution caused by anthropogenic activities has received extensive attention because of its hazardous nature and Hg is generally regarded as a global pollutant. Although geochemical explorations have been conducted in the Sino-Mongolian border region, there have been relatively few studies with detailed and consolidated analyses of single elements such as Hg. To address this research gap we have examined the concentration and spatial distribution of Hg in this region. Influencing factors including the geological background, geomorphological landscapes, mineral distribution, development and use, and human activities, were also explored. The Hg concentration in the catchment sediment samples had a median value of 16.5 ng/g considered the geochemical background value, mean value of 18.9 ng/g, and a range of 0.5–712.5 ng/g. Low Hg concentrations are distributed in the Gobi Desert areas of Xinjiang, Inner Mongolia in China, and in the southern Gobi region of Mongolia because of the poor adsorption capacity for Hg by quartz-rich windblown sand. Areas with high Hg concentrations are distributed in the eastern and northwestern parts of the study region. Fifteen anomalous Hg enriched areas were delineated in the study region, among which 24 high Hg concentration centers (>25.4 ng/g, > Q85%) reached the scale of geochemical provinces. The high concentration of Hg in these centers is attributed to the widespread distribution of minerals containing elevated concentrations of Hg, minerals that produce substantial quantities of Hg as a by-product, and anthropogenic contributions such as mining and smelting activities. The enrichment of Hg in certain areas is an effective indicator of the occurrence of ore deposits. The Hg geochemical data obtained from the Sino-Mongolian border region is an important reference for further studies on geology, mineral exploration, and environmental protection.