The Bayan Obo giant REE–Nb–Fe deposit in Inner Mongolia is the most famous and largest REE deposit in the world. Its genesis has been long disputed. Carbonate rocks of this deposit are divided into four categories based on their texture, mineral assemblage and geological occurrence: (1) Sedimentary limestone and dolostone, (2) Deformed mineralized coarse-grained dolomite marble, (3) Fine-grained dolomite marble, and (4) Carbonatite dykes. In this study, major and trace elements of different dolomite marbles are analyzed by XRF and ICP-AES and mineral compositions are analyzed by EPMA. Results show that major, trace and rare earth elements of coarse-grained dolomite marble samples are comparable to those of sedimentary carbonate rocks. In comparison, geochemical features of the fine-grained dolomite marbles are comparable with those of carbonatites. The SrO and MnO concentrations analyzed by EPMA in carbonate minerals from fine-grained dolomite marble are similar to carbonatite, and where least altered, carbonates from coarse-grained dolomite marble retain SrO signatures of sedimentary carbonates. We find that coarse-grained dolomite marbles are the result of weak mineralization of sedimentary carbonate rocks, and fine-grained dolomite marbles are the product of intensive mineralization by REE-rich fluids derived from carbonatitic magma. Moreover, SrO content in carbonate minerals is a more sensitive indicator than MnO to distinguish carbonatite from sedimentary carbonate rock in the Bayan Obo REE–Nb–Fe deposit.