Sedimentary phosphate rocks are characteristically rich in organic matter, and contain sulfides and a high concentration of trace elements, including cadmium (Cd), which is harmful to the human body. The mining of phosphate rock and phosphogypsum at Qingping has expanded the release of Cd into groundwater and farmland soil. To prevent and control Cd pollution it is critical to reveal the carrier mineral phase(s) and migration mechanism of Cd and other elements in phosphate rock and phosphogypsum. The elemental and mineral composition of bulk samples were analysed by XRF, ICP-MS, and XRD, respectively. The results showed that from phosphate rock to phosphogypsum, the main constituent elements changed from CaO (49.43%) and P2O5 (36.63%) to CaO (33.65%) and SO3 (>34%), and the main mineral changed from fluorapatite to gypsum. Among all the elements, the element transfer factor (ETF) of P2O5, F, Co, U, Cd, and other elements was low; the ETF (Cd) was only 10.85%, and only a small amount of Cd entered the acidic phosphogypsum during the production process. Raman spectroscopy analysis revealed two types of apatite: the brown–black apatite with organic matter (type 1) and the yellow–light brown apatite without organic matter (type 2). LA-ICP-MS analysis showed that the Cd element content in type 2 was lower, while the organic matter and Cd element content in type 1 were higher, suggesting that Cd may be controlled by organic matter, and the relationship with apatite is not apparent. Electron probe analysis and XRD semiquantitative results show that the content of Cd in pyrite is higher (511 ppm), which is significantly higher than that of bulk rock. In addition, pyrite is rich in Co, As, Ni, Zn, and other elements. The content of Cd in phosphate ore shows a good correlation with that of pyrite. Cd in Qingping phosphate rock is mainly controlled by organic matter and pyrite, and only a small amount of Cd is transferred to phosphogypsum. Reducing the environment leads to the enrichment of Cd in phosphate rock.