The Maodeng deposit, located in Xilinhot, Inner Mongolia, is a typical Mo-Bi-Sn-Cu deposit in the southern segment of the Great Xing’an Range. It is genetically related to the Alubaogeshan granitic complex, which is composed of porphyritic monzogranite (G1), granite porphyry (G2), and alkali feldspar granite (G3). However, the comparison of physical–chemical condition and magmatic evolution of different lithofacies has not been conducted. We present in-situ major-trace elements and Sm-Nd isotopes of apatite, in-situ major-trace elements of plagioclase, as well as in-situ major elements of magnetite and ilmenite from the G1 and G2, with the goal of discussing magma genesis and its constraints on mineralization. The apatites from the two granites are characterized by relative enrichment in F and light rare earth elements (LREEs) but depletion in Cl, S, and heavy rare earth elements (HREEs), as well as obviously negative Eu, Sr, Nb, Ba, Zr, and Hf and positive Y anomalies, indicating that the two granites have similar sources and experienced significant crystallization differentiation. When compared to G1, apatites from G2 exhibit more significant Eu, Sr, and Y anomalies, as well as lower (La/Sm)N and (La/Yb)N ratios, likely due to intensive crystallization of plagioclase and exsolution of Cl-bearing magmatic fluids. The presence of core-rim zoning patterns with BSE-bright and BSE-grey domains in some G2 apatite crystals indicates extensive hydrothermal alteration. The plagioclases within G1 belong to andesine–oligoclase series corresponding to magmatic water contents of 1.4 to 1.9 wt%, and those in G2 belong to oligoclase–albite series with calculated magmatic water contents of 2.1 to 2.2 wt%. The plagioclase crystals from G1 are characterized by prominently positive Eu and moderately positive Sr and Ba anomalies, with crystals from G2 showing strongly negative Eu, Ba and Sr anomalies, indicating more significant fractionating plagioclase and apatite in G2 than G1. The magnetite compositions are distinguished by high FeO, TiO2, and MnO contents. Ilmenite usually cuts and dissolves magnetite, and contains a lot of FeO, TiO2, MnO, and Nb2O3. The magnetite and ilmenite from the two granites have a coherent negative correlation between Ti and Fe3+ but a consistent positive correlation between Ti and Fe2+, implying that they crystallized from a homologous silicate magma. During the crystallization of apatites, magmatic oxygen fugacities of the two granites were slightly above the ΔNNO buffer line. However, these were near the FMQ buffer line when the magnetite and ilmenite were in equilibrium at temperatures ranging from 561 to 666 °C, indicating a decreasing trend in oxygen fugacity during magma evolution. The apatites of both G1 and G2 have high εNd(t) values (+0.59 to + 1.94 and − 2.13 to + 0.42, respectively), and young two-stage Nd model ages (tDM2 = 775–884 Ma and 897–1106 Ma, respectively), suggesting the predominant origin of these rocks by the partial melting of a juvenile mafic lower crust and “old” pre-existing crustal components, having a substantial proportion of continental crust for G2. The decline in oxygen fugacity, enrichment of F element, highly crystallization differentiation, intense melt/fluid interaction, and the strong hydrothermal alteration for the ore-related granite could contribute to the formation of the Maodeng deposit.