The Lingshan granite complex, located in the northeast part of the Qin-Hang Belt, Jiangxi Province, southeast China, has produced the largest Ta-Nb deposits in China, i.e., the Huangshan Nb-Ta and Songshugang Ta-Nb-(W-Sn) deposits. This complex is spatially a ring-like multiphase batholith, and comprises a series of granitic rocks from the central to marginal units. Mafic microgranular enclaves (MMEs) occur only in the central unit. Geochronologic results show that the magmatic activities at Lingshan district took place at Early Cretaceous, with similar emplacement ages (133–134 Ma) between the central and marginal units. The Lingshan granites are metaluminous to weakly peraluminous (most A/CNK < 1.1), and display typical A-type affinity. However, the whole-rock REE contents and Zr/Hf and (La/Yb)N ratios exhibit an abnormal increase from the central to marginal units. The Li, Rb and Cs elements of biotite as well as the Zr/Hf ratios of zircon do not exhibit a continuous enrichment trend from the central towards marginal granites. All these features suggest that the generation of Lingshan granites is more complex than the fractional crystallization from a single parental magma. Alternatively, a crystal mush model could be applied to better interpret the geochemical and isotopic contrasts between different granite phases. The marginal granites were likely formed by later crystallization events after the rejuvenation and extraction from the crystal mush reservoirs, while the central granites were formed from the residual crystal-rich mush with a relatively low terminal porosity. Moreover, the Nb and Ta contents of biotite exhibit a gradual enrichment with increasing fluorine contents in the marginal granites. The enrichment of lithium and fluorine in the granitic melt can significantly promote the proportion of non-bridging oxygens (NBOs) for coordinating Nb and Ta in the melt, further enhancing the solubility of columbite and tantalite, and eventually causing the Nb and Ta elements either incorporate into the mineral lattice of micas in the early marginal granites or form columbite-group minerals in the pegmatite/altered granite. In summary, we propose that the Ta-Nb enrichment event in the Lingshan district is spatially and genetically related to the extensive fractionation evolution from central to marginal units.