The Devonian Sanhekou Group phyllite in the Yangshan gold belt includes multiple stages of pyrite development, and Au is mainly distributed within the pyrite. Bodies within the Gaoloushan ore section of the gold belt are located in the broken and altered carbonaceous phyllite and granitic rock dykes, where quartz vein-type gold ores are more common as compared to other ore sections (Anba, Getiaowan). Studying the trace element distribution characteristics of gold-bearing and non-gold-bearing pyrite can constrain the evolution of ore-forming fluids and the types of ore deposit genesis. Based on detailed fieldwork and microscopic identification, 48 pyrite samples from the 69th exploration line of the Gaoloushan ore section were analyzed in situ by LA-ICP-MS. The following conclusions were drawn: (1) The gold-bearing pyrite formed during the ore-forming stage while the non-gold-bearing pyrite formed during the sedimentary and metamorphic period. The morphology of the pyrites differs, with gold-bearing pyrite mainly characterized by medium-fine grains, impregnation, fine veins, and reticulate structure, with crystal morphology mainly pentagonal dodecahedra or pentagonal octahedra, whereas non-gold-bearing pyrite is mainly characterized by a medium or medium-coarse grains, with stellate, strawberry, or cluster structures, and crystal morphology mainly in the form of cubes or pentagonal octahedra; (2) The As and Te content of the sedimentary pyrite is low, locally below the detection limit, and the Co and Ni contents and ratios vary greatly, with a distribution of 117–945 ppm and ratio distribution of 0.11–1.92. Pyrite from the ore-forming stage is primarily characterized by high As, Pb, Co, Ni, Cu, Sb, Zn, and Se concentrations, with Bi, Au, and trace amounts of Tl, Te, and Sn observed in later formations; (3) Au and As are positively correlated. The average Au content of 60.53 ppm in pyrite from granite dykes is higher than the average Au 15.97 ppm in carbonaceous phyllite. Pyrite particles with Au contents below the detection limit are sourced in carbonaceous phyllite. In addition, the higher As enrichment is observed in pyrite from granite dykes, averaging 35 770.60 ppm with a maximum value of 72 930.19 ppm. In comparison, the average As content in carbonaceous phyllite is 8974.66 ppm, with a maximum of 69 800.50 ppm, and the Au-As diagram indicates that gold in the Gaoloushan ore section is mainly in the form of solid solution gold within the pyrite lattice; (4) Statistical analysis of the Co/Ni ratio variation in the different lithologies and pyrite stages reveals an initial increase followed by a decrease. The median Co/Ni ratio variations during the mineralization stages of Py1s (0.51–0.7), Py1m (1.35–2.25), Py2 (1.43–1.87), Py1-2 m (1.53), Py2a (1.59), Py2b (1.4), Py2s (1.2), Py2m (1.63), and Py3 (0.19–1.40) suggest that magmatic-hydrothermal fluids dominated during the early stage of mineralization in the gold belt and metamorphic fluids during the main mineralization stage, with deep-seated metamorphic fluids playing a significant role in the late mineralization stage. The varying proportions of ore-forming fluids are likely to have been crucial in controlling gold mineralization in the Yangshan gold belt.