Two series of silica glasses [(95.9−x)SiO2–4Al2O3–xP2O5–0.1Yb2O3 (in mol%, x=0–10) and (92−y) SiO2–4Al2O3–4P2O5–yYb2O3 (in mol%, y=0–0.2)] were prepared by sol–gel method combined with high-temperature sintering. The relationship between the glass structure and spectroscopic properties is investigated. Significant alterations in density, refractive index, absorption and emission cross sections, fluorescence lifetime, and scalar crystal field NJ are observed at a molar ratio of P5+/Al3+=1 due to the transformation of P5+ structure type from AlPO4-like to PO4 units. This structural change is clearly observed in Raman spectra. In addition, NMR experimental results suggest that Yb3+ is mainly located in AlPO4-like but not SiO4 units at a molar ratio of P5+/Al3+=1, and also Raman spectra reveals the presence of P=O linkage in samples with molar ratio of P5+/Al3+>1. This further demonstrates that it is due to the formation of P=O site but not the simple addition of P2O5 in Yb3+/Al3+/P5+ co-doped silica glass to significantly decrease the absorption and emission cross sections of Yb3+. It reflects in the change of Yb3+ Stark splitting, and is revealed by the decline of Yb3+ asymmetry degree.