Mineral fillers are an essential component of asphalt concrete. Significant amounts of mineral fillers are used in constructing asphalt pavement. However, mineral fillers are an important non-renewable natural resource which needs to be preserved. In addition, the production of mineral fillers also consumes much energy and affects environment negatively. This study set out to explore the feasibility of upcycling waste phosphogypsum as an alternative filler for asphalt concrete. The phosphogypsum was first pretreated at 60, 160, and 800 °C, and the crystalline phase of phosphogypsum was evaluated by X-ray diffraction (XRD). Phosphogypsum with three crystalline phases and limestone powder were then mixed with base asphalt binders to prepare asphalt mastic at the filler-asphalt ratio of 1:1 at 150 °C. The chemical structural changes and thermal stability features of phosphogypsum asphalt mastic were analyzed using Fourier transform infrared spectroscopy (FTIR) and thermogravimetry (TG). The viscoelasticity, high-temperature performance, and fatigue characteristics of different asphalt mastics were measured by frequency sweep, multiple stress creep and recovery (MSCR), and linear amplitude sweep (LAS), respectively. The results indicate that phosphogypsum had different crystalline phases after pretreatment at different temperatures. The addition of phosphogypsum as fillers in asphalt is mainly a physical hardening effect. The 600 °C residual mass ratio of asphalt mastic prepared from phosphogypsum pretreated at 60, 160, and 800 °C was 53.9%, 56.2%, and 58.6%. As the temperature increased, the different crystallized water states in phosphogypsum were decomposed. The replacement of mineral powder with phosphogypsum can improve the high-temperature properties of asphalt mastic and the percent recovery (R), while the nonrecoverable creep compliance (Jnr) was reduced. At the extremely high-temperature of 70 °C, the improvement of the high-temperature performance of phosphogypsum asphalt mastic is more evident than that of 46 °C. Furthermore, the addition of phosphogypsum also reduced the fatigue life of asphalt mastic. This study is expected to paving the road for the upcycling of waste phosphogypsum as an alternative filler in asphalt pavement.