Perennial freeze-thaw cycles have caused plenty of defects on asphalt pavements. To analyze the effect on asphalt binder and mastic under freeze-thaw cycles, the multiple stress creep recovery (MSCR) tests, frequency sweep tests, linear amplitude sweep (LAS) tests, and bending beam rheological (BBR) tests were performed first. The rheological property of asphalt binder and mastic was evaluated through the Burgers model, 2S2P1D (two springs, two parabolic elements, and one dashpot) model, Sigmoidal model, and viscoelastic continuum damage (VECD) theory. The Einstein coefficient and K.D.Ziegel−B−G∗ (K−B−G∗) model were adopted to investigate the asphalt binder-filler interaction subsequently. Finally, Fourier transform infrared spectroscopy (FTIR) test was conducted to analyze the variation of chemical composition of asphalt binder before and after freeze-thaw cycles. The results showed that as freeze-thaw cycle increased, the percent recovery of asphalt binder increased, and the non-recoverable creep compliance of asphalt binder decreased and that of asphalt mastic increased first but decreased. The rutting factor and fatigue factor of asphalt binder and mastic increased after freeze-thaw cycles. The asphalt mastic showed poorer fatigue resistance than asphalt binder at a given freeze-thaw cycle. The low-temperature stiffness of asphalt binder and mastic increased as freeze-thaw cycle increased. K−B−G∗ value showed that the asphalt binder-filler interaction damaged seriously after fifteen freeze-thaw cycles, and the Einstein coefficient decreased by 52.73% after twenty freeze-thaw cycles. The IS and IC indexes increased, and the IB index decreased after twenty freeze-thaw cycles. The design and construction of asphalt pavement in seasonal frozen regions may benefit from this study.