The current study emphasizes the impact of styrene-butadiene-styrene (SBS) molecular structures on the extent of performance deterioration in asphalt mixes after short-term and long-term laboratory aging. First, modified binders were prepared using linear (L-SBS), branched (B-SBS), high vinyl (HV-SBS), and di-block (DB-SB) SBS copolymers, and consequent asphalt mixes were fabricated using these modified binders. The influence of aging on the asphalt mixture performance has been evaluated through Cantabro mass loss, fracture properties, fatigue, and rut resistance behavior. Results showed that irrespective of the type of SBS polymer in the binder, the short-term aging (STA) of asphalt mixes had an insignificant effect on their performance properties. However, a substantial drop in all mix performances was observed after long-term aging (LTA) with L-SBS and B-SBS modified binders due to the degradation of the polymer chains. Mixes prepared with L-SBS and B-SBS copolymers depicted higher variation in the aging index after LTA than their counterparts and were also unable to protect the base binder from stiffening. In contrast, HV-SBS and DB-SB generate significantly lower temperature susceptibility in the asphalt mixture due to higher vinyl content (≥30%), i.e., location of the C=C group as the side chain in the butadiene segment. Fourier-transform infrared spectroscopy (FTIR) results reveal that the increase of carbonyl and sulphoxide compounds, and the lowering of polybutadiene compounds validate the adverse effect of LTA on L-SBS and B-SBS copolymers. On the contrary, such behavior was not reflected in HV-SBS and DB-SB copolymers. Thus, the current study presents the major limitation of using branched and linear SBS polymers having comparatively lower vinyl content, and recommends the mandatory use of high-vinyl SBS copolymers for ensuring enhanced long-term performance.