The objective of this paper is to investigate the microstructure of high-viscosity modified asphalt binder (HVA) containing high-content polymers and its effect on fatigue performance, as well as the applicability of different analytical methods and failure criteria of linear amplitude sweep (LAS) test to HVA. In this paper, two types of HVAs with different polymer contents, a commercial HVA, a conventional polymer-modified asphalt binder, and a base asphalt binder were first prepared or selected. Fluorescence microscopy tests, time sweep tests, and LAS fatigue tests were then performed on these binders. Finally, the fatigue properties of the different binders were analyzed in depth using different analytical methods and failure criteria. The results show that the polymers in HVA crosslink with each other to form a strong cross-linking structure, and the phase inversion of the polymers renders them into a continuous phase, which significantly improves the fatigue damage resistance of the HVA; moreover, the evolutions of stress with shear strain for the fatigue process of HVAs is related to the polymer network. Besides, the two-dimensional damage characteristic curve (DCC) cannot accurately reflect the evolution of the fatigue performance of HVA, and this paper suggests the use of a three-dimensional DCC curve with strain information. Furthermore, the 35% reduction in C0 in the dissipative energy method is not a suitable failure criterion, a more accurate fatigue performance ranking can be obtained by using the yield stress as the failure criterion. Although the fatigue resistance of HVA after yield stress is theoretically considered using the pseudo-strain energy approach, HVA does not show a peak of stored pseudo-strain energy even when the strain is increased to 60% in the LAS test. Therefore, the maximum strain value (30%) in the standard LAS test is not sufficient in characterizing high-polymer HVAs because there is not enough fatigue accumulation of HVAs in this range, consequently, the maximum shear strain parameter in the LAS experiment needs to be reconsidered.