In order to improve the safety of the tunnel asphalt pavement in the event of a fire, and reduce the deterioration of the low temperature crack resistance of the asphalt by the flame retardant. The research uses aluminum hydroxide (ATH) as a smoke suppressant, diethyl aluminum hypophosphite (ADP) as a flame retardant, and halloysite nanotubes (HNTs) as a synergist to modified styrene-butadiene-styrene block copolymer (SBS) modified asphalt (MA). First, the content of ATH, ADP, and HNTs was used as the response variable. The physical properties (Penetration, Softening point, Ductility) and static flame retardant properties (Limiting oxygen index meter, Ignition point) of the asphalt modified by nanocomposite flame-retardant (HNTs-CFRMA) were the response variables. The response surface methodology was used to design the test, and regression models were established to analyze the influence of flame retardants on the performance of asphalt. Then, comprehensively considering the effects of physical properties and flame retardant properties, the normalized desirability function was used to perform a multi-objective optimization design on the components of the nanocomposite flame retardant modifier to obtain the best flame retardant formula. Finally, the rheological properties of MA, conventional flame-retardant modified asphalt (CFRMA), and HNTs-CFRMA were tested based on Dynamic shear rheometer, Multiple stress creep test, Force ductility tester, and Bending beam rheometer. The performance of flame-retardant and smoke suppression were tested by the Cone calorimeter tests. The result shows that ATH, ADP, and HNTs can enhance the high temperature performance of asphalt, reduce the penetration. The addition of HNTs can increase significantly the softening point and reduce the deteriorating effect of flame retardants on the low temperature performance of asphalt; the addition of ATH and HNTs can improve significantly the flame retardancy of asphalt. Based on the desirability function of power exponent, the formulation of the nanocomposite flame retardant with better physical properties and flame retardant properties is ATH:ADP:HNTs = 3:5:1, and the total content is 9 wt%. Nanocomposite flame retardants can improve obviously the high temperature rheological properties of asphalt. The rutting factor and the cracking factor of HNTs-CFRMA improve markedly, and the irrecoverable creep compliance is reduced, compared with MA and CFRMA. Nanocomposite flame retardant can make up for the deterioration of conventional flame retardants on asphalt’s low temperature performance. At the same time, it has better flame-retardant performance and smoke suppression performance.