<p indent="0mm">Time delay is one of the key factors damaging the driving safety in a vehicle lateral motion control system. This paper establishes a nonlinear vehicle model considering yaw and lateral motion, and two types of nonlinear steering characteristics and motion stability conditions are analyzed using handling diagrams. Next, a PD-based active yaw moment controller with time delay is applied to construct a complete hybrid vehicle system with delayed control. Firstly, the effect of time delay on the general stability and stability boundary of the vehicle system is studied using the location of the characteristic roots and the frequency domain method, wherein the results indicate that the lateral stability of the vehicle is significantly weakened under time delay, and the instability boundary can be reached under a substantial time delay. Furthermore, the optimization factor characterizing the system decay rate is introduced, and the optimization method for the control parameter of the vehicle lateral system is designed. The stability region is optimally divided to select the optimal control gain set under some delay. The optimization is compared and verified based on the steady-state simulation and dynamic test, and the results validate its effectiveness in improving the lateral stability and suppressing the effect of time delay, wherein the optimized dynamic responses of the system follow the target trajectory. Based on the relations between the optimization factor and vehicle characteristics, the critical maximum of the optimization factor is proposed as the quantitative index to evaluate the vehicle lateral stability under time delay, where its general parametric expressions are derived by exploring two types of critical instability conditions. Finally, the effectiveness and efficiency of the index in quantifying the delayed system stability and predicting the time-delay boundary of instability are sufficiently verified in steady-state and dynamic simulation.