This paper proposes a robust lateral control scheme for the path tracking of autonomous vehicles. Considering the discrepancies between the model parameters and the actual values of the vehicle and the fluctuation of parameters during driving, the norm-bounded uncertainty is utilized to deal with the uncertainty of model parameters. Because some state variables in the model are difficult to measure, an H∞ observer is designed to estimate state variables and provide accurate state information to improve the robustness of path tracking. An H∞ state feedback controller is proposed to suppress system nonlinearity and uncertainty and produce the desired steering wheel angle to solve the path tracking problem. A feedforward control is designed to deal with road curvature and further reduce tracking errors. In summary, a path tracking method with H∞ performance is established based on the linear matrix inequality (LMI) technique, and the gains in observer and controller can be obtained directly. The hardware-in-the-loop (HIL) test is built to validate the real-time processing performance of the proposed method to ensure excellent practical application potential, and the effectiveness of the proposed control method is validated through the utilization of urban road and highway scenes. The experimental results indicate that the suggested control approach can track the desired trajectory more precisely compared with the model predictive control (MPC) method and make tracking errors within a small range in both urban and highway scenarios.