The path following control of an under-actuated unmanned surface vehicle (USV) under the constraints of rudder angle, rudder velocity, and rudder response time in the disturbing environment is studied, and a cascaded path following control system based on guidance law and heading control law is designed. First, the guidance law is designed on the basis of integral line-of-sight, while the tracking error state is introduced to design a variable gain disturbance observer, which not only ensures the stability of the convergence section but also takes into account the tracking accuracy of the stable section. The stability of the system is analyzed. Subsequently, the rudder maneuverability constraint and rudder effect delay are further imposed after fully considering the limited range of rudder angle, rudder velocity, and rudder response time in the process of path following. In addition, the heading control law is, therefore, designed on the basis of the rolling optimization strategy, which effectively reduces the oscillation while ensuring the convergence speed. The stability of the control law is further proved. Thereafter, a simulation experiment proves the effectiveness and advancement of the algorithm designed in this paper. In the end, based on the software and hardware design of the control system, “Sea Sturgeon” USV is used for the lake test of the proposed control algorithm to verify its feasibility in practical engineering applications.