AbstractThis paper treats the design of a decentralized nonlinear robust control system for formation flying of multiple unmanned aerial vehicles (UAVs). In close formation, it is assumed that vortex of any UAV affects the motion of all the UAVs behind it. The forces produced by these vortices are complex functions of relative position co‐ordinates of the UAVs. In this paper, these forces are treated as unknown functions, which cannot be parameterized. Since the system is not invertible in the wind axes system, a simplified co‐ordinate system obtained from the wind axes system for which the velocity roll (bank angle) is zero, is considered for the design of the control system. A nonlinear robust control system for the separation trajectory control of the wing aircraft in the simplified wind coordinate system is derived. Uncertain functions and unmeasured variables are estimated using a high‐gain observer for the synthesis of the control system. Each wing UAV synthesizes its control law using its own state variables and the relative position of the preceding UAV with respect to the wing UAV. Thus the control system is decentralized since each UAV has to communicate (depending on sensors for position measurement) with at most one (preceding) UAV, and no data transmission from the remaining vehicles is required. Simulation results for two UAVs are presented which show precise separation trajectory control of each wing UAV in spite of the presence of unknown and unstructured vortex forces, while the lead aircraft maneuvers. Furthermore, these results confirm that when the wing aircraft is positioned properly in the vortex of the lead aircraft, it experiences reduction in its required flight power. Copyright © 2003 John Wiley & Sons, Ltd.