As an increasing number of unmanned aerial vehicles (UAVs) have been widely applied in many aspects, controllers with higher performance are preferred. In this paper, a new varying-parameter adaptive multi-layer neural dynamic based controller (termed as VP-AMND controller) design method is proposed and applied to controllers of multi-rotor UAVs. First, a varying-parameter convergent neural dynamic (VP-CND) based controller is proposed and its convergence and robustness are theoretically proven. Second, by incorporating the adaptive control method into the VP-CND controller, the VP-AMND controller design method is proposed, of which the global stability, fast convergence speed and strong robustness can be guaranteed. Different from traditional triple zeroing dynamic (TZD) and VP-CND controllers, the proposed VP-AMND controller with self-tuning rates can estimate the unknown disturbances and enhance the stability of the system in the face of uncertainty. Third, computer simulation results verify that the multi-rotor UAVs with VP-AMND controllers can track time-varying trajectories quickly and solve the parameter uncertainty and disturbances problems effectively.