This article introduces a robust controller designed to facilitate consensus among leader-follower agents in a swarm system. The method employs algebraic graph theory to establish a consensus framework and incorporates a sliding surface-based uncertainty and disturbance estimator (SUDE) to mitigate the effects of nonlinearities, external disturbances, and parametric uncertainty inherent in swarm systems. The stability of the SUDE controller is examined using Lyapunov theory. Importantly, this approach operates without prior knowledge of disturbance bounds, relying instead on the dynamics of disturbance error. Simulation studies demonstrate the effectiveness of the proposed control methodology, showing that the SUDE controller expedites consensus achievement while ensuring smooth and non-chattering control behavior. The proposed SUDE controller demonstrates a notable improvement in achieving position and velocity consensus in both the X and Y directions, with a 75% and 66% faster convergence rate, respectively, when compared to the conventional sliding mode controller (SMC). Additionally, the performance of the proposed controller is assessed under various external disturbances and parametric uncertainty conditions. Comparative analyses using root mean square error (RMSE) and integral absolute time error (IATE), both based on consensus error, confirm the effectiveness of the proposed SUDE consensus controller.