The adaptive robust size-independent consensus problem of leader following multi-agent systems (MASs) with unknown dynamics based on only onboard relative displacement information is studied. It is assumed that all followers only can measure the relative displacement regarding their adjacent agents. Due to lack of communication tools, each follower must estimate its uncertain dynamics and the leader acceleration simultaneously. A decentralized adaptive robust consensus law is presented to achieve the global state consensus for uncertain second-order multi-agent systems by employing only relative displacement information. According to the Lyapunov theorem, it is proved that without any knowledge about the number of followers and communication graph, the global state consensus is attained. The control and adaptive gains are tuned independently of the number of followers and communication topology. Therefore, the controller is robust against size-changing and can be applied to large-scale uncertain MASs. Afterward, it is verified that if the basis vectors are persistently exciting (PE), the estimation errors of uncertain vectors tend to zero. Several numerical scenarios are prepared to confirm the efficiency of this method.