This paper studies the universal control for multiple nonholonomic unicycles, i.e., finding a control law that allows for both rendezvous and tracking uses. The reduced-order approach and optimal technique are applied to design an exponential observer that estimates the leader's states. Using the observer signal, we convert the pose errors into the form of nonholonomic integrators and derive a local tracking control law. The obtained control law guarantees exponential convergence of tracking errors and maintains the solution trajectory of converted states in an invariant set. More importantly, the original pose error between each unicycle and the leader is proven exponentially convergent to zero, whether or not the leader satisfies persistency of excitation (PE) conditions. The extension of the proposed control algorithm to leader-following formation control is also illustrated. Numerical simulations validate the control design.