There is an increasing interest in direction-of-arrival (DOA) estimation using nested arrays composed of vector sensors. Considering acoustic vector sensors (AVSs) commonly used in underwater applications, this paper proposes a novel algorithm, called augmented nested quaternion-MUSIC (ANQ-MUSIC), to perform DOA estimation. By judiciously arranging the multi-component outputs of AVSs, we model the received signals from the entire nested AVS array as a quaternion observation vector in a compact way to reduce the computational complexity. Next, we formulate a quaternion-based difference co-array (QDCA) model via vectorizing the quaternion covariance matrix (QCM). Based on the obtained insights from the QDCA model, we derive a suitable QCM, which is constructed by applying the spatial smoothing technique. Finally, classical quaternion-MUSIC is logically introduced to estimate the DOA parameters. In simulations, we take into account non-uniform received noise and inter-component correlated noise, which may occur in practical underwater environments. The results demonstrate that the proposed method shows superiority in angular resolution and achieves a desirable trade-off between estimate accuracy and computational burden, besides showing robust performance in the above test scenarios.