Continuous identity authentication is critical for privacy protection throughout an entire user login session. In this paper, we propose a continuous user authentication mechanism namely, which employs the vibration responses from hand biometrics and is passively activated by natural user-device interaction. Hand vibration responses are embedded in the mechanical vibration of a force-bearing body consisting of one mobile device and one user hand. A built-in accelerometer of the device can capture hand-dependent vibration signals. Considering the concealment of vibration generation and the non-replicability of hand structure, it’s difficult for attackers to counterfeit user identity. Moreover, for ensuring the robustness of authentication performance to tapping behavior interference, we construct a data augmentation module jointly leveraging a signal processing and learning-based pipeline. It can generate enough vibration responses representing hand structure biometrics under various behaviors, thereby making comprehensively understand vibration response variation. We prototype on smartphones, and extensive experiments demonstrate that can achieve satisfactory authentication accuracy.