With metaverse attracting increasing attention from both academic and industry, the application of virtual reality (VR) has extended beyond 3D immersive viewing/gaming to a broader range of areas, such as banking, shopping, tourism, education, etc., which involves a growing amount of sensitive and private user data into VR systems. However, with current password-based user authentication schemes in mainstream VR devices, studies demonstrate that side-channel attacks can pose a severe threat to VR user privacy. To mitigate the threat, we propose a novel Panoramic-image-based VR user authentication system, i.e., Pivot , to defend against such attacks, yet maintain high usability. Specifically, in Pivot , we design an image-random-pivoting-based user interaction mechanism to assist users in quickly and securely selecting memorable points of interest in a panoramic image. Then an image region segmentation algorithm is designed to automatically scatter the points to regions to form the customized graphic password for the user, which could ensure a sufficiently large password space and also reduce the near-region point misclicks. Afterward, the region indexes are used to generate the hashed password for authentication. Both theoretical security analysis and extensive user studies demonstrate that Pivot is secure and user-friendly in practice.