Metaverse is a vast virtual environment parallel to the physical world in which users enjoy a variety of services acting as an avatar. To build a secure living habitat, it’s vital to ensure the virtual-physical traceability that tracking a malicious player in the physical world via his avatars in virtual space. In this paper, we propose a two-factor authentication framework based on biometric-based authentication and chameleon signature. First, aiming at disguise in virtual space, we design an avatar’s two-factor identity model to ensure the verifiability of avatar’s virtual identity and physical identity. Second, facing at authentication efficiency and keys holding cost, we propose a chameleon collision signature algorithm to efficiently ensure that the avatar’s virtual identity is associated with its physical identity. Finally, aiming at impersonation in the physical world, we design two decentralized authentication protocols based on the avatar’s identity model and the chameleon collision signature to achieve real-time authentication on the avatar’s identity. Security analysis indicates that the proposed authentication framework guarantees the consistency and traceability of the avatar’s identity. Simulation experiments show that the framework not only completes the decentralized authentication between avatars but also achieves virtual-physical tracking.