Optical phenomena associated with an extremely localized field should be understood with considerations of nonlocal and quantum effects, which pose a hurdle to conceptualize the physics with a picture of eigenmodes. Here we first propose a generalized Lorentz model to describe general nonlocal media under linear mean-field approximation and formulate source-free Maxwell's equations as a linear eigenvalue problem to define the quasinormal modes. Then we introduce an orthonormalization scheme for the modes and establish a canonical quasinormal mode framework for general nonlocal media. Explicit formalisms for metals described by a quantum hydrodynamic model and polar dielectrics with nonlocal response are exemplified. The framework enables for the first time a direct modal analysis of mode transition in the quantum tunneling regime and provides physical insights beyond usual far-field spectroscopic analysis. Applied to nonlocal polar dielectrics, the framework also unveils the important roles of longitudinal phonon polaritons in optical response.