The problem of finding the optimal current distribution supported by small radiators yielding the minimum quality (Q) factor is a fundamental problem in electromagnetism. Q factor bounds constrain the maximum operational bandwidth of devices including antennas, metamaterials, and nanoresonators, and have been featured in seminal papers in the past decades. Here, we determine the lower bounds of Q factors of small-size plasmonic and high-permittivity dielectric resonators, which are characterized by quasi-electrostatic and quasi-magnetostatic natural modes, respectively. We expand the induced current density field in the resonator in terms of these modes, leading to closed-form analytical expressions for the electric and magnetic polarizability tensors, whose largest eigenvalue is directly linked to the minimum Q factor. Our results allow also to determine in closed form the corresponding optimal current density field. In particular, when the resonator exhibits two orthogonal reflection symmetries the minimum Q factor can be simply obtained from the Q factors of the single current modes with non-vanishing dipole moments aligned along the major axis of the resonator. Overall, our results open exciting opportunities in the context of nano-optics and metamaterials, facilitating the analysis and design of optimally shaped resonators for enhanced and tailored light-matter interactions.