We investigate the quantum Mpemba effect from the perspective of nonequilibrium quantum thermodynamics by studying relaxation dynamics of quantum systems coupled to a Markovian heat bath, which are described by Davies maps. Starting from a state with coherences in the energy eigenbasis, we demonstrate that an exponential speedup to equilibrium will always occur if the state is transformed to a diagonal state in the energy eigenbasis, provided that the spectral gap of the generator is defined by a complex eigenvalue. When the transformed state has a higher nonequilibrium free energy, we argue using thermodynamic reasoning that this is a genuine quantum Mpemba effect. Furthermore, we show how a unitary transformation on an initial state can always be constructed to yield the effect and demonstrate our findings by studying the dynamics of both the nonequilibrium free energy and the irreversible entropy production in single and multiqubit examples.