Measuring fundamental stellar parameters is key to fully comprehending the evolution of stars. However, current theoretical models overpredict effective temperatures, and underpredict radii, compared to observations of K and M dwarfs (radius inflation problem). In this work, we have developed a model-independent method to infer precise radii of single FGK and M dwarfs using Gaia DR3 parallaxes and photometry, and used it to study the radius inflation problem. We calibrated nine surface brightness–color relations for the three Gaia magnitudes and colors using a sample of stars with angular diameter measurements. We achieved an accuracy of 4% in our angular diameter estimations, which Gaia’s parallaxes allow us to convert to physical radii. We validated our method by comparing our radius measurements with literature samples and the Gaia DR3 catalog, which confirmed the accuracy of our method and revealed systematic offsets in the Gaia measurements. Moreover, we used a sample with measured Hα equivalent width (Hα EW), a magnetic activity indicator, to study the radius inflation problem. We demonstrated that active stars have larger radii than inactive stars, showing that radius inflation is correlated with magnetic activity. We found a correlation between the radius inflation of active stars and Hα EW for the mass bin 0.5 < M[M ⊙] ≤ 0.6, but we found no correlation for lower masses. This could be due to lack of precision in our radius estimation or a physical reason. Radius measurements with smaller uncertainties are necessary to distinguish between the two scenarios.