Cavity quantum-electrodynamics (QED) is a rich area of optical physics, where extreme light–matter coupling can give rise to ultrastrong coupling. The ultrastrong coupling regime presents some fascinating uniquely quantum mechanical effects, such as ground state virtual photons and vacuum squeezing. Focusing on the widely adopted Hopfield model with cavity dissipation, we show how the linear spectrum of an ultrastrong coupled cavity and a dipole can be described either classically or quantum mechanically, but only when the quantum model includes (i) corrections to maintain gauge invariance, and (ii) a specific type of cavity bath coupling, which has so far not been identified. We also show the impact of this bath model on the quantum Rabi model, which has no classical analog in ultrastrong coupling. These results can be used to guide emerging experiments and significantly impact current models and interpretations of ultrastrong coupling between light and matter.