This Letter derives explicit factors linking mode-mismatch-induced power losses in Hermite-Gauss optical modes to the losses of the fundamental spatial mode. Higher-order modes are found to be more sensitive to beam parameter mismatches. This is particularly relevant for gravitational-wave detectors, where lasers employing higher-order optical modes have been proposed to mitigate thermal noise, and quantum-enhanced detectors are very susceptible to losses. This work should inform mode matching and squeezing requirements for Advanced+ and third generation detectors.