Optical cavities are an enabling technology of modern quantum science: From their essential role in the operation of lasers to applications as flywheels in atomic clocks and interaction-enhancing components in quantum optics experiments, developing a quantitative understanding of the mode shapes and energies of optical cavities has been crucial for the growth of the field. Nonetheless, the standard treatment using paraxial quadratic optics fails to capture the influence of optical aberrations present in modern cavities with high finesse, small waist, and/or many degenerate modes. In this work, we compute the mode spectrum of optical resonators, allowing for both nonparaxial beam propagation and beyond-quadratic mirrors and lenses. Generalizing prior works, we develop a complete theory of resonator aberrations, including intracavity lenses, nonplanar geometries, and arbitrary mirror forms. Harnessing these tools, we reconcile the near absence of aberration in the work of Schine et al. [N. Schine et al., Nature (London) 534, 671 (2016)] with the strongly evident aberrations in the seemingly similar cavity of Clark et al. [L. W. Clark et al., Nature (London) 582, 41 (2020)]. We further validate our approach by comparison to a family of nonplanar lens cavities realized in the laboratory, finding good quantitative agreement. This work opens prospects for cavities with smaller waists and more degenerate modes.