We present a theoretical study of vibrational and vibronic properties of a point defect in the dilute limit by means of first-principles density functional theory calculations. As an exemplar we choose the negatively charged nitrogen-vacancy center, a solid-state system that has served as a testbed for many protocols of quantum technology. We achieve low effective concentrations of defects by constructing dynamical matrices of large supercells containing tens of thousands of atoms. The main goal of the paper is to calculate luminescence and absorption lineshapes due to coupling to vibrational degrees of freedom. The coupling to symmetric $a_1$ modes is computed via the Huang-Rhys theory. Importantly, to include a nontrivial contribution of $e$ modes we develop an effective methodology to solve the multi-mode $E \otimes e$ Jahn-Teller problem. Our results show that for NV centers in diamond a proper treatment of $e$ modes is particularly important for absorption. We obtain good agreement with experiment for both luminescence and absorption. Finally, the remaining shortcomings of the theoretical approach are critically reviewed. The presented theoretical approach will benefit identification and future studies of point defects in solids.