Elliptic flow ($v_2$) fluctuations in central heavy-ion collisions are direct probes of the fluctuating geometry of the quark-gluon plasma, and, as such, are strongly sensitive to any deviation from spherical symmetry in the shape of the colliding nuclei. We investigate the consequences of nuclear deformation for $v_2$ fluctuations, and we assess whether current models of medium geometry are able to predict and capture such effects. Assuming linear hydrodynamic response between $v_2$ and the eccentricity of the medium, $\varepsilon_2$, we perform accurate comparisons between model calculations of $\varepsilon_2$ fluctuations and STAR data on cumulants of elliptic flow, in central Au+Au and U+U collisions. From these comparisons, we evince that the most distinct signatures of nuclear deformation appear in the non-Gaussianities of $v_2$ fluctuation, and we show, in particular, that the non-Gaussian $v_2$ fluctuations currently observed in central Au+Au collisions are incompatible with model calculations that implement a quadrupole coefficient of order 12\% in the $^{197}$Au nuclei. Finally, we make robust predictions for the behavior of higher-order cumulants of $v_2$ in collisions of non-spherical nuclei.