Coupling electromagnetic radiation with matter, e.g., by resonant light fields in external optical cavities, is highly promising for tailoring the optoelectronic properties of functional materials on the nanoscale. Here, we demonstrate that even internal fields induced by coherent lattice motions can be used to control the transient excitonic optical response in CsPbBr3 halide perovskite crystals. Upon resonant photoexcitation, two-dimensional electronic spectroscopy reveals an excitonic peak structure oscillating persistently with a 100-fs period for up to ~2 ps which does not match the frequency of any phonon modes of the crystals. Only at later times, beyond 2 ps, two low-frequency phonons of the lead-bromide lattice dominate the dynamics. We rationalize these findings by an unusual exciton-phonon coupling inducing off-resonant 100-fs Rabi oscillations between 1s and 2p excitons driven by the low-frequency phonons. As such, prevailing models for the electron-phonon coupling in halide perovskites are insufficient to explain these results. We propose the coupling of characteristic low-frequency phonon fields to intra-excitonic transitions in halide perovskites as the key to control the anharmonic response of these materials in order to establish new routes for enhancing their optoelectronic properties.