We use a combination of first-principles density-functional calculations and spin-dynamics simulations to explain the unusual diffuse inelastic neutron scattering in the hexagonal multiferroic yttrium manganite, YMnO3. Using symmetry considerations, we construct a model spin Hamiltonian with parameters derived from our density-functional calculations and show that it captures the measured behavior. We then show that the observed directionality in the structured diffuse scattering in momentum space is a hallmark of the triangular geometry, and that its persistence across a wide range of temperatures, both above and below the Néel temperature TN, is a result of the strong magnetic frustration. We predict that this diffuse scattering exists in a yet-to-be-observed modulated continuum of energies, that its associated spin excitations have distinct in-plane and out-of-plane character, and that vestiges of the magnetic frustration persist into the sub−TN state. Finally, we show that visualizing the magnetic order in terms of composite trimer magnetoelectric monopoles and toroidal moments, rather than individual spins, provides insight into the real-space fluctuations, revealing clusters of emerging order in the paramagnetic state, as well as collective short-range excitations in the ordered antiferromagnetic phase. Our understanding of this directional diffuse scattering in such a wide temperature range, both below and above TN, provides new insight into the magnetic phase transitions in classical frustrated systems. Published by the American Physical Society 2024