We observe magnetic effects in ultra-high quality factor crystalline quartz Bulk Acoustic Wave resonators at milli-Kelvin temperature. The study reveals existence of hysteresis loops, jumps and memory effects of acoustical resonance frequencies. These loops arise as a response to the external magnetic field and span over few Hertz range for modes with linewidths of about $25$mHz, which constitute a frequency shift of order 60 linewidths. The effects are broadband but get stronger towards higher frequencies where both nonlinear effects and losses are limited by two level systems. This suggests that the observed effects are due to ferromagnet-like phase of a spin ensemble coupled to mechanical modes. The observed coupling between mechanical and spin degrees of freedom in the ultra low loss regime brings new possibilities for the emerging class of quantum hybrid systems.