The nature of the nematic state in FeSe remains one of the major unsolved mysteries in Fe- based superconductors. Both spin and orbital physics have been invoked to explain the origin of this phase. Here we present experimental evidence for frustrated, short-range magnetic order, as suggested by several recent theoretical works, in the nematic state of FeSe. We use a combination of magnetostriction, susceptibility and resistivity measurements to probe the in-plane anisotropies of the nematic state and its associated fluctuations. Despite the absence of long-range magnetic order in FeSe, we observe a sizable in-plane magnetic susceptibility anisotropy, which is responsible for the field-induced in-plane distortion inferred from magnetostriction measurements. Further we demonstrate that all three anisotropies in FeSe are very similar to those of BaFe2As2, which strongly suggests that the nematic phase in FeSe is also of magnetic origin.