This work used two approaches of mechanical alloying before high temperature solid state reaction to synthesize BaFe12-xScxO19 (x = 0.5 to 6.0) system. Rietveld refinement of X-ray diffraction patterns showed the formation of extra phases at higher Sc doping, along with the main magnetoplumbite structure. Raman spectroscopy suggested random distribution of Sc ions at the Fe-sites of hexaferrite structure. X-ray photoelectron spectroscopy (XPS) confirmed the +3 charge state at the Fe/Sc-sites. The Fe 3s band suggested dilution of ferromagnetic 3s-3d spin exchange interactions. The neutron diffraction study re-confirmed random occupancy of the non-magnetic Sc3+ ions at Fe3+ sites of the Sc doped Ba hexaferrite system. The transformation of magnetic spin order from collinear to non-collinear state has shown spin glass-like features at low temperature regime in the dc magnetization curves. This is an effect of dilution of Fe–O–Fe ferrimagnetic super exchange interactions and spin frustration in Sc-doped samples.