This paper presents an investigation into the structural, magnetic, and dielectric characteristics of BaFe12-xYxO19 (x = 0.0, 0.2, 0.4, 0.6) nanoparticles (NPs) produced by the sol-gel method at 800, 900, 1000, 1100 °C annealing. Rietveld refinement analysis shows that the lattice parameter ratio (c/a) lies within the range of 3.93–3.96 provides the confirmation of M type hexaferrite with space group P63/mmc. The average crystallite size for BaFe12O19 is found to decrease (53.0–23.4 nm) with increasing annealing temperature while for doped BaFe12-xYxO19 it is increased from 30.7 to 96.2 nm. The magnetization found to decrease (82–17 emu/g) with Y3+ content for 800 and 900 °C annealing. However, for 1000 and 1100 °C annealing the magnetization increases and reaches to maximum 100.2 emu/g for x = 0.2 at 1100 °C annealing. The site preference of the Fe3+ ions is responsible to change the value of magnetization. The coercivity of undoped BaFe12O19 varies from 4039 to 2028 with doping content. However, maximum coercivity 4586 Oe is found for x = 0.4 at 900 °C. The dielectric function is found to vary with annealing temperature and Y content as well. The wide range of magnetic properties (soft and hard magnetic) can be used in data storage, spintronic and microwave device applications.