The effect of Mg substitution on structural, magnetic, and dielectric properties of Y-type hexaferrite Ba0.5Sr1.5Zn2-xMgxFe11AlO22 (x = 0, 0.4, 0.8, 1.2, 1.6, 2.0) prepared by the solid state reaction method was investigated. The X-ray diffraction patterns are analyzed by Rietveld refinement method and the hexagonal structure with R3−m was confirmed. The magnetic transition temperatures associated with the longitudinal conical to proper screw phase transition (T1) and proper screw to ferrimagnetic phase transition (T2) can be modulated by varying Mg content. For x ≤ 1.2 samples, the transition temperature T2 monotonically increases with increasing Mg substitution, suggesting the possibility that the helical spin order of Mg substituted Y-type hexaferrites is sustained up to higher temperatures than that of undoped samples. The saturation magnetization at room temperature increases to a maximum with Mg content x = 1.2 and then decreases with x. The variations of magnetic properties can be attributed to the difference in the preferential site for Zn and Mg and different distribution of Fe3+ ions over tetrahedral and octahedral sites. The results of magnetic field dependence of dielectric constant, i.e., magnetodielectric effect, at various temperatures imply that the doped samples show magnetically induced ferroelectricity up to 250 K. The intrinsic magnetodielectric effect is observed in all the samples. It is worthy to note that the magnetodielectric effect measured below 200 K increases with increasing Mg content, which is probably attributed to the variation of magnetic structures. The Mg-doped hexaferrites Ba0.5Sr1.5Zn2-xMgxFe11AlO22 may be potential candidates for application in magnetoelectric devices.