Yb-substituted Ni0.5Zn0.5YbxFe2−xO4 (0 ≤ x ≤ 0.20 in the step of 0.04) ferrites have been synthesized using sol–gel auto combustion method. The structural characterization of the compositions has been performed by X-ray diffraction (XRD) analysis, field emission scanning electron microscopy (FESEM), quantum design physical properties measurement system (PPMS) that ensured the formation of single phase cubic spinel structure. Crystallite and average grain size are calculated and found to decrease with increasing Yb3+ contents. Saturation magnetization (Ms) and Bohr magnetic moment (µB) decrease while the coercivity increases with the increase in Yb3+ contents and successfully explained by the Neel’s collinear two sub-lattice model and critical size effect, respectively. Critical particle size has been estimated at 6.4 nm from the DXRD vs. Ms, Hc plot, the transition point between single domain regime (below the critical size) and multi-domain regime (beyond the critical size). Curie temperature (Tc) reduces due to the weakening of A–O–B super exchange interaction and redistribution of cations, confirmed by the M–T graph. The compositions retain ferromagnetic ordered structured below Tc and above Tc, it becomes paramagnetic, making them plausible candidates for high temperature magnetic device applications. The relative quality factor (RQF) peak is obtained at a very high frequency (≥ 108 MHz), indicating the compositions could also be applicable for high frequency magnetic device applications.