Abstract A series of NiTb x Fe 2− x O 4 ( x = 0.0, 0.1, 0.2, 0.3) nanoparticles with small size have been prepared using sol–gel technique. The effect of Tb 3+ substitution on the crystalline phase, crystallite size and magnetic, electrical and dielectric properties has been investigated in detail. The lattice constant shows an increasing trend with increase in Tb 3+ ion concentration. The high frequency tetrahedral vibration band ν 1 and the low frequency octahedral vibration band ν 2 in the FTIR spectra are in the reported range. The observed shift in ν 2 towards lower wave number side with increase in terbium concentration suggests the occupation of Tb 3+ ions predominantly on octahedral sites. Saturation magnetization is found to decrease with terbium addition. Magnetic hysteresis loss is observed to decrease up to x = 0.2 and then increases. The room temperature resistivity values are found to increase with terbium content. The observed dielectric dispersion with frequency is explained on the basis of Maxwell–Wagner two-layer model and electron hopping mechanism which is responsible for conduction and polarization. Dielectric constant and dielectric loss are found to decrease by terbium addition and is explained in terms of the decrease in hopping rate due to the substitution of Fe 3+ ions by Tb 3+ ions on the octahedral sites. The increased resistivity and reduced dielectric loss attained in terbium doped ferrite is promising as it is desirable for high frequency applications.