Nano Zn0.9Ni0.1Co2-xFexO4 samples were prepared using the hydrothermal procedure in order to enhance the functional characteristics of ZnCo2O4 sample. The structural and microstructural parameters of the prepared samples were determined through Rietveld and FTIR analyses. The morphologies and compositions of the samples were examined by SEM and EDS techniques. Upon Ni2+ doping, the bandgap energy of nano ZnCo2O4 decreased from 2.22 eV to 3.61 eV–2.18 eV and 3.51 eV. Doping with Fe caused a slight decrease in bandgap energy for x = 0.05, then an increase for a higher content of Fe. Several empirical equations were used to determine the refractive index from the corresponding obtained optical energy gap values. As the Fe content rose to 5 %, the value of refractive index and nonlinear optical parameters increased; thereafter, it declined with additional Fe doping. The photoluminescence technique was used to analyze defects and processes of charge carrier recombination in the samples. ZnCo2O4 sample exhibits paramagnetic performance. With the addition of Ni to ZnCo2O4, the magnetic characteristics of the sample became superparamagnetic. Samples doped with Ni and various quantities of Fe exhibit a weak ferromagnetic behavior. The effect of doping on the magnetic transition, coercivity, saturation magnetization and remanent magnetization was explored.