In this research, the third-order nonlinear characteristics of some novel azo dyes based on 5-hydroxyquinoline (containing quinoline skeleton in their structure) were studied via Z-scan technique and by means of a continuous wave of a diode-pumped laser at the wavelength 532 nm with an intensity of 38.72 × 103 W/m2. The nonlinear absorption coefficient and nonlinear refractive index (β and n2, respectively) were calculated by fitting the theoretical model and experimental data obtained from the Z-scan technique. The open aperture component of this technique illustrated reverse saturable absorption phenomenon. Thermal nonlinear phase shift affected the shapes of closed-aperture Z-scan curves, and they did not fit well with the common theoretical models. The forms of peak-to-valley in these plots did behave like a modified aberrant thermal lens model. The substituent effect on the values of β and n2 was also investigated and demonstrated that the azo dyes bearing electron-withdrawing groups offer higher values of β and n2 than the other azo dyes with or without electron-donating groups. The calculated values of third-order susceptibility of these azo dyes were from the order 10-6esu, indicating their potential capability for the use in optical devices.