The optical nonlinearity of Rhodamine B dye at different concentration (10−3 M and 5 × 10−4 M) within reverse micelle prepared by water-in-decane microemulsion having constant molar ratio of water-to-surfactant molecules of 40 (W = [H2O]/[AOT] = 40) as well as function of mass fraction of nano-droplet (MFD) values of 0.01,0.04, 0.07, and 0.1 has been studied by Z-scan method, Fluorescence, FTIR, and UV–Visible. The optical responses of RhB were characterized by measuring the intensity dependent refractive index (n2) of the micellar environment using Z-scan technique. The open aperture Z-scan trace of the dye in bulk water and in AOT micelle displayed both saturable and reverse saturable absorption depending upon different parameters.The closed apertures Z-scan trace also vary as a function of environment and concentrations of dye. The nonlinearity of absorption comes due to droplets interactions and nonlinearity of refractive index appears due to thermal effect.