Nonlinear optical properties of 3-styryl and 3-vinyl coumarin dyes have been investigated with Density Functional Theory (DFT) using global hybrid (GH) and range-separated hybrid (RSH) functionals. The performance of GHs - B3LYP, BHHLYP, PBE0, M06, M06L, M062X, and M06HF and RSHs - CAM-B3LYP, HISSbPBE, HSEH1PBE, wB97, wB97X, and wB97XD in combination with 6-311++G(d,p) basis set has been analyzed. Estimated (hyper) polarizability ( $$\alpha _{0}, \beta _{0})$$ obtained from the GHs - M06, M062X and PBE0 are in agreement with each other. The RSHs - wB97 and wB97X estimate very close values of $$\beta _{0}$$ . The $$\beta _{0}$$ value of 3-styryl and 3-vinyl coumarins reaches the maximum as the bond length alternation and bond order alternation parameters tend to zero. Natural bond orbital analysis shows there is extensive charge transfer in the excited state leading to large value of $$\beta _{0}$$ . The vibrational contribution to $$\alpha _{0}$$ and $$\beta _{0}$$ is significantly less when the donor is methoxy group and acceptor is nitro group in the 3-styryl coumarin. The dye, (E)-7-(diethylamino)-3-(4-nitrostyryl)-2H-chromen-2-one (3c) is found to give the highest NLO response. An increasing electrophilicity originating from the decreased HOMO-LUMO band gap leads to an increase in $$\alpha _{0} $$ and $$\beta _{0}$$ in all the cases. JCSC-D-17-00839R1 Synopsis - The NLO properties of 3-styryl and 3-vinyl coumarin dyes have been investigated thoroughly with the help of DFT using global hybrid (GH) and range-separated hybrid (RSH) functionals. The performance of GHs and RSHs in combination with 6-311++G(d,p) basis set has been analyzed. Calculated BLA/BOA index, NBO analysis, electrophilicity index and HOMO-LUMO gap are good agreement with the $$\alpha _{0}$$ and $$\beta _{0}$$ .