An analytical and numerical study has been carried out for self-focusing of an intense cosh-Gaussian laser beam in collisionless plasma and its impact on the excitation of ion-acoustic wave and stimulated Brillouin backscattering process. The analytical model has been developed under Wentzel–Kramers–Brillouin and paraxial ray approximations. The nonlinearities of ponderomotive force on electron and the relativistic oscillation of the electron mass have been used in this study. The nonlinear differential equations have been set up for the beam width parameters of the main beam, ion-acoustic wave, backscattered wave and back reflectivity of stimulated Brillouin scattering (SBS). These equations have been solved numerically for different values of decentred parameter (b), relative plasma density (ωp0/ω0) and incident laser intensity (a). The results have been compared with only relativistic nonlinearity and Gaussian profile of laser beam. The focusing of laser beam, ion-acoustic wave and scattered wave are found to be strong under relativistic-ponderomotive regime compared to only relativistic regime. Further, it is observed that focusing/intensity of main laser beam, ion acoustic wave and SBS back reflectivity increases with increasing the values of b and ωp0/ω0. Itis also found that back reflectivity of SBS process gets suppressed with the increase in the value of a. This study may be useful in laser induced fusion scheme where back scattering of SBS plays very important role.