AbstractThe present analysis mathematically investigates the thermohaline convection problem in viscoelastic fluid layer saturating porous media by utilizing the modified Boussinesq approximation. By performing linear stability analysis, the Darcy–Rayleigh numbers for stationary and oscillatory modes of convection are derived. The effects of different parameters describing the problem are studied numerically. In nonlinear stability analysis, the heat and mass transfer rates in the form of Nusselt and Sherwood numbers, respectively, are obtained for oscillatory convection using the derived Ginzburg–Landau equation. From the results, it is observed that overstability is the preferred mode of instability in linear stability. It is found that in linear double‐diffusive convection problems, the stress relaxation imparts a destabilizing effect whereas the strain retardation time, the coefficient of specific heat variation due to temperature, and the concentration gradient have a stabilizing effect on the system's stability. The numerical values of heat and mass transfer rates varied with the coefficient of specific heat showing that the heat transport decreases while the mass transport increases. Also, the stress relaxation time, the concentration gradient, and the gravity modulation's amplitude increase while the strain retardation time decreases the heat and mass transfer rates. The wavelength of oscillations remains unaltered with the variation of specific heat variation due to temperature. The modulation frequency does not affect the heat/mass transfer rate; though, the wavelength of oscillations decreases with increasing frequency.