The deformation and failure of soft rock affected by hydro-mechanical (HM) effect are one of the most concerns in geotechnical engineering, which are basically attributed to the grain sliding of soft rock. This study tried to develop a dissolution-diffusion sliding model for the typical red bed soft rock in South China. Based on hydration film, mineral dissolution and diffusion theory, and geochemical thermodynamics, a dissolution-diffusion sliding model with the HM effect was established to account for the sliding rate. Combined with the digital image processing technology, the relationship between the grain size of soft rock and the amplitude of sliding surface was presented. An equation for the strain rate of soft rocks under steady state was also derived. The reliability of the dissolution-diffusion sliding model was verified by triaxial creep tests on the soft rock with the HM coupling effect and by the relationship between the inversion average disjoining pressure and the average thickness of the hydration film. The results showed that the sliding rate of the soft rock grains was affected significantly by the waviness of sliding surface, the shear stress, and the average thickness of hydration film. The average grain size is essential for controlling the steady-state creep rate of soft rock. This study provides a new idea for investigating the deformation and failure of soft rock with the HM effect.