The repeated wetting–drying (WD) cycles can cause the deterioration of rock and thus affect the service performance of the related rock engineering, e.g., the crushed-rock embankment and the reservoir. The study comprehensively investigated the deterioration mechanism of sandstone under the WD cycles based on a series of multi-scale experiments, including the microstructure, the meso–micro pore characteristics, and the macroscopic physical and mechanical properties. The results indicate that the content of dissolved minerals and the permeability are two key factors that determine the deterioration effect caused by the WD cycles. During the WD cycles, the pore size range of sandstone becomes wider due to the formation of new small pores and the connection of original pores. Thus, low-density area forms within the samples, which causes the increasing of density dispersion. The structure deterioration weakens the physical and mechanical properties of the samples. The ultrasonic test indicates that the decreasing rate of the P-wave velocity is larger than that of the S-wave velocity, especially in the first 5 cycles. Besides, the exponential equations of uniaxial compressive strength and elastic modulus with the number of WD cycles are established to describe the deterioration of mechanical characteristics.