The excavation of deep tunnel in rock mass undergoes complex loading and unloading stress paths, resulting in rib spalling, flaking, and even severe rockburst disasters. Based on the variation law of the stress path of the surrounding rock, laboratory tests of rock mechanics are designed, and the deformation and strength behavior of marble with different initial confining pressure and unloading rates are systematically studied. By introducing strain increment, the characteristic stress, and the dilatancy index, the rock’s dilatancy and brittleness under different unloading conditions are quantitatively analyzed. During unloading, the energy transformation mechanism of rock is described, and the law of deformation and failure is discussed based on characteristic energy. The rock failure strength fitting formula is given by applying the Mogi–Coulomb criterion and elastic strain energy criterion. The advantages of the elastic strain energy criterion are theoretically explained. This study shows that comprehensive consideration of the complex stress paths, confining pressure levels, and the loading-unloading rates of surrounding rock is an effective way to accurately study unloading rock characteristics. The results can provide theoretical basis for stability analysis of high-stress underground engineering.