Excavation, river incision and anchor cable relaxation would result in unloading of stress in at least one direction of rock masses in caverns or slopes, and the unloading failure is prominent and it even shows remarkable tensile failure. The previous direct shear tests with constant normal stress can no longer meet these circumstances. However, the experimental study of rock shear behavior under unloading normal stress condition is rare. This paper presents an innovative experimental method to study the mechanical behaviors of sandstone containing a pre-existing flaw under unloading normal stress with constant shear stress. Five failure patterns were identified based on the analysis of crack propagation and their mechanical properties. The failure pattern transforms from mixed tensile–shear failure to tensile failure then to shear failure with the increase of flaw angle. Initial normal stress is greater, the propagating cracks behave as stronger tensile fractures. The peak dilatancy angle increases and decreases with the increase of the initial normal and shear stresses, respectively. Internal friction angle and cohesion both decrease first and then increase with the increase of flaw angle. The branch crack, extent of exfoliation, shear scratch and failure pattern are different from that in the traditional direct shear tests. Unloading normal stress significantly weakens the shear strength compared to traditional direct shear tests, suggesting that the shear strength parameters used in stability evaluation of rock excavation engineering should be determined by unloading tests. The results enrich the basic theory of rock mechanics.