The rotating detonation rocket engine (RDRE) fueled by hydrogen/oxygen propellant represents a promising propulsion technology due to its high thermodynamic efficiency and propellant superior specific impulse. The rotating detonation wave (RDW) must propagate in a specific propagation mode while maintaining the self-sustaining state to ensure stable operation. An experimental system of hydrogen/oxygen fueled RDRE was developed in the present study. The operation of RDRE and propagation mode of RDW were investigated under atmospheric pressure conditions, and both hollow and annular combustors were tested. The high-frequency pressure fluctuations in the RDRE were measured by the dynamic pressure transducer, while a high-speed camera was used to capture images of flame luminescence at the rear end of the RDRE. The experimental results showed that the RDW could be initiated and reached a self-sustaining propagation state with hydrogen/oxygen propellant in the hollow and annular RDRE. A single-wave mode, a two-wave co-rotating mode, and a three-wave co-rotating mode were visualized under different conditions. With the increase in the equivalence ratio, the number of rotating detonation fronts decreased, and the variations in the RDW propagation modes were consistent in the hollow and annular RDRE. However, when the equivalence ratio exceeds 1.2, the propagation velocity decreases sharply in the annular combustor, while in the hollow combustor the RDW propagates stably, revealing a higher upper limit for the equivalence ratio. Also, the dominant frequency distribution was more concentrated in the hollow combustor. The findings provide valuable insight into the variations in detonation modes related to the equivalence ratio and combustor configuration.