Due to their high thermal cycle efficiency and compact combustor, oblique detonation engines hold great promise for hypersonic propulsion. Previous numerical simulations of oblique detonation waves have predominantly solved the Euler equations, disregarding the influence of viscosity and boundary layers. This work aims to study how the interaction between the oblique detonation wave and the boundary layer influences the detonation wave structures in confined spaces. Two-dimensional numerical simulations considering detailed chemistry are performed in a stoichiometric H2/air mixture. The results indicate that the wedge-induced oblique detonation wave generates a strong adverse pressure gradient upon impacting the upper wall, leading to boundary layer separation. The separation zone subsequently induces an oblique shock wave near the upper wall, and an increase in separation angle will cause the transition from an oblique shock wave to an oblique detonation wave. The formation of the separation zone reduces the actual flow area and may even lead to flow choking; its obstructive effect is similar to that of the Mach stem in inviscid flow. To establish a connection between the viscous recirculation zone and the inviscid Mach stem, we introduce a dimensionless parameter, η, based on the inviscid assumption. It is defined as the ratio of the inviscid Mach stem height to the channel entrance height. This parameter can be used to identify three wave systems in a viscous flow field: separation shock-dominated wave systems, separation detonation-dominated wave systems, and unstable Mach stem-dominated wave systems. Among these, the appearance of detonation Mach stems leads to flow choking, and the shock-detonation wave system continually moves upstream, ultimately causing the failure of the oblique detonation combustion. The findings of this study provide new insights into the investigation of the influence of viscosity on the flow structure of oblique detonation waves.