Abstract New phase diagrams for the dynamic structure of clay-laden open-channel flows are proposed. These diagrams can be used to distinguish between turbulent Newtonian, transitional, and laminar non-Newtonian flow behavior, on the basis of the balance between turbulent forces (approximated by the horizontal components of flow velocity and turbulence intensity) and cohesive forces (approximated by the suspended clay concentration and rheology). Stability regimes for five different flow types are defined using a comprehensive series of laboratory flume experiments at depth-averaged flow velocities ranging from 0.13 m s−1 to 1.47 m s−1, and at volumetric kaolinite clay concentrations ranging from 0.03% (= 0.8 g L−1) to 16.7% (= 434 g L−1). As clay concentration increases, five flow types can be distinguished: turbulent flow, turbulence-enhanced transitional flow, lower and upper transitional plug flow, and quasi-laminar plug flow. The turbulent properties of transitional flows are shown to be considerably more complex than the common notion of gradual turbulence damping. Turbulence-enhanced transitional flows display higher turbulence intensity than turbulent flows of similar velocity, with such enhancement originating from development of a highly turbulent basal internal shear layer within ~ 0.01 m of the bed. In lower transitional plug flows, the basal internal shear layer separates a lower region of high vertical gradient in horizontal velocity and strong turbulence from an upper region of plug flow with a much gentler velocity gradient and lower turbulence intensity. Kelvin-Helmholtz shear instabilities within the highly turbulent shear layer are expressed as distinct second-scale oscillations in the time series of downstream velocity. Turbulence damping dominates upper transitional plug flows, because strong cohesive forces, inferred to be caused by gelling of the high-concentration clay suspension, start to outbalance turbulent forces. In quasi-laminar plug flows, gelling is pervasive and turbulence is fully suppressed, apart from some minor residual turbulence near the base of these flows. With very few exceptions, all flows pass through the same development stages as clay concentration increases, regardless of their velocity, but the threshold concentrations for turbulence enhancement, gelling, and development of internal shear layers and plug flows are proportional to flow velocity. At flow velocities below ~ 0.5 m s−1, only low concentrations (< 0.75%) of kaolinite are required to induce transitional flow behavior, thus potentially affecting many slow-moving and decelerating clay flows in natural sedimentary environments. However, at flow velocities above 1 m s−1, clay concentrations of at least 6% are required in order for flows to enter the transitional flow phase, but even at these velocities the transitional flow phases make up a significant proportion of the flow phase space. By converting the experimental data to nondimensional Froude number (momentum term) and Reynolds number (cohesive term), it is shown that each boundary between the turbulent, transitional, and laminar flow phases can be described by a specific narrow range of Reynolds numbers. Within the duration of the experiments, settling of clay particles occurred only in plug flows of low flow velocity (and low Froude number), when the flows lacked the strength to support the entire clay suspension load.