Local scour often causes pier instability; however, the characteristics and mechanism of downflow, representing one of the crucial flow structures, are still unclear. In this paper, the interaction between the downflow and the horseshoe vortex system and the role of the downflow under clear-water local scour conditions are discussed, based on the stress distribution obtained via experiments and simulations. In the present experiment, more accurate data are measured by installing suitable sensors on 3D-printed models that reproduce the scour hole conditions at various times. The obtained results reveal that the downflow exhibits a strong interaction with the horseshoe vortex system. From the perspective of flow structures, the flow structures collide and rub against each other, which weakens the effect of the downflow. From the perspective of energy transfer, the horseshoe vortex system absorbs the energy carried by the downflow to develop and reduce the energy introduced into the sediment. In addition, shear stress is a crucial factor in maintaining a high tangent slope. When the shear stress is down to a minimum and is stable, the tangent slope rises with the growth of the pressure stress, which means that the downflow is able to promote scour depth development.