Flow instability is a common issue encountered by high-speed compressors when they operate outside of their optimal range, especially in highly loaded compressors. This study investigates the potential of an unsteady passive flow control technique, self-excited bleed (SEB), which involves casing modification, to improve the base flow and stability characteristics of an ultra-highly loaded low reaction transonic compressor rotor. Through transient computational fluid dynamics simulations, we demonstrate that SEB can extend the rotor's operating range by up to 14.07%. The physical mechanism underlying this stability enhancement is the suppression of the shock-induced breakdown of the tip leakage vortex (TLV) near the blade leading edge and the attenuation of the double leakage flow by SEB. The unsteady excitation of the bleed effect dominates the tip flow and eliminates the spontaneous closed-loop feedback process based on the dynamic interaction between the TLV breakdown, the tip secondary vortex, and the blade loading. Time-resolved tip-region flow patterns elucidate the self-organization and reconstruction of this feedback mechanism. Frequency spectral analysis further reveals that the self-induced oscillation frequency of the tip leakage flow formed during the feedback process disappears, and the bleed excitation frequency replaces it as the main frequency of the tip flow field. However, increasing the bleed flow rate causes the boundary layer on the suction surface to migrate radially outward, resulting in increased flow blockage at the rear of the tip passage. These two influences of SEB are quantified by a blockage factor, and determining the optimal bleed flow rate requires a trade-off between beneficial and detrimental impacts.