A combined numerical-experimental investigation is presented with focus on the effects of boundary-layer instabilities and transition on the wall cooling performance in a Mach 5 low-enthalpy flow over a flat plate, with coolant injection achieved through a row of slots. The numerical study has been performed through direct numerical simulation (DNS) of the compressible Navier-Stokes equations, and is supported by results from linear stability analysis (LST) for the considered boundary layer. The experiments have been conducted in the High Density Tunnel (HDT) of the Oxford Thermofluids Institute, and include several blowing ratio conditions of injected air for the same freestream conditions. Surface heat transfer and pressure measurements, film effectiveness measurements, and Schlieren images are presented. The analysis links the wall cooling performance to the growth of imposed unstable boundary layer modes. Results indicate that 2D and 3D unstable modes, pertaining to the class of first instability modes, exist in the laminar boundary layer, and that imposition of these modes at different amplitudes leads to different states of the boundary layer, which we refer to as a perturbed state and a transitional state for medium and high amplitude respectively. As confirmed by comparison with experimental data, the perturbed and transitional states of the boundary layer significantly affect the wall cooling performance, providing an increase of the wall heat flux that results in a reduction of the beneficial effects of cooling.