Electrical current pulses can be used to manipulate magnetization efficiently via spin-orbit torques. Pulse durations as short as a few picoseconds have been used to switch the magnetization of ferromagnetic films, reaching the terahertz regime. However, little is known about the reversal mechanisms and energy requirements in the ultrafast switching regime. In this work we quantify the energy cost for magnetization reversal over seven orders of magnitude in pulse duration, in both ferromagnetic and ferrimagnetic samples, bridging quasi-static spintronics and femtomagnetism. To this end, we develop a method to stretch picosecond pulses generated by a photoconductive switch by an order of magnitude. Thereby we can create current pulses from picoseconds to durations approaching the pulse width available with commercial instruments. We show that the energy cost for spin-orbit torque switching decreases by more than an order of magnitude in all samples when the pulse duration enters the picosecond range. We project an energy cost of 9 fJ for a 100 × 100 nm2 ferrimagnetic device. Micromagnetic and macrospin simulations unveil a transition from a non-coherent to a coherent magnetization reversal with a strong modification of the magnetization dynamical trajectories as pulse duration is reduced. Our results show the potential for high-speed magnetic spin-orbit torque memories and highlight alternative magnetization reversal pathways at fast timescales.