Ultracompact chiral plasmonic nanostructures with unique chiral light-matter interactions are vital for future photonic technologies. However, previous studies are limited to reporting their steady-state performance, presenting a fundamental obstacle to the development of high-speed optical devices with polarization sensitivity. Here, a comprehensive analysis of ultrafast chiroptical response of chiral gold nano-oligomers using time-resolved polarimetric measurements is provided. Significant differences are observed in terms of the absorption intensity, thus hot electron generation, and hot carrier decay time upon polarized photopumping, which are explained by a phenomenological model of the helicity-resolved optical transitions. Moreover, the chiroptical signal is switchable by reversing the direction of the pump pulse, demonstrating the versatile modulation of polarization selection in a single device. The results offer fundamental insights into the helicity-resolved optical transitions in photoexcited chiral plasmonics and can facilitate the development of high-speed polarization-sensitive flat optics with potential applications in nanophotonics and quantum optics.