All-optical modulation is an entangled part of ultrafast nonlinear optics with promising impacts on tunable optical devices in the future. Current advancements in all-optical control predominantly offer modulation by means of altering light intensity, while the ultrafast manipulation of other attributes of light have yet to be further explored. Here, we demonstrate the active modulation of the phase, polarization, and amplitude of light through the nonlinear modification of the optical response of a plasmonic crystal that supports subradiant, high Q, and polarization-selective resonance modes. The designed mode is exclusively accessible via TM-polarized light, which enables significant phase modulation and polarization conversion within the visible spectrum. To tailor the device performance in the time domain, we exploit the ultrafast transport dynamics of hot electrons at the interface of plasmonic metals and charge acceptor materials to facilitate an ultrafast switching speed. In addition, the operating wavelength of the proposed device can be tuned through the control of the in-plane momentum of light. Our work reveals the viability of dynamic phase and polarization control in plasmonic systems for all-optical switching and data processing.