Abstract Nonlinear optical materials are cornerstones of modern optics including ultrafast lasers, optical computing, and harmonic generation. The nonlinear coefficients of optical materials suffer from limitations in strength and bandwidth. Also, the nonlinear performance is typically monotonous without polarization selectivity, and to date, no natural material has been found to possess nonlinear coefficients with positive or negative signs simultaneously at a specific wavelength, all of which impede practical applications in the specific scenario. Here, we realize broadband large optical nonlinearity accompanied with ultrafast dynamics in a coupled system composed of gold dolmens and an epsilon-near-zero material for dual orthogonal polarizations simultaneously. The system also shows the polarization-selected nonlinearity transition properties, where the sign of the optical nonlinear refractive indexes can be converted via polarization switching. This guarantees active transitions from self-focusing to self-defocusing by polarization rotation without tuning wavelength in practical utilizations. The measured nonlinear refractive index and susceptibility demonstrate more than three orders of magnitude enhancement over a 400-nm-bandwidth compared with the constituents, while maintaining the sub-1 ps time response. The realized enhanced, ultrafast response, and the polarization tunability ensure the designed system a promising platform for the development of integrated ultrafast laser sources, all-optical circuits and quantum chips.