A novel type of nonlinear symmetry breaking in symmetric plasmonic oligomers is reported. By monitoring the strength of the second-harmonic signal while changing the polarization angle of the pump, we observe nonlinear symmetry breaking as a large variation in the generated nonlinear signal. Importantly, the strongly anisotropic nonlinear response is produced by a symmetric structure with isotropic linear response when rotating polarization. We provide theoretical analysis to describe and characterize this effect. Our finding opens new avenues to reveal and characterize the symmetry of nanoscale structures and molecules and also to remotely monitor variations of near-field patterns produced by symmetric nanostructures.