The transverse photo-induced voltages generated by the photon drag effect under normally and obliquely incident circularly polarized light across the plasmonic symmetry-breaking isosceles-triangle holes and right-triangle holes have been characterized. It is observed that the sign of transverse photovoltage flips when the incident circular polarization is switched for both types of plasmonic triangle holes. However, the unbalanced photovoltage between two circular polarizations is achieved across the plasmonic right-triangle holes, compared to the balanced photovoltage in the plasmonic isosceles-triangle holes. Such manipulation of the sign and the amplitude of transverse photovoltage is enabled by the broken symmetries of the electric and magnetic field patterns supported in the asymmetric triangle holes due to the interplay between the light helicity of circular polarization and the shape symmetries of triangle holes, together with the incident angle. These results will create opportunities for many applications relevant to nonlinear optics, photodetection, and chiral sensing.