This Rapid Communication presents a structure and full theoretical analysis to exploit the photon drag effect for THz signal generation in a graphene layer integrated with a plasmonic structure. The plasmonic structure is composed of a periodic array of asymmetric nanoparticles patterned over a graphene layer. The nanoparticles are designed to accomplish two goals: field localization due to the plasmonic resonance and manipulating the phase of the near field to effectively drag the quasiparticles in graphene. Combining the asymmetry with the plasmon resonances of nanoparticles, we show that an enhancement as large as three orders of magnitude is attainable in the power of the generated THz wave. This level of unprecedented enhancement mostly stems from the phase manipulation of the near field caused by asymmetric nanoparticles. Using the achieved enhancement, it is demonstrated that an ultra-wideband THz signal carrying the power of $1\phantom{\rule{0.28em}{0ex}}\ensuremath{\mu}\mathrm{W}$ can be generated using a commercially available femtosecond pulsed laser.