The performance uniformity of each pixel integrated with a patch antenna in a terahertz plasmon detector array is very important in building the large array necessary for a real-time imaging system. We found a parasitic antenna effect in the terahertz plasmon detector whose response is dependent on the position of the detector pixel in the illumination area of the terahertz beam. It was also demonstrated that the parasitic antenna effect is attributed to the physical structure consisting of signal pads, bonding wires, and interconnection lines on a chip and a printed circuit board. Experimental results show that the performance of the detector pixel is determined by the sum of the effects of each parasitic antenna and the on-chip integrated antenna designed to detect signals at the operating frequency. The parasitic antenna effect can be minimized by blocking the interconnections with a metallic shield.