Passive millimeter-wave and terahertz (MMW/THz) imaging is a significant technique used in personal security inspection and scene surveillance. However, a human body generally generates an obvious reflection on the floor, which may have negative influences on target detection and observer vision. In this article, we present a physically based method using dual-polarization feature and saliency information to solve the problem of reflection removal. The physical principle of a special dual-polarization phenomenon has been revealed by theoretical calculation and simulation analysis. By combining the second Stokes parameter (TQ) and modified linear polarization ratio (mLPR), a new feature parameter has been created to extract reflections of human bodies. By using a saliency analysis, the false positive alarms are eliminated completely. The experimental results demonstrate the robustness and effectiveness of the proposed method using various measured MMW/THz images.