Passive millimeter wave (PMMW) imaging which owes a lot of advantages over infa-red and optical imaging has been widely applied in many fields. But due to the confinement of the antenna aperture size, the acquired PMMW image is often noisy and in low resolution. A novel PMMW image denoising algorithm which combines the high-dimensional mean median filter and the adaptive manifolds is proposed in this paper. Our method firstly applies the weighted traversal algorithm to the input image for generating a high-dimensional image data. To accelerate the algorithm, the PCA based dimension reduction is performed subsequently to extract the main information and. Then the PCA retained high-dimensional data is processed by the proposed mean median filter which computes the mean of pixels near the centered median pixel in each dimension. The adaptive manifolds are generated based on the input image and the PCA retained image data. Finally, the adaptive manifolds filter is used to produce the final response. The experiment results indicate that our method has a good ability in removing the noise with edge preserving and is efficient for both the simulated images and the real PMMW image.