Long-wavelength infrared (LWIR) photodetection is important for heat-seeking technologies, such as thermal imaging, all-weather surveillance, and missile guidance. Among various detection techniques, photothermoelectric (PTE) detectors are promising in that they can realize ultra-broadband photodetection at room temperature without an external power supply. However, their performance in terms of speed, responsivity, and noise level in the LWIR regime still needs further improvement. Here, we demonstrated a high-performance PTE photodetector based on low-symmetry palladium selenide (PdSe2) with asymmetric van der Waals contacts. The temperature gradient induced by asymmetric van der Waals contacts even under global illumination drives carrier diffusion to produce a photovoltage via the PTE effect. A responsivity of over 13 V/W, a response time of ∼50 μs, and a noise equivalent power of less than 7 nW/Hz1/2 are obtained in the 4.6-10.5 μm regime at room temperature. Furthermore, due to the anisotropic absorption of PdSe2, the detector exhibits a linear polarization angle sensitive response with an anisotropy ratio of 2.06 at 4.6 μm and 1.21 at 10.5 μm, respectively. Our proposed device architecture provides an alternative strategy to design high-performance photodetectors in the LWIR regime by utilizing van der Waals layered materials.