Infrared photodetectors are finding widespread applications in telecommunication, motion detection, chemical sensing, thermal imaging and bio‐medical imaging, etc. The nanostructured materials and architectures are attracting extensive interests in photodetectors in view of the potential benefits from confined light‐matter interaction, fast carrier dynamics and ultrahigh photoconductive gains. This review concentrates on the photodetection in the infrared spectrum and recent progresses in constructing advanced infrared photodetectors based on quantum wells, dots, and the rapidly evolving 1D and 2D materials are summarized. The recent achievements in exploring nanostructured plasmonic metamaterials for the intriguing subwavelength photon confinement and waveguides in devices are also surveyed considering their importance in device integration. An outlook of infrared photodetection is given in the end as a guideline for this vigorous field.