Imagers with pre-processing functions, such as image recognition and classification, contrast enhancement, and noise reduction, play a critical role in the neuromorphic visual system. Optoelectronic plasticity is a prerequisite to achieve these functions. In this study, we demonstrate a nonvolatile reconfigurable broadband photodetector based on a ferroelectric heterostructure composed of BP (black phosphorus)/α-In2Se3. The plasticity of the device comes from the ferroelectric polarization of α-In2Se3 that can tune the built-in potential of the p–n junction. As a result, the rectification ratio and responsivity increase almost one order when changing the gate voltage pulse from +16 V to −16 V. Due to the introduction of BP, the device has a wide spectral response covering 473–1550 nm. In addition, our devices show excellent performance in terms of a high responsivity of up to 4.73 × 104 A/W, a large specific detectivity of ∼2.09 × 1012 Jones, a high external quantum efficiency of 9.21 × 106%, and a notable photo-on-off ratio of 4.82 × 103. Due to its high performance, reconfigurability, and broadband response, our device shows considerable potential in neuromorphic visual systems even in the infrared region.