2D sliding ferroelectric semiconductors have greatly expanded the ferroelectrics family with the flexibility of bandgap and material properties, which hold great promise for ultrathin device applications that combine ferroelectrics with optoelectronics. Besides the induced different resistance states for non-volatile memories, the switchable ferroelectric polarizations can also modulate the photogenerated carriers for potentially ultrafast optoelectronic devices. Here, it is demonstrated that the room temperature sliding ferroelectricity can be used for ultrafast switchable photovoltaic response in ε-InSe layers. By first-principles calculations and experimental characterizations, it is revealed that the ferroelectricity with out-of-plane (OOP) polarization only exists in even layer ε-InSe. The ferroelectricity is also demonstrated in ε-InSe-based vertical devices, which exhibit high on-off ratios (≈104) and non-volatile storage capabilities. Moreover, the OOP ferroelectricity enables an ultrafast (≈3ps) bulk photovoltaic response in the near-infrared band, rendering it a promising material for self-powered reconfigurable and ultrafast photodetector. This work reveals the essential role of ferroelectric polarization on the photogenerated carrier dynamics and paves the way for hybrid multifunctional ferroelectric and optoelectronic devices.