All‐polymer photodetectors possess unique mechanical flexibility and are ideally suitable for the application in next‐generation flexible, wearable short‐wavelength infrared (SWIR, 1000–2700 nm) photodetectors. However, all‐polymer photodetectors commonly suffer from low sensitivity, high noise, and low photoresponse speed in the SWIR region, which significantly diminish their application potential in wearable electronics. Herein, two polymer acceptors with absorption beyond 1000 nm, namely P4TOC‐DCBT and P4TOC‐DCBSe, were designed and synthesized. The two polymers possess rigid structure and good conformational stability, which is beneficial for reducing energetic disorder and suppressing dark current. Owing to the efficient charge generation and ultralow noise current, the P4TOC‐DCBT‐based all‐polymer photodetector achieved a specific detectivity () of over 1012 Jones from 650 (visible) to 1070 nm (SWIR) under zero bias, with a response time of 1.36 μs. These are the best results for reported all‐polymer SWIR photodetectors in photovoltaic mode. More significantly, the all‐polymer blend films exhibit good mechanical durability, and hence the P4TOC‐DCBT‐based flexible all‐polymer photodetectors show a small performance attenuation (<4%) after 2000 cycles of bending to a 3 mm radius. The all‐polymer flexible SWIR organic photodetectors are successfully applied in pulse signal detection, optical communication and image capture.