Broad‐response and high‐detectivity for all‐polymer photodetectors based on p‐ and n‐type semiconducting polymers have been achieved through optimization of polymer property and film microstructure. The electron‐donating units in the p‐type polymers affect a great deal of the polymer properties such as solubility, absorption spectra, and electronic energy levels, which in turn can influence the device performance. The polymer (P3) based on dithienopyrrole and diketopyrrolopyrrole is most promising for photodetector applications, as it possesses the suitable energy level with regard to the n‐type polymer and exhibits appropriate film morphology and molecular stacking with aid of 1,8‐diiodooctane as an additive during film processing. The photodetector based on P3/poly{[N,N′‐bis(2‐octyldodecyl)‐naphthalene‐1,4,5,8‐bis(dicarboximide)‐2,6‐diyl]‐alt‐5,5′‐(2,2′‐bithiophene)} (PNDI) exhibits good response from 300 to 1100 nm and nearly constant detectivity (D*) above 1012 Jones at 330–980 nm, rendering a great potential of all‐polymer photodetectors for practical applications. image