Four conjugated copolymers with phthalimide (PhI) or thieno[3,4-c]pyrrole-4,6-dione (TPD) as the acceptor, thiophene (T) or selenophene (Se) as the spacer and 3,3′-didodecyl-2,2′-bithiophene (BT) as the common donor, namely, PPhI-T, PPhI-Se, PTPD-T and PTPD-Se, have been synthesized and the effects of intra- and intermolecular interactions on the optical properties, molecular stacking, and organic electronic device performances were investigated. The intramolecular S(Se)⋯O (carbonyl) interactions between the spacer and the PhI's or TPD's carbonyl and the intermolecular reciprocity between the polymeric backbones differ from each other as the spacer and the acceptor were varied. Among the four polymers, PPhI-T with the weakest intramolecular S⋯O interaction and intermolecular backbone reciprocity exhibited the poorest photovoltaic performance with a PCE of 0.31%. When the T spacer was replaced by the more polarized Se spacer, the resultant copolymer PPhI-Se exhibited stronger intra- and intermolecular interactions, resulting in better optical properties with a PCE of 0.94% when blended with PC71BM. When PhI is replaced with the more polarized TPD unit, the TPD-based polymers, PTPD-T and PTPD-Se, showed even better coplanarity compared to that of the PhI-based polymers, with a PCE of 2.04% for PTPD-T and 1.52% for PTPD-Se blended with PC71BM. To the best of our knowledge, this is the first systematic study on the influences of structure-manipulated molecular stacking on solid-state optical properties and electronic device performance through modulations of the intramolecular and intermolecular interactions.