The development of organic materials with high thermoelectric performance requires the exploration of organic molecules having various structures and a deep understanding of the relationship between their structure and thermoelectric properties. In this study, a series of donor–acceptor-type conjugated copolymers with an isoindigo (IID) unit as a building block are synthesized and compared to investigate the effect of the molecular structure of the IID-based conjugated polymers on the microstructure and thermoelectric properties of polymer films. The crystallinity and molecular packing of the polymers change significantly depending on the symmetry of the donor unit copolymerized with the IID unit, resulting in considerable differences in the doping and charge transport characteristics of the polymers. Importantly, the polymer with high crystallinity and fibril-like crystalline network shows excellent thermoelectric performance, exhibiting a high power factor of 212 µW m−1 K−2 upon doping; this is attributed to the high electrical conductivity and high Seebeck coefficient, which result from its highly ordered structure. This study not only demonstrates the potential of the IID unit as a promising building block for high-performance conjugated polymers for thermoelectric applications but also provides important guidelines for the design of IID-based polymers having high thermoelectric performance.