Two N-annulated perylene diimide (PDI) dimers, termed as, TVT-(PPDI-B)2 and TYT-(PPDI-B)2, selecting (E)-1,2-di(thien-2-yl)ethylene (TVT) and 1,2-di(thien-2-yl)acetylene (TYT) as the middle linkers, N-butyl-[1,12-b,c,d]pyrrole-N,Nʹ-bis(2-hexyldecyl)perylene diimide (PPDI-B) fused aromatic moiety as two wings, were developed to investigate the influence of linking core. It was exhibited that the improved absorption, 0.05 eV up-shifted ELUMO, drastically reduced twisting angle from 68.0o to 27.9o between two N-annulated PDI moieties and enhanced molecular aggregation were found after substituting TVT linking core with TYT. Accordingly, under the best processing condition, TVT-(PPDI-B)2-based device gained the VOC of 0.92, JSC of 3.61 mA‧cm−2, FF of 30.3% and consequently PCE of 1.02% when blending with PTB7-Th. In contrast, TYT-(PPDI-B)2-based device afforded 4.35% increased VOC of 0.96 V, 86.98% enhanced JSC of 6.75 mA‧cm−2 and 55.45% raised FF of 47.1%, collectively contributed to a 211% raised PCE as high as 3.17%. This efficiency enhancement was mainly attributed to improved absorption, raised ELUMO, more efficient exciton dissociation and electron mobility, weakened charge recombination, and more suitable morphology. Interestingly, 0.12⁓0.16 V up-shifted VOC was achieved after applying N-annulation strategy at the bay-position of PDI building block. This indicates that precisely tuning the linking core in the N-annulated PDI-based dimers can effectively affect optoelectronic property, molecular geometry and morphology, and thus further boost the corresponding device efficiency.