A high power conversion efficiency (PCE) and long-term stability are crucial for the commercialization of all-polymer solar cells (all-PSCs). In this study, we develop two new polymerized small-molecule acceptors (PSMAs) that include vinylene and fluorinated vinylene linkers (PYBO-VH and PYBO-VF, respectively), to achieve all-PSCs with high PCE (> 18.5%) and thermal stability (T80% lifetime > 1800 h at 120 °C). The PYBO-VF with fluorinated vinylene linkers has more planar backbones than PYBO-VH with vinylene linkers. Thus, PYBO-VF exhibits an 1.5-fold increase in electron mobility (μe = 4.8 × 10−4 cm2 V−1 s−1) and higher glass transition temperature (Tg = 200 °C) compared with PYBO-VH (μe = 3.1 × 10−4 cm2 V−1 s−1 and Tg = 158 °C). As a result, all-PSCs based on PYBO-VF demonstrate superior PCE (17.16%) and thermal stability (T80% lifetime = 2270 h) than those based on PYBO-VH (PCE = 16.44% and T80% lifetime = 1385 h). Importantly, when both PYBO-VH and PYBO-VF are used for constructing ternary all-PSCs, we successfully demonstrate a high-performance and stable all-PSC with a PCE of 18.53% and thermal stability at 120 °C (T80% lifetime = 1840 h). The ternary all-PSCs have one of the best performances among all-PSCs reported to date.