The emerging non-fullerene acceptors with low voltage losses have pushed the power conversion efficiency of organic solar cells (OSCs) to ≈20% with auxiliary morphology optimization. Thermal annealing (TA), as the most widely adopted post-treatment method, has been playing an essential role in realizing the potential of various material systems. However, the procedure of TA, i.e., the way that TA is performed, is almost identical among thousands of OSC papers since ≈30 years ago other than changes in temperature and annealing time. Herein, a reverse thermal annealing (RTA) technique is developed, which can enhance the dielectric constant of active layer film, thereby producing a smaller Coulomb capture radius (14.93 nm), meanwhile, forming a moderate nano-scale phase aggregation and a more favorable face-on molecular stacking orientation. Thus, this method can reduce the decline in open circuit voltage of the conventional TA method by achieving decreased radiative (0.334 eV) and non-radiative (0.215 eV) recombination loss. The power conversion efficiency of the RTA PM6:L8-BO-X device increases to 19.91% (certified 19.42%) compared to the TA device (18.98%). It is shown that this method exhibits a superb universality in 4 other material systems, revealing its dramatic potential to be employed in a wide range of OSCs.