Inverted polymer solar cells (PSCs) with high open-circuit voltages of 1.00-1.06 V are fabricated by using an indenofluorene-containing copolymer (PIFTBT8) as an electron donor material and [6,6]-phenyl-C71-butyric acid methyl ester (PC71BM) as an electron acceptor material. To improve the photovoltaic performance, interface control of various low-temperature processed ZnO films as cathode buffer layers is systematically investigated for effective electron transportation, while transition metal oxides including MoO3, WO3, NiO, and Cu2O are employed as anode buffer layers for hole-extraction. Incorporation of optimized semiconducting metal oxide interlayers can minimize interfacial power losses, which thus affords large open-circuit voltages (Voc), increased short-circuit current densities (Jsc), and fill factors (FF), eventually contributing to higher power conversion efficiencies (PCEs) as well as better device stability. Due to the improved interfacial contacts and fine-matching energy levels, inverted PSCs with a device configuration of ITO/ZnO/PIFTBT8:PC71BM/MoO3/Ag exhibit a high PCE of 5.05% with a large Voc of 1.04 V, a Jsc of 9.74 mA cm(-2), and an FF of 50.1%. For the single junction inverted PSCs with efficiencies over 5.0%, 1.04 V is the largest Voc ever achieved. By controlling the processing conditions of the active layer, the Voc can further be improved to 1.05 and 1.06 V, with PCEs of 4.70% and 4.18%, respectively. More importantly, the inverted PSCs are ascertained to maintain a PCE of 4.55% (>90% of its initial efficiency) and a Voc of 1.05 V over 180 days, demonstrating good long-term stability, which is much better than that of the conventional devices. The results suggest that the interface engineering of metal oxide interlayers is an important strategy to develop PSCs with good performance.