Hole transporting layer-free perovskite solar cells (HTL-free PVSCs) have great potential in the commercialization process due to their simple device structure and low processing temperature. However, the photovoltaic performance of the HTL-free PVSCs is limited by severe carrier recombination and energy loss (Eloss) caused by the large energy level mismatching at the ITO/MAPbI3 interface. In this work, we introduce three dipole interlayers, including 4-Fluorophenylacetic acid, 4-(Trifluoromethyl) phenylacetic acid, and 3,5-Bis(trifluoromethyl)phenylacetic acid, to tune the surface work function (WF) of the ITO substrates by forming interfacial dipoles and shifting the Fermi level of the ITO substrates. The carboxyl groups of these dipole interlayers can passivate the surface terminal –OH groups on the ITO surface. The optimized energy level alignment and reduced defect states at the ITO/MAPbI3 interface are beneficial to suppressing the carrier recombination and Eloss at the ITO/MAPbI3 interface. Besides, the dipole interlayers change the wettability of the ITO substrates and facilitate the formation of high-quality and fewer trap state perovskite films. The dipole interlayers modified HTL-free PVSCs achieve the highest power conversion efficiency (PCE) of 20.19%, which is higher than the pristine PVSCs with the highest PCE of 14.94%. The dipole interlayers modified flexible and large area HTL-free PVSCs also exhibit enhanced PCE compared with the pristine PVSCs. Besides, the stability of the dipole interlayer modified PVSCs is better than the pristine PVSCs. This work provides an in-depth study on the Eloss at the ITO/MAPbI3 interface and promotes the fabrication of efficient and stable HTL-free PVSCs.