Oil-based mud is widely used in shale oil well drilling because of its good stability and high temperature resistance at high temperature. However, during the drilling process of shale oil wells, fluid displacement between the annulus and the formation occurs. The existing temperature field model cannot predict the temperature distribution of the wellbore annulus under this condition. Based on the equations of heat conduction and conservation of energy, a new model of wellbore heat transfer in shale oil wells considering the displacement between downhole fluids is established in this paper. By comparing the temperature distribution of annular fluid under normal circulation, lost circulation and overflow, the accuracy of this model is confirmed. Taking the parameters of a shale oil well in a certain area as an example, the factors that affect the temperature of the annular fluid are analyzed and solved. Numerical analysis and simulation results show that the flow rate remains unchanged, and the annular temperature decreases by about 1.42 °C/L when the flow rate of the annulus increases. As the specific heat capacity of shale oil increases, the annular temperature increases by about 0.0012 °C/J/(kg·°C). In addition, when displacement between fluids occurs in the upper open-hole segment, there is an inflection point on the temperature distribution curve of the annular fluid, and the location of the inflection point is consistent with the location of the fluid displacement point. The results can provide a theoretical reference for drilling in complex formations with frequent formation-annulus fluid displacement, leakage and overflow.