Pipeline pullback load is a crucial basis for drill rig selection and pipeline strength design. This paper presents a new pullback load calculation model from the perspective of pipe–borehole wall contact. The pipe–borehole wall contact analysis includes the distribution of contact pressure and the relationship between the external load and compressive displacement. The friction force between the pipe and the borehole wall was calculated based on the pipe–borehole wall contact analysis and adhesion theory without depending on the empirical friction coefficient. The effects of the eccentricity were also considered when calculating the fluid drag force. Through case studies, we verified the applicability of the model and discussed the possible reasons for the errors between the theoretical and field-measured results. This study can provide a helpful tool for analyzing the pipe–borehole wall contact and pullback loads for horizontal directional drilling.