A geometric equation of borehole deformation under stress was deduced based on the basic theory of elasticity. Subsequently, we established the quantitative relationship between the in situ stress and geometrical parameters of borehole deformation. Furthermore, we proposed an in situ stress prediction model based on borehole deformation. Additionally, numerical simulations of borehole morphology in different lithologies under in situ stress were conducted to analyze the deformation effect. Logging parameters that are sensitive to the shear wave time difference, such as longitudinal wave time difference, density, and natural gamma radiation, were selected for training using an artificial neural network (ANN) to predict the shear wave time difference. The results demonstrated that 1) combining the theoretical derivation and numerical simulation, the borehole geometry under stress was quasi-elliptic, and 2) compared with the existing shear wave time difference curve, the predicted geometry by the ANN was consistent with the actual geometry. Consequently, compared with the tested data from acoustic emission, the overall error of the in situ stress predicted using the new method was less than 9.2%. Moreover, the accuracy of the coal seam was the highest, wherein the average errors of the maximum and minimum horizontal principal stresses were 2.01 and 2.56%, respectively, which confirms the feasibility of the proposed method.