A network model was proposed for identifying the growth posture of young peach fruits in orchards (YOLON) with the YOLOv5 network model in view of identifying the growth posture of young peach fruits in the natural growth state of orchards. First of all, Coordinate Attention (CA) was added based on the original backbone network due to the influence of peach young fruits on the near-color background to enhance the feature extraction ability of the image. Then, a small-target detection layer was added to the backbone network to optimize the detection effect of smaller young fruits. Finally, the rotating frame was selected to replace the original horizontal frame to detect the young fruits with directional attributes. Experiments showed that proposed YOLON in the visual recognition of the growth posture of multi-directional peach young fruits had the average accuracy rate of the recognition of peach young fruit and the estimation error of its growth angle of 93.04% and ±3.94°, respectively. It has a good recognition effect, which provides a reference for the visual recognition of the growth posture of other young fruits and vegetables.