Abstract The Indonesian oil palm plantation experiences a production rate of fresh fruit bunches and crude palm oil (CPO) influenced by the quality of fruits with excellent and uniform maturity accepted by the palm oil processing factory. The uniformity of fruit quality produced from the harvesting process through fruit maturity criteria, namely fractions 00, fraction 1, fraction 3, and fraction 4, necessitates the need for maturity detection based on deep learning with a pre-trained YOLOv5 model capable of detecting the maturity of new oil palm fruit bunches. The aim is to develop a deep learning-based maturity detection model based on visual images and analyse the deep learning algorithm’s performance by classifying the maturity level of new oil palm fruit bunches. The method collects a dataset of new oil palm fruit bunches with 4180 images. The collected dataset will be annotated with Roboflow, so the annotation results will undergo training, validation, and testing processes using the deep learning script. The result obtained for the data found to detect the ripeness levels of Fresh Fruit Bunches (FFB), shows that the YOLOv5 model demonstrates strong performance with an accuracy reaching 72.5% in testing, which falls into the category of fair annotation. These results indicate that the system created and used is functioning well.