A visual system for classifying different types of dates is extremely important for the buyer and the customer. Practical success in this field is still limited, and better outcomes are essentially required. This paper aims to classify all kinds of date fruits to help the customer and the buyer identify these types. The framework of this paper consists of a single model for the classification of date fruits, which is based on using the CNN method. This method is utilized with the transfer of learning and fine-tuning depending on the previous models and new models. To build an accurate and robust visual system, a rich image data set of the date fruit varieties is generated. The dataset contains significant differences, including differences in angles, dimensions, and lighting. In this paper, the CNN method is used to identify and classify various types of date fruits. Using this method produced high-accuracy results and a few errors. In this work, the database includes nine types of dates, whereby only six types were taken for training. High accuracy results were obtained from training these types of dates. The achieved accuracy of the results is 99%.