Identifying olive cultivars and maturity stages is crucial in the olive industry, as these traits significantly impact the nutritional and sensory properties of olive products and extracted oil. For this purpose, this study presents a novel automatic computer vision system that applies state-of-the-art deep learning technology to sort and classify two Iranian olive cultivars, Zard and Roghani, in five maturity stages, resulting in a total of ten distinct classes. The model was developed by evaluating multiple user-defined and standard structures. It was based on a dual-path lightweight convolutional neural network that uses both regular and dilated convolution operators. Dilated convolutions were used to extract more information and capture different properties by providing larger receptive fields. With a significantly lower number of trainable parameters than standard architectures, the lightweight nature of the model would enhance its potential for delivering fast responses in on-the-go applications. Four optimizers (RMSProp, SGD, Adam, and Nadam) were tested on the developed model to enhance its performance, and Nadam exhibited the greatest accuracy. The proposed model achieved a total classification accuracy of 95.79 % and a loss of 0.2214. The proposed model was completely accurate for some classes, and the classification metrics for all categories were high, ranging from 88 % to 100 % for precision, 83–100 % for recall, and 86–100 % for F1-score. The accuracy of classification within the Roghani cultivar classes stood at 98.28 %, while for the Zard cultivar classes, it achieved 97.76 %. The study found that the proposed model can be efficiently incorporated into an olive sorting system, facilitating the identification of olives with different cultivars and varying levels of maturity, thereby enhancing the production of post-harvest products and superior quality of oil.