While digital ocular fundus images are commonly used for diagnosing ocular tumors, interpreting these images poses challenges due to their complexity and the subtle features specific to tumors. Automated detection of ocular tumors is crucial for timely diagnosis and effective treatment. This study investigates a robust deep learning system designed for classifying ocular tumors. The article introduces a novel optimizer that integrates the Caputo fractional gradient descent (CFGD) method with the cuckoo search algorithm (CSA) to enhance accuracy and convergence speed, seeking optimal solutions. The proposed optimizer’s performance is assessed by training well-known Vgg16, AlexNet, and GoogLeNet models on 400 fundus images, equally divided between benign and malignant classes. Results demonstrate the significant potential of the proposed optimizer in improving classification accuracy and convergence speed. In particular, the mean accuracy attained by the proposed optimizer is 86.43%, 87.42%, and 87.62% for the Vgg16, AlexNet, and GoogLeNet models, respectively. The performance of our optimizer is compared with existing approaches, namely stochastic gradient descent with momentum (SGDM), adaptive momentum estimation (ADAM), the original cuckoo search algorithm (CSA), Caputo fractional gradient descent (CFGD), beetle antenna search with ADAM (BASADAM), and CSA with ADAM (CSA-ADAM). Evaluation criteria encompass accuracy, robustness, consistency, and convergence speed. Comparative results highlight significant enhancements across all metrics, showcasing the potential of deep learning techniques with the proposed optimizer for accurately identifying ocular tumors. This research contributes significantly to the development of computer-aided diagnosis systems for ocular tumors, emphasizing the benefits of the proposed optimizer in medical image classification domains.