Early diagnosis of oral cancer is crucial for improving patient outcomes and saving lives. However, inaccurate and improper diagnosis can hinder effective treatment. This paper presents a novel method for detecting oral cancer using an optimized version of Convolutional Neural Network (CNN). While basic CNNs have been widely used for image classification tasks, the incorporation of the Seagull Optimization Algorithm and Particle Swarm Optimization Algorithm in optimizing the CNN architecture specifically for oral cancer detection is a unique approach that is provided in this study. By combining these algorithms, the proposed method optimizes the CNN's architecture, parameters, and training process specifically for oral cancer detection. This optimization enhances the performance and accuracy of the CNN in identifying cancerous regions in oral images. Unlike previous approaches, our method incorporates advanced image processing techniques, including noise reduction, contrast enhancement, and data augmentation, to enhance the quality of input data extracted from the Oral Cancer (Lips and Tongue) images (OCI) dataset. The optimized CNN architecture uses its ability to learn intricate patterns and features from the enhanced images, enabling more accurate identification of cancerous regions. To evaluate the effectiveness of our approach, we compare it against Textural analysis, FCM, CNN, R-CNN, and ResNet-101 using four measurement indices. Results demonstrate that our proposed CSOA-based CNN system achieves the highest accuracy rate (96.94%) compared to other methods, indicating its superior performance in oral cancer detection. Furthermore, our precision rate of 94.65% and recall rate of 91.60% highlight the model's high correctness and positive classification ability. Finally, our proposed method achieves the highest F1-score (88.55%), emphasizing its superiority over other comparative methods. Through our innovative integration of the Seagull Optimization Algorithm and Particle Swarm Optimization Algorithm with CNN, coupled with advanced image processing techniques, we provide a reliable and effective solution for early detection of oral cancer.