Lung cancer is the uncontrolled growth of cells in the lung that is made up of two spongy organs located in the chest. These cells may penetrate outside the lung in a process called metastasis and spread to the tissues and organs of the body and increase the risk of death from this disease. CT scan of pulmonary nodules is one of the methods of early detection of lung cancer. One of the main challenges in diagnosing pulmonary nodules is the difficulty of identifying and distinguishing pulmonary nodules from lung components. In this study, a computer-aided detection system is introduced to identify these nodules. In the study, after image preprocessing, an image segmentation based on Otsu followed by mathematical morphology is proposed. Then, optimal features are selected based on a new metaheuristic method. Consequently, the characteristics are injected into an improved convolutional neural network (CNN)-based classifier to provide a high accuracy diagnosis system. The optimization of the Otsu method, feature selection, and CNN classifier is established by a new modified version of the Red Fox Optimizer (RFO) algorithm. The approach is then applied to three popular lung cancer datasets and the results are compared with three state-of-the-art methods to show the proposed method's higher efficiency.