COVID-19 disease is a major health calamity in twentieth century, in which the infection is spreading at the global level. Developing countries like Bangladesh, India, and others are still facing a delay in recognizing COVID-19 cases. Hence, there is a need for immediate recognition with perfect identification of infection. This clear visualization helps to save the life of suspected COVID-19 patients. With the help of traditional RT-PCR testing, the combination of medical images and deep learning classifiers delivers more hopeful results with high accuracy in the prediction and recognition of COVID-19 cases. COVID-19 disease is recently researched through sample chest X-ray images, which have already proven its efficiency in lung diseases. To emphasize corona virus testing methods and to control the community spreading, the automatic detection process of COVID-19 is processed through the detailed medication reports from medical images. Although there are numerous challenges in the manual understanding of traces in COVID-19 infection from X-ray, the subtle differences among normal and infected X-rays can be traced by the data patterns of Convolutional Neural Network (CNN). To improve the detection performance of CNN, this paper plans to develop an Ensemble Learning with CNN-based Deep Features (EL-CNN-DF). In the initial phase, image scaling and median filtering perform the pre-processing of the chest X-ray images gathered from the benchmark source. The second phase is lung segmentation, which is the significant step for COVID detection. It is accomplished by the Adaptive Activation Function-based U-Net (AAF-U-Net). Once the lungs are segmented, it is subjected to novel EL-CNN-DF, in which the deep features are extracted from the pooling layer of CNN, and the fully connected layer of CNN are replaced with the three classifiers termed “Support Vector Machine (SVM), Autoencoder, Naive Bayes (NB)”. The final detection of COVID-19 is done by these classifiers, in which high ranking strategy is utilized. As a modification, a Self Adaptive-Tunicate Swarm Algorithm (SA-TSA) is adopted as a boosting algorithm to enhance the performance of segmentation and detection. The overall analysis has shown that the precision of the enhanced CNN by using SA-TSA was 1.02%, 4.63%, 3.38%, 1.62%, 1.51% and 1.04% better than SVM, autoencoder, NB, Ensemble, RNN and LSTM respectively. The comparative performance analysis on existing model proves that the proposed algorithm is better than other algorithms in terms of segmentation and classification of COVID-19 detection.