Convolutional Neural Networks Architecture Design Research Articles

Automated 3D convolutional neural network architecture design using genetic algorithm for pulmonary nodule classification

Cancer of the lungs is considered one of the primary causes of death among patients globally. Early detection contributes significantly to the success of pulmonary cancer treatment. To aid the pulmonary nodule classification, many models for the analysis of medical image utilizing deep learning have been developed. Convolutional neural network (CNN) recently, has attained remarkable results in various image classification tasks. Nevertheless, the CNNs performance is heavily dependent on their architectures which still heavily reliant on human domain knowledge. This study introduces a cutting-edge approach that leverages genetic algorithms (GAs) to automatically design 3D CNN architectures for differentiation between benign and malignant pulmonary nodules. The suggested algorithm utilizes the dataset of lung nodule analysis 2016 (LUNA16) for evaluation. Notably, our approach achieved exceptional model accuracy, with evaluations on the testing dataset yielding up to 95.977%. Furthermore, the algorithm exhibited high sensitivity, showcasing its robust performance in distinguishing between benign and malignant nodules. Our findings demonstrate the outstanding capabilities of the proposed algorithm and show an outstanding performance and attain a state of art solution in lung nodule classification.

A Cell-Based Fast Memetic Algorithm for Automated Convolutional Neural Architecture Design.

Neural architecture search (NAS) has attracted much attention in recent years. It automates the neural network construction for different tasks, which is traditionally addressed manually. In the literature, evolutionary optimization (EO) has been proposed for NAS due to its strong global search capability. However, despite the success enjoyed by EO, it is worth noting that existing EO algorithms for NAS are often very computationally expensive, which makes these algorithms unpractical in reality. Keeping this in mind, in this article, we propose an efficient memetic algorithm (MA) for automated convolutional neural network (CNN) architecture search. In contrast to existing EO algorithms for CNN architecture design, a new cell-based architecture search space, and new global and local search operators are proposed for CNN architecture search. To further improve the efficiency of our proposed algorithm, we develop a one-epoch-based performance estimation strategy without any pretrained models to evaluate each found architecture on the training datasets. To investigate the performance of the proposed method, comprehensive empirical studies are conducted against 34 state-of-the-art peer algorithms, including manual algorithms, reinforcement learning (RL) algorithms, gradient-based algorithms, and evolutionary algorithms (EAs), on widely used CIFAR10 and CIFAR100 datasets. The obtained results confirmed the efficacy of the proposed approach for automated CNN architecture design.

Comparative Study of Metaheuristic Optimization of Convolutional Neural Networks Applied to Face Mask Classification

The preventive measures taken to curb the spread of COVID-19 have emphasized the importance of wearing face masks to prevent potential infection with serious diseases during daily activities or for medical professionals working in hospitals. Due to the mandatory use of face masks, various methods employing artificial intelligence and deep learning have emerged to detect whether individuals are wearing masks. In this paper, we utilized convolutional neural networks (CNNs) to classify the use of face masks into three categories: no mask, incorrect mask, and proper mask. Establishing the appropriate CNN architecture can be a demanding task. This study compares four swarm intelligent metaheuristics: particle swarm optimization (PSO), grey wolf optimizer (GWO), bat algorithm (BA), and whale optimization algorithm (WOA). The CNN architecture design involves determining the essential hyperparameters of the CNNs. The results indicate the effectiveness of the PSO and BA in achieving an accuracy of 100% when using 10% of the images for testing. Meanwhile, when 90% of the images were used for testing, the results were as follows: PSO 97.15%, WOA 97.14%, BA 97.23%, and GWO 97.18%. These statistically significant differences demonstrate that the BA allows better results than the other metaheuristics analyzed in this study.

Wavelet analysis model inspired convolutional neural networks for image denoising

In recent years, many image denoising methods have been proposed based on convolutional neural networks (CNNs). While these methods have shown continuous performance improvement by introducing various mechanisms and structures, their computational cost tends to become increasingly expensive, owing to the resulting complex network architectures. This paper aims at winning the trade-off between computational efficiency and denoising performance for CNN-based image denoisers. Towards this end, we draw inspirations from traditional variational models with wavelet analysis operators for CNN architecture design. A model-inspired CNN is proposed with four key modules: iterative encoding-decoding units inspired by the iterative denoising process, directional convolutions inspired by the separable wavelet filters, inception modules inspired by the multi-scale analysis of wavelets, and stage-wise connections inspired by the adding noise back operation. In experiments, our CNN shows high computational efficiency in both training and test, with competitive results to state-of-the-art approaches.

A Metaheuristic Harris Hawks Optimization Algorithm for Weed Detection Using Drone Images

There are several major threats to crop production. As herbicide use has become overly reliant on weed control, herbicide-resistant weeds have evolved and pose an increasing threat to the environment, food safety, and human health. Convolutional neural networks (CNNs) have demonstrated exceptional results in the analysis of images for the identification of weeds from crop images that are captured by drones. Manually designing such neural architectures is, however, an error-prone and time-consuming process. Natural-inspired optimization algorithms have been widely used to design and optimize neural networks, since they can perform a blackbox optimization process without explicitly formulating mathematical formulations or providing gradient information to develop appropriate representations and search paradigms for solutions. Harris Hawk Optimization algorithms (HHO) have been developed in recent years to identify optimal or near-optimal solutions to difficult problems automatically, thus overcoming the limitations of human judgment. A new automated architecture based on DenseNet-121 and DenseNet-201 models is presented in this study, which is called “DenseHHO”. A novel CNN architecture design is devised to classify weed images captured by sprayer drones using the Harris Hawk Optimization algorithm (HHO) by selecting the most appropriate parameters. Based on the results of this study, the proposed method is capable of detecting weeds in unstructured field environments with an average accuracy of 98.44% using DenseNet-121 and 97.91% using DenseNet-201, the highest accuracy among optimization-based weed-detection strategies.

Embedding channel pruning within the CNN architecture design using a bi-level evolutionary approach.

Remarkable advancements have been achieved in machine learning and computer vision through the utilization of deep neural networks. Among the most advantageous of these networks is the convolutional neural network (CNN). It has been used in pattern recognition, medical diagnosis, and signal processing, among other things. Actually, for these networks, the challenge of choosing hyperparameters is of utmost importance. The reason behind this is that as the number of layers rises, the search space grows exponentially. In addition, all known classical and evolutionary pruning algorithms require a trained or built architecture as input. During the design phase, none of them consider the process of pruning. In order to assess the effectiveness and efficiency of any architecture created, pruning of channels must be carried out before transmitting the dataset and computing classification errors. For instance, following pruning, an architecture of medium quality in terms of classification may transform into an architecture that is both highly light and accurate, and vice versa. There exist countless potential scenarios that could occur, which prompted us to develop a bi-level optimization approach for the entire process. The upper level involves generating the architecture while the lower level optimizes channel pruning. Evolutionary algorithms (EAs) have proven effective in bi-level optimization, leading us to adopt the co-evolutionary migration-based algorithm as a search engine for our bi-level architectural optimization problem in this research. Our proposed method, CNN-D-P (bi-level CNN design and pruning), was tested on the widely used image classification benchmark datasets, CIFAR-10, CIFAR-100 and ImageNet. Our suggested technique is validated by means of a set of comparison tests with regard to relevant state-of-the-art architectures.

Lightweight convolutional neural network architecture design for music genre classification using evolutionary stochastic hyperparameter selection

AbstractConvolutional neural networks (CNNs) have succeeded in various domains, including music information retrieval (MIR). Music genre classification (MGC) is one such task in the MIR that has gained attention over the years because of the massive increase in online music content. Accurate indexing and automatic classification of these large volumes of music content require high computational resources, which pose a significant challenge to building a lightweight system. CNNs are a popular deep learning‐based choice for building systems for MGC. However, finding an optimal CNN architecture for MGC requires domain knowledge both in CNN architecture design and music. We present MGA‐CNN, a genetic algorithm‐based approach with a novel stochastic hyperparameter selection for finding an optimal lightweight CNN‐based architecture for the MGC task. The proposed approach is unique in automating the CNN architecture design for the MGC task. MGA‐CNN is evaluated on three widely used music datasets and compared with seven peer rivals, which include three automatic CNN architecture design approaches and four manually designed popular CNN architectures. The experimental results show that MGA‐CNN surpasses the peer approaches in terms of classification accuracy, parameter numbers, and execution time. The optimal architectures generated by MGA‐CNN also achieve classification accuracy comparable to the manually designed CNN architectures while spending fewer computing resources.

Is it enough to optimize CNN architectures on ImageNet?

Classification performance based on ImageNet is the de-facto standard metric for CNN development. In this work we challenge the notion that CNN architecture design solely based on ImageNet leads to generally effective convolutional neural network (CNN) architectures that perform well on a diverse set of datasets and application domains. To this end, we investigate and ultimately improve ImageNet as a basis for deriving such architectures. We conduct an extensive empirical study for which we train 500 CNN architectures, sampled from the broad AnyNetX design space, on ImageNet as well as 8 additional well-known image classification benchmark datasets from a diverse array of application domains. We observe that the performances of the architectures are highly dataset dependent. Some datasets even exhibit a negative error correlation with ImageNet across all architectures. We show how to significantly increase these correlations by utilizing ImageNet subsets restricted to fewer classes. These contributions can have a profound impact on the way we design future CNN architectures and help alleviate the tilt we see currently in our community with respect to over-reliance on one dataset.

Evolving Optimised Convolutional Neural Networks for Lung Cancer Classification

Detecting pulmonary nodules early significantly contributes to the treatment success of lung cancer. Several deep learning models for medical image analysis have been developed to help classify pulmonary nodules. The design of convolutional neural network (CNN) architectures, however, is still heavily reliant on human domain knowledge. Manually designing CNN design solutions has been shown to limit the data’s utility by creating a co-dependency on the creator’s cognitive bias, which urges the development of smart CNN architecture design solutions. In this paper, an evolutionary algorithm is used to optimise the classification of pulmonary nodules with CNNs. The implementation of a genetic algorithm (GA) for CNN architectures design and hyperparameter optimisation is proposed, which approximates optimal solutions by implementing a range of bio-inspired mechanisms of natural selection and Darwinism. For comparison purposes, two manually designed deep learning models, FractalNet and Deep Local-Global Network, were trained. The results show an outstanding classification accuracy of the fittest GA-CNN (91.3%), which outperformed both manually designed models. The findings indicate that GAs pose advantageous solutions for diagnostic challenges, the development of which may to be fully automated in the future using GAs to design and optimise CNN architectures for various clinical applications.

A Self-Adaptive Mutation Neural Architecture Search Algorithm Based on Blocks

Recently, convolutional neural networks (CNNs) have achieved great success in the field of artificial intelligence, including speech recognition, image recognition, and natural language processing. CNN architecture plays a key role in CNNs' performance. Most previous CNN architectures are hand-crafted, which requires designers to have rich expert domain knowledge. The trial-and-error process consumes a lot of time and computing resources. To solve this problem, researchers proposed the neural architecture search, which searches CNN architecture automatically, to satisfy different requirements. However, the blindness of the search strategy causes a 'loss of experience' in the early stage of the search process, and ultimately affects the results of the later stage. In this paper, we propose a self-adaptive mutation neural architecture search algorithm based on ResNet blocks and DenseNet blocks. The self-adaptive mutation strategy makes the algorithm adaptively adjust the mutation strategies during the evolution process to achieve better exploration. In addition, the whole search process is fully automatic, and users do not need expert knowledge about CNNs architecture design. In this paper, the proposed algorithm is compared with 17 state-of-the-art algorithms, including manually designed CNN and automatic search algorithms on CIFAR10 and CIFAR100. The results indicate that the proposed algorithm outperforms the competitors in terms of classification performance and consumes fewer computing resources.

Designing Convolutional Neural Network Pipeline for Near‐Fault Earthquake Catalog Extension Using Single‐Station Waveforms

AbstractIn this study, we developed an end‐to‐end two‐stage pipeline using 1D convolutional neural networks (CNNs) to detect, localize, and characterize earthquakes from single‐station three‐component waveforms. We are presenting here the insights of what makes the difference in developing a deep learning algorithm by performing an extensive hyperparameter grid search for model training, tackling the question of the optimal number of classes, the importance of training data sets as well as the CNN architecture design in terms of optimal length of the CNN model. Moreover, our pipeline is robust and does not need any preprocessing of the seismograms (e.g., filtering) or any prior knowledge of the region. Training, validation, and evaluation of the CNN models is performed on data recorded at the AQU station placed in the city of L’Aquila in the Abruzzo region (Central Italy). Before MW 6.3 2009 L’Aquila earthquake that occurred near the city of L’Aquila the local catalog of events is sparse. Therefore, we provide a methodological pipeline on how one can extend near‐fault local catalog of earthquakes by applying our two‐stage pipeline on unseen continuous data. Our results show that we are able to design a CNN model that is detecting the earthquake events among random noise waveforms with 97% accuracy (first stage: detection). Additionally, we are able to determine the events that are close to the station (<10 km) with a 94% accuracy as well as identify their belonging to four magnitude classes with a 68% accuracy (second stage: characterization).

Deep convolutional neural network architecture design as a bi-level optimization problem

During the last decade, deep neural networks have shown a great performance in many machine learning tasks such as classification and clustering. One of the most successful networks is the CNN (Convolutional Neural Network), which has been applied in many application domains such as pattern recognition, medical diagnosis, and signal processing. Despite the very interesting performance of CNNs, their architecture design is still so far a major challenge for researchers and practitioners. Several works have been proposed in the literature with the aim to find optimized architectures such as ResNet and VGGNet. Unfortunately, most of these architectures are either manually defined by experts or automatically designed by greedy induction algorithms. Recent works suggest the use of Evolutionary Algorithms (EAs) thanks to their ability to escape locally-optimal architectures. Despite the fact that EAs have shown interesting performance, researchers in this direction have considered the design task as a single-level optimization problem; which represents the main research gap we tackle in this paper. The main contribution behind our work consists in the fact that CNN architecture design has a hierarchical nature and thus could be seen as a Bi-Level Optimization Problem (BLOP) where: (1) the upper level minimizes the network complexity defined by the number of blocks and the number of nodes per block; and (2) the lower level optimizes the convolution block ‘graphs’ topologies by maximizing the classification accuracy. Motivated by the originality of our observation with respect to the state of the art, we frame for the first time the CNN architecture design problem as a BLOP and then solve it using an adapted version of an existing efficient bi-level EA; through the definition of the solution encoding, the fitness function, and the variation operators at each level. The adapted EA is named BLOP-CNN and is assessed on the image classification task using the commonly employed CIFAR-10 and CIFAR-100 benchmark data sets. The analysis of our experimental results show the merits of our proposed method in providing the user with optimized architectures that outperform many recent and prominent architectures coming from the three different approaches, namely: manual design, reinforcement learning-based generation, and evolutionary optimization. Moreover, to show the applicability of our approach, we have conducted a case study on the detection of the COVID-19 using a set of benchmark chest X-ray and Computed Tomography (CT) images.

Multi-Branch Neural Architecture Search for Lightweight Image Super-Resolution

Deep convolutional neural networks (CNNs) are widely used to improve the performance of image restoration tasks, including single-image super-resolution (SISR). Generally, researchers are manually designing more complex and deeper CNNs to further increase the given problems’ performance. Instead of this hand-crafted CNN architecture design, neural architecture search (NAS) methods have been developed to find an optimal architecture for a given task automatically. For example, NAS-based SR methods find optimized network connections and operations by reinforcement learning (RL) or evolutionary algorithms (EA). These methods enable finding an optimal system automatically, but most of them need a very long search time. In this paper, we propose a new search method for the SISR that can significantly reduce the overall design time by applying a weight-sharing scheme. We also employ a multi-branch structure to enlarge the search space for capturing multi-scale features, resulting in better reconstruction on the textured region. Experiments show that the proposed method finds an optimal SISR network about twenty times faster than the existing methods, while showing comparable performance in terms of PSNR vs. parameters. Comparison of visual quality validates that the obtained SISR network reconstructs texture areas better than the previous methods because of the enlarged search space to find multi-scale features.

Convolutional Neural Network Based Approach to in Silico Non-Anticipating Prediction of Antigenic Distance for Influenza Virus.

Evaluation of the antigenic similarity degree between the strains of the influenza virus is highly important for vaccine production. The conventional method used to measure such a degree is related to performing the immunological assays of hemagglutinin inhibition. Namely, the antigenic distance between two strains is calculated on the basis of HI assays. Usually, such distances are visualized by using some kind of antigenic cartography method. The known drawback of the HI assay is that it is rather time-consuming and expensive. In this paper, we propose a novel approach for antigenic distance approximation based on deep learning in the feature spaces induced by hemagglutinin protein sequences and Convolutional Neural Networks (CNNs). To apply a CNN to compare the protein sequences, we utilize the encoding based on the physical and chemical characteristics of amino acids. By varying (hyper)parameters of the CNN architecture design, we find the most robust network. Further, we provide insight into the relationship between approximated antigenic distance and antigenicity by evaluating the network on the HI assay database for the H1N1 subtype. The results indicate that the best-trained network gives a high-precision approximation for the ground-truth antigenic distances, and can be used as a good exploratory tool in practical tasks.

Automatically Designing CNN Architectures Using the Genetic Algorithm for Image Classification.

Convolutional neural networks (CNNs) have gained remarkable success on many image classification tasks in recent years. However, the performance of CNNs highly relies upon their architectures. For the most state-of-the-art CNNs, their architectures are often manually designed with expertise in both CNNs and the investigated problems. Therefore, it is difficult for users, who have no extended expertise in CNNs, to design optimal CNN architectures for their own image classification problems of interest. In this article, we propose an automatic CNN architecture design method by using genetic algorithms, to effectively address the image classification tasks. The most merit of the proposed algorithm remains in its "automatic" characteristic that users do not need domain knowledge of CNNs when using the proposed algorithm, while they can still obtain a promising CNN architecture for the given images. The proposed algorithm is validated on widely used benchmark image classification datasets, compared to the state-of-the-art peer competitors covering eight manually designed CNNs, seven automatic + manually tuning, and five automatic CNN architecture design algorithms. The experimental results indicate the proposed algorithm outperforms the existing automatic CNN architecture design algorithms in terms of classification accuracy, parameter numbers, and consumed computational resources. The proposed algorithm also shows the very comparable classification accuracy to the best one from manually designed and automatic + manually tuning CNNs, while consuming fewer computational resources.

The Deep Convolutional Neural Network for NOx Emission Prediction of a Coal-Fired Boiler

This paper presents a methodology for predicting NOx emissions of a coal-fired boiler by using real operation data, coal properties and CNN (Convolutional Neural Network). Two building blocks are carefully designed following the practical guidelines for the light weight CNN architecture design. Furthermore, the building blocks are used to develop the deep CNN-based model for NOx prediction. A comprehensive comparison among different prediction models based on DL (Deep Learning) shows that the proposed deep CNN-based prediction model outperforms other prediction models in terms of RMSE (Root Mean Square Error) criteria. The results indicate that the developed deep CNN-based prediction model has more excellent accuracy and better numerical stability. Besides, the architecture design of the DL-based prediction model has a significant impact on the performance of the prediction model.

Satellite-ground integrated destriping network: A new perspective for EO-1 Hyperion and Chinese hyperspectral satellite datasets

From the EO-1 Hyperion imaging spectrometer to the newly launched Chinese satellite hyperspectral imagers, stripe noise is a ubiquitous phenomenon that seriously degrades the data quality and usability. Although previous efforts have achieved inspiring results, hyperspectral image (HSI) destriping remains a challenging task, as the stripe degradations are sometimes more complicated than the predefined assumptions, i.e., the preselected reference, filter, or handcrafted priors. With the rapid advances in deep learning technologies, convolutional neural networks (CNNs) provide a new potential to learn essential priors in an automatic manner. However, the training of CNNs is highly reliant on a large high-quality standard dataset, which is difficult to acquire for hyperspectral spaceborne sensors. In this paper, an innovative approach termed the satellite-ground integrated destriping network (SGIDN) is proposed for HSIs. Rather than using self-training, a satellite-ground integrated strategy is proposed, for the first time, to mitigate the data dependency, so that a large set of striped-clean pairs is generated from the ground-based HSIs. Considering the varied stripes among different bands, a unique CNN architecture design, including the combination of 3D convolution and 2D convolution, residual learning, and supplementary gradient channels, is integrated to capture the intrinsic spectral-spatial features in the HSIs and the unidirectional property of stripe noise. Compared with the traditional methods, SGIDN can be flexibly extended to specific HSI destriping tasks, e.g., coexisting horizontal and vertical stripes, and generalizes well to different hyperspectral satellite sensors. Given the same study area (Shanghai, China), three HSIs acquired by the EO-1 Hyperion imaging spectrometer, the Chinese HJ-1A HSI sensor, and the wide-range hyperspectral imager onboard the Chinese SPARK spectral micro-nano satellite are adopted to assess the proposed SGIDN model. Both simulated and real-data experiments confirm that SGIDN can consistently outperform the benchmark methods, with a higher degree of efficiency. Moreover, the land-cover mapping results further demonstrate the necessity of destriping and the suitability of the destriped results for use in further applications.

Completely Automated CNN Architecture Design Based on Blocks.

The performance of convolutional neural networks (CNNs) highly relies on their architectures. In order to design a CNN with promising performance, extensive expertise in both CNNs and the investigated problem domain is required, which is not necessarily available to every interested user. To address this problem, we propose to automatically evolve CNN architectures by using a genetic algorithm (GA) based on ResNet and DenseNet blocks. The proposed algorithm is completely automatic in designing CNN architectures. In particular, neither preprocessing before it starts nor postprocessing in terms of CNNs is needed. Furthermore, the proposed algorithm does not require users with domain knowledge on CNNs, the investigated problem, or even GAs. The proposed algorithm is evaluated on the CIFAR10 and CIFAR100 benchmark data sets against 18 state-of-the-art peer competitors. Experimental results show that the proposed algorithm outperforms the state-of-the-art CNNs hand-crafted and the CNNs designed by automatic peer competitors in terms of the classification performance and achieves a competitive classification accuracy against semiautomatic peer competitors. In addition, the proposed algorithm consumes much less computational resource than most peer competitors in finding the best CNN architectures.

Rejecting Character Recognition Errors Using CNN Based Confidence Estimation

Although Optical character recognition (OCR) technology has achieved huge progress in recent years, character misrecognition is inevitable. In order to realize high fidelity content of document digitalization, we propose a new Convolutional neural networks (CNN) based confidence estimation method. We detect the misrecognized characters through comparing the confidence value with a preset threshold, so as to leave the recognition errors as embedded images in the output digital documents. We adopted sofmax as the estimation of posteriori probability, overlap pooling and maxout with dropout technologies in CNN architecture design. Experimental results show that our method has achieved an explicit improvement compared to baseline system.