Design Of Neural Network Architectures Research Articles

AutoInit: Analytic Signal-Preserving Weight Initialization for Neural Networks

Neural networks require careful weight initialization to prevent signals from exploding or vanishing. Existing initialization schemes solve this problem in specific cases by assuming that the network has a certain activation function or topology. It is difficult to derive such weight initialization strategies, and modern architectures therefore often use these same initialization schemes even though their assumptions do not hold. This paper introduces AutoInit, a weight initialization algorithm that automatically adapts to different neural network architectures. By analytically tracking the mean and variance of signals as they propagate through the network, AutoInit appropriately scales the weights at each layer to avoid exploding or vanishing signals. Experiments demonstrate that AutoInit improves performance of convolutional, residual, and transformer networks across a range of activation function, dropout, weight decay, learning rate, and normalizer settings, and does so more reliably than data-dependent initialization methods. This flexibility allows AutoInit to initialize models for everything from small tabular tasks to large datasets such as ImageNet. Such generality turns out particularly useful in neural architecture search and in activation function discovery. In these settings, AutoInit initializes each candidate appropriately, making performance evaluations more accurate. AutoInit thus serves as an automatic configuration tool that makes design of new neural network architectures more robust. The AutoInit package provides a wrapper around TensorFlow models and is available at https://github.com/cognizant-ai-labs/autoinit.

Online evolutionary neural architecture search for multivariate non-stationary time series forecasting

Time series forecasting (TSF) is one of the most important tasks in data science. TSF models are usually pre-trained with historical data and then applied on future unseen datapoints. However, real-world time series data is usually non-stationary and models trained offline usually face problems from data drift. Models trained and designed in an offline fashion can not quickly adapt to changes quickly or be deployed in real-time. To address these issues, this work presents the Online NeuroEvolution-based Neural Architecture Search (ONE-NAS) algorithm, which is a novel neural architecture search method capable of automatically designing and dynamically training recurrent neural networks (RNNs) for online forecasting tasks. Without any pre-training, ONE-NAS utilizes populations of RNNs that are continuously updated with new network structures and weights in response to new multivariate input data. ONE-NAS is tested on real-world, large-scale multivariate wind turbine data as well as the univariate Dow Jones Industrial Average (DJIA) dataset. Results demonstrate that ONE-NAS outperforms traditional statistical time series forecasting methods, including online linear regression, fixed long short-term memory (LSTM) and gated recurrent unit (GRU) models trained online, as well as state-of-the-art, online ARIMA strategies. Additionally, results show that utilizing multiple populations of RNNs which are periodically repopulated provide significant performance improvements, allowing this online neural network architecture design and training to be successful.

Automatic food bio-hazard detection system

This paper presents the design of a convolutional neural network architecture oriented to the detection of food waste, to generate a low, medium, or critical-level alarm. An architecture based on four convolution layers is used, for which a database of 100 samples is prepared. The database is used with the different hyperparameters that make up the final architecture, after the training process. By means of confusion matrix analysis, a 100% performance of the network is obtained, which delivers its output to a fuzzy system that, depending on the duration of the detection time, generates the different alarm levels associated with the risk.

A Neural Network Approach for Homogenization of Multiscale Problems

We propose a neural network-based approach to the homogenization of multiscale problems. The proposed method uses a derivative-free formulation of a training loss, which incorporates Brownian walkers to find the macroscopic description of a multiscale PDE solution. Compared with other network-based approaches for multiscale problems, the proposed method is free from the design of hand-crafted neural network architecture and the cell problem to calculate the homogenization coefficient. The exploration neighborhood of the Brownian walkers affects the overall learning trajectory. We determine the bounds of micro- and macro-time steps that capture the local heterogeneous and global homogeneous solution behaviors, respectively, through a neural network. The bounds imply that the computational cost of the proposed method is independent of the microscale periodic structure for the standard periodic problems. We validate the efficiency and robustness of the proposed method through a suite of linear and nonlinear multiscale problems with periodic and random field coefficients.

Thrust Prediction of Aircraft Engine Enabled by Fusing Domain Knowledge and Neural Network Model

Accurate prediction of aircraft engine thrust is crucial for engine health management (EHM), which seeks to improve the safety and reliability of aircraft propulsion. Thrust prediction is implemented using an on-board adaptive model for EHM. However, the conventional methods for building such a model are often tedious or overly data-dependent. To improve the accuracy of thrust prediction, domain knowledge can be leveraged. Hence, this study presents a strategy for building an on-board adaptive model that can predict aircraft engine thrust in real-time. The strategy combines engine knowledge and neural network architecture to construct a prediction model. The whole-model architecture is divided into separate modules that are mapped in a one-to-one form using a domain decomposition approach. The engine domain knowledge is used to guide feature selection and the neural network architecture design in the method. Furthermore, this study explains the relationships between aircraft engine features and how the model can predict engine thrust in flight condition. To demonstrate the effectiveness and robustness of the architecture, four different testing datasets were used for validation. The results show that the thrust prediction model created by the given architecture has maximum relative deviations below 4.0% and average relative deviations below 2.0% on all testing datasets. In comparison to the performance of the models created by conventional neural network architecture on the four testing datasets, the model created by the presented architecture proves more suitable for aircraft propulsion.

Using (1+1)D quantum cellular automata for exploring collective effects in large-scale quantum neural networks.

Central to the field of quantum machine learning is the design of quantum perceptrons and neural network architectures. A key question in this regard is the impact of quantum effects on the way such models process information. Here, we establish a connection between (1+1)D quantum cellular automata, which implement a discrete nonequilibrium quantum many-body dynamics through successive applications of local quantum gates, and quantum neural networks (QNNs), which process information by feeding it through perceptrons interconnecting adjacent layers. Exploiting this link, we construct a class of QNNs that are highly structured-aiding both interpretability and helping to avoid trainability issues in machine learning tasks-yet can be connected rigorously to continuous-time Lindblad dynamics. We further analyze the universal properties of an example case, identifying a change of critical behavior when quantum effects are varied, showing their potential impact on the collective dynamical behavior underlying information processing in large-scale QNNs.

A Self-adaptive neuroevolution approach to constructing Deep Neural Network architectures across different types

Neuroevolution has greatly promoted Deep Neural Network (DNN) architecture design and its applications, while there is a lack of methods available across different DNN types concerning both their scale and performance. In this study, we propose a self-adaptive neuroevolution (SANE) approach to automatically construct various lightweight DNN architectures for different tasks. One of the key settings in SANE is the search space defined by cells and organs self-adapted to different DNN types. Based on this search space, a constructive evolution strategy with uniform evolution settings and operations is designed to grow DNN architectures gradually. SANE is able to self-adaptively adjust evolution exploration and exploitation to improve search efficiency. Moreover, a speciation scheme is developed to protect evolution from early convergence by restricting selection competition within species. To evaluate SANE, we carry out neuroevolution experiments to generate different DNN architectures including convolutional neural network, generative adversarial network and long short-term memory. The results illustrate that the obtained DNN architectures could have smaller scale with similar performance compared to existing DNN architectures. Our proposed SANE provides an efficient approach to self-adaptively search DNN architectures across different types. Our Code is available at https://doi.org/10.24433/CO.2989985.v1.

Lightweight convolutional neural network architecture design for music genre classification using evolutionary stochastic hyperparameter selection

AbstractConvolutional neural networks (CNNs) have succeeded in various domains, including music information retrieval (MIR). Music genre classification (MGC) is one such task in the MIR that has gained attention over the years because of the massive increase in online music content. Accurate indexing and automatic classification of these large volumes of music content require high computational resources, which pose a significant challenge to building a lightweight system. CNNs are a popular deep learning‐based choice for building systems for MGC. However, finding an optimal CNN architecture for MGC requires domain knowledge both in CNN architecture design and music. We present MGA‐CNN, a genetic algorithm‐based approach with a novel stochastic hyperparameter selection for finding an optimal lightweight CNN‐based architecture for the MGC task. The proposed approach is unique in automating the CNN architecture design for the MGC task. MGA‐CNN is evaluated on three widely used music datasets and compared with seven peer rivals, which include three automatic CNN architecture design approaches and four manually designed popular CNN architectures. The experimental results show that MGA‐CNN surpasses the peer approaches in terms of classification accuracy, parameter numbers, and execution time. The optimal architectures generated by MGA‐CNN also achieve classification accuracy comparable to the manually designed CNN architectures while spending fewer computing resources.

Leveraging image complexity in macro-level neural network design for medical image segmentation

Recent progress in encoder–decoder neural network architecture design has led to significant performance improvements in a wide range of medical image segmentation tasks. However, state-of-the-art networks for a given task may be too computationally demanding to run on affordable hardware, and thus users often resort to practical workarounds by modifying various macro-level design aspects. Two common examples are downsampling of the input images and reducing the network depth or size to meet computer memory constraints. In this paper, we investigate the effects of these changes on segmentation performance and show that image complexity can be used as a guideline in choosing what is best for a given dataset. We consider four statistical measures to quantify image complexity and evaluate their suitability on ten different public datasets. For the purpose of our illustrative experiments, we use DeepLabV3+ (deep large-size), M2U-Net (deep lightweight), U-Net (shallow large-size), and U-Net Lite (shallow lightweight). Our results suggest that median frequency is the best complexity measure when deciding on an acceptable input downsampling factor and using a deep versus shallow, large-size versus lightweight network. For high-complexity datasets, a lightweight network running on the original images may yield better segmentation results than a large-size network running on downsampled images, whereas the opposite may be the case for low-complexity images.

Optimal Design of Convolutional Neural Network Architectures Using Teaching–Learning-Based Optimization for Image Classification

Convolutional neural networks (CNNs) have exhibited significant performance gains over conventional machine learning techniques in solving various real-life problems in computational intelligence fields, such as image classification. However, most existing CNN architectures were handcrafted from scratch and required significant amounts of problem domain knowledge from designers. A novel deep learning method abbreviated as TLBOCNN is proposed in this paper by leveraging the excellent global search ability of teaching–learning-based optimization (TLBO) to obtain an optimal design of network architecture for a CNN based on the given dataset with symmetrical distribution of each class of data samples. A variable-length encoding scheme is first introduced in TLBOCNN to represent each learner as a potential CNN architecture with different layer parameters. During the teacher phase, a new mainstream architecture computation scheme is designed to compute the mean parameter values of CNN architectures by considering the information encoded into the existing population members with variable lengths. The new mechanisms of determining the differences between two learners with variable lengths and updating their positions are also devised in both the teacher and learner phases to obtain new learners. Extensive simulation studies report that the proposed TLBOCNN achieves symmetrical performance in classifying the majority of MNIST-variant datasets, displays the highest accuracy, and produces CNN models with the lowest complexity levels compared to other state-of-the-art methods due to its promising search ability.

Neural architecture search using property guided synthesis

Neural architecture search (NAS) has become an increasingly important tool within the deep learning community in recent years, yielding many practical advancements in the design of deep neural network architectures. However, most existing approaches operate within highly structured design spaces, and hence (1) explore only a small fraction of the full search space of neural architectures while also (2) requiring significant manual effort from domain experts. In this work, we develop techniques that enable efficient NAS in a significantly larger design space. In particular, we propose to perform NAS in an abstract search space of program properties. Our key insights are as follows: (1) an abstract search space can be significantly smaller than the original search space, and (2) architectures with similar program properties should also have similar performance; thus, we can search more efficiently in the abstract search space. To enable this approach, we also introduce a novel efficient synthesis procedure, which performs the role of concretizing a set of promising program properties into a satisfying neural architecture. We implement our approach, αNAS, within an evolutionary framework, where the mutations are guided by the program properties. Starting with a ResNet-34 model, αNAS produces a model with slightly improved accuracy on CIFAR-10 but 96% fewer parameters. On ImageNet, αNAS is able to improve over Vision Transformer (30% fewer FLOPS and parameters), ResNet-50 (23% fewer FLOPS, 14% fewer parameters), and EfficientNet (7% fewer FLOPS and parameters) without any degradation in accuracy.

How does momentum benefit deep neural networks architecture design? A few case studies

We present and review an algorithmic and theoretical framework for improving neural network architecture design via momentum. As case studies, we consider how momentum can improve the architecture design for recurrent neural networks (RNNs), neural ordinary differential equations (ODEs), and transformers. We show that integrating momentum into neural network architectures has several remarkable theoretical and empirical benefits, including (1) integrating momentum into RNNs and neural ODEs can overcome the vanishing gradient issues in training RNNs and neural ODEs, resulting in effective learning long-term dependencies; (2) momentum in neural ODEs can reduce the stiffness of the ODE dynamics, which significantly enhances the computational efficiency in training and testing; (3) momentum can improve the efficiency and accuracy of transformers.

Cancer-Net SCa: tailored deep neural network designs for detection of skin cancer from dermoscopy images

BackgroundSkin cancer continues to be the most frequently diagnosed form of cancer in the U.S., with not only significant effects on health and well-being but also significant economic costs associated with treatment. A crucial step to the treatment and management of skin cancer is effective early detection with key screening approaches such as dermoscopy examinations, leading to stronger recovery prognoses. Motivated by the advances of deep learning and inspired by the open source initiatives in the research community, in this study we introduce Cancer-Net SCa, a suite of deep neural network designs tailored for the detection of skin cancer from dermoscopy images that is open source and available to the general public. To the best of the authors’ knowledge, Cancer-Net SCa comprises the first machine-driven design of deep neural network architectures tailored specifically for skin cancer detection, one of which leverages attention condensers for an efficient self-attention design.ResultsWe investigate and audit the behaviour of Cancer-Net SCa in a responsible and transparent manner through explainability-driven performance validation. All the proposed designs achieved improved accuracy when compared to the ResNet-50 architecture while also achieving significantly reduced architectural and computational complexity. In addition, when evaluating the decision making process of the networks, it can be seen that diagnostically relevant critical factors are leveraged rather than irrelevant visual indicators and imaging artifacts.ConclusionThe proposed Cancer-Net SCa designs achieve strong skin cancer detection performance on the International Skin Imaging Collaboration (ISIC) dataset, while providing a strong balance between computation and architectural efficiency and accuracy. While Cancer-Net SCa is not a production-ready screening solution, the hope is that the release of Cancer-Net SCa in open source, open access form will encourage researchers, clinicians, and citizen data scientists alike to leverage and build upon them.

Assessing the effects of convolutional neural network architectural factors on model performance for remote sensing image classification: An in-depth investigation

Although the application of deep learning in remote sensing (RS) has achieved fruitful results, systematic research on exploring the model performance and guiding the design of new convolutional neural network (CNN) architectures is still lacking. This subject is of great concern to researchers or practitioners in this field because existing CNN structures may not be adequate to deal with complex RS scenarios. In this study, an empirical formula of CNN model performance is delivered based on a literature review. Extensive experiments are conducted on six public RS data sets to investigate the influences of three architectural factors, namely, network depth, width, and cardinality. Two types of CNN architectures, i.e., VGG and ResNet, are adopted as baselines. We monitor and visualize the data distributions and gradients of the utilized CNNs to prevent the gradient vanishing or exploding problem. Grad-CAM is adopted to open the black box of CNNs and to illustrate the effects of adjusting architectural factors. Our experiments indicate that (1) increasing the network depth is beneficial to the semantic feature learning capacity of a CNN model, but excessive depth also leads to a decline of overall accuracy; (2) a partly widening strategy is effective because it can improve the model performance while maintaining the network complexity; and (3) network cardinality has huge potential in achieving a balance between model efficiency and accuracy. Suggestions for improving the CNN model performance and developing new structures are summarized in this paper.

A Novel Hyperparameter Optimization Aided Hand Gesture Recognition Framework Based on Deep Learning Algorithms

The recognition of hand gestures in cluttered or complex environments is a vital research area in the human-computer interaction and computer vision fields due to its various potential applications, such as hand action analysis, driver hand behaviour monitoring, virtual reality, pose estimation, human action recognition, and sign language recognition. In order to create more reliable and efficient algorithms in this research field, various approaches have been suggested in recent years. However, a robust system is still elusive. For this reason, a new deep learning-based architecture for classifying hand gestures is suggested in this study; it is based on a hybrid model. The study makes two main contributions to the literature. The first is the creation of a new database for hand gesture recognition. The second is a novel hybrid architecture that combines two widely used pre-trained network models in an optimised manner, using a genetic algorithm for hyperparameter optimization. The proposed method comprises five main phases, namely, data acquisition, pre-processing, the design of the deep neural network architecture, hyperparameter optimization, the training of the proposed deep neural network architecture. The proposed method was tested on three comprehensive datasets. The experimental results reveal that the suggested method can effectively classify hand gestures with a high accuracy rate and that it outperforms the state-of-the-art methods in the literature.

Fast Approximations for Job Shop Scheduling: A Lagrangian Dual Deep Learning Method

The Jobs Shop Scheduling problem (JSP) is a canonical combinatorial optimization problem that is routinely solved for a variety of industrial purposes. It models the optimal scheduling of multiple sequences of tasks, each under a fixed order of operations, in which individual tasks require exclusive access to a predetermined resource for a specified processing time. The problem is NP-hard and computationally challenging even for medium-sized instances. Motivated by the increased stochasticity in production chains, this paper explores a deep learning approach to deliver efficient and accurate approximations to the JSP. In particular, this paper proposes the design of a deep neural network architecture to exploit the problem structure, its integration with Lagrangian duality to capture the problem constraints, and a post-processing optimization, to guarantee solution feasibility. The resulting method, called JSP-DNN, is evaluated on hard JSP instances from the JSPLIB benchmark library and is shown to produce JSP approximations of high quality at negligible computational costs.

Optimal artificial neural network architecture design for modeling an industrial ethylene oxide plant

Optimum selection of input variables, number of hidden neurons and connections among the network elements deliver the best configuration of an ANN, usually resulting in reduced over-fitting and improved test performance. This study focuses on the development of a superstructure-oriented feedforward ANN design and training algorithm whose impacts are demonstrated on an industrial Ethylene Oxide (EO) plant for the prediction of product related variables. Proposed method brings about a mixed integer nonlinear programming problem (MINLP) to be solved, which takes the existence of inputs, neurons, and connections among the network elements into account by binary variables in addition to continuous weights of existing connections. Our investigations show that almost 85% of the ANN connections are removed compared to the fully connected ANN (FC-ANN) with 50% decrease in the number of inputs of the ANN. The modified ANN delivers a better prediction performance over FC-ANN, since FC-ANN suffers from over-fitting.

Evolving Optimised Convolutional Neural Networks for Lung Cancer Classification

Detecting pulmonary nodules early significantly contributes to the treatment success of lung cancer. Several deep learning models for medical image analysis have been developed to help classify pulmonary nodules. The design of convolutional neural network (CNN) architectures, however, is still heavily reliant on human domain knowledge. Manually designing CNN design solutions has been shown to limit the data’s utility by creating a co-dependency on the creator’s cognitive bias, which urges the development of smart CNN architecture design solutions. In this paper, an evolutionary algorithm is used to optimise the classification of pulmonary nodules with CNNs. The implementation of a genetic algorithm (GA) for CNN architectures design and hyperparameter optimisation is proposed, which approximates optimal solutions by implementing a range of bio-inspired mechanisms of natural selection and Darwinism. For comparison purposes, two manually designed deep learning models, FractalNet and Deep Local-Global Network, were trained. The results show an outstanding classification accuracy of the fittest GA-CNN (91.3%), which outperformed both manually designed models. The findings indicate that GAs pose advantageous solutions for diagnostic challenges, the development of which may to be fully automated in the future using GAs to design and optimise CNN architectures for various clinical applications.

SOLVING POISSON EQUATION WITH CONVOLUTIONAL NEURAL NETWORKS

Context. The Poisson equation is the one of fundamental differential equations, which used to simulate complex physical processes, such as fluid motion, heat transfer problems, electrodynamics, etc. Existing methods for solving boundary value problems based on the Poisson equation require an increase in computational time to achieve high accuracy. The proposed method allows solving the boundary value problem with significant acceleration under the condition of acceptable loss of accuracy.
 Objective. The aim of our work is to develop artificial neural network architecture for solving a boundary value problem based on the Poisson equation with arbitrary Dirichlet and Neumann boundary conditions.
 Method. The method of solving boundary value problems based on the Poisson equation using convolutional neural network is proposed. The network architecture, structure of input and output data are developed. In addition, the method of training dataset generation is described.
 Results. The performance of the developed artificial neural network is compared with the performance of the numerical finite difference method for solving the boundary value problem. The results showed an acceleration of the computational speed in x10–700 times depending on the number of sampling nodes.
 Conclusions. The proposed method significantly accelerated speed of solving a boundary value problem based on the Poisson equation in comparison with the numerical method. In addition, the developed approach to the design of neural network architecture allows to improve the proposed method to achieve higher accuracy in modeling the process of pressure distribution in areas of arbitrary size.

IGWO-SS: Improved Grey Wolf Optimization Based on Synaptic Saliency for Fast Neural Architecture Search in Computer Vision

Neural Architecture Search (NAS) is the process of automating the design of neural network architectures for a given task. Although NAS provides automates the process of finding suitable neural network architectures for a specific task, the existing NAS algorithms are immensely time-consuming. The main bottleneck in NAS algorithms is the training time for each architecture. This study proposes an Improved Grey Wolf Optimization based on Synaptic Saliency (IGWO-SS), which is much faster than the existing NAS algorithms and provides better final performance. The IGWO-SS algorithm skips training the less promising architectures by creating a relative rank between the architectures based on synaptic saliency. The architectures that are lower in rank are considered less promising than the architectures that are higher in rank. Since the calculation of synaptic saliency is a very fast process, a significant amount of time is saved by skipping training of less promising architectures. We performed extensive experiments to determine the efficacy of synaptic saliency in improving NAS. Our experimental results suggest that the synaptic saliency of an untrained neural network positively correlates with its final accuracy. Hence, it can be used to identify untrained promising neural networks. The experimental results suggest that the IGWO-SS algorithm is almost 10<i>x</i> faster and achieves better final performance than five other bio-inspired algorithms. The IGWO-SS algorithm achieves higher mean accuracy than state-of-the-art NAS algorithms, including - REA, RS, RL, BOHB, DARTSV1, DARTSV2, GDAS, SETN, and ENAS. We hope that our work will make NAS more accessible and useful to researchers by reducing the time and resources required to perform NAS.