Neural Network Tuning Research Articles

Classification of COVID-19 chest X-rays with deep learning: new models or fine tuning?

Background and objectivesChest X-ray data have been found to be very promising for assessing COVID-19 patients, especially for resolving emergency-department and urgent-care-center overcapacity. Deep-learning (DL) methods in artificial intelligence (AI) play a dominant role as high-performance classifiers in the detection of the disease using chest X-rays. Given many new DL models have been being developed for this purpose, the objective of this study is to investigate the fine tuning of pretrained convolutional neural networks (CNNs) for the classification of COVID-19 using chest X-rays. If fine-tuned pre-trained CNNs can provide equivalent or better classification results than other more sophisticated CNNs, then the deployment of AI-based tools for detecting COVID-19 using chest X-ray data can be more rapid and cost-effective.MethodsThree pretrained CNNs, which are AlexNet, GoogleNet, and SqueezeNet, were selected and fine-tuned without data augmentation to carry out 2-class and 3-class classification tasks using 3 public chest X-ray databases.ResultsIn comparison with other recently developed DL models, the 3 pretrained CNNs achieved very high classification results in terms of accuracy, sensitivity, specificity, precision, F_1 score, and area under the receiver-operating-characteristic curve.ConclusionAlexNet, GoogleNet, and SqueezeNet require the least training time among pretrained DL models, but with suitable selection of training parameters, excellent classification results can be achieved without data augmentation by these networks. The findings contribute to the urgent need for harnessing the pandemic by facilitating the deployment of AI tools that are fully automated and readily available in the public domain for rapid implementation.

Improving molecular design through a machine learning approach

We present a study for predicting 15 molecular properties through the combination of a quantum mechanical database, taken from the quantum chemistry QM9 database, and feed forward deep neural networks approaches. The aim of the work is to show that the combination of a priori computed ab-initio information and machine learning can support experimental work to speed up the discovery and formulation of novel compounds. The importance of this work also relies on the fact that through this computer-aided molecular design approach no approximate or heuristic contribution methods are needed for physical and thermodynamic properties information. We show that using proper hyper-parameters tuning of deep neural networks is possible, even with modest computational resources, to design the chemical structure of compounds matching target molecular properties making them feasible for practical industrial applications in diverse areas such as energy, water, food, health and transport economical sectors.

Accelerating flash calculations in unconventional reservoirs considering capillary pressure using an optimized deep learning algorithm

An increasing focus was placed in the past few decades on accelerating flash calculations and a variety of acceleration strategies have been developed to improve its efficiency without serious compromise in accuracy and reliability. Recently, as machine learning becomes a powerful tool to handle complicated and time-consuming problems, it is increasingly appealing to replace the iterative flash algorithm, due to the strong nonlinearity of flash problem, by a neural network model. In this study, an NVT flash calculation scheme is established with a thermodynamically stable evolution algorithm to generate training and testing data for the proposed deep neural network. With a modified network structure, the deep learning algorithm is optimized by carefully tuning neural network hyperparameters. Numerical tests indicate that the trained model is capable of accurately estimating phase compositions and states for complex reservoir fluids under a wide range of environmental conditions, while the effect of capillary pressure can be captured well. Thermodynamic rules are preserved well through our algorithm, and the trained model can be used for various fluid mixtures, which significantly accelerates flash calculations in unconventional reservoirs.

Evaluation, Tuning, and Interpretation of Neural Networks for Working with Images in Meteorological Applications

AbstractThe method of neural networks (aka deep learning) has opened up many new opportunities to utilize remotely sensed images in meteorology. Common applications include image classification, e.g., to determine whether an image contains a tropical cyclone, and image-to-image translation, e.g., to emulate radar imagery for satellites that only have passive channels. However, there are yet many open questions regarding the use of neural networks for working with meteorological images, such as best practices for evaluation, tuning, and interpretation. This article highlights several strategies and practical considerations for neural network development that have not yet received much attention in the meteorological community, such as the concept of receptive fields, underutilized meteorological performance measures, and methods for neural network interpretation, such as synthetic experiments and layer-wise relevance propagation. We also consider the process of neural network interpretation as a whole, recognizing it as an iterative meteorologist-driven discovery process that builds on experimental design and hypothesis generation and testing. Finally, while most work on neural network interpretation in meteorology has so far focused on networks for image classification tasks, we expand the focus to also include networks for image-to-image translation.

Novel suboptimal approaches for hyperparameter tuning of deep neural network [under the shelf of optical communication

Grid search is the most effective method for tuning hyperparameters in machine learning (ML). However, it has high computational complexity, and is not appropriate when here are many hyperparameters to be tuned. In this paper, two novel suboptimal grid search methods are presented, which search the grid marginally and alternatively. In order to show the efficiency of hyperparameter tuning by the proposed methods four datasets are used. Two datasets were collected by simulating FSO and fiber OC links by MATLAB software, and two other datasets were collected by experimental setups for FSO and fiber OC links in Optisystem software. Results indicate that despite greatly reducing computational complexity, the proposed methods achieve a favorable performance. The proposed structures are compared with some of the recently published most relevant works, and the efficiency of the proposed methods is proved.

Energy and Policy Considerations for Modern Deep Learning Research

The field of artificial intelligence has experienced a dramatic methodological shift towards large neural networks trained on plentiful data. This shift has been fueled by recent advances in hardware and techniques enabling remarkable levels of computation, resulting in impressive advances in AI across many applications. However, the massive computation required to obtain these exciting results is costly both financially, due to the price of specialized hardware and electricity or cloud compute time, and to the environment, as a result of non-renewable energy used to fuel modern tensor processing hardware. In a paper published this year at ACL, we brought this issue to the attention of NLP researchers by quantifying the approximate financial and environmental costs of training and tuning neural network models for NLP (Strubell, Ganesh, and McCallum 2019). In this extended abstract, we briefly summarize our findings in NLP, incorporating updated estimates and broader information from recent related publications, and provide actionable recommendations to reduce costs and improve equity in the machine learning and artificial intelligence community.

Hessian with Mini-Batches for Electrical Demand Prediction

The steepest descent method is frequently used for neural network tuning. Mini-batches are commonly used to get better tuning of the steepest descent in the neural network. Nevertheless, steepest descent with mini-batches could be delayed in reaching a minimum. The Hessian could be quicker than the steepest descent in reaching a minimum, and it is easier to achieve this goal by using the Hessian with mini-batches. In this article, the Hessian is combined with mini-batches for neural network tuning. The discussed algorithm is applied for electrical demand prediction.

Neural Network-Based Optimal Tracking Control of Continuous-Time Uncertain Nonlinear System via Reinforcement Learning

In this note, optimal tracking control for uncertain continuous-time nonlinear system is investigated by using a novel reinforcement learning (RL) scheme. The uncertainty here refers to unknown system drift dynamics. Based on the nonlinear system and reference signal, we firstly formulate the tracking problem by constructing an augmented system. The optimal tracking control problem for original nonlinear system is thus transformed into solving the Hamilton–Jacobi–Bellman (HJB) equation of the augmented system. A new single neural network (NN)-based online RL method is proposed to learn the solution of tracking HJB equation while the corresponding optimal control input that minimizes the tracking HJB equation is calculated in a forward-in-time manner without requiring any value, policy iterations and the system drift dynamics. In order to relax the dependence of the RL method on traditional Persistence of Excitation (PE) conditions, a concurrent learning technique is adopted to design the NN tuning laws. The Uniformly Ultimately Boundedness of NN weight errors and closed-loop augmented system states are rigorous proved. Three numerical simulation examples are given to demonstrate the effectiveness of the proposed scheme.

Tumor Classification using Block wise fine tuning and Transfer learning of Deep Neural Network and KNN classifier on MR Brain Images

Tumor Classification using Block wise fine tuning and Transfer learning of Deep Neural Network and KNN classifier on MR Brain Images

Oven Controlled Crystal Oscillator Control Based on BP Neural Network Tuning PID

Oven Controlled Crystal Oscillator Control Based on BP Neural Network Tuning PID

Auto-Ensemble: An Adaptive Learning Rate Scheduling Based Deep Learning Model Ensembling

Ensembling deep learning models is a shortcut to promote its implementation in new scenarios, which can avoid tuning neural networks, losses and training algorithms from scratch. However, it is difficult to collect sufficient accurate and diverse models through once training. This paper proposes Auto-Ensemble (AE) to collect checkpoints of deep learning model and ensemble them automatically by adaptive learning rate scheduling algorithm. The advantage of this method is to make the model converge to various local optima by scheduling the learning rate in once training. When the number of lo-cal optimal solutions tends to be saturated, all the collected checkpoints are used for ensemble. Our method is universal, it can be applied to various scenarios. Experiment results on multiple datasets and neural networks demonstrate it is effective and competitive, especially on few-shot learning. Besides, we proposed a method to measure the distance among models. Then we can ensure the accuracy and diversity of collected models.

Computer diagnostics of resistance spot welding based on Hamming neural network

The article deals with utilizing the method of qualitative assessment of welding zone dynamic resistance based on a Hamming neural network to increase versatility and reliability of computer diagnostics for resistance spot welding. We propose a mechanism for encoding information on dynamic resistance into bipolar signals required for the neural network tuning and operation. The algorithm of welding diagnostics was developed and implemented with specialized software. The results of the neural network training and testing are presented. As the analysis shows, the relative error in predicting destruction force does not exceed 10%. The approach proposed in this article complies with the requirements of ISO 9000:2015 standard for continuous monitoring and documentation of each welded connection and allows for increased accuracy of computer diagnostics of welds.

Deep Learning for Emotion Recognition in Affective Virtual Reality and Music Applications

This paper presents a deep learning approach to emotion recognition as applied to virtual reality and music predictive analytics. Firstly, it investigates the deep parameter tuning of the multi-hidden layer neural networks, which are also commonly referred to simply as deep networks that are used to conduct emotion detection in virtual reality (VR)- electroencephalography (EEG) predictive analytics. Deep networks have been studied extensively over the last decade and have shown to be among the most accurate methods for predictive analytics in image recognition and speech processing domains. However, most predictive analytics deep network studies focus on the shallow parameter tuning when attempting to boost prediction accuracies, which includes deep network tuning parameters such as number of hidden layers, number of hidden nodes per hidden layer and the types of activation functions used in the hidden nodes. Much less effort has been put into investigating the tuning of deep parameters such as input dropout ratios, L1 (lasso) regularization and L2 (ridge regularization) parameters of the deep networks. As such, the goal of this study is to perform a parameter tuning investigation on these deep parameters of the deep networks for predicting emotions in a virtual reality environment using electroencephalography (EEG) signal obtained when the user is exposed to immersive content. The results show that deep tuning of deep networks in VR-EEG can improve the accuracies of predicting emotions. The best emotion prediction accuracy was improved to over 96% after deep tuning was conducted on the deep network parameters of input dropout ratio, L1 and L2 regularization parameters. Secondly, it investigates a similar possible approach when applied to 4-quadrant music emotion recognition. Recent studies have been characterizing music based on music genres and various classification techniques have been used to achieve the best accuracy rate. Several researches on deep learning have shown outstanding results in relation to dimensional music emotion recognition. Yet, there is no concrete and concise description to express music. In regards to this research gap, a research using more detailed metadata on twodimensional emotion annotations based on the Russell’s model is conducted. Rather than applying music genres or lyrics into machine learning algorithm to MER, higher representation of music information, acoustic features are used. In conjunction with the four classes classification problem, an available dataset named AMG1608 is feed into a training model built from deep neural network. The dataset is first preprocessed to get full access of variables before any machine learning is done. The classification rate is then collected by running the scripts in R environment. The preliminary result showed a classification rate of 46.0%.

Pavo: A RNN-Based Learned Inverted Index, Supervised or Unsupervised?

The booms of big data and graphic processing unit technologies have allowed us to explore more appropriate data structures and algorithms with smaller time complexity. However, the application of machine learning as a potential alternative for the traditional data structures, especially using deep learning, is a relatively new and largely uncharted territory. In this paper, we propose a novel recurrent neural network-based learned inverted index, called Pavo, to efficiently and flexibly organize inverted data. The basic hash function in the traditional inverted index is replaced by a hierarchical neural network, which makes Pavo be able to well adapt for various data distributions while showing lower collision rate as well as higher space utilization rate. A particular feature of our approach is that a novel unsupervised learning strategy to construct the hash function is proposed. To the best of our knowledge, there are no similar results, in which the unsupervised learning strategy is employed to design hash functions, in the existing literature. Extensive experimental results show that the unsupervised model owns some advantages than the supervised one. Our approaches not only demonstrate the feasibility of deep learning-based data structures for index purpose but also provide benefits for developers to make more accurate decisions on both the design and the configuration of data organization, operation, and parameters tuning of neural network so as to improve the performance of information searching.

RBF neural network tuning PID control based on UMAC

RBF neural network tuning PID control based on UMAC

Cloud‐based video analytics using convolutional neural networks

SummaryObject classification is a vital part of any video analytics system, which could aid in complex applications such as object monitoring and management. Traditional video analytics systems work on shallow networks and are unable to harness the power of distributed processing for training and inference. We propose a cloud‐based video analytics system based on an optimally tuned convolutional neural network to classify objects from video streams. The tuning of convolutional neural network is empowered by in‐memory distributed computing. The object classification is performed by comparing the target object with the prestored trained patterns, generating a set of matching scores. The matching scores greater than an empirically determined threshold reveal the classification of the target object. The proposed system proved to be robust to classification errors with an accuracy and precision of 97% and 96%, respectively, and can be used as a general‐purpose video analytics system.

Energy‐based tuning of convolutional neural networks on multi‐GPUs

SummaryDeep Learning (DL) applications are gaining momentum in the realm of Artificial Intelligence, particularly after GPUs have demonstrated remarkable skills for accelerating their challenging computational requirements. Within this context, Convolutional Neural Network (CNN) models constitute a representative example of success on a wide set of complex applications, particularly on datasets where the target can be represented through a hierarchy of local features of increasing semantic complexity. In most of the real scenarios, the roadmap to improve results relies on CNN settings involving brute force computation, and researchers have lately proven Nvidia GPUs to be one of the best hardware counterparts for acceleration. Our work complements those findings with an energy study on critical parameters for the deployment of CNNs on flagship image and video applications, ie, object recognition and people identification by gait, respectively. We evaluate energy consumption on four different networks based on the two most popular ones (ResNet/AlexNet), ie, ResNet (167 layers), a 2D CNN (15 layers), a CaffeNet (25 layers), and a ResNetIm (94 layers) using batch sizes of 64, 128, and 256, and then correlate those with speed‐up and accuracy to determine optimal settings. Experimental results on a multi‐GPU server endowed with twin Maxwell and twin Pascal Titan X GPUs demonstrate that energy correlates with performance and that Pascal may have up to 40% gains versus Maxwell. Larger batch sizes extend performance gains and energy savings, but we have to keep an eye on accuracy, which sometimes shows a preference for small batches. We expect this work to provide a preliminary guidance for a wide set of CNN and DL applications in modern HPC times, where the GFLOPS/w ratio constitutes the primary goal.

Optimal hyperparameter tuning of convolutional neural networks based on the parameter-setting-free harmony search algorithm

Hyperparameters determine layer architecture in the feature extraction step of a convolutional neural network (CNN), and this affects classification accuracy and learning time. In this paper, we propose a method to improve CNN performance by hyperparameter tuning in the feature extraction step of CNN. In the proposed method, the hyperparameter is adjusted using a parameter-setting-free harmony search (PSF-HS) algorithm, which is a metaheuristic optimization method. In the PSF-HS algorithm, the hyperparameter to be adjusted is set as the harmony, and harmony memory is generated after generating the harmony. Harmony memory is updated based on the loss of a CNN. A simulation using CNN architecture with reference to LeNet-5 and a MNIST dataset, and a simulation using the CNN architecture with reference to CifarNet and a Cifar-10 dataset are performed. By two simulations, it is possible to improve the performance by tuning the hyperparameters in CNN architectures proposed in the past.

Gain parameter and dropout-based fine tuning of deep networks

Training of deep neural networks can involve two phases: unsupervised pre-training and supervised fine tuning. Unsupervised pre-training is used to learn the initial parameter values of deep networks, while as supervised fine tuning improves upon what has been learned in the pre-training stage. Backpropagation algorithm can be used for supervised fine tuning of deep neural networks. In this paper we evaluate the use of backpropagation with gain parameter algorithm for fine tuning of deep networks. We further propose a modification where backpropagation with gain parameter algorithm is integrated with the dropout technique and evaluate its performance in fine tuning of deep networks. The effectiveness of fine tuning done by proposed technique is also compared with other variants of backpropagation algorithm on benchmark datasets. The experimental results show that the fine tuning of deep networks using the proposed technique yields promising results among all the studied methods on the tested datasets.

Experimental Parameter Tuning of Artificial Neural Network in Customer Churn Prediction

Purpose of the article: The ability of the company to predict customer churn and retain customers is considered to be worthy competitive advantage since it improves cost allocation in customer retention programs, retaining future revenue and profits. In addition, it has several positive indirect impacts such as increasing customer’s loyalty. Therefore, the focus of the article is on building highly reliable and robust classification model, which deals with such a task. Methodology/methods: The analysis is carried out on labelled ecommerce retail dataset describing 10 000 most valuable customers with the highest CLV (Customer Lifetime Value). To obtain the best performing ANN (Artificial Neural Network) classification model, proposed hyperparameter search space is explored with genetic algorithm to find suitable parameter settings. ANN classification performance is measured with regard to prediction ability, which is understood as point estimate of AUC (Area Under Curve) mean on 4fold cross-validation set. Explored part of hyperparameter search space is analyzed with conditional inference tree structure addressing underlying fundamental context of given optimization which results in identification of critical factors leading to well performing ANN classification model. Scientific aim: To present and execute experimental design for performance evaluation and hyperparameter optimization of classification models, which are used for customer churn prediction. Findings: It is concluded and statistically proven that in experimental context described, regularization parameter as well as training function have significant influence on classifiers AUC performance contrasting other properties of ANN. More specifically, well performing ANN classification models have regularization parameter set to 0, adaptation function set to trainlm or trainscg and more than 100 training epochs. Global optimum is identified for solution with regularization parameter set to 0, trainlm adaptation function, 350 training epochs and 7-4-2 architecture. Conclusions: Results imply that placing hyperparameter optimization to ANN classification model leads to improved customer churn prediction ability. The article describes design and execution of machine learning pipeline, hyperparameter optimization and original meta-analysis of the results with conditional inference tree structure, which are considered beneficial for further research.