Optimization Of Deep Neural Networks Research Articles

Analysis and design of optimal deep neural network model for image recognition using hybrid cuckoo search with self-adaptive particle swarm intelligence

Analysis and design of optimal deep neural network model for image recognition using hybrid cuckoo search with self-adaptive particle swarm intelligence

DNN-BP: a novel framework for cuffless blood pressure measurement from optimal PPG features using deep learning model.

Continuous blood pressure (BP) provides essential information for monitoring one's health condition. However, BP is currently monitored using uncomfortable cuff-based devices, which does not support continuous BP monitoring. This paper aims to introduce a blood pressure monitoring algorithm based on only photoplethysmography (PPG) signals using the deep neural network (DNN). The PPG signals are obtained from 125 unique subjects with 218 records and filtered using signal processing algorithms to reduce the effects of noise, such as baseline wandering, and motion artifacts. The proposed algorithm is based on pulse wave analysis of PPG signals, extracted various domain features from PPG signals, and mapped them to BP values. Four feature selection methods are applied and yielded four feature subsets. Therefore, an ensemble feature selection technique is proposed to obtain the optimal feature set based on major voting scores from four feature subsets. DNN models, along with the ensemble feature selection technique, outperformed in estimating the systolic blood pressure (SBP) and diastolic blood pressure (DBP) compared to previously reported approaches that rely only on the PPG signal. The coefficient of determination ( ) and mean absolute error (MAE) of the proposed algorithm are 0.962 and 2.480 mmHg, respectively, for SBP and 0.955 and 1.499 mmHg, respectively, for DBP. The proposed approach meets the Advancement of Medical Instrumentation standard for SBP and DBP estimations. Additionally, according to the British Hypertension Society standard, the results attained Grade A for both SBP and DBP estimations. It concludes that BP can be estimated more accurately using the optimal feature set and DNN models. The proposed algorithm has the potential ability to facilitate mobile healthcare devices to monitor continuous BP.

Power Law in Deep Neural Networks: Sparse Network Generation and Continual Learning With Preferential Attachment.

Training deep neural networks (DNNs) typically requires massive computational power. Existing DNNs exhibit low time and storage efficiency due to the high degree of redundancy. In contrast to most existing DNNs, biological and social networks with vast numbers of connections are highly efficient and exhibit scale-free properties indicative of the power law distribution, which can be originated by preferential attachment in growing networks. In this work, we ask whether the topology of the best performing DNNs shows the power law similar to biological and social networks and how to use the power law topology to construct well-performing and compact DNNs. We first find that the connectivities of sparse DNNs can be modeled by truncated power law distribution, which is one of the variations of the power law. The comparison of different DNNs reveals that the best performing networks correlated highly with the power law distribution. We further model the preferential attachment in DNNs evolution and find that continual learning in networks with growth in tasks correlates with the process of preferential attachment. These identified power law dynamics in DNNs can lead to the construction of highly accurate and compact DNNs based on preferential attachment. Inspired by the discovered findings, two novel applications have been proposed, including evolving optimal DNNs in sparse network generation and continual learning tasks with efficient network growth using power law dynamics. Experimental results indicate that the proposed applications can speed up training, save storage, and learn with fewer samples than other well-established baselines. Our demonstration of preferential attachment and power law in well-performing DNNs offers insight into designing and constructing more efficient deep learning.

Optimal Deep Neural Network Modified by Advanced Seasons Optimizer for Classification of Olympic Sports Images

Sports image classification has gained increasing significance in various domains, including athlete training, event management, and broadcasting. Nonetheless, accurately classifying these images remains a formidable challenge due to factors such as variations in lighting conditions, pose, and attire. This paper proposes an advanced deep learning approach to address this issue. The proposed method uses a modified Inception-V4 (IV4) architecture, which has exhibited remarkable success in image classification tasks. To further enhance the performance of the IV4 model, an advanced version of the seasons optimizer (ASO), a potent optimization algorithm, has been introduced, which adjusts the learning rate during training. This modified optimizer enables the IV4 model to converge more rapidly and achieve higher accuracy. An evaluation of the proposed method has been conducted using a comprehensive dataset comprising Olympic sports images. The experimental results demonstrate significant enhancements over existing approaches, attaining superior performance across multiple evaluation metrics. These findings underscore the potential of combining cutting-edge deep neural networks with advanced optimization techniques for the precise and efficient classification of sports images.

A Deep Learning Approach for Chromium Detection and Characterization from Soil Hyperspectral Data.

High levels of chromium (Cr) in soil pose a significant threat to both humans and the environment. Laboratory-based chemical analysis methods for Cr are time consuming and expensive; thus, there is an urgent need for a more efficient method for detecting Cr in soil. In this study, a deep neural network (DNN) approach was applied to the Land Use and Cover Area frame Survey (LUCAS) dataset to develop a hyperspectral soil Cr content prediction model with good generalizability and accuracy. The optimal DNN model was constructed by optimizing the spectral preprocessing methods and DNN hyperparameters, which achieved good predictive performance for Cr detection, with a correlation coefficient value of 0.79 on the testing set. Four important hyperspectral bands with strong Cr sensitivity (400-439, 1364-1422, 1862-1934, and 2158-2499 nm) were identified by permutation importance and local interpretable model-agnostic explanations. Soil iron oxide and clay mineral content were found to be important factors influencing soil Cr content. The findings of this study provide a feasible method for rapidly determining soil Cr content from hyperspectral data, which can be further refined and applied to large-scale Cr detection in the future.

Optimizing DNN training with pipeline model parallelism for enhanced performance in embedded systems

Deep Neural Networks (DNNs) have gained widespread popularity in different domain applications due to their dominant performance. Despite the prevalence of massively parallel multi-core processor architectures, adopting large DNN models in embedded systems remains challenging, as most embedded applications are designed with single-core processors in mind. This limits DNN adoption in embedded systems due to inefficient leveraging of model parallelization and workload partitioning. Prior solutions attempt to address these challenges using data and model parallelism. However, they lack in finding optimal DNN model partitions and distributing them efficiently to achieve improved performance.This paper proposes a DNN model parallelism framework to accelerate model training by finding the optimal number of model partitions and resource provisions. The proposed framework combines data and model parallelism techniques to optimize the parallel processing of DNNs for embedded applications. In addition, it implements the pipeline execution of the partitioned models and integrates a task controller to manage the computing resources. The experimental results for image object detection demonstrate the applicability of our proposed framework in estimating the latest execution time and reducing overall model training time by almost 44.87% compared to the baseline AlexNet convolutional neural network (CNN) model.

A deep neural network-based method to predict J-integral for surface cracked plates under biaxial loading

As surface cracks may cause potential failure for engineering structures, it is of vital significance to carry out the fracture assessment for cracked structures. The crack driving force in the form of J-integral is an important input parameter in fracture assessment. This paper focuses on the determination of J-integral for surface cracked plates under biaxial loading. A method for predicting the J-integral based on the deep neural network (DNN) is proposed to estimate J-integral efficiently and accurately. Firstly, extensive three-dimensional (3D) elastic–plastic finite element (FE) analysis is conducted to compute the J-integral along the crack front for developing a FE J-integral datasets containing 1600 cases. Various crack aspect ratios, crack depth ratios, strain hardening exponents, ratios of the loading perpendicular to the crack plane to yield stress, and biaxial ratios, are strategically considered. Secondly, DNN models for predicting the J-integral are constructed and trained according to the obtained datasets by FE analysis. Then, the evaluation criteria for DNN models and the grid search method are utilized to determine the optimal DNN model structure. Finally, test cases are employed to verify the feasibility and prediction accuracy of the determined DNN model. Based on validation results, the determined DNN model exhibits sufficiently accurate prediction performance. This research provides an idea for engineering applications, through building software based on continuously optimized DNN models, the J-integral for surface cracked plates under biaxial loading can be accurately and efficiently predicted for performing fracture assessments.

Infrared microspectroscopy and machine learning: A novel approach to determine the origin and variety of individual rice grains

Accurately distinguishing the origin and variety of rice types is of paramount importance to conducting research on this staple crop. While various methods are currently employed for this purpose, few approaches can verify the identity of single grains rapidly and accurately. In this study, we present a method that integrates machine learning with infrared (IR) microspectroscopy for swift detection of the origin and variety of a single rice grain. To establish the validity of our approach, we assembled a diverse collection of rice samples, comprising 14 distinct types with different origins or varieties. Each rice sample yielded 100 microspectroscopy spectra, resulting in 1400 spectra. We applied two deep learning algorithms, deep neural network (DNN) and convolutional neural network (CNN), for spectral analysis. The 1400 spectra were randomly partitioned into calibration and validation sets at a ratio of 3:1. These datasets were subjected to both DNN and CNN analysis for classification of samples by origin and variety. Following 10,000 iterations, we selected optimal DNN and CNN models. The predication accuracies of the optimal DNN model for calibration and validation sets were 95.4% and 90.0%, respectively. In comparison, the optimal CNN model demonstrated superior accuracy, with 99.8% for the calibration set and 92.0% for the validation set. Based on these results, we selected the CNN model as the final model for field use in rice grain classification.

Optimizing a deep learning framework for accurate detection of the Earth’s ionospheric plasma structures from all-sky airglow images

The ionosphere, part of the Earth’s atmosphere, where different plasma irregularities/structures are generated through various electrodynamical processes. Airglow imaging is one of the tools to observe various morphological features of these irregularities. We aim to help all-sky airglow imaging-based ionospheric research by proposing novel deep learning at the edge framework for the detection of different types of mid-latitude ionospheric plasma structures (medium scale traveling ionospheric disturbances with single and multiple bands, and plasma bubbles). The primary challenge was to find an optimal deep neural network by considering the trade-off between accuracy and inference time. In order to address this, a novel hyperparameter optimization technique has been used that integrates the approaches of random and grid search mechanisms as a compact function. The random function generates the subspace for hyperparameters to check the convergence of the model while the grid search creates possible combinations to tune these hyperparameters. Our novel optimization method for deep learning inference at the edge led us to a low-cost, high-accuracy convolutional neural network (CNN) model, which outperformed the complex state-of-the-art deep learning models such as Inception-v3, DenseNet169, VGG16, and VGG19. For airglow image-based plasma structure detection, the proposed model recorded an accuracy of 99.9% with an inference time of 5.8 s. This was an improvement of about 4% in accuracy while 85% reduction in inference time over the best-performing baseline. There was about 93% reduction in the number of model parameters with respect to the best-performing baseline. Uncertainty quantification has also been performed in the present work through bootstrapping to validate the robustness of the proposed model. The findings of our study show the promise of deep learning at the edge for all-sky airglow imaging systems by demonstrating an alternate low-cost, high-accuracy deep neural network.

Deep neural networks optimization for resource-constrained environments: techniques and models

This paper aims to present a comprehensive review of advanced techniques and models with a specific focus on deep neural network (DNN) for resource-constrained environments (RCE). The paper contributes by highlighting the RCE devices, analyzing challenges, reviewing a broad range of optimization techniques and DNN models, and offering a comparative assessment. The findings provide potential optimization techniques and recommend a baseline model for future development. It encompasses a broad range of DNN optimization techniques, including network pruning, weight quantization, knowledge distillation, depthwise separable convolution, residual connections, factorization, dense connections, and compound scaling. Moreover, the review analyzes the established optimization models which utilizes the above optimization techniques. A comprehensive analysis is conducted for each technique and model, considering its specific attributes, usability, strengths, and limitations in the context of effective deployment in RCEs. The review also presents a comparative assessment of advanced DNN models’ deployment for image classification, employing key evaluation metrics such as accuracy and efficiency factors like memory and inference time. The article concludes with the finding that combining depthwise separable convolution, weight quantization, and pruning represents potential optimization techniques, while also recommending EfficientNetB1 as a baseline model for the future development of optimization models in RCE image classification.<p> </p>

High-performing neural network models of visual cortex benefit from high latent dimensionality.

Geometric descriptions of deep neural networks (DNNs) have the potential to uncover core representational principles of computational models in neuroscience. Here we examined the geometry of DNN models of visual cortex by quantifying the latent dimensionality of their natural image representations. A popular view holds that optimal DNNs compress their representations onto low-dimensional subspaces to achieve invariance and robustness, which suggests that better models of visual cortex should have lower dimensional geometries. Surprisingly, we found a strong trend in the opposite direction-neural networks with high-dimensional image subspaces tended to have better generalization performance when predicting cortical responses to held-out stimuli in both monkey electrophysiology and human fMRI data. Moreover, we found that high dimensionality was associated with better performance when learning new categories of stimuli, suggesting that higher dimensional representations are better suited to generalize beyond their training domains. These findings suggest a general principle whereby high-dimensional geometry confers computational benefits to DNN models of visual cortex.

Optimal Deep Convolutional Neural Network based Fusion Model for Soil Nutrient Analysis

The vast majority of people in India, agriculture is their main line of work, and it has a large economic impact.. Soil is important for supplying vital nutrients to crops for better yield. Determining soil nutrients is certainly essential for selecting appropriate crops and monitoring growth. Common methods used by agriculturalists are inadequate to satisfy increasing demands and have to obstruct cultivating soil. For a better crop yield, agriculturalists must have an awareness regarding the soil nutrients for a specific crop. There comes a need for using Deep learning methods in soil analysis that would help farmers in the domain. This study introduces an Optimal Deep Convolutional Neural Network Fusion Model for Soil nutrient Type Classification (ODCNNF-STC) technique. The presented ODCNNF-STC technique examines the input soil images to classify them into different nutrients present in the soil. In this approach, the noise present in the soil images are initially filtered using a bilateral filter (BF) followed by contrast enhancement. The preprocessed soil images are fed to the model formed by the fusion of DenseNet201 and InceptionResNetV2 models extracting the soil images that can successfully differentiate soil nutrients. Finally, classification of soil nutrients were performed by three classifiers namely extreme learning machine (ELM), RMSProp optimizer-based 1DCNN, and RMSProp optimizer-based Stacked Auto Encoder (SAE). The experimental validation of ODCNNF-STC method is examined on real-time dataset of soil images with a maximum accuracy of 99.39% over recent methods.

A2DWQPE: Adaptive and automated data-driven water quality parameter estimation

Accurate remote sensing estimation of inland water quality parameters (WQPs) plays a crucial role in guiding water resource management. To achieve this, researchers have explored various data-driven approaches utilizing machine learning (ML) techniques. However, there are two major challenges in WQPs estimation for inland waters. Firstly, current data-driven approaches focus on building a unified estimation model for an entire study area, which underestimates the complex dynamics of water constituents and optical properties. Secondly, ML models, particularly neural networks, require extensive hyperparameter tuning and are not user-friendly for researchers lacking relevant background and experience. In this paper, we propose an innovative method called adaptive and automated data-driven water quality parameter estimation (A2DWQPE) to address both challenges. Our method operates under the assumption that water bodies with similar spectral characteristics should share the same WQP estimation model. A2DWQPE is composed of three phases. Firstly, water types are automatedly classified by unsupervised hierarchical clustering according to spectral similarity. Then, optimal Deep Neural Network (DNN) models for estimating WQPs from multi-spectral satellite images are customized for each water type utilizing Bayesian optimization (BO). Finally, the target WQP is estimated based on the type-specific estimates and degree of membership of each water type. To evaluate the effectiveness of A2DWQPE, we applied it to estimate Secchi disk depth (SDD) in Lake Erie with in situ measurements and Moderate Resolution Imaging Spectroradiometer (MODIS) images. The results demonstrate that A2DWQPE outperforms the traditional approaches of developing a unified model for the entire study area. A2DWQPE achieved high accuracy with coefficient of determination (R2) over 0.72 and root mean square error (RMSE) below 1.4 m. Our method also outperforms the methods that applied Genetic Algorithm (GA) and Particle Swarm Optimization (PSO) instead of BO, and several traditional ML algorithms. We firmly believe that A2DWQPE holds great potential for accurate inland water quality estimation and will contribute significantly to various applications in water quality monitoring and pollution prevention.

Intrusion detection system using metaheuristic fireworks optimization based feature selection with deep learning on Internet of Things environment

AbstractInternet of Things (IoT), cloud computing, and other significant advancements in communication have created new security challenges. Due to these advancements and the ineffectiveness of the current security measures, cyber‐attacks are also increasing quickly. Recently, several artificial intelligence (AI)–based solutions have been presented for various secure applications, such as intrusion detection. This article proposes an intrusion detection system using dynamic search fireworks optimization–based feature selection with optimal deep recurrent neural network (DFWAFS‐ODRNN) model in IoT environment. The presented DFWAFS‐ODRNN model follows a two‐stage process, namely, feature selection and intrusion classification. In the first phase, the DFWAFS‐ODRNN model elects an optimal subset of features using the dynamic search fireworks optimization algorithm (DFWAFS) technique. Next, in the second stage, the intrusions are identified and categorized using the DRNN model. At last, the hyperparameters of the DRNN are optimally chosen by the Nadam optimizer. A detailed simulation analysis of the DFWAFS‐ODRNN model is validated on benchmark intrusion detection system (IDS) dataset, and the outcomes show the efficacy of intrusion detection. The proposed model efficiently detects the intrusion detection with an accuracy of 96.11%.

Low power high speed FPGA design of lossless medical image compression using optimal deep neural network

Low power high speed FPGA design of lossless medical image compression using optimal deep neural network

Hardware-aware approach to deep neural network optimization

Deep neural networks (DNNs) have been a pivotal technology in a myriad of fields, boasting remarkable achievements. Nevertheless, their substantial workload and inherent redundancies pose ongoing challenges for both practitioners and academia. While numerous researchers endeavor to optimize DNNs, the inherent parallelism features of hardware are generally underutilized, resulting in inefficient use of hardware resources. To address this deficit, the paper unveils a hardware-aware mechanism, IHSOpti, which incorporates hardware characteristics with software algorithms for DNN optimization. IHSOpti endeavors to exploit the full potential of modern hardware parallelism, with significant emphasis on pipelining mechanisms. Specifically, IHSOpti formulates an advanced sparse training algorithm Polar_HSPG which incorporates the newly-proposed layer-wise refined polarization regularizer (LWPolar), grounded on the half-space project gradient (HSPG). Subsequently, IHSOpti pioneeringly introduces the residual strategy for optimizing the layer-level redundancies of neural networks, capitalizing on the pipelining attributes inherent in current hardware. Experimental findings demonstrate that IHSOpti attains outstanding pruning ratios in both parameters and FLOPs. Specifically, IHSOpti achieves up to 96.90% and 82.73% pruning ratios with the accuracy of 93.34% for VGGBN, 97.69% and 95.24% pruning ratios with the accuracy of 94.69% for ResNet, 98.07% and 97.80% pruning ratios with the accuracy of 95.73% for the cutting-edge network RegNet, respectively. Notably, the running efficiency exhibits remarkable improvements with accelerations ranging from 3.63× to 8.20× for CPUs and 1.22× to 2.25× for GPUs, respectively. These outcomes surpass the latest advances in the field. Through the incorporation of specific hardware characteristics, IHSOpti provides a comprehensive and effective approach to harness the intrinsic parallelism of contemporary hardware platforms for DNNs.

Fault diagnosis in asynchronous motors based on an optimal deep bidirectional long short-term memory networks

Fault diagnosis of asynchronous motors has become a pressing need in the metallurgical industry. Due to the complex structure of asynchronous motors, fault types and fault characteristics are diverse, with strong nonlinear relationships between them, which leads to the difficulty of fault diagnosis. To efficiently and accurately diagnose various motor faults, we propose a fault diagnosis method based on an optimal deep bidirectional long short-term memory neural network. First, the three-phase current, multidimensional vibrational signal, and acoustic signal of the asynchronous motor are collected and construct diverse and robust data sample set to enhance the generalization ability of the model. Next, a modified 3D logistics-sine complex chaotic map (3D LSCCM) is constructed to improve the global and local search capabilities of the pigeon swarm optimization algorithm (PIO). Then, we construct a deep bidirectional long short-term memory network (Bid-LSTM) with attention mechanism to mine high-value fault characteristic information. Meanwhile, the optimal hyper-parameters of the deep ABid-LSTM are explored using the modified PIO to improve the training performance of the model. Finally, the fault data samples of asynchronous motor are induced to train and test the proposed framework. By fusing diverse data samples, the proposed method outperforms conventional deep Bid-LSTM and achieves fault diagnosis accuracy of 99.13%. It provides a novel diagnostic strategy for motor fault diagnosis.

Optimal Deep Recurrent Neural Networks for IoT-enabled Human Activity Recognition in Elderly and Disabled Persons

Aging is related to a decrease in the ability to execute activities of day-to-day routine and decay in physical exercise, which affect mental and physical health. Elderly patients or people can depend on a human activity recognition (HAR) system, which monitors the activity interventions and patterns if any critical event or behavioral changes occur. A HAR system incorporated with the Internet of Things (IoT) environment might allow these people to live independently. While the number of groups of activities and sensor measurements is enormous, the HAR problem could not be resolved deterministically. Hence, machine learning (ML) algorithm was broadly applied for the advancement of the HAR system to find the patterns of human activity from the sensor data. Therefore, this study presents an Optimal Deep Recurrent Neural Networks for Human Activity Recognition (ODRNN-HAR) on Elderly and Disabled Persons technique in the IoT platform. The intension of the ODRNN-HAR approach lies in the recognition and classification of various kinds of human activities in the IoT environment. Primarily, the ODRNN-HAR technique enables IoT devices to collect human activity data and employs Z-score normalization as a preprocessing step. For effectual recognition of human activities, the ODRNN-HAR technique uses the DRNN model. At the final stage, the optimal hyperparameter adjustment of the DRNN model takes place using the mayfly optimization (MFO) algorithm. The result analysis of the ODRNN-HAR algorithm takes place on benchmark HAR dataset, and the outcomes are examined. The comprehensive simulation outcomes highlighted the improved recognition results of the ODRNN-HAR approach in terms of different measures.

PMP: A partition-match parallel mechanism for DNN inference acceleration in cloud–edge collaborative environments

To address the challenges of delay-sensitive deep learning tasks, Deep Neural Network (DNN) models are often partitioned and deployed to the cloud–edge environment for parallel and collaborative inference. However, existing parallel coordination mechanisms are not suitable for the cloud–edge environment, as the high inter-layer dependence of DNNs can increase transmission latency and wait times for inference, which contradicts the advantage of low latency in edge computing. To resolve this contradiction, the PMP mechanism takes into account the inter-layer transfer dependence of partitioning solutions and employs a multi-objective equalization algorithm to derive DNN model partitioning strategies suitable for multi-way parallel computing. Moreover, the mechanism establishes a DNN inference time prediction model based on these partitions and utilizes an iterative matching algorithm to approximate an optimal DNN inference workflow. Extensive evaluations of the proposed mechanism are conducted using various DNN models, and the results demonstrate its superiority over existing schemes, including local, CoEdge, and EdgeFlow. Notably, PMP achieves significant reductions in total inference latency compared to these schemes, with reductions of 80.9%, 37.9%, and 9.1%, respectively.

Neural Architecture Search (NAS) for designing optimal power quality disturbance classifiers

Deep learning techniques have recently demonstrated outstanding success when used for Power Quality Disturbance (PQD) classification. However, a core obstacle is that deep neural networks (DNN)s are complex models, and their architecture is designed using trial and error processes. Accordingly, the problem of finding the optimal architecture can be considered as a problem that consists of high-dimensional solutions. Meanwhile, in the last couple of years, Neural Architecture Search (NAS) techniques have been developed to efficiently find the best possible performance architecture for a specific task. In this light, the goal of this research is to develop a method to find optimal PQD classifiers using the NAS technique, based on an evolutionary algorithm. This method can converge efficiently to an optimal DNN architecture. Thus, a classifier that achieves high accuracy for PQDs classification is provided using limited resources and with minimal human intervention. This idea is demonstrated on two different DNN typologies—convolutional neural networks (CNN) and Bi-directional long short-term memory (Bi-LSTM). By adopting this method, the results of the generated PQD classifiers are more accurate when compared to recently developed classifiers.