Application Of Deep Neural Networks Research Articles

Application of novel deep neural network on prediction of compressive strength of fly ash based concrete

Application of novel deep neural network on prediction of compressive strength of fly ash based concrete

Speech Emotion Recognition Using Feedforward Neural Network

Speech Emotion Recognition (SER) has gained increasing attention due to its application in fields such as human-computer interaction, healthcare, customer service automation, and affective computing. This review focuses on the application of Deep Neural Networks (DNNs), specifically Feedforward Neural Networks (FNNs), in detecting and classifying emotions from speech signals. Recent advancements in deep learning have significantly enhanced the ability of machines to interpret emotional cues from auditory data. This paper discusses the motivation, objectives, and scope of employing DNN FNN architectures for SER. We also address the technical feasibility and potential of this approach for real-time applications. By reviewing existing literature, this study aims to provide insights into the current progress, challenges, and future prospects of SER systems.

Robust neural networks using stochastic resonance neurons.

Various successful applications of deep artificial neural networks are effectively facilitated by the possibility to increase the number of layers and neurons in the network at the expense of the growing computational complexity. Increasing computational complexity to improve performance makes hardware implementation more difficult and directly affects both power consumption and the accumulation of signal processing latency, which are critical issues in many applications. Power consumption can be potentially reduced using analog neural networks, the performance of which, however, is limited by noise aggregation. Following the idea of physics-inspired machine learning, we propose here a type of neural network using stochastic resonances as a dynamic nonlinear node and demonstrate the possibility of considerably reducing the number of neurons required for a given prediction accuracy. We also observe that the performance of such neural networks is more robust against the impact of noise in the training data compared to conventional networks.

Repairs and Breaks Prediction for Deep Neural Networks

With the increasing prevalence of software incorporating deep neural networks (DNNs), quality assurance for these software systems has become a crucial concern. To this end, various methods have been proposed to repair the misbehavior of DNNs by modifying their weights. However, these repair methods may not meet the developer's needs for a given dataset and model. In this study, we build prediction models for repair outcomes (i.e., repairs and breaks) to help determine whether the repair method is likely to work. By using our prediction models, developers and operators of DNNs can decide whether or not to apply a repair method, and if so, which method to use. Our prediction models utilize four metrics as explanatory metrics that represent the confidence or ambiguity in the DNN predictions. We experimented with four repair methods and 10 datasets. The experimental results demonstrate that our prediction models successfully select a repair method that meets developers’ needs in 16 out of 24 cases, resulting in an average time saving of 16.29% compared to the naive method. Based on these results, our prediction models can reduce costs for developers and operators when deciding whether to employ repair methods for real-world applications of DNNs.

Jaynes machine: The universal microstructure of deep neural networks

Despite the recent stunning progress in large-scale deep neural network applications, our understanding of their microstructure, ‘energy’ functions, and optimal design remains incomplete. Here, we present a new game-theoretic framework, called statistical teleodynamics, that reveals important insights into these key properties. The optimally robust design of such networks inherently involves computational benefit-cost trade-offs that physics-inspired models do not adequately capture. These trade-offs occur as neurons and connections compete to increase their effective utilities under resource constraints during training. In a fully trained network, this results in a state of arbitrage equilibrium, where all neurons in a given layer have the same effective utility, and all connections to a given layer have the same effective utility. The equilibrium is characterized by the emergence of two lognormal distributions of connection weights and neuronal output as the universal microstructure of large deep neural networks. We call such a network the Jaynes Machine. Our theoretical predictions are shown to be supported by empirical data from seven large-scale deep neural networks. We also show that the Hopfield network and the Boltzmann Machine are the same special case of the Jaynes Machine.

Invisible backdoor attack with attention and steganography

Recently, with the development and widespread application of deep neural networks (DNNs), backdoor attacks have posed new security threats to the training process of DNNs. Backdoor attacks on neural networks undermine the security and trustworthiness of DNNs by implanting hidden, unauthorized triggers, leading to benign behavior on clean samples while exhibiting malicious behavior on samples containing backdoor triggers. Existing backdoor attacks typically employ triggers that are sample-agnostic and identical for each sample, resulting in poisoned images that lack naturalness and are ineffective against existing backdoor defenses. To address these issues, this paper proposes a novel stealthy backdoor attack, where the backdoor trigger is dynamic and specific to each sample. Specifically, we leverage spatial attention on images and pre-trained models to obtain dynamic triggers, which are then injected using an encoder–decoder network. The design of the injection network benefits from recent advances in steganography research. To demonstrate the effectiveness of the proposed steganographic network, we design two backdoor attack modes named ASBA and ATBA, where ASBA utilizes the steganographic network for attack, while ATBA is a backdoor attack without steganography. Subsequently, we conducted attacks on Deep Neural Networks (DNNs) using four standard datasets. Our extensive experiments show that ASBA surpasses ATBA in terms of stealthiness and resilience against current defensive measures. Furthermore, both ASBA and ATBA demonstrate superior attack efficiency.

Digital Grading Company: Deep Neural Networks for Predictive Analytics for Alternative Asset Portfolio Management

In this study, we investigate the application of deep neural networks for predictive analytics in the realm of alternative asset portfolio management, focusing on Trading Card Games (TCGs) like Pokemon. Our primary objective is to understand the optimal trading cards for investment and to determine the appropriate timing for divestment and retention of these assets. Drawing on the successful implementation of machine learning predictive analytics in traditional financial markets, we aim to extend these techniques to alternative investments. To achieve this, we employ a two-pronged approach using a comprehensive dataset of historical sales values for Pokemon TCGs spanning five years, comprising over 200 million unlabeled data points. The first approach utilizes a semi-supervised sequence learning method, where a Long Short-Term Memory (LSTM) model is initially trained as a sequence autoencoder. The pre-trained model's weights are then used in a supervised phase to fine-tune the predictions for future trading card values. The second approach involves heuristic pretraining, which leverages market-derived indicators from existing literature to bridge the gap between traditional feature engineering and deep learning methods. Our technology stack is designed to handle end-to-end data collection, analytics generation, and execution of recommended trading strategies. Simulated backtesting results demonstrate that our Digital Grading Company (DGC) portfolio model outperforms traditional indices, achieving a 357% return compared to the 61% return of the S&P 500 Index over the same period. Real-time analysis further indicates that our model consistently identifies profitable trading opportunities in the TCG market using hourly data and proprietary algorithmic trade logic, executing trades 24/7. The DGC Predictive Portfolio, based on ungraded Pokemon TCG trading cards, also shows superior performance when compared to the Nasdaq 100 Index (Invesco QQQ ETF). Our analytics framework not only predicts the direction of price movement but also provides insights into the magnitude of changes, enabling a more dynamic and responsive trading strategy. The findings suggest that the integration of deep learning models with heuristic indicators has the potential to revolutionize portfolio management in alternative investments, paving the way for more accurate and profitable decision-making in niche markets. This research highlights the promising role of machine learning in enhancing alternative asset management strategies, suggesting future applications in other asset classes. We conclude that deep neural networks, when combined with data-driven heuristics, can significantly enhance trading outcomes, making them valuable tools for investors in the evolving landscape of alternative investments.

UNeXt: An Efficient Network for the Semantic Segmentation of High-Resolution Remote Sensing Images

The application of deep neural networks for the semantic segmentation of remote sensing images is a significant research area within the field of the intelligent interpretation of remote sensing data. The semantic segmentation of remote sensing images holds great practical value in urban planning, disaster assessment, the estimation of carbon sinks, and other related fields. With the continuous advancement of remote sensing technology, the spatial resolution of remote sensing images is gradually increasing. This increase in resolution brings about challenges such as significant changes in the scale of ground objects, redundant information, and irregular shapes within remote sensing images. Current methods leverage Transformers to capture global long-range dependencies. However, the use of Transformers introduces higher computational complexity and is prone to losing local details. In this paper, we propose UNeXt (UNet+ConvNeXt+Transformer), a real-time semantic segmentation model tailored for high-resolution remote sensing images. To achieve efficient segmentation, UNeXt uses the lightweight ConvNeXt-T as the encoder and a lightweight decoder, Transnext, which combines a Transformer and CNN (Convolutional Neural Networks) to capture global information while avoiding the loss of local details. Furthermore, in order to more effectively utilize spatial and channel information, we propose a SCFB (SC Feature Fuse Block) to reduce computational complexity while enhancing the model’s recognition of complex scenes. A series of ablation experiments and comprehensive comparative experiments demonstrate that our method not only runs faster than state-of-the-art (SOTA) lightweight models but also achieves higher accuracy. Specifically, our proposed UNeXt achieves 85.2% and 82.9% mIoUs on the Vaihingen and Gaofen5 (GID5) datasets, respectively, while maintaining 97 fps for 512 × 512 inputs on a single NVIDIA GTX 4090 GPU, outperforming other SOTA methods.

How can geostatistics help us understand deep learning? An exploratory study in SAR-based aircraft detection

Deep Neural Networks (DNNs) have garnered significant attention across various research domains due to their impressive performance, particularly Convolutional Neural Networks (CNNs), known for their exceptional accuracy in image processing tasks. However, the opaque nature of DNNs has raised concerns about their trustworthiness, as users often cannot understand how the model arrives at its predictions or decisions. This lack of transparency is particularly problematic in critical fields such as healthcare, finance, and law, where the stakes are high. Consequently, there has been a surge in the development of explanation methods for DNNs. Typically, the effectiveness of these methods is assessed subjectively via human observation on the heatmaps or attribution maps generated by eXplanation AI (XAI) methods. In this paper, a novel GeoStatistics Explainable Artificial Intelligence (GSEAI) framework is proposed, which integrates spatial pattern analysis from Geostatistics with XAI algorithms to assess and compare XAI understandability. Global and local Moran’s I indices, commonly used to assess the spatial autocorrelation of geographic data, assist in comprehending the spatial distribution patterns of attribution maps produced by the XAI method, through measuring the levels of aggregation or dispersion. Interpreting and analyzing attribution maps by Moran’s I scattergram and LISA clustering maps provide an accurate global objective quantitative assessment of the spatial distribution of feature attribution and achieves a more understandable local interpretation. In this paper, we conduct experiments on aircraft detection in SAR images based on the widely used YOLOv5 network, and evaluate four mainstream XAI methods quantitatively and qualitatively. By using GSEAI to perform explanation analysis of the given DNN, we could gain more insights about the behavior of the network, to enhance the trustworthiness of DNN applications. To the best of our knowledge, this is the first time XAI has been integrated with geostatistical algorithms in SAR domain knowledge, which expands the analytical approaches of XAI and also promotes the development of XAI within SAR image analytics.

A Distance Transformation Deep Forest Framework With Hybrid-Feature Fusion for CXR Image Classification.

Detecting pneumonia, especially coronavirus disease 2019 (COVID-19), from chest X-ray (CXR) images is one of the most effective ways for disease diagnosis and patient triage. The application of deep neural networks (DNNs) for CXR image classification is limited due to the small sample size of the well-curated data. To tackle this problem, this article proposes a distance transformation-based deep forest framework with hybrid-feature fusion (DTDF-HFF) for accurate CXR image classification. In our proposed method, hybrid features of CXR images are extracted in two ways: hand-crafted feature extraction and multigrained scanning. Different types of features are fed into different classifiers in the same layer of the deep forest (DF), and the prediction vector obtained at each layer is transformed to form distance vector based on a self-adaptive scheme. The distance vectors obtained by different classifiers are fused and concatenated with the original features, then input into the corresponding classifier at the next layer. The cascade grows until DTDF-HFF can no longer gain benefits from the new layer. We compare the proposed method with other methods on the public CXR datasets, and the experimental results show that the proposed method can achieve state-of-the art (SOTA) performance. The code will be made publicly available at https://github.com/hongqq/DTDF-HFF.

Robust estimation of hydrogeological parameters from wireline logs usingsemi-supervised deep neural networks assisted with global optimization-based regression methods

Understanding the distribution of hydrogeological properties of the aquifers is crucial for sustainable groundwater resource development. This research explores the application of deep autoencoder neural networks (AE-NN), assisted with global optimization methods for estimating hydrogeological parameters in the Quaternary aquifer system in the Debrecen area, Hungary. Traditional methods for estimating aquifer parameters typically depend on field experiments and laboratory analyses, which are both costly and time-consuming, and often fail to account for the heterogeneity of groundwater formations. In this study, deep AE-NN models are trained to extract latent space (LS) representations that capture key features from the available well logs, including spontaneous potential (SP), natural gamma ray (NGR), shallow resistivity (RS), and deep resistivity (RD). The LS log is then correlated with shale volume and hydraulic conductivity, as determined by the Larionov and Csókás methods, respectively. Regression analysis revealed a Gaussian relationship between the LS log and shale volume and a negative nonlinear relationship with hydraulic conductivity. Global optimization methods, including simulated annealing (SA) and particle swarm optimization (PSO), were used to refine the regression parameters, enhancing the predictive capabilities of the models. The results demonstrated that AE-NN assisted with global optimization methods can be effectively used to estimate shale volume and hydraulic conductivity, proposing a novel and independent approach for estimating hydrogeological parameters critical to groundwater flow and contaminant transport modeling.

Deep neural networks-prescribed performance optimal control for stochastic nonlinear strict-feedback systems

This article explores the application of deep neural networks (DNNs) for optimized backstepping control design in a category of stochastic nonlinear strict-feedback systems with unknown dynamics, focusing on prescribed performance. DNNs, distinguished by their ability to handle high-complexity functions and enhance function approximation, are employed to estimate unknown nonlinear functions. The weight updating strategy for each layer of DNNs is determined through a Lyapunov function analysis. Tomitigate potential issues such as the tracking error exploding at uncertain moments, prescribed performance control (PPC) is introduced to constrain the tracking error within a predetermined range. Subsequently, a novel optimal tracking control scheme, integrating the backstepping method and Hamilton–Jacobi–Bellman (HJB) equation, is presented. The results indicate that all signals in the closed-loop system are bounded in probability, and the tracking error is maintained within a set of predefined arbitrarily small residuals. Simulation outcomes affirm the efficacy of the proposed control scheme.

Development of data-driven models to predict seismic drift response of RC wall structures: An application of deep neural networks

This research aimed to develop data-driven models using deep neural networks (DNNs) that can rapidly predict the seismic drift responses of reinforced concrete (RC) wall structures in building frame systems, aiming to overcome the challenges of costly seismic performance evaluation of building structures. A total of 46 RC wall structures ranging from four to 40 stories were analyzed and subjected to 1,000 ground motions including far-field, near-field-pulse, and near-field-no-pulse types. Two input sets were investigated initially. The first input set comprises peak ground acceleration (PGA), spectral accelerations at 1 s–6 s with 1-s intervals, and the first five natural periods of the wall structure. The second input set contains PGA and spectral accelerations at the first five natural periods of the wall structure. The DNN model developed based on the first input set demonstrated superior accuracy achieving an R2 value of 0.880, and slightly outperformed other reputable machine learning models. To enhance the applicability of the developed model, the ground motion type was incorporated as an additional variable to the first input set, which led to an R2 value of 0.869. Ultimately, two DNN models, one based on the first input set and the other considering the ground motion type, were proposed in this study. Finally, the two models were applied to quickly predict seismic drift responses of a new 24-story RC wall structure, and the predicted results were then utilized to efficiently construct the fragility function. The findings highlight the potential of DNNs as an efficient solution to address complex and costly challenges in the earthquake engineering domain.

Crack detection in buildings using the YOLO v8 network

The objective of this study is to develop and apply deep neural networks for the automation of crack detection in buildings. The methodology involved training the YOLO v8 network with images collected from the internet, aiming to identify and locate cracks in real time. The model obtained 80% accuracy in validation with images not used in training, despite performance limitations in Google Collab. These limitations included restrictions on the execution environment, and the model is specific to cracks. The originality of the tool lies in its relevance for the automated detection of cracks, with the potential to extend to other pathological manifestations. It is concluded that the application of deep neural networks offers an efficient solution for the identification of problems in buildings.

Gender and Age Detection using Deep Learning

The demand for automatic gender and age prediction has surged with the proliferation of social media and online platforms. Despite advancements in related fields like facial recognition, the effectiveness of current technologies in real-world scenarios remains limited. This paper explores the application of deep convolutional neural networks (CNNs) to address this challenge. Our proposed method involves a five-step process: facial recognition, background removal, face alignment, application of multiple CNN models, and a voting mechanism to enhance prediction accuracy. We evaluate our approach using the recent Audience-Face benchmark dataset, focusing on gender detection and age estimation. The implementation is carried out using Python. Keywords: Convolutional neural networks (CNN), Deep learning, Facial recognition, Computer vision, Region of interest (ROI).

Designing Efficient Bit-Level Sparsity-Tolerant Memristive Networks.

With the rapid progress of deep neural network (DNN) applications on memristive platforms, there has been a growing interest in the acceleration and compression of memristive networks. As an emerging model optimization technique for memristive platforms, bit-level sparsity training (with the fixed-point quantization) can significantly reduce the demand for analog-to-digital converters (ADCs) resolution, which is critical for energy and area consumption. However, the bit sparsity and the fixed-point quantization will inevitably lead to a large performance loss. Different from the existing training and optimization techniques, this work attempts to explore more sparsity-tolerant architectures to compensate for performance degradation. We first empirically demonstrate that in a certain search space (e.g., 4-bit quantized DARTS space), network architectures differ in bit-level sparsity tolerance. It is reasonable and necessary to search the architectures for efficient deployment on memristive platforms by the neural architecture search (NAS) technology. We further introduce bit-level sparsity-tolerant NAS (BST-NAS), which encapsulates low-precision quantization and bit-level sparsity training into the differentiable NAS, to explore the optimal bit-level sparsity-tolerant architectures. Experimentally, with the same degree of sparsity and experiment settings, our searched architectures obtain a promising performance, which outperform the normal NAS-based DARTS-series architectures (about 5.8% higher than that of DARTS-V2 and 2.7% higher than that of PC-DARTS) on CIFAR10.

Efficient Model Updating of a Prefabricated Tall Building by a DNN Method.

The significance of model updating methods is becoming increasingly evident as the demand for greater precision in numerical models rises. In recent years, with the advancement of deep learning technology, model updating methods based on various deep learning algorithms have begun to emerge. These methods tend to be complicated in terms of methodological architectures and mathematical processes. This paper introduces an innovative model updating approach using a deep learning model: the deep neural network (DNN). This approach diverges from conventional methods by streamlining the process, directly utilizing the results of modal analysis and numerical model simulations as deep learning input, bypassing any additional complex mathematical calculations. Moreover, with a minimalist neural network architecture, a model updating method has been developed that achieves both accuracy and efficiency. This distinctive application of DNN has seldom been applied previously to model updating. Furthermore, this research investigates the impact of prefabricated partition walls on the overall stiffness of buildings, a field that has received limited attention in the previous studies. The main finding was that the deep neural network method achieved a Modal Assurance Criterion (MAC) value exceeding 0.99 for model updating in the minimally disturbed 1st and 2nd order modes when compared to actual measurements. Additionally, it was discovered that prefabricated partitions exhibited a stiffness ratio of about 0.2-0.3 compared to shear walls of the same material and thickness, emphasizing their role in structural behavior.

Oil and gas flow anomaly detection on offshore naturally flowing wells using deep neural networks

The oil and gas industry is changing. The drive towards cleaner and safer operations is becoming increasingly important. Researchers are looking for more efficient and accurate ways to detect faults that could lead to environmental and sustainability issues. This study aims to enhance the safety and sustainability of the oil and gas industry by improving existing artificial intelligence approaches to automate monitoring and detection of malfunctions. This article explores the application of deep neural networks for anomaly detection in monitoring oil and gas flow in natural flow offshore wells, proposing an innovative approach that takes advantage of the power of Genetic Algorithms and Gated Recurrent Units (GRU). The study aims to enhance the safety and sustainability of the oil and gas industry by leveraging artificial intelligence to automate the monitoring and detection malfunctions. Utilizing a comprehensive dataset from the 3W Petrobras project, which includes real-time data from 21 wells collected between 2012 and 2018, the research focuses on detecting various anomalies such as abrupt increases in basic sediment and water, spurious closures of downhole safety valves, severe slugging, flow instability, rapid productivity loss, quick restrictions in the production choke, scaling, and hydrate formation in production lines. The methodology integrates Long Short-Term Memory (LSTM) networks and GRU backbones with genetic algorithms to optimise model performance. Several hyperparameter optimisation tools were explored innovatively, focusing mainly on Genetic Algorithms, and it was possible to obtain an algorithm with 2 stacked GRU with better comparative performance compared to what is reported in the literature and producing an F1 equal to 0.97. The findings demonstrate the potential of AI to improve real-time anomaly detection, thereby reducing operational risks and contributing to the industry's transition towards greener practices. It also underscores the importance of open data and collaborative efforts in advancing AI applications in the oil and gas sector, aligning with the United Nations' Sustainable Development Goals to mitigate climate impact and promote responsible consumption and production.

Fast Loosely-Timed Deep Neural Network Models with Accurate Memory Contention

The emergence of data-intensive applications, such as Deep Neural Networks (DNN), exacerbates the well-known memory bottleneck in computer systems and demands early attention in the design flow. Electronic System-Level (ESL) design using SystemC Transaction Level Modeling (TLM) enables effective performance estimation, design space exploration (DSE), and gradual refinement. However, memory contention is often only detectable after detailed TLM-2.0 approximately-timed or cycle-accurate RTL models are developed. A memory bottleneck detected at such a late stage can severely limit the available design choices or even require costly redesign. In this work, we propose a novel TLM-2.0 loosely-timed contention-aware (LT-CA) modeling style that offers high-speed simulation close to traditional loosely-timed (LT) models, yet shows the same accuracy for memory contention as low-level approximately-timed (AT) models. Thus, our proposed LT-CA modeling breaks the speed/accuracy tradeoff between regular LT and AT models and offers fast and accurate observation and visualization of memory contention. Our extensible SystemC model generator automatically produces desired TLM-1 and TLM-2.0 models from a DNN architecture description for design space exploration focusing on memory contention. We demonstrate our approach with a real-world industry-strength DNN application, GoogLeNet. The experimental results show that the proposed LT-CA modeling is 46× faster in simulation than equivalent AT models with an average error of less than 1% in simulated time. Early detection of memory contentions also suggests that local memories close to computing cores can eliminate memory contention in such applications.

On the applications of neural ordinary differential equations in medical image analysis

Medical image analysis tasks are characterized by high-noise, volumetric, and multi-modality, posing challenges for the model that attempts to learn robust features from the input images. Over the last decade, deep neural networks (DNNs) have achieved enormous success in medical image analysis tasks, which can be attributed to their powerful feature representation capability. Despite the promising results reported in numerous literature, DNNs are also criticized for several pivotal limits, with one of the limitations is lack of safety. Safety plays an important role in the applications of DNNs during clinical practice, helping the model defend against potential attacks and preventing the model from silent failure prediction. The recently proposed neural ordinary differential equation (NODE), a continuous model bridging the gap between DNNs and ODE, provides a significant advantage in ensuring the model’s safety. Among the variants of NODE, the neural memory ordinary differential equation (nmODE) owns the global attractor theoretically, exhibiting superiority in prompting the model’s performance and robustness during applications. While NODE and its variants have been widely used in medical image analysis tasks, there is a lack of a comprehensive review of their applications, hindering the in-depth understanding of NODE’s working principle and its potential applications. To mitigate this limitation, this paper thoroughly reviews the literature on the applications of NODE in medical image analysis from the following five aspects: segmentation, reconstruction, registration, disease prediction, and data generation. We also summarize both the strengths and downsides of the applications of NODE, followed by the possible research directions. To the best of our knowledge, this is the first review regards the applications of NODE in the field of medical image analysis. We hope this review can draw the researchers’ attention to the great potential of NODE and its variants in medical image analysis.