Application Of Deep Neural Networks Research Articles

Graph Autoencoders for Embedding Learning in Brain Networks and Major Depressive Disorder Identification.

Brain functional connectivity (FC) networks inferred from functional magnetic resonance imaging (fMRI) have shown altered or aberrant brain functional connectome in various neuropsychiatric disorders. Recent application of deep neural networks to connectome-based classification mostly relies on traditional convolutional neural networks (CNNs) using input FCs on a regular Euclidean grid to learn spatial maps of brain networks neglecting the topological information of the brain networks, leading to potentially sub-optimal performance in brain disorder identification. We propose a novel graph deep learning framework that leverages non-Euclidean information inherent in the graph structure for classifying brain networks in major depressive disorder (MDD). We introduce a novel graph autoencoder (GAE) architecture, built upon graph convolutional networks (GCNs), to embed the topological structure and node content of large fMRI networks into low-dimensional representations. For constructing the brain networks, we employ the Ledoit-Wolf (LDW) shrinkage method to efficiently estimate high-dimensional FC metrics from fMRI data. We explore both supervised and unsupervised techniques for graph embedding learning. The resulting embeddings serve as feature inputs for a deep fully-connected neural network (FCNN) to distinguish MDD from healthy controls (HCs). Evaluating our model on resting-state fMRI MDD dataset, we observe that the GAE-FCNN outperforms several state-of-the-art methods for brain connectome classification, achieving the highest accuracy when using LDW-FC edges as node features. The graph embeddings of fMRI FC networks also reveal significant group differences between MDD and HCs. Our framework demonstrates the feasibility of learning graph embeddings from brain networks, providing valuable discriminative information for diagnosing brain disorders.

Research on Adversarial Sample Detection Method Based on Image Similarity

<p>With the widespread application of deep neural networks in image detection, adversarial sample attacks have gradually become a hot issue of concern for researchers. In this paper we propose a new adversarial sample detection approach called AdvDetector, which combines image generation through label fusion with image similarity detection. AdvDetector enhances sample quality and effectively identifies adversarial samples. Specifically, the method generates images by selecting seed pixels, the labels of deep neural network classification, and the pixel distribution learned from training data, and detects them using image similarity comparison methods. During the sample generation process, we introduce the AdvDetector method for adversarial sample detection to improve the quality of generated samples. We evaluated the effectiveness of the method on three publicly available image datasets, MNIST, Cifar-10, and GTSR, and the results show that the method is superior to existing baseline methods in terms of adversarial sample detection rate and sample generation quality.</p> <p> </p>

Efficient Application of Deep Neural Networks for Identifying Small and Multiple Weed Patches Using Drone Images

Efficient Application of Deep Neural Networks for Identifying Small and Multiple Weed Patches Using Drone Images

A Distance Transformation Deep Forest Framework With Hybrid-Feature Fusion for CXR Image Classification.

Detecting pneumonia, especially coronavirus disease 2019 (COVID-19), from chest X-ray (CXR) images is one of the most effective ways for disease diagnosis and patient triage. The application of deep neural networks (DNNs) for CXR image classification is limited due to the small sample size of the well-curated data. To tackle this problem, this article proposes a distance transformation-based deep forest framework with hybrid-feature fusion (DTDF-HFF) for accurate CXR image classification. In our proposed method, hybrid features of CXR images are extracted in two ways: hand-crafted feature extraction and multigrained scanning. Different types of features are fed into different classifiers in the same layer of the deep forest (DF), and the prediction vector obtained at each layer is transformed to form distance vector based on a self-adaptive scheme. The distance vectors obtained by different classifiers are fused and concatenated with the original features, then input into the corresponding classifier at the next layer. The cascade grows until DTDF-HFF can no longer gain benefits from the new layer. We compare the proposed method with other methods on the public CXR datasets, and the experimental results show that the proposed method can achieve state-of-the art (SOTA) performance. The code will be made publicly available at https://github.com/hongqq/DTDF-HFF.

Designing Efficient Bit-Level Sparsity-Tolerant Memristive Networks.

With the rapid progress of deep neural network (DNN) applications on memristive platforms, there has been a growing interest in the acceleration and compression of memristive networks. As an emerging model optimization technique for memristive platforms, bit-level sparsity training (with the fixed-point quantization) can significantly reduce the demand for analog-to-digital converters (ADCs) resolution, which is critical for energy and area consumption. However, the bit sparsity and the fixed-point quantization will inevitably lead to a large performance loss. Different from the existing training and optimization techniques, this work attempts to explore more sparsity-tolerant architectures to compensate for performance degradation. We first empirically demonstrate that in a certain search space (e.g., 4-bit quantized DARTS space), network architectures differ in bit-level sparsity tolerance. It is reasonable and necessary to search the architectures for efficient deployment on memristive platforms by the neural architecture search (NAS) technology. We further introduce bit-level sparsity-tolerant NAS (BST-NAS), which encapsulates low-precision quantization and bit-level sparsity training into the differentiable NAS, to explore the optimal bit-level sparsity-tolerant architectures. Experimentally, with the same degree of sparsity and experiment settings, our searched architectures obtain a promising performance, which outperform the normal NAS-based DARTS-series architectures (about 5.8% higher than that of DARTS-V2 and 2.7% higher than that of PC-DARTS) on CIFAR10.

APPLICATION OF DEEP NEURAL NETWORK ON AUTONOMOUS UNDERWATER VEHICLES (AUV) IN SUBSEA PIPELINE INSPECTION

The purpose of this study is to investigate the application of deep neural network for the Automated Underwater Vehicle in the subsea pipeline inspection. In today’s modern world, we see more and more sophisticated and precise computer vision object detection technology being implemented in our daily lives. To name a few, security cameras, self-driving cars, drones and more. This research suggests that computer vision pipeline defect detection is an attractive solution for the future subsea pipeline defect detection as it relies less human intervention and is more reliable. In this paper, review of the current methods of subsea pipeline inspection was done. Some of the methods with visual detection and their limitations are also discussed. Apart from that, the machine learning algorithm of our focus, Faster RCNN is studied. Beyond that, we have explained the methods of our experimentation and the process of training as well as validating the custom dataset. The outcomes are separated into different sections, training curve, visual data representation as well as the model accuracy in different operating environment. In conclusion, we found that the model is able to detect the underwater pipeline with up to 1mm leak size. However, the low accuracy due to the insufficient dataset is recognized as a bottleneck and some of the recommendations are suggested for future improvement.

Mapping of Deep Neural Network Accelerators on Wireless Multistage Interconnection NoCs

In the last few decades, the concept of Wireless Network-on-chip (WiNoC) has emerged as a promising alternative for Multiprocessor Systems on Chip (MPSOC) to achieve reliable and scalable communication. Worth recalling in this regard is that our research team has already designed, verified and evaluated Multistage Interconnection Networks (MIN) in this field. With respect to the present work, we consider proceeding with further exploring our thoughts on this research area. Firstly, we propose the design and performance evaluation of a hybrid (wireless/wired) MIN, analysing how this augmented network can potentially improve not only the average delay, but also energy consumption. Secondly, we continue with examining the implementation of our advanced DELTA-based MIN architecture on Deep Neural Network (DNN) accelerators, while accounting for its potential regularity and scalability in simultaneously maintaining an effective power efficiency and lower latency throughout the DNN operating process. In this context, several metrics have been evaluated in regard to three DNN application cases through implementation of their main respective modules.

NeuroCool: Dynamic Thermal Management of 3D DRAM for Deep Neural Networks through Customized Prefetching

Deep neural network (DNN) implementations are typically characterized by huge datasets and concurrent computation, resulting in a demand for high memory bandwidth due to intensive data movement between processors and off-chip memory. Performing DNN inference on general-purpose cores/edge is gaining attraction to enhance user experience and reduce latency. The mismatch in the CPU and conventional DRAM speed leads to under-utilization of the compute capabilities, causing increased inference time. 3D DRAM is a promising solution to effectively fulfill the bandwidth requirement of high-throughput DNNs. However, due to high power density in stacked architectures, 3D DRAMs need dynamic thermal management (DTM), resulting in performance overhead due to memory-induced CPU throttling. We study the thermal impact of DNN applications running on a 3D DRAM system, and make a case for a memory temperature-aware customized prefetch mechanism to reduce DTM overheads and significantly improve performance. In our proposed NeuroCool DTM policy, we intelligently place either DRAM ranks or tiers in low power state, using the DNN layer characteristics and access rate. We establish the generalization of our approach through training and test datasets comprising diverse data points from widely used DNN applications. Experimental results on popular DNNs show that NeuroCool results in a average performance gain of 44% (as high as 52%) and memory energy improvement of 43% (as high as 69%) over general-purpose DTM policies.

Immunohistochemical Stain-Aided Annotation Accelerates Machine Learning and Deep Learning Model Development in the Pathologic Diagnosis of Nasopharyngeal Carcinoma.

Nasopharyngeal carcinoma (NPC) is an epithelial cancer originating in the nasopharynx epithelium. Nevertheless, annotating pathology slides remains a bottleneck in the development of AI-driven pathology models and applications. In the present study, we aim to demonstrate the feasibility of using immunohistochemistry (IHC) for annotation by non-pathologists and to develop an efficient model for distinguishing NPC without the time-consuming involvement of pathologists. For this study, we gathered NPC slides from 251 different patients, comprising hematoxylin and eosin (H&E) slides, pan-cytokeratin (Pan-CK) IHC slides, and Epstein-Barr virus-encoded small RNA (EBER) slides. The annotation of NPC regions in the H&E slides was carried out by a non-pathologist trainee who had access to corresponding Pan-CK IHC slides, both with and without EBER slides. The training process utilized ResNeXt, a deep neural network featuring a residual and inception architecture. In the validation set, NPC exhibited an AUC of 0.896, with a sensitivity of 0.919 and a specificity of 0.878. This study represents a significant breakthrough: the successful application of deep convolutional neural networks to identify NPC without the need for expert pathologist annotations. Our results underscore the potential of laboratory techniques to substantially reduce the workload of pathologists.

Computer-Aided Diagnosis of Laryngeal Cancer Based on Deep Learning with Laryngoscopic Images.

Laryngeal cancer poses a significant global health burden, with late-stage diagnoses contributing to reduced survival rates. This study explores the application of deep convolutional neural networks (DCNNs), specifically the Densenet201 architecture, in the computer-aided diagnosis of laryngeal cancer using laryngoscopic images. Our dataset comprised images from two medical centers, including benign and malignant cases, and was divided into training, internal validation, and external validation groups. We compared the performance of Densenet201 with other commonly used DCNN models and clinical assessments by experienced clinicians. Densenet201 exhibited outstanding performance, with an accuracy of 98.5% in the training cohort, 92.0% in the internal validation cohort, and 86.3% in the external validation cohort. The area under the curve (AUC) values consistently exceeded 92%, signifying robust discriminatory ability. Remarkably, Densenet201 achieved high sensitivity (98.9%) and specificity (98.2%) in the training cohort, ensuring accurate detection of both positive and negative cases. In contrast, other DCNN models displayed varying degrees of performance degradation in the external validation cohort, indicating the superiority of Densenet201. Moreover, Densenet201's performance was comparable to that of an experienced clinician (Clinician A) and outperformed another clinician (Clinician B), particularly in the external validation cohort. Statistical analysis, including the DeLong test, confirmed the significance of these performance differences. Our study demonstrates that Densenet201 is a highly accurate and reliable tool for the computer-aided diagnosis of laryngeal cancer based on laryngoscopic images. The findings underscore the potential of deep learning as a complementary tool for clinicians and the importance of incorporating advanced technology in improving diagnostic accuracy and patient care in laryngeal cancer diagnosis. Future work will involve expanding the dataset and further optimizing the deep learning model.

Application of deep neural survival networks to the development of risk prediction models for diabetes mellitus, hypertension, and dyslipidemia.

: Although numerous risk prediction models have been proposed, few such models have been developed using neural network-based survival analysis. We developed risk prediction models for three cardiovascular disease risk factors (diabetes mellitus, hypertension, and dyslipidemia) among a working-age population in Japan using DeepSurv, a deep feed-forward neural network. : Data were obtained from the Japan Epidemiology Collaboration on Occupational Health Study. A total of 51 258, 44 197, and 31 452 individuals were included in the development of risk models for diabetes mellitus, hypertension, and dyslipidemia, respectively; two-thirds of whom were used to develop prediction models, and the rest were used to validate the models. We compared the performances of DeepSurv-based models with those of prediction models based on the Cox proportional hazards model. : The area under the receiver-operating characteristic curve was 0.878 [95% confidence interval (CI) = 0.864-0.892] for diabetes mellitus, 0.835 (95% CI = 0.826-0.845) for hypertension, and 0.826 (95% CI = 0.817-0.835) for dyslipidemia. Compared with the Cox proportional hazards-based models, the DeepSurv-based models had better reclassification performance [diabetes mellitus: net reclassification improvement (NRI) = 0.474, P ≤ 0.001; hypertension: NRI = 0.194, P ≤ 0.001; dyslipidemia: NRI = 0.397, P ≤ 0.001] and discrimination performance [diabetes mellitus: integrated discrimination improvement (IDI) = 0.013, P ≤ 0.001; hypertension: IDI = 0.007, P ≤ 0.001; and dyslipidemia: IDI = 0.043, P ≤ 0.001]. : This study suggests that DeepSurv has the potential to improve the performance of risk prediction models for cardiovascular disease risk factors.

Application of Deep Neural Network and Human-Computer Interaction Technology Based on Multimodal Perception in Art Design

Abstract The essence of art design is the process of emotional interaction with humans. In this paper, emotions are classified using multimodal fusion features, and abstract fusion features are obtained by sampling the multimodal matching tensor using average pooling. The matching fusion matrix in the tensor operator is used to convert from two-modal to multi-modal matching. Virtual interaction model in art design controls the design objectives, and the optimized virtual world is constructed by using virtual reality technology, so as to build an immersive art design model. Finally, a study was conducted to examine the impact of the use of emotion perception and interaction technology in art product design with students from the School of Design and Art, University of G. The results show that the liking degree of three-color matching in the cognitive experiment test is 0.307, which is higher than the liking degree of two-color matching of 0.223, indicating that overall three-color matching samples are more popular in art design. This study provides effective evaluation and guidance for designing art products that are emotional and innovative.

Semantic-Layout-Guided Image Synthesis for High-Quality Synthetic-Aperature Radar Detection Sample Generation

With the widespread application and functional complexity of deep neural networks (DNNs), the demand for training samples is increasing. This elevated requirement also extends to DNN-based SAR object detection. Most public SAR object detection datasets are oriented to marine targets such as ships, while data sets oriented to land targets are relatively rare, though they are an effective way to improve the land object detection capability of deep models through SAR sample generation. In this paper, a synthesis generation collaborative SAR sample augmentation framework is proposed to achieve flexible and diverse high-quality sample augmentation. First, a semantic-layout-guided image synthesis strategy is proposed to generate diverse detection samples. The issues of object location rationality and object layout diversity are also addressed. Meanwhile, a pix2pixGAN network guided by layout maps is utilized to achieve diverse background augmentation. Second, a progressive training strategy of diffusion models is proposed to achieve semantically controllable SAR sample generation to further improve the diversity of scene clutter. Finally, a sample cleaning method considering distribution migration and network filtering is employed to further improve the quality of detection samples. The experimental results show that this semantic synthesis generation method can outperform existing sample augmentation methods, leading to a comprehensive improvement in the accuracy metrics of classical detection networks.

Hashing Fake: Producing Adversarial Perturbation for Online Privacy Protection Against Automatic Retrieval Models

The wide application of deep neural networks (DNNs) has significantly improved the performance of hashing models on multimodal retrieval issues. DNN-based deep models can automatically learn semantic features from raw data to make human-level decisions. However, the superior generalization leads to potential privacy leakage risks. Strong DNN-based retrieval models enable malicious crawlers to search for nontag private information based on semantic similarity matching. Hence, executing effective privacy protection mechanisms against those retrieval software is essential for reliable social website construction. In this article, we propose a retrieval task-based adversarial perturbation generation method called Hashing Fake to meet this request. Specifically, DNNs are recently found to be vulnerable to a specific set of attacks called adversarial perturbations, which denote some magnitude-restricted signals added on objective samples to misguide well-crafted DNN models, and perturbations’ magnitudes are small enough that will not induce humans’ perception. Moreover, since existing adversarial perturbation generation methods are designed for supervised tasks, Hashing Fake constructs a differential approximation substitution for perturbation production on unsupervised retrieval tasks. Through extensive experiments on several deep retrieval benchmarks, we demonstrate that well-crafted perturbations using Hashing Fake can effectively misguide objective models’ recognitions to make false predictions. The added norm-restricted perturbations on objective samples will not alter humans’ perception; hence, Hashing Fake can be applied on real-world social websites to protect subscribers’ privacy against malicious retrieval software.

Two‐stage architectural fine‐tuning for neural architecture search in efficient transfer learning

AbstractIn many deep neural network (DNN) applications, the difficulty of gathering high‐quality data in industry fields hinders the practical use of DNN. Thus, the concept of transfer learning (TL) has emerged, which leverages the pretrained knowledge of the DNN which was built based on large‐scale datasets. For this TL objective, this paper suggests two‐stage architectural fine‐tuning for reducing the costs and time while exploring the most efficient DNN model, inspired by neural architecture search (NAS). The first stage is mutation, which reduces the search costs using a priori architectural information. Moreover, the next stage is early‐stopping, which reduces NAS costs by terminating the search process in the middle of computation. The data‐intensive experimental results verify that the proposed method outperforms benchmarks.

Intracranial hemorrhage detection: how much can windowing, transfer learning and data augmentation affect a deep learning model’s performance?

The application of Deep Neural Networks to detect intracranial hemorrhage from computed tomography images has been widely used in clinical medicine. In general, building these solutions combines three types of approaches: Preprocessing, Transfer Learning, and Data Augmentation. This study’s goal was to measure the contribution of each of these approaches and thus highlight which approach has more margin of improvement. An experimental study was conducted on a public dataset containing computerized tomography images. The comparison used the stratified ten-fold cross-validation process to set confidence intervals evaluating performance measured by the area under the receiver operating characteristic curve. The ResNet-50 was the deep learning model selected. The results showed that all the approaches raise the generalization power when applied in isolation, and Data Augmentation offers the most significant gain to the baseline. The experiment also showed an opportunity to improve the detection of intracranial hemorrhage by applying new preprocessing techniques since this was the approach that showed the smallest increase in discriminatory power among the investigated approaches. The paired Wilcoxon’s Signed-Rank Test showed that not all the differences were statistically significant, with a confidence level of 95%.

Application of Deep Neural Network with Frequency Domain Filtering in the Field of Intrusion Detection

In the field of intrusion detection, existing deep learning algorithms have limited capability to effectively represent network data features, making it challenging to model the complex mapping relationship between network data and attack behavior. This limitation, in turn, impacts the detection accuracy of intrusion detection systems. To address this issue and further enhance detection accuracy, this paper proposes an algorithm called the Fourier Neural Network (FNN). The core of FNN consists of a Deep Fourier Neural Network Block (DFNNB), which is composed of a Hadamard Neural Network (HNN) and a Fourier Neural Network Layer (FNNL). In a DFNNB, the HNN is responsible for sampling the network intrusion data samples in different time domain spaces. The FNNL, on the other hand, performs a Fourier transform on the samples outputted by the HNN and maps them to the frequency domain space, followed by a filtering process. Finally, the data processed by filtering are transformed back to the time domain space for subsequent feature extraction work by the DFNNB. Additionally, to enhance the algorithm’s detection accuracy and filter out noise signals, this paper also introduces a High-energy Filtering Process (HFP), which eliminates noise signals from the data signal and reduces interference on the final detection result. Due to the ability of FNN to process network data in both the time domain space and the frequency domain space, it possesses a stronger capability in expressing data features. Finally, this paper conducts performance evaluations on the KDD Cup99, NSL-KDD, UNSW-NB15, and CICIDS2017 datasets. The results demonstrate that the proposed FNN-based IDS model achieves higher detection rates, lower false alarm rates, and better detection performance than classical deep learning and machine learning methods.

Unifying Gradients to Improve Real-World Robustness for Deep Networks

The wide application of deep neural networks (DNNs) demands an increasing amount of attention to their real-world robustness, i.e., whether a DNN resists black-box adversarial attacks, among which score-based query attacks (SQAs) are the most threatening since they can effectively hurt a victim network with only access to model outputs. Defending against SQAs requires a slight but artful variation of outputs due to the service purpose for users, who share the same output information with SQAs. In this article, we propose a real-world defense by Unifying Gradients (UniG) of different data so that SQAs could only probe a much weaker attack direction that is similar for different samples. Since such universal attack perturbations have been validated as less aggressive than the input-specific perturbations, UniG protects real-world DNNs by indicating to attackers a twisted and less informative attack direction. We implement UniG efficiently by a Hadamard product module, which is plug-and-play. According to extensive experiments on 5 SQAs, 2 adaptive attacks and 7 defense baselines, UniG significantly improves real-world robustness without hurting clean accuracy on CIFAR10 and ImageNet. For instance, UniG maintains a model of 77.80% accuracy under a 2500-query Square attack while the state-of-the-art adversarially trained model only has 67.34% on CIFAR10. Simultaneously, UniG outperforms all compared baselines in terms of clean accuracy and achieves the smallest modification of the model output. The code is released at https://github.com/snowien/UniG-pytorch .

Application of Multiple Deep Neural Networks to Multi-Solution Synthesis of Linkage Mechanisms

This paper studies the problem of linkage-bar synthesis by means of multiple deep neural networks (DNNs), which requires the inverse solution of linkage parameters based on a desired trajectory curve. This problem is highly complex due to the fact that the solution space is nonlinear and may contain multiple solutions, while a good quality of learning cannot be obtained by a single neural network approach. Therefore, this paper proposes employing Fourier descriptors to represent trajectory curves in a systematic and normalized form, developing a multi-solution distribution evaluation by random restart local searches (MDE-RRLS) to examine a better solution-space partitioning scheme, utilizing multiple DNNs to learn subspace regions separately, and creating a multi-facet query (MFQuery) to cooperatively predict multiple solutions. The experiments demonstrate that the proposed approach can obtain better or at least competitive outcomes compared to previous work in the literature. Furthermore, to verify the effectiveness and applicability, this paper investigates the design problem of an industrial six-linkage-bar ladle mechanism used in a die-casting system, and the proposed method can obtain several superior design solutions and offer alternatives in a short period of time when faced with redesign requirements.

FTG: Score-based black-box watermarking by fragile trigger generation for deep model integrity verification

Deep neural networks (DNNs) are widely used in real-world applications, thanks to their exceptional performance in image recognition. However, their vulnerability to attacks, such as Trojan and data poison, can compromise the integrity and stability of DNN applications. Therefore, it is crucial to verify the integrity of DNN models to ensure their security. Previous research on model watermarking for integrity detection has encountered the issue of overexposure of model parameters during embedding and extraction of the watermark. To address this problem, we propose a novel score-based black-box DNN fragile watermarking framework called fragile trigger generation (FTG). The FTG framework only requires the prediction probability distribution of the final output of the classifier during the watermarking process. It generates different fragile samples as the trigger, based on the classification prediction probability of the target classifier and a specified prediction probability mask to watermark it. Different prediction probability masks can promote the generation of fragile samples in corresponding distribution types. The whole watermarking process does not affect the performance of the target classifier. When verifying the watermarking information, the FTG only needs to compare the prediction results of the model on the samples with the previous label. As a result, the required model parameter information is reduced, and the FTG only needs a few samples to detect slight modifications in the model. Experimental results demonstrate the effectiveness of our proposed method and show its superiority over related work. The FTG framework provides a robust solution for verifying the integrity of DNN models, and its effectiveness in detecting slight modifications makes it a valuable tool for ensuring the security and stability of DNN applications.