Baseline Deep Neural Network Research Articles

A Deep-Learning and Transfer-Learning Hybrid Aerosol Retrieval Algorithm for FY4-AGRI: Development and Verification over Asia

The Advanced Geosynchronous Radiation Imager (AGRI) is a mission-critical instrument for the Fengyun series of satellites. AGRI acquires full-disk images every 15 min and views East Asia every 5 min through 14 spectral bands, enabling the detection of highly variable aerosol optical depth (AOD). Quantitative retrieval of AOD has hitherto been challenging, especially over land. In this study, an AOD retrieval algorithm is proposed that combines deep learning and transfer learning. The algorithm uses core concepts from both the Dark Target (DT) and Deep Blue (DB) algorithms to select features for the machine-learning (ML) algorithm, allowing for AOD retrieval at 550 nm over both dark and bright surfaces. The algorithm consists of two steps: ① A baseline deep neural network (DNN) with skip connections is developed using 10 min Advanced Himawari Imager (AHI) AODs as the target variable, and ② sunphotometer AODs from 89 ground-based stations are used to fine-tune the DNN parameters. Out-of-station validation shows that the retrieved AOD attains high accuracy, characterized by a coefficient of determination (R2) of 0.70, a mean bias error (MBE) of 0.03, and a percentage of data within the expected error (EE) of 70.7%. A sensitivity study reveals that the top-of-atmosphere reflectance at 650 and 470 nm, as well as the surface reflectance at 650 nm, are the two largest sources of uncertainty impacting the retrieval. In a case study of monitoring an extreme aerosol event, the AGRI AOD is found to be able to capture the detailed temporal evolution of the event. This work demonstrates the superiority of the transfer-learning technique in satellite AOD retrievals and the applicability of the retrieved AGRI AOD in monitoring extreme pollution events.

A two-stage data quality improvement strategy for deep neural networks in fault severity estimation

As fault diagnosis based on deep neural network (DNN) has been developing from laboratory to industrial application, creating the appropriate data pipeline to train and evaluate the models has become a noticeable challenge. For mechanical fault diagnosis, high-quality data should be consistent with comprehensive fault information. In this paper, a two-stage data quality improvement strategy is proposed to sculpt the data for deep learning models in fault severity estimation. In the first stage, the spatial information reconstruction (SIR) approach is developed to transform the sensing data from time domain into spatial domain to establish the relationship between fault symptom and spatial position. In the second stage, spatial domain signals are organized based on fault characteristics and types of DNN models, which aims to convert data into useful information. The proposed strategy has been verified by experimental cases, and a comprehensive evaluation on the performance of DNNs trained by different input data has been applied. The experimental results proved that with thoughtfully engineered data, baseline DNN models can achieve high accuracy and robustness under different speed conditions. This paper provides a way for engineering practitioners to design industrial DNN applications with their domain knowledge.

DOSnet as a non-black-box PDE solver: When deep learning meets operator splitting

Deep neural networks (DNNs) recently emerged as a promising tool for analyzing and solving complex differential equations arising in science and engineering applications. Alternative to traditional numerical schemes, learning-based solvers utilize the representation power of DNNs to approximate the input-output relations in an automated manner. However, the lack of physics-in-the-loop often makes it difficult to construct a neural network solver that simultaneously achieves high accuracy, low computational burden, and interpretability. In this work, focusing on a class of evolutionary PDEs characterized by decomposable operators, we show that the classical “operator splitting” technique can be adapted to design neural network architectures. This gives rise to a learning-based PDE solver, which we name Deep Operator-Splitting Network (DOSnet). Such non-black-box network design is constructed from the physical rules and operators governing the underlying dynamics, and is more efficient and flexible than the classical numerical schemes and standard DNNs. To demonstrate the advantages of our new AI-enhanced PDE solver, we train and validate it on several types of operator-decomposable differential equations. We also apply DOSnet to nonlinear Schrödinger equations which have important applications in the signal processing for modern optical fiber transmission systems, and experimental results show that our model has better accuracy and lower computational complexity than numerical schemes and the baseline DNNs.

Autoencoder and restricted Boltzmann machine for transfer learning in functional magnetic resonance imaging task classification

Deep neural networks (DNNs) have been adopted widely as classifiers for functional magnetic resonance imaging (fMRI) data, advancing beyond traditional machine learning models. Consequently, transfer learning of the pre-trained DNN becomes crucial to enhance DNN classification performance, specifically by alleviating an overfitting issue that occurs when a substantial number of DNN parameters are fitted to a relatively small number of fMRI samples. In this study, we first systematically compared the two most popularly used, unsupervised pretraining models for resting-state fMRI (rfMRI) volume data to pre-train the DNNs, namely autoencoder (AE) and restricted Boltzmann machine (RBM). The group in-brain mask used when training AE and RBM displayed a sizable overlap ratio with Yeo's seven functional brain networks (FNs). The parcellated FNs obtained from the RBM were fine-grained compared to those from the AE. The pre-trained AE and RBM served as the weight parameters of the first of the two hidden DNN layers, and the DNN fulfilled the task classifier role for fMRI (tfMRI) data in the Human Connectome Project (HCP). We tested two transfer learning schemes: (1) fixing and (2) fine-tuning the DNN's pre-trained AE or RBM weights. The DNN with transfer learning was compared to a baseline DNN, trained using random initial weights. Overall, DNN classification performance from the transfer learning proved superior when the pre-trained RBM weights were fixed and when the pre-trained AE weights were fine-tuned (average error rates: 14.8% for fixed RBM, 15.1% fine-tuned AE, and 15.5% for the baseline model) compared to the alternative scenarios of DNN transfer learning schemes. Moreover, the optimal transfer learning scheme between the fixed RBM and fine-tuned AE varied according to seven task conditions in the HCP. Nonetheless, the computational load reduced substantially for the fixed-weight-based transfer learning compared to the fine-tuning-based transfer learning (e.g., the number of weight parameters for the fixed-weight-based DNN model reduced to 1.9% compared with a baseline/fine-tuned DNN model). Our findings suggest that weight initialization at the DNN's first layer using RBM-based pre-trained weights provides the most promising approach when the whole-brain fMRI volume supports associated task classification. We believe that our proposed scheme could be applied to a variety of task conditions to improve their classification performance and to utilize computational resources efficiently using our AE/RBM-based pre-trained weights compared to random initial weights for DNN training.

Multisite algal bloom predictions in a lake using graph attention networks

The algal blooms caused by eutrophication is a major concern in water and aquatic resource management, and various attempts have been made to accurately predict it. Algal blooms and the factors influencing them exhibit high spatial variability depending on the characteristics of the water body and water flow. However, traditional machine learning and deep learning methods have limitations to account for the spatial interactions of various influencing factors across multiple monitoring sites. In addition, attempts to predict multiple sites simultaneously using a single model are limited. In this study, we proposed a model that considers spatial interactions and performs multisite predictions based on a graph attention network (GAT). The GAT–DNN, which combines a deep neural network (DNN) after GAT layer, was applied to forecast chlorophyll-a levels at multiple sites. The proposed model accurately captured the high variability and peak chlorophyll-a levels. Moreover, the GAT–DNN consistently outperformed two baseline DNNs in both cases. Additionally, we examined the optimal forecast horizon by comparing the performance of the model across various forecast horizons. Therefore, the proposed model can be applied to a wide range of prediction models to capture spatial interactions and obtain the benefits of performance outcomes for each site.

Assessment of Transfer Learning Capabilities for Fatigue Damage Classification and Detection in Aluminum Specimens with Different Notch Geometries

Fatigue damage detection and its classification in metallic materials are persistently challenging the structural health monitoring community. The mechanics of fatigue damage is difficult to analyze and is further complicated because of the presence of notches of different geometries. These notches act as possible crack-nucleation sites resulting in failure mechanisms that are drastically different from one another. Often, sensor-based tools are used to monitor and detect fatigue damage in critical metallic materials such as aluminum alloys. Through deep neural networks (DNNs), such a sensor-based approach can be ubiquitously extended for a variety of geometries as appropriate for different applications. To that end, this paper presents a DNN-based transfer learning framework that can be used to classify and detect fatigue damage across candidate notch geometries. The DNNs are built upon ultrasonic time-series data obtained during fatigue testing of Al7075-T6 specimens with two types of notch geometries, namely, a U-notch and a V-notch. The baseline U-notch DNN is shown to achieve an accuracy of 96.1% while the baseline V-notch DNN has an accuracy of 95.8%. Both baseline DNNs are, thereafter, subjected to a transfer learning process by keeping a certain number of layers frozen and retraining only the remaining layers with a small volume of data obtained from the other notch geometry. When a layer of the baseline U-notch DNN is retrained with just 10% of the total V-notch data, an accuracy above 90% is observed for fatigue damage detection of V-notch specimens. Similar results are also obtained when the baseline V-notch DNN is retrained and interrogated to detect damage for U-notch specimens. These results, in summary, demonstrate the data-thrifty quality of combining the concepts of transfer learning and DNN for fatigue damage detection in different geometries of specimens made of high-performance aluminum alloys.

Novel informed deep learning-based prognostics framework for on-board health monitoring of lithium-ion batteries

This paper proposes a novel, informed deep-learning-based prognostics framework for on-board state of health and remaining useful life estimations of lithium-ion batteries, which are critical components for strategizing energy and power used in electric vehicles. The framework comprises three phases. First, reliable and online accessible impedance-related features are collected from discharge curves. Second, these features are inputted into the proposed knowledge-infused recurrent neural network, a hybrid model that combines an empirical model with a deep neural network. Third, Monte Carlo dropout, a deep learning method for obtaining a probabilistic prediction of a neural network, is addressed to secure robustness in estimating the state of health and remaining useful life. Layer-wise relevance propagation, a deep learning technique for tracking the evolution of feature importance and offering scientific reasoning of the output, confirms that impedance-related features significantly contribute to the estimation accuracy compared to other features investigated in previous studies. Moreover, the hybrid model improves the estimation accuracy and robustness, whereas Monte Carlo dropout ensures robustness and reliability. Specifically, the estimation results for the public degradation data reveal that the proposed model can output significantly more accurate state of health and remaining useful life estimations than the baseline deep neural networks. The findings of this study provide insight into the explicable and uncertainty-based pipeline of deep neural networks with respect to battery health monitoring, which are highly recommendable features for decision-making and corrective planning of power and energy used in lithium-ion battery cells and packs.

Hyperspectral Image Classification Promotion Using Clustering Inspired Active Learning

Deep neural networks (DNNs) have promoted much of the recent progress in hyperspectral image (HSI) classification, which depends on extensive labeled samples and deep network structure and has achieved surprisingly good generalization capacity. However, due to the expensive labeling cost, the labeled samples are scarce in most practice cases, which causes these DNN-based methods to be prone to over-fitting and influences the classification result. To mitigate this problem, we present a clustering-inspired active learning method for enhancing the HSI classification result, which mainly contributes to two aspects. On one hand, the modified clustering by fast search and find of peaks clustering method is utilized to select highly informative and diverse samples from unlabeled samples in the candidate set for manual labeling, which empowers us to appropriately augment the limited training set (i.e., labeled samples) and thus improves the generalization capacity of the baseline DNN model. On the other hand, another K-means clustering-based pseudo-labeling scheme is utilized to pre-train the DNN model with all samples in the candidate set. By doing this, the pre-trained model can be effectively generalized to unlabeled samples in the testing set after being fine tuned-based on the augmented training set. The experiment accuracies on two benchmark HSI datasets show the effectiveness of the proposed method.

Cross-View Matching for Vehicle Localization by Learning Geographically Local Representations

Cross-view matching aims to learn a shared image representation between ground-level images and satellite or aerial images at the same locations. In robotic vehicles, matching a camera image to a database of geo-referenced aerial imagery can serve as a method for self-localization. However, existing work on cross-view matching only aims at global localization, and overlooks the easily accessible rough location estimates from GNSS or temporal filtering. We argue that the availability of coarse location estimates at test time should already be considered during training. We adopt a simple but effective adaptation to the common triplet loss, resulting in an image representation that is more discriminative within the geographically local neighborhood, without any modifications to a baseline deep neural network. Experiments on the CVACT dataset confirm that the improvements generalize across spatial regions. On a new benchmark constructed from the Oxford RobotCar dataset, we also show generalization across recording days within the same region. Finally, we validate that improvements on these image-retrieval benchmarks also translate to a real-world localization task. Using a particle filter to fuse the cross-view matching scores of a vehicle's camera stream with real GPS measurements, our learned geographically local representation reduces the mean localization error by 17% compared to the standard global representation learned by the current state-of-the-art.

A calibrated deep learning ensemble for abnormality detection in musculoskeletal radiographs

Musculoskeletal disorders affect the locomotor system and are the leading contributor to disability worldwide. Patients suffer chronic pain and limitations in mobility, dexterity, and functional ability. Musculoskeletal (bone) X-ray is an essential tool in diagnosing the abnormalities. In recent years, deep learning algorithms have increasingly been applied in musculoskeletal radiology and have produced remarkable results. In our study, we introduce a new calibrated ensemble of deep learners for the task of identifying abnormal musculoskeletal radiographs. Our model leverages the strengths of three baseline deep neural networks (ConvNet, ResNet, and DenseNet), which are typically employed either directly or as the backbone architecture in the existing deep learning-based approaches in this domain. Experimental results based on the public MURA dataset demonstrate that our proposed model outperforms three individual models and a traditional ensemble learner, achieving an overall performance of (AUC: 0.93, Accuracy: 0.87, Precision: 0.93, Recall: 0.81, Cohen’s kappa: 0.74). The model also outperforms expert radiologists in three out of the seven upper extremity anatomical regions with a leading performance of (AUC: 0.97, Accuracy: 0.93, Precision: 0.90, Recall:0.97, Cohen’s kappa: 0.85) in the humerus region. We further apply the class activation map technique to highlight the areas essential to our model’s decision-making process. Given that the best radiologist performance is between 0.73 and 0.78 in Cohen’s kappa statistic, our study provides convincing results supporting the utility of a calibrated ensemble approach for assessing abnormalities in musculoskeletal X-rays.

Spam Email Detection Using Deep Learning Techniques

Abstract Unsolicited emails such as phishing and spam emails cost businesses and individuals millions of dollars annually. Several models and techniques to automatically detect spam emails have been introduced and developed yet non showed 100% predicative accuracy. Among all proposed models both machine and deep learning algorithms achieved more success. Natural language processing (NLP) enhanced the models’ accuracy. In this work, the effectiveness of word embedding in classifying spam emails is introduced. Pre-trained transformer model BERT (Bidirectional Encoder Representations from Transformers) is fine-tuned to execute the task of detecting spam emails from non-spam (HAM). BERT uses attention layers to take the context of the text into its perspective. Results are compared to a baseline DNN (deep neural network) model that contains a BiLSTM (bidirectional Long Short Term Memory) layer and two stacked Dense layers. In addition results are compared to a set of classic classifiers k-NN (k-nearest neighbors) and NB (Naive Bayes). Two open-source data sets are used, one to train the model and the other to test the persistence and robustness of the model against unseen data. The proposed approach attained the highest accuracy of 98.67% and 98.66% F1 score.

COVID-19-CT-CXR: A Freely Accessible and Weakly Labeled Chest X-Ray and CT Image Collection on COVID-19 From Biomedical Literature.

The latest threat to global health is the COVID-19 outbreak. Although there exist large datasets of chest X-rays (CXR) and computed tomography (CT) scans, few COVID-19 image collections are currently available due to patient privacy. At the same time, there is a rapid growth of COVID-19-relevant articles in the biomedical literature, including those that report findings on radiographs. Here, we present COVID-19-CT-CXR, a public database of COVID-19 CXR and CT images, which are automatically extracted from COVID-19-relevant articles from the PubMed Central Open Access (PMC-OA) Subset. We extracted figures, associated captions, and relevant figure descriptions in the article and separated compound figures into subfigures. Because a large portion of figures in COVID-19 articles are not CXR or CT, we designed a deep-learning model to distinguish them from other figure types and to classify them accordingly. The final database includes 1,327 CT and 263 CXR images (as of May 9, 2020) with their relevant text. To demonstrate the utility of COVID-19-CT-CXR, we conducted four case studies. (1) We show that COVID-19-CT-CXR, when used as additional training data, is able to contribute to improved deep-learning (DL) performance for the classification of COVID-19 and non-COVID-19 CT. (2) We collected CT images of influenza, another common infectious respiratory illness that may present similarly to COVID-19, and fine-tuned a baseline deep neural network to distinguish a diagnosis of COVID-19, influenza, or normal or other types of diseases on CT. (3) We fine-tuned an unsupervised one-class classifier from non-COVID-19 CXR and performed anomaly detection to detect COVID-19 CXR. (4) From text-mined captions and figure descriptions, we compared 15 clinical symptoms and 20 clinical findings of COVID-19 versus those of influenza to demonstrate the disease differences in the scientific publications. Our database is unique, as the figures are retrieved along with relevant text with fine-grained descriptions, and it can be extended easily in the future. We believe that our work is complementary to existing resources and hope that it will contribute to medical image analysis of the COVID-19 pandemic. The dataset, code, and DL models are publicly available at https://github.com/ncbi-nlp/COVID-19-CT-CXR.

Mixture of Expert/Imitator Networks: Scalable Semi-Supervised Learning Framework

The current success of deep neural networks (DNNs) in an increasingly broad range of tasks involving artificial intelligence strongly depends on the quality and quantity of labeled training data. In general, the scarcity of labeled data, which is often observed in many natural language processing tasks, is one of the most important issues to be addressed. Semisupervised learning (SSL) is a promising approach to overcoming this issue by incorporating a large amount of unlabeled data. In this paper, we propose a novel scalable method of SSL for text classification tasks. The unique property of our method, Mixture of Expert/Imitator Networks, is that imitator networks learn to “imitate” the estimated label distribution of the expert network over the unlabeled data, which potentially contributes a set of features for the classification. Our experiments demonstrate that the proposed method consistently improves the performance of several types of baseline DNNs. We also demonstrate that our method has the more data, better performance property with promising scalability to the amount of unlabeled data.

Stop Memorizing: A Data-Dependent Regularization Framework for Intrinsic Pattern Learning

Deep neural networks (DNNs) typically have enough capacity to fit random data by brute force even when conventional data-dependent regularizations focusing on the geometry of the features are imposed. We find out that the reason for this is the inconsistency between the enforced geometry and the standard softmax cross entropy loss. To resolve this, we propose a new framework for data-dependent DNN regularization, the Geometrically-Regularized-Self-Validating neural Networks (GRSVNet). During training, the geometry enforced on one batch of features is simultaneously validated on a separate batch using a validation loss consistent with the geometry. We study a particular case of GRSVNet, the Orthogonal-Low-rank Embedding (OLE)-GRSVNet, which is capable of producing highly discriminative features residing in orthogonal low-rank subspaces. Numerical experiments show that OLE-GRSVNet outperforms DNNs with conventional regularization when trained on real data, especially when the training samples are scarce. More importantly, unlike conventional DNNs, OLE-GRSVNet refuses to memorize random data or random labels, suggesting that it only learns intrinsic patterns by reducing the memorizing capacity of the baseline DNN.

AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild

Automated affective computing in the wild setting is a challenging problem in computer vision. Existing annotated databases of facial expressions in the wild are small and mostly cover discrete emotions (aka the categorical model). There are very limited annotated facial databases for affective computing in the continuous dimensional model (e.g., valence and arousal). To meet this need, we collected, annotated, and prepared for public distribution a new database of facial emotions in the wild (called AffectNet). AffectNet contains more than 1,000,000 facial images from the Internet by querying three major search engines using 1250 emotion related keywords in six different languages. About half of the retrieved images were manually annotated for the presence of seven discrete facial expressions and the intensity of valence and arousal. AffectNet is by far the largest database of facial expression, valence, and arousal in the wild enabling research in automated facial expression recognition in two different emotion models. Two baseline deep neural networks are used to classify images in the categorical model and predict the intensity of valence and arousal. Various evaluation metrics show that our deep neural network baselines can perform better than conventional machine learning methods and off-the-shelf facial expression recognition systems.

깊은 신경망 기반의 전이학습을 이용한 사운드 이벤트 분류

ABSTRACT: Deep neural network that effectively capture the characteristics of data has been widely used in various applications. However, the amount of sound database is often insufficient for learning the deep neural network properly, so resulting in overfitting problems. In this paper, we propose a transfer learning framework that can effectively train the deep neural network even with insufficient sound event data by employing rich speech or music data. A series of experimental results verify that proposed method performs significantly better than the baseline deep neural network that was trained only with small sound event data.Keywords: Sound event classification, Transfer learning, Deep neural networkPACS numbers: 43.60.Bf, 43.72.Bs†Corresponding author: Hoirin Kim (hoirkim@kaist.ac.kr) School of Electrical Engineering, Korea Advanced Institute of Science and Technology, 291 Daehak-ro, Yuseong-gu, Daejeon 34141, Republic of Korea(Tel: 82-42-350-7617, Fax: 82-42-350-7619) 한국음향학회지 제 35권 제2호 pp. 143～148 (2016)The Journal of the Acoustical Society of Korea Vol.35, No.2 (2016)http://dx.doi.org/10.7776/ASK.2016.35.2.143