Deep Neural Network Method Research Articles

Survey of brain tumor segmentation with deep neural networks

A brain tumour is a serious condition that, if not diagnosed and treated early on, can lead to death. Researchers have proposed a variety of traditional and recently developed deep learning based segmentation and classification approaches for determining the condition of the tumor. Deep learning is found to be efficient and robust for classification and segmentation as it detects the fine-to-coarse information about the tumors. The main component of deep learning is layered neural network architecture popularly known as convolutional neural network. Distinct information from brain images can be acquired and analysed depending on the architecture. In order to achieve high segmentation and classification accuracy, more research is required in this area. In this paper presents a review of state-of-the-art deep learning methods for brain tumor segmentation and deep learning neural networks, clearly highlighting their building blocks and different strategies. Finally, this article implying about present status on segmentation and classification of tumor-based image processing through deep learning models.

Doğal dil işleme ve makine öğrenme algoritmaları ile makine dili modellemesi

Language is one of the most important elements of communication for people and all living things. A large number of studies have been conducted in the literature on the formation of languages and natural language processing processes. It has been observed that these studies contain differences in terms of analysis, approach and method. In the study, a model has been proposed for machines to create their own language as a contribution to the literature. With this point of view, it has been contributed to the idea that the efficiency of all processes will increase by communicating with each other in their own language like humans. With the aim of developing an approach for machines to produce their own languages, it is aimed to distinguish the sounds of living things by classifying them and to generate new sounds by using convolutional deep neural network method. In the study, the applied alphabet was decided by the system. In next step, Regenerative Artificial Neural Networks were used together with Mel Frequency Cepstral Coefficient and Dynamic Time Warp methodologies to create similar sounds and the living things were named with their respective sounds. Unlike similar studies, a visual data was applied which was converted from the audio files that was obtained from Kaggle open data repository "Audio Cats and Dogs" dataset. In addition, the model was supported with CNN networks to achieve effective performance.

Forest Fire Prediction with Imbalanced Data Using a Deep Neural Network Method

Forests suffer from heavy losses due to the occurrence of fires. A prediction model based on environmental condition, such as meteorological and vegetation indexes, is considered a promising tool to control forest fires. The construction of prediction models can be challenging due to (i) the requirement of selection of features most relevant to the prediction task, and (ii) heavily imbalanced data distribution where the number of large-scale forest fires is much less than that of small-scale ones. In this paper, we propose a forest fire prediction method that employs a sparse autoencoder-based deep neural network and a novel data balancing procedure. The method was tested on a forest fire dataset collected from the Montesinho Natural Park of Portugal. Compared to the best prediction results of other state-of-the-art methods, the proposed method could predict large-scale forest fires more accurately, and reduces the mean absolute error by 3–19.3 and root mean squared error by 0.95–19.3. The proposed method can better benefit the management of wildland fires in advance and the prevention of serious fire accidents. It is expected that the prediction performance could be further improved if additional information and more data are available.

Hyper-clustering enhanced spatio-temporal deep learning for traffic and demand prediction in bike-sharing systems

Bike traffic prediction is essential to operations management in bike-sharing systems (BSS). The spatio-temporal distribution of shared bikes is imbalanced; therefore, it is necessary to predict the number of bikes that are available at each station to facilitate operations. Many researchers have developed deep neural network methods to improve predictive performance. Predicting bike demand from a higher level by clustering mobility trends has been shown to increase prediction accuracy. However, complex iterative optimization problems need to be solved to stabilize the trend iteration, and such solutions are missing in previous works. In this paper, we propose a novel hyper-clustering approach that enhances a spatio-temporal deep neural network for traffic prediction in bike-sharing systems. In particular, our hyper-clustering approach captures mobility trends among individuals and clusters. Experimental results show that the designed hyper-clustering-enhanced spatio-temporal deep neural network model can predict the number of available bikes more accurately. Compared with ten state-of-the-art benchmark methods, our model demonstrates better performance in reducing prediction error by 10.9%–43.1%.

Estimation of Greenhouse Tomato Foliage Temperature Using DNN and ML Models

Since leaf temperature (LT) is not a trivial measurement, deep-neural networks (DNN) and machine learning (ML) models were evaluated in this study as tools for estimating foliage temperature. Two DNN methods were used. The first DNN used convolutional layers, while the second DNN was based on fully-connected layers and was trained by cross-validation techniques. The machine learning used the K-nearest neighbors (KNN) method for LT estimation. All models used the meteorological and microclimatic parameters (hereafter referred to as features) of the examined greenhouses to determine the average foliage temperature. The models were trained on 75% of the collected data and tested on the remaining 25%. RMS and absolute error were used to evaluate the performance of the different models compared to the LT values measured by a thermal camera. In addition, after finding the correlation of each feature to the leaf temperature, the models were trained based on the high-correlated features only. The machine learning model was superior to DNN when all available features were used and when only high-correlated features were used, resulting in errors of 0.7 °C and 0.8 °C, respectively.

Explainable deep transfer learning model for disease risk prediction using high-dimensional genomic data.

Building an accurate disease risk prediction model is an essential step in the modern quest for precision medicine. While high-dimensional genomic data provides valuable data resources for the investigations of disease risk, their huge amount of noise and complex relationships between predictors and outcomes have brought tremendous analytical challenges. Deep learning model is the state-of-the-art methods for many prediction tasks, and it is a promising framework for the analysis of genomic data. However, deep learning models generally suffer from the curse of dimensionality and the lack of biological interpretability, both of which have greatly limited their applications. In this work, we have developed a deep neural network (DNN) based prediction modeling framework. We first proposed a group-wise feature importance score for feature selection, where genes harboring genetic variants with both linear and non-linear effects are efficiently detected. We then designed an explainable transfer-learning based DNN method, which can directly incorporate information from feature selection and accurately capture complex predictive effects. The proposed DNN-framework is biologically interpretable, as it is built based on the selected predictive genes. It is also computationally efficient and can be applied to genome-wide data. Through extensive simulations and real data analyses, we have demonstrated that our proposed method can not only efficiently detect predictive features, but also accurately predict disease risk, as compared to many existing methods.

Odor clustering using a gas sensor array system of chicken meat based on temperature variations and storage time

Shelf life and temperature are two things that affect the freshness of meat. Generally, people identify the freshness of meat by looking at the texture, color, and even aroma of meat. These methods have less effective approaches to identify the freshness of meat. The limitations of the human sense of smell have led to the development of gas sensor array system technology. Research has been done on odor cluster analysis using gas sensor array with variations in shelf life and temperature in classifying the smell of chicken meat. The study used a sample of 20 g of chicken meat in a 150 ml bottle which was sensed using a gas sensor array system at a certain storage period and temperature. The shelf life used is a shelf life of 0 h, 6 h, 12 h, 18 h, and 24 h as well as variations in temperature 4 °C, 30 °C, 35 °C, 40 °C, 45 °C, 50 °C. The analysis is carried out using machine learning in the form of principal component analysis and deep neural network. In this study using the principal component analysis and deep neural network method, it can be seen that the gas sensor array is able to classify well. Meanwhile, the results of deep neural network model can be classified as fresh and unfresh chicken meat with a testing accuracy of 98.70%. The result showed that gas sensor array could classify chicken meat with high accuracy and the proposed method provides a significant improvement.

Multi-center validation of machine learning model for preoperative prediction of postoperative mortality

Accurate prediction of postoperative mortality is important for not only successful postoperative patient care but also for information-based shared decision-making with patients and efficient allocation of medical resources. This study aimed to create a machine-learning prediction model for 30-day mortality after a non-cardiac surgery that adapts to the manageable amount of clinical information as input features and is validated against multi-centered rather than single-centered data. Data were collected from 454,404 patients over 18 years of age who underwent non-cardiac surgeries from four independent institutions. We performed a retrospective analysis of the retrieved data. Only 12–18 clinical variables were used for model training. Logistic regression, random forest classifier, extreme gradient boosting (XGBoost), and deep neural network methods were applied to compare the prediction performances. To reduce overfitting and create a robust model, bootstrapping and grid search with tenfold cross-validation were performed. The XGBoost method in Seoul National University Hospital (SNUH) data delivers the best performance in terms of the area under receiver operating characteristic curve (AUROC) (0.9376) and the area under the precision-recall curve (0.1593). The predictive performance was the best when the SNUH model was validated with Ewha Womans University Medical Center data (AUROC, 0.941). Preoperative albumin, prothrombin time, and age were the most important features in the model for each hospital. It is possible to create a robust artificial intelligence prediction model applicable to multiple institutions through a light predictive model using only minimal preoperative information that can be automatically extracted from each hospital.

DeepPulse: An Uncertainty-aware Deep Neural Network for Heart Rate Estimations from Wrist-worn Photoplethysmography.

Wearable Photoplethysmography (PPG) has gained prominence as a low cost, unobtrusive and continuous method for physiological monitoring. The quality of the collected PPG signals is affected by several sources of interference, predominantly due to physical motion. Many methods for estimating heart rate (HR) from PPG signals have been proposed with Deep Neural Networks (DNNs) gaining popularity in recent years. However, the "black-box" and complex nature of DNNs has caused a lack of trust in the predicted values. This paper contributes DeepPulse, an uncertainty-aware DNN method for estimating HR from PPG and accelerometer signals, with aims of increasing trust of the predicted HR values. To the best of the authors' knowledge no PPG HR estimation method has considered aleatoric and epistemic uncertainty metrics. The results show DeepPulse is the most accurate method for DNNs with smaller network sizes. Finally, recommendations are given to reduce epistemic uncertainty, validate uncertainty estimates, improve the accuracy of DeepPulse as well as reduce the model size for resource-constrained edge devices.

Robust Explainability: A tutorial on gradient-based attribution methods for deep neural networks

With the rise of deep neural networks, the challenge of explaining the predictions of these networks has become increasingly recognized. While many methods for explaining the decisions of deep neural networks exist, there is currently no consensus on how to evaluate them. On the other hand, robustness is a popular topic for deep learning research; however, it is hardly talked about in explainability until very recently. In this tutorial paper, we start by presenting gradient-based interpretability methods. These techniques use gradient signals to assign the burden of the decision on the input features. Later, we discuss how gradient-based methods can be evaluated for their robustness and the role that adversarial robustness plays in having meaningful explanations. We also discuss the limitations of gradient-based methods. Finally, we present the best practices and attributes that should be examined before choosing an explainability method. We conclude with the future directions for research in the area at the convergence of robustness and explainability.

Transfer Learning Featured Short-Term Combining Forecasting Model for Residential Loads With Small Sample Sets

In recent years, a large variety of short-term load forecasting methodologies have been proposed, but a common drawback of them is that require a large amount of historical data to train the model. However, in reality, the cases in limited historical data would be more frequently occurred, for example, a new house or substation is built. To tackle this issue, a combining forecasting method of deep residual neural network (ResNet)-based transfer learning for residential loads is proposed in this article. To improve the accuracy and reliability of transfer learning, a novel deep ResNet with dual skip connections (ResNet-DSC) is proposed as the base model. Then with sparse training data, a Bayesian probability weighted averaging (approach is proposed, to address the model combination parameters estimation problem. In addition, probability density function forecasting is also delivered by the method. For demonstration, a series of experiments, with actual residential load data, was performed. With the comparison with other transfer learning and nontransfer learning approaches, the effectiveness and improvement of the proposed method have been validated.

Diffractive Deep Neural Network for Optical Orbital Angular Momentum Multiplexing and Demultiplexing

Vortex beams (VBs), characterized by helical phase front and orbital angular momentum (OAM), have shown perspective potential in improving communication capacity density for providing an additional multiplexing dimension. Here, we propose a diffractive deep neural network (D <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> NN) method for OAM mode multiplexing and demultiplexing. By designing the D <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> NN model and simulating light propagation through multiple diffractive screens, the phase and amplitude values can be automatically adjusted to manipulate the wavefront of light beams. Training the D <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> NN model with mode coupler and separator functions, we convert VBs into target light fields with the diffraction efficiency exceeds 97%, and the mode purities are over 97%. Constructing an OAM multiplexing link, we successfully multiplex and demultiplex two OAM channels that carry 16-QAM signals in simulation, and the demodulated bit-error-rates are below 1×10 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">-4</sup> . It is anticipated that the D <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> NN can perform flexible modulation of multiple OAM modes, which may open a new avenue for high-capacity OAM communication and all-optical information processing, etc.

Wearable Inertial Sensor-Based Limb Lameness Detection and Pose Estimation for Horses

Accurate objective, automated limb lameness detection and pose estimation play an important role for animal well-being and precision livestock farming. We present a wearable sensor-based limb lameness detection and pose estimation for horse walk and trot locomotion. The gait event and lameness detection are first built on a recurrent neural network (RNN) with long short-term memory (LSTM) cells. Its outcomes are used in the limb pose estimation. A learned low-dimensional motion manifold is parameterized by a phase variable with a Gaussian process dynamic model. We compare the RNN-LSTM-based lameness detection method with a feature-based multi-layer classifier (MLC) and a multi-class classifier (MCC) that are built on support vector machine/K-nearest-neighbors and deep convolutional neural network methods, respectively. Experimental results show that using only accelerometer measurements, the RNN-LSTM-based approach achieves 95% lameness detection accuracy and also outperforms the feature-based MLC or MCC in terms of several assessment criteria. The pose estimation scheme can predict the 24 limb joint angles in the sagittal plane with average errors less than 5 and 10 degs under normal and induced lameness conditions, respectively. The presented work demonstrate the successful use of machine learning techniques for high performance lameness detection and pose estimation in equine science. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —Automation technologies are increasingly used for precision agriculture but few have focused on monitoring individual animals in open field for precision livestock farming. Limb lameness detection and pose estimation in open field is labor-intensive, unsafe for farmers, and inefficient. The presented machine learning-enabled, wearable inertial sensor-based design provides an effective and efficient approach for horse limb lameness detection and pose estimation applications. We present an RNN-LSTM for lameness detection and an integrated manifold learning model is used to predict the horse limb joint angles in walk and trot gaits under normal and induced lameness conditions. We also present a systematic analysis and experiments to demonstrate the impacts of the wearable sensor locations and signal information on lameness detection and pose estimation performance. Several other machine learning-based lameness detection methods are also presented and compared. The extensive multi-horse testing results are presented to demonstrate the superior accuracy and higher performance than other types of machine learning methods. One attractive feature of the proposed design lies in its high performance and fast computational capability for potential real-time applications in open field.

The Global LAnd Surface Satellite (GLASS) evapotranspiration product Version 5.0: Algorithm development and preliminary validation

• GLASS ET Algorithm v5.0 (DNN) outperforms GLASS ET Algorithm v4.0 (BMA). • GLASS ET Algorithm v5.0 is a robust method for global ET estimation. • DNN has enhanced the performance of ET estimation for extreme cases. An accurate estimation of spatially and temporally continuous global terrestrial evapotranspiration (ET) is essential in the assessment of surface energy, water and carbon cycles. The Global LAnd Surface Satellite (GLASS) ET product Version 4.0 (v4.0) based on the Bayesian model averaging (BMA) method was generated to estimate global terrestrial ET. However, certain uncertainty for the GLASS ET product v4.0 limits its application. In this study, we introduced the deep neural networks (DNN) merging framework to improve terrestrial ET estimation for GLASS ET product Version 5.0 (v5.0) generation by integrating five satellite-derived ET products [Moderate Resolution Imaging Spectroradiometer (MODIS) ET product (MOD16), Shuttleworth–Wallace dual-source ET product (SW), Priestley–Taylor-based ET product (PT-JPL), modified satellite-based Priestley–Taylor ET product (MS-PT) and simple hybrid ET product (SIM)]. We compared the performance of DNN method against other merging methods, including GLASS ET algorithm v4.0 (BMA), the gradient boosting regression tree (GBRT) method and the random forest (RF) method, based on 195 global eddy covariance (EC) flux towers covering observations from 2000 through 2015. Validations indicated that the DNN had the highest accuracy among four merging methods across different land cover types, yielding the highest average determination coefficients (R 2 , 0.62), root-mean-squared-error (RMSE, 24.1 W/m 2 ) and Kling–Gupta efficiency (KGE, 0.77) with a of 99% confidence interval. Compared with GLASS ET algorithm v4.0, the DNN improved on the R 2 by approximately 7% ( p < 0.01) and the KGE by 10%. Based on the DNN, we then generated 8-day GLASS ET product v5.0 globally with a 1 km spatial resolution from 2001 to 2015 driven by GLASS vegetation and surface net radiation (R n ) datasets and Modern-Era Retrospective Analysis for Research and Applications, Version 2 (MERRA2) datasets. This global terrestrial ET product provides a valuable dataset for monitoring regional and global water resources and environmental changes.

Electrocardiogram analysis of post-stroke elderly people using one-dimensional convolutional neural network model with gradient-weighted class activation mapping

Stroke is the second leading cause of death globally after ischemic heart disease, also a risk factor of cardioembolic stroke. Thus, we postulate that heartbeats encapsulate vital signals related to stroke. With the rapid advancement of deep neural networks (DNNs), it emerges as a powerful tool to decipher intriguing heartbeat patterns associated with post-stroke patients. In this study, we propose the use of a one-dimensional convolutional network (1D-CNN) architecture to build a binary classifier that distinguishes electrocardiograms (ECGs) between the post-stroke and the stroke-free. We have built two 1D-CNNs that were used to identify distinct patterns from an openly accessible ECG dataset collected from elderly post-stroke patients. In addition to prediction accuracy, which is the primary focus of existing ECG deep neural network methods, we have utilized Gradient-weighted Class Activation Mapping (GRAD-CAM) to facilitate model interpretation by uncovering subtle ECG patterns captured by our model. Our stroke model has achieved ~90 % accuracy and 0.95 area under the Receiver Operating Characteristic curve. Findings suggest that the core PQRST complex alone is important but not sufficient to differentiate the post-stroke and the stroke-free. In conclusion, we have developed an accurate stroke model using the latest DNN method. Importantly, our work has illustrated an approach to enhance model interpretation, overcoming the black-box issue confronting DNNs, fostering higher user confidence and adoption of DNNs in medicine.

Understanding Stochastic Optimization Behavior at the Layer Update Level (Student Abstract)

Popular first-order stochastic optimization methods for deep neural networks (DNNs) are usually either accelerated schemes (e.g. stochastic gradient descent (SGD) with momentum) or adaptive step-size methods (e.g. Adam/AdaMax, AdaBelief). In many contexts, including image classification with DNNs, adaptive methods tend to generalize poorly compared to SGD, i.e. get stuck in non-robust local minima; however, SGD typically converges slower. We analyze possible reasons for this behavior by modeling gradient updates as vectors of random variables and comparing them to probabilistic bounds to identify "meaningful" updates. Through experiments, we observe that only layers close to the output have "definitely non-random" update behavior. In the future, the tools developed here may be useful in rigorously quantifying and analyzing intuitions about why some optimizers and particular DNN architectures perform better than others.

Self-normalized density map (SNDM) for counting microbiological objects

The statistical properties of the density map (DM) approach to counting microbiological objects on images are studied in detail. The DM is given by U^2-Net. Two statistical methods for deep neural networks are utilized: the bootstrap and the Monte Carlo (MC) dropout. The detailed analysis of the uncertainties for the DM predictions leads to a deeper understanding of the DM model’s deficiencies. Based on our investigation, we propose a self-normalization module in the network. The improved network model, called Self-Normalized Density Map (SNDM), can correct its output density map by itself to accurately predict the total number of objects in the image. The SNDM architecture outperforms the original model. Moreover, both statistical frameworks—bootstrap and MC dropout—have consistent statistical results for SNDM, which were not observed in the original model. The SNDM efficiency is comparable with the detector-base models, such as Faster and Cascade R-CNN detectors.

Seismic multiple suppression based on a deep neural network method for marine data

Seismic multiples in marine seismic data can affect the identification of oil and gas reservoirs. The efficiency of traditional multiple suppression methods, such as the Radon transform, depends on the accuracy of the velocity model for primaries and multiples, and the assumption of random background noise. To attenuate multiples with background noise, a method of primary reconstruction using a deep neural network based on data augmentation training is proposed. The designed deep neural network (DNN) includes convolutional encoding and decoding processes. The convolutional encoding process uses convolutional layers and maximum pooling layers to learn the features of the primaries, multiples, and background noise in a seismic data set. The convolutional decoding process uses these features to reconstruct the primaries and suppress the multiples and background noise. Using the data augmentation training method in the training phase, the full-wavefield data and the predicted multiples are added to background noise and then rotated to constitute the augmented data sets. This allows the DNN to have a better multiple suppression effect and better robustness. A well-trained DNN can be used for primary reconstruction in the same work area directly or in other work areas with a transfer learning method. Three examples of synthetic data with two simple models and a Pluto model verify the effectiveness, efficiency, stability, and good generalization of the proposed method for primary reconstruction and multiple suppression. Another example from field data finds that the proposed method can efficiently suppress seismic multiples under complex conditions.

Path-ATT-CNN: A Novel Deep Neural Network Method for Key Pathway Identification of Lung Cancer.

Attention convolutional neural networks (ATT-CNNs) have got a huge gain in picture operating and nature language processing. Shortage of interpretability cannot remain the adoption of deep neural networks. It is very conspicuous that is shown in the prediction model of disease aftermath. Biological data are commonly revealed in a nominal grid data structured pattern. ATT-CNN cannot be applied directly. In order to figure out these issues, a novel method which is called the Path-ATT-CNN is proposed by us, making an explicable ATT-CNN model based on united omics data by making use of a recently characterized pathway image. Path-ATT-CNN shows brilliant predictive demonstration difference in primary lung tumor symptom (PLTS) and non-primary lung tumor symptom (non-PLTS) when applied to lung adenocarcinomas (LADCs). The imaginational tool adoption which is linked with statistical analysis enables the status of essential pathways which finally exist in LADCs. In conclusion, Path-ATT-CNN shows that it can be effectively put into use elucidating omics data in an interpretable mode. When people start to figure out key biological correlates of disease, this mode makes promising power in predicting illness.

A supervised deep neural network approach with standardized targets for enhanced accuracy of IVIM parameter estimation from multi-SNR images.

Extraction of intravoxel incoherent motion (IVIM) parameters from noisy diffusion‐weighted (DW) images using a biexponential fitting model is computationally challenging, and the reliability of the estimated perfusion‐related quantities represents a limitation of this technique. Artificial intelligence can overcome the current limitations and be a suitable solution to advance use of this technique in both preclinical and clinical settings. The purpose of this work was to develop a deep neural network (DNN) approach, trained on numerical simulated phantoms with different signal to noise ratios (SNRs), to improve IVIM parameter estimation. The proposed approach is based on a supervised fully connected DNN having 3 hidden layers, 18 inputs and 3 targets with standardized values. 14 × 103 simulated DW images, based on a Shepp–Logan phantom, were randomly generated with varying SNRs (ranging from 10 to 100). 7 × 103 images (1000 for each SNR) were used for training. Performance accuracy was assessed in simulated images and the proposed approach was compared with the state‐of‐the‐art Bayesian approach and other DNN algorithms. The DNN approach was also evaluated in vivo on a high‐field MRI preclinical scanner. Our DNN approach showed an overall improvement in accuracy when compared with the Bayesian approach and other DNN methods in most of the simulated conditions. The in vivo results demonstrated the feasibility of the proposed approach in real settings and generated quantitative results comparable to those obtained using the Bayesian and unsupervised approaches, especially for D and f, and with lower variability in homogeneous regions. The DNN architecture proposed in this work outlines two innovative features as compared with other studies: (1) the use of standardized targets to improve the estimation of parameters, and (2) the implementation of a single DNN to enhance the IVIM fitting at different SNRs, providing a valuable alternative tool to compute IVIM parameters in conditions of high background noise.