Deep Convolutional Recurrent Neural Network Research Articles

Image Caption using VGG model and LSTM

Deep convolutional networks and recurrent neural networks have gained significant popularity in the field of image captioning tasks in recent times. As we all know the performance and the architecture of models are still eternal topic. We constructed the model using a new method to enhance its performance and accuracy. In our model, we make use of pretrained CNN model VGG (Visual Geometry Group) to extract image features, and learn caption sentence features using bidirectional LSTM(Long-Short-Term-Memory) which can better understand the meaning of sentences in the text. Then we combine the image features and caption features to predict captions for images. The dataset Flickr8K is used to train and test the model. Additionally, the model has the ability to produce captions that are shorter than a specified caption length. We evaluated our model with Bilingual Evaluation Understudy (BLEU) score which measures the similarity of predicted text and to the real text. After evaluation and comparison, our model is proved to be well-done on some conditions.

EDLNet: ensemble deep learning network model for automatic brain tumor classification and segmentation

The brain’s abnormal and uncontrollable cell partitioning is a severe cancer disease. The tissues around the brain or the skull induce this tumor to develop spontaneously. For the treatment of a brain tumor, surgical techniques are typically preferred. Deep learning models in the biomedical field have recently attracted a lot of attention for detecting and treating diseases. This article proposes a new Ensemble Deep Learning Network (EDLNet) model. This research uses the Modified Faster RCNN approach to classify brain MRI scan images into cancerous and non-cancerous. A deep recurrent convolutional neural network (DRCNN)-based diagnostic method for early-stage brain tumor segmentation is presented. The evaluation outcomes show that the proposed tumor classification and segmentation model’s performance accurately segments tissues from MRI images. For the analysis of the proposed model, two different publicly available datasets (D1&D2) are used. For D1 and D2 datasets, a total of 99.76% and 99.87% accuracies are achieved by the proposed model. The performance results of the proposed model are more effective than the state-of-the-art network models as per the experimental results. Communicated by Ramaswamy H. Sarma

Automatic ICD-10-CM coding via Lambda-Scaled attention based deep learning model

The International Classification of Diseases (ICD) serves as a global healthcare administration standard, with one of its editions being ICD-10-CM, an enhanced diagnostic classification system featuring numerous new codes for specific anatomic sites, co-morbidities, and causes. These additions facilitate conveying the complexities of various diseases. Currently, ICD-10 coding is widely adopted worldwide. However, public hospitals in Pakistan have yet to implement it and automate the coding process. In this research, we implemented ICD-10-CM coding for a private database and named it Clinical Pool of Liver Transplant (CPLT). Additionally, we proposed a novel deep learning model called Deep Recurrent-Convolution Neural Network with a lambda-scaled Attention module (DRCNN-ATT) using the CPLT database to achieve automatic ICD-10-CM coding. DRCNN-ATT combines a bi-directional long short-term memory network (bi-LSTM), a multi-scale convolutional neural network (MS-CNN), and a lambda-scaled attention module. Experimental results demonstrate that deep recurrent convolutional neural network (DRCNN) faces attention score vanishing problem with a standard attention module for automatic ICD coding. However, adding a lambda-scaled attention module resolves this issue. We evaluated DRCNN-ATT model using two distinct datasets: a private CPLT dataset and a public MIMIC III top 50 dataset. The results indicate that the DRCNN-ATT model outperformed various baselines by generating 0.862 micro F1 and 0.25 macro F1 scores on CPLT dataset and 0.705 micro F1 and 0.655 macro F1 scores on MIMIC III top 50 dataset. Furthermore, we also deployed our model for automatic ICD-10-CM coding using ngrok and the Flask APIs, which receives input, processes it, and then returns the results.

Hybrid ARMA-GARCH-Neural Networks for intraday strategy exploration in high-frequency trading

The frequency of armed conflicts increased during the last 20 years. The problems of the emergence of military disputes, not only concern social parameters, but also economic and financial dimensions. This study examines the potential impact of global geopolitical events on the stock market prices of the Dow Jones U.S. Aerospace & Defense Index and Foreign Exchange (FOREX) markets movements. We analyse whether defence stocks and exchange rate perform similarly during military incidents or geopolitical crises. We built an Autoregressive Moving Average Model with a Generalized Autoregressive Conditional Heteroskedasticity process (ARMA-GARCH) with the machine learning methods of Neural Networks, Deep Recurrent Convolutional Neural Networks, Deep Neural Decision Trees, Quantum Neural Networks, and Quantum Recurrent Neural Networks, aimed at detecting intraday patterns for forecasting defence stock market and FOREX markets disturbances in a market microstructure framework. The empirical results provide preliminary findings on the foreseeability of market disturbances and small differences are observed before and during geopolitical events. Additionally, we confirm the effectiveness of the hybrid model ARMA-GARCH with the machine learning approaches, being ARMA-GARCH-Quantum Recurrent Neural Network the technique that achieves the best accuracy results. Our work has a large potential impact on investment market agents and portfolio managers, as shocks from geopolitical events could provide a new methodology to support the decision-making process for trading in High-Frequency Trading.

An internet of things enabled machine learning model for Energy Theft Prevention System (ETPS) in Smart Cities

Energy theft is a significant problem that needs to be addressed for effective energy management in smart cities. Smart meters are highly utilized in smart cities that help in monitoring the energy utilization level and provide information to the users. However, it is not able to detect energy theft or over-usage. Therefore, we have proposed a multi-objective diagnosing structure named an Energy Theft Prevention System (ETPS) to detect energy theft. The proposed system utilizes a combination of machine learning techniques Gated Recurrent Unit (GRU), Grey Wolf Optimization (GWO), Deep Recurrent Convolutional Neural Network (DDRCNN), and Long Short-Term Memory (LSTM). The statistical validation has been performed using the simple moving average (SMA) method. The results obtained from the simulation have been compared with the existing technique in terms of delivery ratio, throughput, delay, overhead, energy conversation, and network lifetime. The result shows that the proposed system is more effective than existing systems.

Deep learning network (DL-Net) based classification and segmentation of multi-class retinal diseases using OCT scans

The most frequent cause of blindness in industrialized nations is age-related macular degeneration (AMD), particularly in persons over 60. In recent years in this field, three key factors have led to an increase in the workload of experts and the healthcare system: 1) population aging worldwide prevalence, 2) AMD's chronic nature, and 3) more widespread usage of the retinal optical coherence tomography (OCT) imaging technology. The establishment of fully automated diagnosis systems has recently been made possible by recent developments in the deep learning field. The main objective of the research is to develop a new deep learning-based model using OCT images. The performance for classification and segmentation of retinal diseases is improved by the proposed model through effectively capturing inter-scale features and combining these features using convolutional blocks. Then, we classify volumetric OCT images via a new deep learning-based model. For multi-class classification and segmentation of ocular diseases, the OCT scans of the human eye are used in this research to demonstrate a Deep Learning Network (DL-Net) model by automatically recognizing normal, diabetic macular edema (DME), choroidal neovascularization (CNV), AMD, and drusen images. This research introduces the Modified ResNet-50 approach and Image Processing for multiple OCT image classification. This research presents an efficient diagnostic technique for image segmentation based on a Bi-LSTM-based deep recurrent convolutional neural network (DRCNN). The modified SqueezeNet model analyzes the volumetric segmentation of OCT images. The volumetric publicly available datasets are used to analyze the proposed model. The proposed multi-scale structure model outperforms several well-known OCT classification and segmentation frameworks. 99.76% accuracy is achieved by the proposed model respectively. In healthcare settings, the proposed methodology can be used as a diagnostic instrument to help medical experts make better-diagnosing decisions, as shown by the encouraging proposed architecture's qualitative evaluations and quantitative results by generating a confusion matrix.

A driver stress detection model via data augmentation based on deep convolutional recurrent neural network

Excessive stress generally leads to degraded driving performance, which increases the risk of road accidents. Therefore, real-time driver stress detection is of great significance to improve road safety. To this end, this paper proposes a deep convolutional recurrent neural network with Wasserstein generative adversarial network-based data augmentation (WGAN-DCRNN) for real-time driver stress detection, where only the pupillary response data are used. The proposed model can learn the sequence features from the input data more effectively and address the class-imbalance problem of the dataset. First, two types of DCRNN models (i.e., CNN-LSTM and CNN-GRU) are established and compared with baseline models (i.e., CNN, LSTM and GRU). The best baseline model (i.e., CNN) achieves 93.02% accuracy on the overall samples of the test set, while the CNN-LSTM and CNN-GRU improve the accuracy to 94.68% and 94.71%, respectively. This suggests that the performance of the CNN can be improved by integrating the RNN to consider the temporal dependencies of the pupillary response data. Second, deep generative model-based data augmentation is used to address the class-imbalance problem of the dataset. Specifically, the WGAN is trained to generate new samples, which are used to expand and balance the training set. To validate the effect of the proposed data augmentation method, comparative experiments are further conducted. The results show that the accuracies of CNN-LSTM and CNN-GRU are further improved to 95.39% and 95.30%, respectively. More importantly, the recall rates of minority categories achieve significant improvement. This indicates that the proposed model overcomes the class-imbalance problem and that the generalization ability of the recognition model has been further improved by using data augmentation. In addition, the WGAN-DCRNN outperforms models in existing driver stress detection studies. The proposed model can be used for different classification tasks to reduce road accident risks caused by drivers.

A Deep Regression Approach for Human Activity Recognition Under Partial Occlusion.

In real-life scenarios, Human Activity Recognition (HAR) from video data is prone to occlusion of one or more body parts of the human subjects involved. Although it is common sense that the recognition of the majority of activities strongly depends on the motion of some body parts, which when occluded compromise the performance of recognition approaches, this problem is often underestimated in contemporary research works. Currently, training and evaluation is based on datasets that have been shot under laboratory (ideal) conditions, i.e. without any kind of occlusion. In this work, we propose an approach for HAR in the presence of partial occlusion, in cases wherein up to two body parts are involved. We assume that human motion is modeled using a set of 3D skeletal joints and also that occluded body parts remain occluded during the whole duration of the activity. We solve this problem using regression, performed by a novel deep Convolutional Recurrent Neural Network (CRNN). Specifically, given a partially occluded skeleton, we attempt to reconstruct the missing information regarding the motion of its occluded part(s). We evaluate our approach using four publicly available human motion datasets. Our experimental results indicate a significant increase of performance, when compared to baseline approaches, wherein networks that have been trained using only nonoccluded or both occluded and nonoccluded samples are evaluated using occluded samples. To the best of our knowledge, this is the first research work that formulates and copes with the problem of HAR under occlusion as a regression task.

Deep Convolutional Attention based Bidirectional Recurrent Neural Network for Measuring Correlated Colour Temperature from RGB images

Information on the connected colour temperature, which affects the image due to the surrounding illumination, is critical, particularly for natural lighting and capturing images. Several methods are introduced to detect colour temperature precisely; however, the majority of them are difficult to use or may generate internal noise. To address these issues, this research developed a hybrid deep model that properly measures temperature from RGB images while reducing noise. The proposed study includes image collection, pre-processing, feature extraction and CCT evaluation. The input RGB pictures are initially generated in the CIE 1931 colour space. After that, the raw input samples are pre-processed to improve picture quality by performing image cropping and scaling, denoising by hybrid median-wiener filtering and contrast enhancement via Rectified Gamma-based Quadrant Dynamic Clipped Histogram Equalisation (RG_QuaDy_CHE). The colour and texture features are eliminated during pre-processing to obtain the relevant CCT-based information. The Local Intensity Grouping Order Pattern (LIGOP) operator extracts the texture properties. In contrast, the colour properties are extracted using the RGB colour space’s mean, standard deviation, skewness, energy, smoothness and variance. Finally, using the collected features, the CCT values from the submitted images are estimated using a unique Deep Convolutional Attention-based Bidirectional Recurrent Neural Network (DCA_BRNNet) model. The Coati Optimisation Algorithm (COA) is used to improve the performance of a recommended classifier by modifying its parameters. In the Result section, the suggested model is compared to various current techniques, obtaining an MAE value of 529K and an RMSE value of 587K, respectively.

Premature Infant Cry Classification via Deep Convolutional Recurrent Neural Network Based on Multi-class Features

Premature Infant Cry Classification via Deep Convolutional Recurrent Neural Network Based on Multi-class Features

Disturbance Robust Abnormality Diagnosis of Fused Magnesium Furnaces Using Deep Neural Networks

Operation abnormalities of fused magnesium furnaces (FMFs), e.g., semi-molten, can degrade the product quality and operation performance. The abnormalities can even lead to accidents caused by leakage of the fusing fluids with ultra-high temperatures. Therefore, it is essential to identify the semi-molten abnormality timely and accurately. In view of the spatiotemporal characteristics of the image sequences of the furnace shell under abnormal conditions of the FMFs, and the existence of strong disturbances caused by water mist, white spot, and flame fluctuation on top of the furnace, this paper establishes a novel deep learning architecture for operation abnormality diagnosis with robustness to disturbances of the FMFs. The new scheme is composed of two parts, i.e., a predictive neural networks (PredNet) for disturbance processing, and a deep three-dimensional convolutional recurrent neural networks (3DCRNN) for abnormality diagnosis. First, PredNet-based unsupervised learning is incorporated with image residual extraction for disturbance processing. Second, using the clean image sequences after disturbance processing, a new deep 3DCRNN that integrates three-dimensional CNN (3DCNN) and long-short term memory (LSTM) is proposed for enhanced spatiotemporal feature extraction and semi-molten abnormality diagnosis. The 3DCRNN successfully overcomes the limitation of conventional 3DCNN that focuses on local spatiotemporal extraction and loses the opportunities to capture long-term changes. The experimental results using the image sequences collected from a real FMF demonstrate the effectiveness of the proposed method.

DnRCNN: Deep Recurrent Convolutional Neural Network for HSI Destriping.

In spite of achieving promising results in hyperspectral image (HSI) restoration, deep-learning-based methodologies still face the problem of spectral or spatial information loss due to neglecting the inner correlation of HSI. To address this issue, we propose an innovative deep recurrent convolution neural network (DnRCNN) model for HSI destriping. To the best of our knowledge, this is the first study on HSI destriping from the perspective of inner band and interband correlation explorations with the recurrent convolution neural network. In the novel DnRCNN, a selective recurrent memory unit (SRMU) is designed to respectively extract the correlative features involved in spectral and spatial domains. Moreover, an innovative recurrent fusion (RF) strategy incorporated with group concatenation is further proposed to remove strip noise and preserve scene details using the complementary features from SRMU. Experimental results on extensive HSI datasets validated that the proposed method achieves a new state-of-the-art (SOTA) HSI destriping performance.

Al-Biruni Earth Radius Optimization with Transfer Learning Based Histopathological Image Analysis for Lung and Colon Cancer Detection.

An early diagnosis of lung and colon cancer (LCC) is critical for improved patient outcomes and effective treatment. Histopathological image (HSI) analysis has emerged as a robust tool for cancer diagnosis. HSI analysis for a LCC diagnosis includes the analysis and examination of tissue samples attained from the LCC to recognize lesions or cancerous cells. It has a significant role in the staging and diagnosis of this tumor, which aids in the prognosis and treatment planning, but a manual analysis of the image is subject to human error and is also time-consuming. Therefore, a computer-aided approach is needed for the detection of LCC using HSI. Transfer learning (TL) leverages pretrained deep learning (DL) algorithms that have been trained on a larger dataset for extracting related features from the HIS, which are then used for training a classifier for a tumor diagnosis. This manuscript offers the design of the Al-Biruni Earth Radius Optimization with Transfer Learning-based Histopathological Image Analysis for Lung and Colon Cancer Detection (BERTL-HIALCCD) technique. The purpose of the study is to detect LCC effectually in histopathological images. To execute this, the BERTL-HIALCCD method follows the concepts of computer vision (CV) and transfer learning for accurate LCC detection. When using the BERTL-HIALCCD technique, an improved ShuffleNet model is applied for the feature extraction process, and its hyperparameters are chosen by the BER system. For the effectual recognition of LCC, a deep convolutional recurrent neural network (DCRNN) model is applied. Finally, the coati optimization algorithm (COA) is exploited for the parameter choice of the DCRNN approach. For examining the efficacy of the BERTL-HIALCCD technique, a comprehensive group of experiments was conducted on a large dataset of histopathological images. The experimental outcomes demonstrate that the combination of AER and COA algorithms attain an improved performance in cancer detection over the compared models.

Joint PMD Tracking and Nonlinearity Compensation With Deep Neural Networks

Overcoming fiber nonlinearity is one of the core challenges limiting the capacity of optical fiber communication systems. Machine learning based solutions such as learned digital backpropagation (LDBP) and the recently proposed deep convolutional recurrent neural network (DCRNN) have been shown to be effective for fiber nonlinearity compensation (NLC). Incorporating distributed compensation of polarization mode dispersion (PMD) within the learned models can improve their performance even further but at the same time, it also couples the compensation of nonlinearity and PMD. Consequently, it is important to consider the time variation of PMD for such a joint compensation scheme. In this paper, we investigate the impact of PMD drift on the DCRNN model with distributed compensation of PMD. We propose a transfer learning based selective training scheme to adapt the learned neural network model to changes in PMD. We demonstrate that fine-tuning only a small subset of weights as per the proposed method is sufficient for adapting the model to PMD drift. Using decision directed feedback for online learning, we track continuous PMD drift resulting from a time-varying rotation of the state of polarization (SOP). We show that transferring knowledge from a pre-trained base model using the proposed scheme significantly reduces the re-training efforts for different PMD realizations. Applying the hinge model for SOP rotation, our simulation results show that the learned models maintain their performance gains while tracking the PMD.

Predicting the non-linear response of composite materials using deep recurrent convolutional neural networks

This paper presents a novel framework to predict the full non-linear response of composite materials using Deep Recurrent Convolutional Neural (DCRN) Networks. The framework is based on a Representative Volume Element (RVE) database populated by sampling the composite design space in terms of layups, defects/variabilities and loading conditions. Several sources of material non-linearity are included in these models such as matrix damage, delamination, fibre failure and shear non-linearity. A DCRN Network architecture is proposed which combines convolutional layers, for spatial features detection, with Long/Short Term Memory layers, for material loading history dependencies. The models’ database consists of images of each RVE model, representing the layup and variables such as the presence of wrinkles, gaps or voids, and the homogenised 3D stress/strain curves. DCRN Networks are trained to predict the full 3D stress response of a laminated composite using the information from the RVE models database. Two formulations for the DCRN Network are studied, a pointwise prediction formulation and a time-marching prediction formulation. The two formulations are compared based on accuracy and robustness. The results show that both approaches can accurately predict the non-linear response of laminated composites.

Integration of the latent variable knowledge into deep image captioning with Bayesian modeling

AbstractAutomatic image captioning systems assign one or more sentences to images to describe their visual content. Most of these systems use attention‐based deep convolutional neural networks and recurrent neural networks (CNN‐RNN‐Att). However, they must optimally use latent variables and side information within the image concepts. This study aims to integrate a latent variable into image captioning using CNN‐RNN‐Att. A Bayesian modeling framework is used for this work. As an instance of a latent variable, High‐Level Semantic Concepts (HLSCs) of tags are used to implement the proposed model. The Bayesian model output interpretation is to localize the entire image description process and breaks it down into sub‐problems. Thus, a baseline descriptor subnet is trained independently for each sub‐problem, and it is the only expert in captioning for a given HLSC. The final output is the caption derived from the subnet; its HLSC is closest to the image content. The results indicate that CNN‐RNN‐Att applied to data localized using HLSCs improves the captioning accuracy of the proposed method, which can be compared to the latest state‐of‐the‐art and most accurate captioning systems.

Autonomous Swallow Segment Extraction Using Deep Learning in Neck-Sensor Vibratory Signals From Patients With Dysphagia.

Dysphagia occurs secondary to a variety of underlying etiologies and can contribute to increased risk of adverse events such as aspiration pneumonia and premature mortality. Dysphagia is primarily diagnosed and characterized by instrumental swallowing exams such as videofluoroscopic swallowing studies. videofluoroscopic swallowing studies involve the inspection of a series of radiographic images for signs of swallowing dysfunction. Though effective, videofluoroscopic swallowing studies are only available in certain clinical settings and are not always desirable or feasible for certain patients. Because of the limitations of current instrumental swallow exams, research studies have explored the use of acceleration signals collected from neck sensors and demonstrated their potential in providing comparable radiation-free diagnostic value as videofluoroscopic swallowing studies. In this study, we used a hybrid deep convolutional recurrent neural network that can perform multi-level feature extraction (localized and across time) to annotate swallow segments automatically via multi-channel swallowing acceleration signals. In total, we used signals and videofluoroscopic swallowing study images of 3144 swallows from 248 patients with suspected dysphagia. Compared to other deep network variants, our network was superior at detecting swallow segments with an average area under the receiver operating characteristic curve value of 0.82 (95% confidence interval: 0.807-0.841), and was in agreement with up to 90% of the gold standard-labeled segments.

Adapting a deep convolutional RNN model with imbalanced regression loss for improved spatio-temporal forecasting of extreme wind speed events in the short to medium range

Abstract. The number of wind farms and amount of wind power production in Europe, both on- and offshore, have increased rapidly in the past years. To ensure grid stability and on-time (re)scheduling of maintenance tasks and to mitigate fees in energy trading, accurate predictions of wind speed and wind power are needed. Particularly, accurate predictions of extreme wind speed events are of high importance to wind farm operators as timely knowledge of these can both prevent damages and offer economic preparedness. This work explores the possibility of adapting a deep convolutional recurrent neural network (RNN)-based regression model to the spatio-temporal prediction of extreme wind speed events in the short to medium range (12 h lead time in 1 h intervals) through the manipulation of the loss function. To this end, a multi-layered convolutional long short-term memory (ConvLSTM) network is adapted with a variety of imbalanced regression loss functions that have been proposed in the literature: inversely weighted, linearly weighted and squared error-relevance area (SERA) loss. Forecast performance is investigated for various intensity thresholds of extreme events, and a comparison is made with the commonly used mean squared error (MSE) and mean absolute error (MAE) loss. The results indicate the inverse weighting method to most effectively shift the forecast distribution towards the extreme tail, thereby increasing the number of forecasted events in the extreme ranges, considerably boosting the hit rate and reducing the root-mean-squared error (RMSE) in those ranges. The results also show, however, that such improvements are invariably accompanied by a pay-off in terms of increased overcasting and false alarm ratio, which increase both with lead time and intensity threshold. The inverse weighting method most effectively balances this trade-off, with the weighted MAE loss scoring slightly better than the weighted MSE loss. It is concluded that the inversely weighted loss provides an effective way to adapt deep learning to the task of imbalanced spatio-temporal regression and its application to the forecasting of extreme wind speed events in the short to medium range.

An innovative deep active learning approach for improving unlabeled audio classification by selectively querying informative instance

Audio classification tasks like speech recognition and acoustic scene analysis require substantial labeled data, which is expensive. This work explores active learning to reduce annotation costs for a sound classification problem with rare target classes where existing datasets are insufficient. A deep convolutional recurrent neural network extracts spectro-temporal features and makes predictions. An uncertainty sampling strategy queries the most uncertain samples for manual labeling by experts and non-experts. A new alternating confidence sampling strategy and two other certainty-based strategies are proposed and evaluated. Experiments show significantly higher accuracy than passive learning baselines with the same labeling budget. Active learning generalizes well in a qualitative analysis of 20,000 unlabeled recordings. Overall, active learning with a novel sampling strategy minimizes the need for expensive labeled data in audio classification, successfully leveraging unlabeled data to improve accuracy with minimal supervision.

Kurdish Speech to Text Recognition System Based on Deep Convolutional-recurrent Neural Networks

In recent years, deep learning has had enormous success in speech recognition and natural language processing. In other languages, recent progress in speech recognition has been quite promising, but the Kurdish language has not seen comparable development. There are extremely few research papers on Kurdish speech recognition. In this paper, investigated Gated Recurrent Units (GRUs) which is one of the popular RNN models to recognize individual Kurdish words, and propose a very simplified deep-learning architecture to get more efficient and high accuracy model. The proposed model consists of a combination of CNN and GRU layers. The Kurdish Sorani Speech KSS dataset was created for the speech recognition system, as its 18799 sound files for 500 formal Kurdish words. Finally, the model proposed was trained with collected data and yielded over %96 accuracy. The combination of CNN an RNN (GURs) for speech recognition achieved superior performance compared to the other feed-forward deep neural network models and other statistical methods.