Multi-stage Convolutional Neural Network Research Articles

Short-Term Wind Power Prediction Based on Feature-Weighted and Combined Models

Accurate wind power prediction helps to fully utilize wind energy and improve the stability of the power grid. However, existing studies mostly analyze key wind power-related features equally without distinguishing the importance of different features. In addition, single models have limitations in fully extracting input feature information and capturing the time-dependent relationships of feature sequences, posing significant challenges to wind power prediction. To solve these problems, this paper presents a wind power forecasting approach that combines feature weighting and a combination model. Firstly, we use the attention mechanism to learn the weights of different input features, highlighting the more important features. Secondly, a Multi-Convolutional Neural Network (MCNN) with different convolutional kernels is employed to extract feature information comprehensively. Next, the extracted feature information is input into a Stacked BiLSTM (SBiLSTM) network to capture the temporal dependencies of the feature sequence. Finally, the prediction results are obtained. This article conducted four comparative experiments using measured data from wind farms. The experimental results demonstrate that the model has significant advantages; compared to the CNN-BiLSTM model, the mean absolute error, mean squared error, and root mean squared error of multi-step prediction at different prediction time resolutions are reduced by 35.59%, 59.84%, and 36.77% on average, respectively, and the coefficient of determination is increased by 1.35% on average.

Efficient model updating of shaft-raft-hull system using multi-stage convolutional neural network combined with sensitivity analysis

Model updating for marine shaft-raft-hull systems presents significant challenges due to the numerous components and the resulting inaccessible parameters. This study introduces an advanced framework for updating the model parameters of these complex coupling systems using convolutional neural networks (CNNs). Rapid analysis technique based on the substructure synthesis method is employed to enhance the efficiency of generating the CNN training dataset. Global sensitivity analysis is then utilized to identify critical parameters across various frequency bands. Informed by parameter sensitivity, the frequency bands are segmented, and a multi-stage CNN model updating strategy is proposed. The effectiveness of this method is validated through numerical simulations and experimental studies on a scaled shaft-raft-hull model. The findings demonstrate that segmenting the dataset based on global sensitivity analysis results markedly improves the convergence of CNNs, providing a robust solution for model updating in complex marine systems.

Classification of osteoarthritic and healthy cartilage using deep learning with Raman spectra

Raman spectroscopy is a rapid method for analysing the molecular composition of biological material. However, noise contamination in the spectral data necessitates careful pre-processing prior to analysis. Here we propose an end-to-end Convolutional Neural Network to automatically learn an optimal combination of pre-processing strategies, for the classification of Raman spectra of superficial and deep layers of cartilage harvested from 45 Osteoarthritis and 19 Osteoporosis (Healthy controls) patients. Using 6-fold cross-validation, the Multi-Convolutional Neural Network achieves comparable or improved classification accuracy against the best-performing Convolutional Neural Network applied to either the raw or pre-processed spectra. We utilised Integrated Gradients to identify the contributing features (Raman signatures) in the network decision process, showing they are biologically relevant. Using these features, we compared Artificial Neural Networks, Decision Trees and Support Vector Machines for the feature selection task. Results show that training on fewer than 3 and 300 features, respectively, for the disease classification and layer assignment task provide performance comparable to the best-performing CNN-based network applied to the full dataset. Our approach, incorporating multi-channel input and Integrated Gradients, can potentially facilitate the clinical translation of Raman spectroscopy-based diagnosis without the need for laborious manual pre-processing and feature selection.

Enhancing Alzheimer's disease diagnosis and staging: a multistage CNN framework using MRI.

This study addresses the pervasive and debilitating impact of Alzheimer's disease (AD) on individuals and society, emphasizing the crucial need for timely diagnosis. We present a multistage convolutional neural network (CNN)-based framework for AD detection and sub-classification using brain magnetic resonance imaging (MRI). After preprocessing, a 26-layer CNN model was designed to differentiate between healthy individuals and patients with dementia. After detecting dementia, the 26-layer CNN model was reutilized using the concept of transfer learning to further subclassify dementia into mild, moderate, and severe dementia. Leveraging the frozen weights of the developed CNN on correlated medical images facilitated the transfer learning process for sub-classifying dementia classes. An online AD dataset is used to verify the performance of the proposed multistage CNN-based framework. The proposed approach yielded a noteworthy accuracy of 98.24% in identifying dementia classes, whereas it achieved 99.70% accuracy in dementia subclassification. Another dataset was used to further validate the proposed framework, resulting in 100% performance. Comparative evaluations against pre-trained models and the current literature were also conducted, highlighting the usefulness and superiority of the proposed framework and presenting it as a robust and effective AD detection and subclassification method.

Memory-less scattering imaging with ultrafast convolutional optical neural networks.

The optical memory effect in complex scattering media including turbid tissue and speckle layers has been a critical foundation for macroscopic and microscopic imaging methods. However, image reconstruction from strong scattering media without the optical memory effect has not been achieved. Here, we demonstrate image reconstruction through scattering layers where no optical memory effect exists, by developing a multistage convolutional optical neural network (ONN) integrated with multiple parallel kernels operating at the speed of light. Training this Fourier optics-based, parallel, one-step convolutional ONN with the strong scattering process for direct feature extraction, we achieve memory-less image reconstruction with a field of view enlarged by a factor up to 271. This device is dynamically reconfigurable for ultrafast multitask image reconstruction with a computational power of 1.57 peta-operations per second (POPS). Our achievement establishes an ultrafast and high energy-efficient optical machine learning platform for graphic processing.

Yoga Pose Detection and Correction Using OpenPose

Abstract: OpenPose serves as a powerful tool for Human Pose Estimation, specializing in localizing anatomical key points or parts with a primary focus on identifying various body parts. Leveraging part affinity fields and a multi-convolutional neural network (CNN) architecture, OpenPose excels in extracting human parts and establishing associations with exceptional performance. The integrated design aims at jointly learning parts detection and their associations, with the part affinity score guiding the connections between different body parts.

Real-time Multi-CNN-based Emotion Recognition System for Evaluating Museum Visitors’ Satisfaction

Conventional studies on the satisfaction of museum visitors focus on collecting information through surveys to provide a one-way service to visitors, and thus it is impossible to obtain feedback on the real-time satisfaction of visitors who are experiencing the museum exhibition program. In addition, museum practitioners lack research on automated ways to evaluate a produced content program's lifecycle and its appropriateness. To overcome these problems, we propose a novel multi-convolutional neural network, called VimoNet, which is able to recognize visitors emotions automatically in real-time based on their facial expressions and body gestures. Furthermore, we design a user preference model of content and a framework to obtain feedback on content improvement for providing personalized digital cultural heritage content to visitors. Specifically, we define seven emotions of visitors and build a dataset of visitor facial expressions and gestures with respect to the emotions. Using the dataset, we proceed with feature fusion of face and gesture images trained on the DenseNet-201 and VGG-16 models for generating a combined emotion recognition model. From the results of the experiment, VimoNet achieved a classification accuracy of 84.10%, providing 7.60% and 14.31% improvement, respectively, over a single face and body gesture-based method of emotion classification performance. It is thus possible to automatically capture the emotions of museum visitors via VimoNet, and we confirm its feasibility through a case study with respect to digital content of cultural heritage.

Ensembled CNN with artificial bee colony optimization method for esophageal cancer stage classification using SVM classifier.

Esophageal cancer (EC) is aggressive cancer with a high fatality rate and a rapid rise of the incidence globally. However, early diagnosis of EC remains a challenging task for clinicians. To help address and overcome this challenge, this study aims to develop and test a new computer-aided diagnosis (CAD) network that combines several machine learning models and optimization methods to detect EC and classify cancer stages. The study develops a new deep learning network for the classification of the various stages of EC and the premalignant stage, Barrett's Esophagus from endoscopic images. The proposed model uses a multi-convolution neural network (CNN) model combined with Xception, Mobilenetv2, GoogLeNet, and Darknet53 for feature extraction. The extracted features are blended and are then applied on to wrapper based Artificial Bee Colony (ABC) optimization technique to grade the most accurate and relevant attributes. A multi-class support vector machine (SVM) classifies the selected feature set into the various stages. A study dataset involving 523 Barrett's Esophagus images, 217 ESCC images and 288 EAC images is used to train the proposed network and test its classification performance. The proposed network combining Xception, mobilenetv2, GoogLeNet, and Darknet53 outperforms all the existing methods with an overall classification accuracy of 97.76% using a 3-fold cross-validation method. This study demonstrates that a new deep learning network that combines a multi-CNN model with ABC and a multi-SVM is more efficient than those with individual pre-trained networks for the EC analysis and stage classification.

Pose Estimation for Human Activity Recognition Using Deep Learning on Video Data

Pose estimation is a critical task in computer vision, aiming to determine the spatial positions and orientations of objects or individuals within an image or video. This paper introduces a novel approach to pose estimation that leverages deep learning techniques to achieve high accuracy and robustness in diverse environments. We propose a multi-stage convolutional neural network (CNN) that refines pose predictions through iterative processing, significantly enhancing the precision of keypoint localization. The network architecture is complemented by a loss function designed to handle occlusions and ambiguous poses, ensuring reliable performance even in complex scenes.

A CNN-Based Chest Infection Diagnostic Model: A Multistage Multiclass Isolated and Developed Transfer Learning Framework

In 2019, a deadly coronaviral infection (COVID-19) that infected millions of people globally was detected in China. This fatal virus affects the respiratory system and currently spreads to more than 200 nations worldwide. COVID-19 may be found using a chest X-ray scan, a reliable imaging method. Although an expert may examine an X-ray scan manually, this process takes a lot of time. Therefore, deep convolutional neural networks (CNNs) may be utilized to automate this procedure. In this work, at the first step, a novel isolated 19-layer CNN model is developed from scratch to detect chest infections using X-rays. Then, the developed model is reutilized to distinguish the type of chest infection, such as COVID-19, fibrosis, pneumonia, and tuberculosis, using the transfer learning approach. Stochastic gradient descent with momentum is utilized to optimize the model. The proposed multistage framework shows 98.85% and 97% classification accuracies for chest infection detection (binary classification between normal and patient) and four-class subclassification (COVID-19, fibrosis, pneumonia, and tuberculosis) for an online chest X-ray dataset. The reliability of the proposed multistage CNN model was further validated through a new dataset, showing an accuracy of 98.5%. The proposed multistage methodology took minimal training time compared to publically available pretrained models. Therefore, the presented multistage deep learning framework can help doctors in clinical practices.

Convolutional Neural Network-Based Automated System for Dog Tracking and Emotion Recognition in Video Surveillance

This paper proposes a multi–convolutional neural network (CNN)-based system for the detection, tracking, and recognition of the emotions of dogs in surveillance videos. This system detects dogs in each frame of a video, tracks the dogs in the video, and recognizes the dogs’ emotions. The system uses a YOLOv3 model for dog detection. The dogs are tracked in real time with a deep association metric model (DeepDogTrack), which uses a Kalman filter combined with a CNN for processing. Thereafter, the dogs’ emotional behaviors are categorized into three types—angry (or aggressive), happy (or excited), and neutral (or general) behaviors—on the basis of manual judgments made by veterinary experts and custom dog breeders. The system extracts sub-images from videos of dogs, determines whether the images are sufficient to recognize the dogs’ emotions, and uses the long short-term deep features of dog memory networks model (LDFDMN) to identify the dog’s emotions. The dog detection experiments were conducted using two image datasets to verify the model’s effectiveness, and the detection accuracy rates were 97.59% and 94.62%, respectively. Detection errors occurred when the dog’s facial features were obscured, when the dog was of a special breed, when the dog’s body was covered, or when the dog region was incomplete. The dog-tracking experiments were conducted using three video datasets, each containing one or more dogs. The highest tracking accuracy rate (93.02%) was achieved when only one dog was in the video, and the highest tracking rate achieved for a video containing multiple dogs was 86.45%. Tracking errors occurred when the region covered by a dog’s body increased as the dog entered or left the screen, resulting in tracking loss. The dog emotion recognition experiments were conducted using two video datasets. The emotion recognition accuracy rates were 81.73% and 76.02%, respectively. Recognition errors occurred when the background of the image was removed, resulting in the dog region being unclear and the incorrect emotion being recognized. Of the three emotions, anger was the most prominently represented; therefore, the recognition rates for angry emotions were higher than those for happy or neutral emotions. Emotion recognition errors occurred when the dog’s movements were too subtle or too fast, the image was blurred, the shooting angle was suboptimal, or the video resolution was too low. Nevertheless, the current experiments revealed that the proposed system can correctly recognize the emotions of dogs in videos. The accuracy of the proposed system can be dramatically increased by using more images and videos for training the detection, tracking, and emotional recognition models. The system can then be applied in real-world situations to assist in the early identification of dogs that may exhibit aggressive behavior.

Improved Detection and Analysis of Wheat Spikes U sing Multi-Stage Convolutional Neural Network

High throughput plant phenotyping is the advanced scientific approach for rapid phenotyping of plant traits, especially high consumable grains or crops, which is designed to process a high volume of data in a short time for plant breeders and cultivars to utilize. Detection and counting of crop traits such as plants, fruits, wheat or rice spikes, sorghum head, and plant diseases is more advanced research in this field, where real-world data are collected using aerial and land-based imaging platforms equipped with a variety of geospatial sensors, and their statistical analysis is conducted using Artificial Intelligence (AI) and Deep Learning-based solutions. In this paper, we contributed to solving such a challenge of phenotyping by detecting and counting wheat spikes from land-based imaging by applying a Region-based Convolutional Neural Network (CNN) model. Our method employs the use of CNN to extract features from the imaging platform and the learning model is trained to detect and count wheat spikes in field images based on these extracted features. Using the publicly available SPIKE dataset to train and test our model, our proposed method achieved 98% average precision and 91% average F1 score on the test set. Our results show a significant improvement of 2.9% and 11.2% in detection accuracy as well as 1% and 3% in average precision metric over state-of-the-art Faster Regionbased Convolutional Neural Network (Faster-RCNN), and RetinaNet, respectively, and have the potential to significantly benefit plant breeders by facilitating the selection of wheat varieties with high yields. DUJASE Vol. 7 (2) 21-30, 2022 (July)

Self-Supervised Learning of Perceptually Optimized Block Motion Estimates for Video Compression.

Block based motion estimation is integral to inter prediction processes performed in hybrid video codecs. Prevalent block matching based methods that are used to compute block motion vectors (MVs) rely on computationally intensive search procedures. They also suffer from the aperture problem, which tends to worsen as the block size is reduced. Moreover, the block matching criteria used in typical codecs do not account for the resulting levels of perceptual quality of the motion compensated pictures that are created upon decoding. Towards achieving the elusive goal of perceptually optimized motion estimation, we propose a search-free block motion estimation framework using a multi-stage convolutional neural network, which is able to conduct motion estimation on multiple block sizes simultaneously, using a triplet of frames as input. This composite block translation network (CBT-Net) is trained in a self-supervised manner on a large database that we created from publicly available uncompressed video content. We deploy the multi-scale structural similarity (MS-SSIM) loss function to optimize the perceptual quality of the motion compensated predicted frames. Our experimental results highlight the computational efficiency of our proposed model relative to conventional block matching based motion estimation algorithms, for comparable prediction errors. Further, when used to perform inter prediction in AV1, the MV predictions of the perceptually optimized model result in average Bjontegaard-delta rate (BD-rate) improvements of -1.73% and -1.31% with respect to the MS-SSIM and Video Multi-Method Assessment Fusion (VMAF) quality metrics, respectively, as compared to the block matching based motion estimation system employed in the SVT-AV1 encoder.

A Novel Framework for Semi-supervised Multiple-label Image Classification using Multi-stage CNN and Visual Attention Mechanism

To train deep neural networks effectively, a lot of labeled data is typically needed. However, real-time applications make it difficult and expensive to acquire high-quality labels for the data because it takes skill and knowledge to accurately annotate multiple label images. In order to enhance classification performance, it is also crucial to extract image features from all potential objects of various sizes as well as the relationships between labels of numerous label images. The current approaches fall short in their ability to map the label dependencies and effectively classify the labels. They also perform poor to label the unlabeled images when small amount of labeled images available for classification. In order to solve these issues, we suggest a new framework for semi-supervised multiple object label classification using multi-stage Convolutional neural networks with visual attention (MSCNN)and GCN for label co-occurrence embedding(LCE) (MSCNN-LCE-MIC), which combines GCN and attention mechanism to concurrently capture local and global label dependencies throughout the entire image classification process. Four main modules make up MSCNN-LCE-MIC: (1) improved multi-label propagation method for labeling largely available unlabeled image; (2) a feature extraction module using multi-stage CNN with visual attention mechanism that focuses on the connections between labels and target regions to extract accurate features from each input image; (3) a label co-existence learning that applies GCN to discover the associations between different items to create embeddings of label co-occurrence; and (4) an integrated multi-modal fusion module. Numerous tests on MS-COCO and PASCAL VOC2007 show that MSCNN-LCE-MIC significantly improves classification efficiency on mAP 84.3% and 95.8% respectively when compared to the most recent existing methods.

Design and Optimization of CNN Architecture to Identify the Types of Damage Imagery

Damage to the surface construction of reinforced concrete (RC) will impact the security of the facility’s structure. Deep learning can effectively identify various types of damage, which is useful for taking protective measures to avoid further deterioration of the structure. Based on deep learning, the multi-convolutional neural network (MCNN) has the potential for identifying multiple RC damage images. The MCNN6 of this study was evaluated by indicators (accuracy, loss, and efficiency), and the optimized architecture was confirmed. The results show that the identification performance for “crack and rebar exposure” (Type B) by MCNN6 is the best, with an accuracy of 96.81% and a loss of 0.07. The accuracy of the other five types of damage combinations is also higher than 80.0%, and the loss is less than 0.44. Finally, the MCNN6 model can be used in the detection of various damage to achieve automated assessment for RC facility surface conditions.

Deep visual social distancing monitoring to combat COVID-19: A comprehensive survey.

Since the start of the COVID-19 pandemic, social distancing (SD) has played an essential role in controlling and slowing down the spread of the virus in smart cities. To ensure the respect of SD in public areas, visual SD monitoring (VSDM) provides promising opportunities by (i) controlling and analyzing the physical distance between pedestrians in real-time, (ii) detecting SD violations among the crowds, and (iii) tracking and reporting individuals violating SD norms. To the authors’ best knowledge, this paper proposes the first comprehensive survey of VSDM frameworks and identifies their challenges and future perspectives. Typically, we review existing contributions by presenting the background of VSDM, describing evaluation metrics, and discussing SD datasets. Then, VSDM techniques are carefully reviewed after dividing them into two main categories: hand-crafted feature-based and deep-learning-based methods. A significant focus is paid to convolutional neural networks (CNN)-based methodologies as most of the frameworks have used either one-stage, two-stage, or multi-stage CNN models. A comparative study is also conducted to identify their pros and cons. Thereafter, a critical analysis is performed to highlight the issues and impediments that hold back the expansion of VSDM systems. Finally, future directions attracting significant research and development are derived.

A deep asymmetric Laplace neural network for deterministic and probabilistic wind power forecasting

Accurate forecasting of wind power faces two challenges: 1) extracting more effective information on power fluctuations from limited input features, and 2) constructing a suitable loss function for model training. This paper proposes a novel deep asymmetric Laplace neural network named AL-MCNN-BiLSTM for wind power forecasting. The maximal information coefficient, which can describe the linear and nonlinear relationships between targeted and historical wind power data, is employed to determine the optimal inputs. Then, a novel multi-convolutional neural network (MCNN) is designed with a multi-scale information fusion block, which helps make full use of the multi-scale information in different convolutional layers. The MCNN extracts local information from inputs, then bidirectional long-short-term memory (BiLSTM) is employed to extract temporal information. An asymmetric Laplace distribution is assumed to characterize the uncertainty in wind power forecasts, such that an asymmetric Laplace-based loss function can be used in the model. The forecasting results on four datasets demonstrate that AL-MCNN-BiLSTM not only generates more precise deterministic wind power forecasts with a maximum coefficient of determination of 0.9803, but also produces more reliable prediction intervals at 85%, 90%, 95%, and 99% confidence levels with minimum values of pinball loss reaching 6.8948, 5.1895, 3.0189, and 0.7645, respectively.

Automated freezing of gait assessment with marker-based motion capture and multi-stage spatial-temporal graph convolutional neural networks

BackgroundFreezing of gait (FOG) is a common and debilitating gait impairment in Parkinson’s disease. Further insight into this phenomenon is hampered by the difficulty to objectively assess FOG. To meet this clinical need, this paper proposes an automated motion-capture-based FOG assessment method driven by a novel deep neural network.MethodsAutomated FOG assessment can be formulated as an action segmentation problem, where temporal models are tasked to recognize and temporally localize the FOG segments in untrimmed motion capture trials. This paper takes a closer look at the performance of state-of-the-art action segmentation models when tasked to automatically assess FOG. Furthermore, a novel deep neural network architecture is proposed that aims to better capture the spatial and temporal dependencies than the state-of-the-art baselines. The proposed network, termed multi-stage spatial-temporal graph convolutional network (MS-GCN), combines the spatial-temporal graph convolutional network (ST-GCN) and the multi-stage temporal convolutional network (MS-TCN). The ST-GCN captures the hierarchical spatial-temporal motion among the joints inherent to motion capture, while the multi-stage component reduces over-segmentation errors by refining the predictions over multiple stages. The proposed model was validated on a dataset of fourteen freezers, fourteen non-freezers, and fourteen healthy control subjects.ResultsThe experiments indicate that the proposed model outperforms four state-of-the-art baselines. Moreover, FOG outcomes derived from MS-GCN predictions had an excellent (r = 0.93 [0.87, 0.97]) and moderately strong (r = 0.75 [0.55, 0.87]) linear relationship with FOG outcomes derived from manual annotations.ConclusionsThe proposed MS-GCN may provide an automated and objective alternative to labor-intensive clinician-based FOG assessment. Future work is now possible that aims to assess the generalization of MS-GCN to a larger and more varied verification cohort.

Low-dose computed tomography image reconstruction via a multistage convolutional neural network with autoencoder perceptual loss network.

Computed tomography (CT) is widely used in medical diagnoses due to its ability to non-invasively detect the internal structures of the human body. However, CT scans with normal radiation doses can cause irreversible damage to patients. The radiation exposure is reduced with low-dose CT (LDCT), although considerable speckle noise and streak artifacts in CT images and even structural deformation may result, significantly undermining its diagnostic capability. This paper proposes a multistage network framework which gradually divides the entire process into 2-staged sub-networks to complete the task of image reconstruction. Specifically, a dilated residual convolutional neural network (DRCNN) was used to denoise the LDCT image. Then, the learned context information was combined with the channel attention subnet, which retains local information, to preserve the structural details and features of the image and textural information. To obtain recognizable characteristic details, we introduced a novel self-calibration module (SCM) between the 2 stages to reweight the local features, which realizes the complementation of information at different stages while refining feature information. In addition, we also designed an autoencoder neural network, using a self-supervised learning scheme to train a perceptual loss neural network specifically for CT images. We evaluated the diagnostic quality of the results and performed ablation experiments on the loss function and network structure modules to verify each module's effectiveness in the network. Our proposed network architecture obtained high peak signal-to-noise ratio (PSNR), structural similarity index measure (SSIM), and visual information fidelity (VIF) values in terms of quantitative evaluation. In the analysis of qualitative results, our network structure maintained a better balance between eliminating image noise and preserving image details. Experimental results showed that our proposed network structure obtained better metrics and visual evaluation. This study proposed a new LDCT image reconstruction method by combining autoencoder perceptual loss networks with multistage convolutional neural networks (MSCNN). Experimental results showed that the newly proposed method has performance than other methods.

Using a multi-convolutional neural network to automatically identify small-sample tea leaf diseases

Tea leaf disease is a factor leading to the decline of tea quality and yield. In this study, a multi-convolutional neural network (CNN) model, MergeModel, combined with diseased leaf segmentation and a weight initialization method, was used for automatic identification of tea leaf diseases in small samples. Diseased leaf segmentation could reduce the impact of complex backgrounds on the identification results. The integration of multiple CNN modules enabled MergeModel to extract a variety of discriminative features. The identification ability of MergeModel was improved compared with a single neural network. The weight initialization method encoded the diseased leaf features into the convolution filter, which helped the model to concentrate on learning important features at the beginning of training. In addition, this study used a small number of diseased tea leaf images as the original training samples and generated new training samples through the unconditional generation model, namely, SinGAN, for data augmentation. Experimental results showed that MergeModel could effectively distinguish diseased tea leaves from healthy tea leaves and identify common tea leaf diseases such as tea white scab, tea leaf blight, tea red scab, and tea sooty mould. Compared with the existing methods, the proposed method had higher accuracy in identifying tea leaf diseases in small samples.