Multi-stream Convolutional Neural Network Research Articles

An analysis of information segregation in parallel streams of a multi-stream convolutional neural network.

Visual information is processed in hierarchically organized parallel streams in the primate brain. In the present study, information segregation in parallel streams was examined by constructing a convolutional neural network with parallel architecture in all of the convolutional layers. Although filter weights for convolution were initially set to random values, color information was segregated from shape information in most model instances after training. Deletion of the color-related stream decreased recognition accuracy of animate images, whereas deletion of the shape-related stream decreased recognition accuracy of both animate and inanimate images. The results suggest that properties of filters and functions of a stream are spontaneously segregated in parallel streams of neural networks.

Deep learning methods for early detection of Alzheimer’s disease using structural MR images: a survey

In this paper, we present an extensive review of the most recent works on Alzheimer’s disease (AD) prediction, focusing on Moderate Cognitive Impairment (MCI) conversion prediction. We aimed to identify the most useful brain-magnetic resonance imaging (MRI) biomarkers and deep learning frameworks used for prediction. To achieve this, we analyzed more than 130 studies and reviewed 7 articles. A closer examination revealed that the hippocampus is an important region of interest (ROI) affected early by AD, and many related features help detect the disease in its early stages. However, when considered alone, this ROI is not sufficient to ensure high prediction performance. Therefore, several other brain regions can also provide additional information to improve prediction accuracy. Concerning state-of-the-art deep neural networks, the U-Net represents the most efficient architecture for hippocampus segmentation. The RESU-Net architecture achieved the highest Dice Similarity Coefficient (DSC) value, equal to 94%.For MCI conversion prediction, the best results were obtained by two models identifying significant landmarks from the entire brain for classification. The multi-stream convolutional neural network achieved the best Area Under the Curve (AUC) and specificity of 94.39% and 99.70%, respectively. Finally, a region ensemble model delivered the highest accuracy of 85.90%, highlighting the need for further research to address this challenging problem.

ClueCatcher: Catching Domain-Wise Independent Clues for Deepfake Detection

Deepfake detection is a focus of extensive research to combat the proliferation of manipulated media. Existing approaches suffer from limited generalizability and struggle to detect deepfakes created using unseen techniques. This paper proposes a novel deepfake detection method to improve generalizability. We observe domain-wise independent clues in deepfake images, including inconsistencies in facial colors, detectable artifacts at synthesis boundaries, and disparities in quality between facial and nonfacial regions. This approach uses an interpatch dissimilarity estimator and a multistream convolutional neural network to capture deepfake clues unique to each feature. By exploiting these clues, we enhance the effectiveness and generalizability of deepfake detection. The experimental results demonstrate the improved performance and robustness of this method.

Radar-Based Air-Writing Gesture Recognition Using a Novel Multistream CNN Approach

Hand gestures, being a convenient and natural way of communication, is getting huge attention for human–computer interface designs. Among these gestures, detecting mid-air writing is one of the most promising applications. Existing radar-based solutions often perform the mid-air writing recognition by tracking the hand trajectory using multiple monostatic or bistatic radars. This article presents a multistream convolutional neural network (MS-CNN)-based in-air digits recognition method using a frequency-modulated continuous-wave (FMCW) radar. With one FMCW radar comprising of two receiving channels, a novel three-stream CNN network with range-time, Doppler-time, and angle-time spectrograms as inputs is constructed and the features are fused together in the later stage before making a final recognition. Unlike the traditional CNN, MS-CNN with multiple independent input layers enables the creation of a multidimensional deep-learning model for FMCW radars. Twelve human volunteers were invited to writing the digits from zero to nine in the air in both home and lab environments. The three-stream CNN architecture-based air writing for digits has shown a promising accuracy of 95%. A comparison of the proposed MS-CNN system was made with 45 different variants of CNN and preliminary results shows that MS-CNN outperforms the other traditional CNN architectures for air-writing application. The gestures radar data have also been made available to the research community.

MSFF-Net: Multi-Stream Feature Fusion Network for surface electromyography gesture recognition.

In the field of surface electromyography (sEMG) gesture recognition, how to improve recognition accuracy has been a research hotspot. The rapid development of deep learning provides a new solution to this problem. At present, the main applications of deep learning for sEMG gesture feature extraction are based on convolutional neural network (CNN) structures to capture spatial morphological information of the multichannel sEMG or based on long short-term memory network (LSTM) to extract time-dependent information of the single-channel sEMG. However, there are few methods to comprehensively consider the distribution area of the sEMG signal acquisition electrode sensor and the arrangement of the sEMG signal morphological features and electrode spatial features. In this paper, a novel multi-stream feature fusion network (MSFF-Net) model is proposed for sEMG gesture recognition. The model adopts a divide-and-conquer strategy to learn the relationship between different muscle regions and specific gestures. Firstly, a multi-stream convolutional neural network (Multi-stream CNN) and a convolutional block attention module integrated with a resblock (ResCBAM) are used to extract multi-dimensional spatial features from signal morphology, electrode space, and feature map space. Then the learned multi-view depth features are fused by a view aggregation network consisting of an early fusion network and a late fusion network. The results of all subjects and gesture movement validation experiments in the sEMG signal acquired from 12 sensors provided by NinaPro's DB2 and DB4 sub-databases show that the proposed model in this paper has better performance in terms of gesture recognition accuracy compared with the existing models.

3D Face Recognition Neural Network for Digital Human Resource Management

Digital human resource management can improve the organizational and operational efficiency of enterprises.In order to improve the efficiency of enterprise digital management and solve the problems of low security level and insufficient stability of 2D face recognition, we introduce 3D face recognition into the digital human resource management system. We propose a face recognition method based on a multistream convolutional neural network and local binary pattern and build a digital face recognition management system. We first build the system computer vision scene. Then a local binary mode facial expression feature extraction scheme is designed according to the depth camera image extraction method. Considering that face 3D features are easy to be missed, we build a multistream convolutional neural network to learn facial 3D features. Finally, we validate the effectiveness of the method in selecting a public face dataset. Experiments prove that our method can reach 98% face recognition accuracy, which is significantly better than other methods.

A Multi-Stream Convolutional Neural Network for Classification of Progressive MCI in Alzheimer's Disease Using Structural MRI Images.

Early diagnosis of Alzheimer's disease and its prodromal stage, also known as mild cognitive impairment (MCI), is critical since some patients with progressive MCI will develop the disease. We propose a multi-stream deep convolutional neural network fed with patch-based imaging data to classify stable MCI and progressive MCI. First, we compare MRI images of Alzheimer's disease with cognitively normal subjects to identify distinct anatomical landmarks using a multivariate statistical test. These landmarks are then used to extract patches that are fed into the proposed multi-stream convolutional neural network to classify MRI images. Next, we train the architecture in a separate scenario using samples from Alzheimer's disease images, which are anatomically similar to the progressive MCI ones and cognitively normal images to compensate for the lack of progressive MCI training data. Finally, we transfer the trained model weights to the proposed architecture in order to fine-tune the model using progressive MCI and stable MCI data. Experimental results on the ADNI-1 dataset indicate that our method outperforms existing methods for MCI classification, with an F1-score of 85.96%.

Driver Identification System Using 2D ECG and EMG Based on Multistream CNN for Intelligent Vehicle

As autonomous driving technologies develop, biometrics technology using driver's bio-signals provides various driver-customized infotainment services. Many studies on driver identification are underway to improve the feature extraction and classification steps to identify drivers with high accuracy. The existing identification systems have used a driver's single biological signal to identify the driver without considering the driver's driving state. The identification error rate was high because electromyography (EMG) is included according to motion artifacts while driving in the electrocardiogram (ECG) acquired by the driver's behavioral characteristics. In this letter, EMG acquired by the driver's motion and ECG acquired by the behavioral characteristics are used simultaneously to improve the identification accuracy. The proposed identification system converts ECG and EMG into 2D constant Q transform (CQT) images and classifies drivers by multistream convolutional neural network (CNN). Our extensive experiments show that a single ECG can achieve 98.1% of the identification accuracy, a single EMG can achieve 84.4% of the identification accuracy, and a multisignal can achieve 98.9% of the identification accuracy with 2D CQT.

Multi-Stream Convolutional Neural Networks for Rotating Machinery Fault Diagnosis under Noise and Trend Items.

In recent years, rotating machinery fault diagnosis methods based on convolutional neural network have achieved much success. However, in real industrial environments, interfering signals are unavoidable, which may reduce the accuracy of fault diagnosis seriously. Most of the current fault diagnosis methods are of single input type, which may lead to the information contained in the vibration signal not being fully utilized. In this study, theoretical analysis and comprehensive comparative experiments are completed to investigate the time domain input, frequency domain input, and two types of time–frequency domain input. Based on this, a new fault diagnosis model, named multi-stream convolutional neural network, is developed. The model takes the time domain, frequency domain, and time–frequency domain images as input, and it automatically fuses the information contained in different inputs. The proposed model is tested based on three public datasets. The experimental results suggested that the model achieved pretty high accuracy under noise and trend items without the help of signal separation algorithms. In addition, the positive implications of multiple inputs and information fusion are analyzed through the visualization of learned features.

Multi-Stream Convolutional Neural Network-Based Wearable, Flexible Bionic Gesture Surface Muscle Feature Extraction and Recognition.

Surface electromyographic (sEMG) signals are weak physiological electrical signals, which are highly susceptible to coupling external noise and cause major difficulties in signal acquisition and processing. The study of using sEMG signals to analyze human motion intention mainly involves data preprocessing, feature extraction, and model classification. Feature extraction is an extremely critical part; however, this often involves many manually designed features with specialized domain knowledge, so the experimenter will spend time and effort on feature extraction. To address this problem, deep learning methods that can automatically extract features are applied to the sEMG-based gesture recognition problem, drawing on the success of deep learning for image classification. In this paper, sEMG is captured using a wearable, flexible bionic device, which is simple to operate and highly secure. A multi-stream convolutional neural network algorithm is proposed to enhance the ability of sEMG to characterize hand actions in gesture recognition. The algorithm virtually augments the signal channels by reconstructing the sample structure of the sEMG to provide richer input information for gesture recognition. The methods for noise processing, active segment detection, and feature extraction are investigated, and a basic method for gesture recognition based on the combination of multichannel sEMG signals and inertial signals is proposed. Suitable filters are designed for the common noise in the signal. An improved moving average method based on the valve domain is used to reduce the segmentation error rate caused by the short resting signal time in continuous gesture signals. In this paper, three machine learning algorithms, K-nearest neighbor, linear discriminant method, and multi-stream convolutional neural network, are used for hand action classification experiments, and the effectiveness of the multi-stream convolutional neural network algorithm is demonstrated by comparison of the results. To improve the accuracy of hand action recognition, a final 10 gesture classification accuracy of up to 93.69% was obtained. The separability analysis showed significant differences in the signals of the two cognitive-behavioral tasks when the optimal electrode combination was used. A cross-subject analysis of the test set subjects illustrated that the average correct classification rate using the pervasive electrode combination could reach 93.18%.

근전도 신호 기반 Multi-Stream CNN 및 경험적 모드 분해를 이용한 사용자 인식

근전도 신호는 골격근이 수축하는 동안 발생하는 생체신호로, 수행한 동작에 따라 다른 신호 파형이 생성된다. 이에 기존 사용자 인식 방법의 등록 정보를 변경할 수 없는 문제 해결이 가능하다. 본 논문에서는 근전도 신호 기반 경험적 모드 분해 및 multi-stream Convolutional Neural Network(CNN)을 이용한 사용자 인식 방법을 제안한다. 제안한 방법은 특징 데이터 증가를 위해 근전도 신호를 경험적 모드 분해를 이용해 내재 모드 함수로 분해한다. 분해된 내재 모드 함수 1-4는 설계한 multi-stream CNN에 입력데이터로 사용해 사용자 인식을 수행한다. 벤치마킹 근전도 데이터인 Ninapro DB2를 이용한 실험 결과, multi-stream CNN 및 내재 모드 함수를 이용한 방법은 98.3%의 정확도로, 기존 CNN을 이용한 연구에 비해 1% 성능이 높게 나타났다.

COVID-19 detection on chest radiographs using feature fusion based deep learning.

The year 2020 will certainly be remembered in human history as the year in which humans faced a global pandemic that drastically affected every living soul on planet earth. The COVID-19 pandemic certainly had a massive impact on human’s social and daily lives. The economy and relations of all countries were also radically impacted. Due to such unexpected situations, healthcare systems either collapsed or failed under colossal pressure to cope with the overwhelming numbers of patients arriving at emergency rooms and intensive care units. The COVID -19 tests used for diagnosis were expensive, slow, and gave indecisive results. Unfortunately, such a hindered diagnosis of the infection prevented abrupt isolation of the infected people which, in turn, caused the rapid spread of the virus. In this paper, we proposed the use of cost-effective X-ray images in diagnosing COVID-19 patients. Compared to other imaging modalities, X-ray imaging is available in most healthcare units. Deep learning was used for feature extraction and classification by implementing a multi-stream convolutional neural network model. The model extracts and concatenates features from its three inputs, namely; grayscale, local binary patterns, and histograms of oriented gradients images. Extensive experiments using fivefold cross-validation were carried out on a publicly available X-ray database with 3886 images of three classes. Obtained results outperform the results of other algorithms with an accuracy of 97.76%. The results also show that the proposed model can make a significant contribution to the rapidly increasing workload in health systems with an artificial intelligence-based automatic diagnosis tool.

Multi-Stream CNN-Based Personal Recognition Method Using Surface Electromyogram for 5G Security

As fifth generation technology standard (5G) technology develops, the possibility of being exposed to the risk of cyber-attacks that exploits vulnerabilities in the 5G environment is increasing. The existing personal recognition method used for granting permission is a password-based method, which causes security problems. Therefore, personal recognition studies using bio-signals are being conducted as a method to access control to devices. Among bio-signal, surface electromyogram (sEMG) can solve the existing personal recognition problem that was unable to the modification of registered information owing to the characteristic changes in its signal according to the performed operation. Furthermore, as an advantage, sEMG can be conveniently measured from arms and legs. This paper proposes a personal recognition method using sEMG, based on a multi-stream convolutional neural network (CNN). The proposed method decomposes sEMG signals into intrinsic mode functions (IMF) using empirical mode decomposition (EMD) and transforms each IMF into a spectrogram. Personal recognition is performed by analyzing time–frequency features from the spectrogram transformed into multi-stream CNN. The database (DB) adopted in this paper is the Ninapro DB, which is a benchmark EMG DB. The experimental results indicate that the personal recognition performance of the multi-stream CNN using the IMF spectrogram improved by 1.91%, compared with the single-stream CNN using the spectrogram of raw sEMG.

Multimodal Data and Multiscale Kernel-Based Multistream CNN for Fine Classification of a Complex Surface-Mined Area

Fine land cover classification (FLCC) of complex landscapes is a popular and challenging task in the remote sensing community. In complex surface-mined areas (CSMAs), researchers have conducted FLCC using traditional machine learning methods and deep learning algorithms. However, convolutional neural network (CNN) algorithms that may be useful for FLCC of CSMAs have not been fully investigated. This study proposes a multimodal remote sensing data and multiscale kernel-based multistream CNN (3M-CNN) model. Experiments based on two ZiYuan-3 (ZY-3) satellite imageries of different times and seasons were conducted in Wuhan, China. The 3M-CNN model had three main features: (1) multimodal data-based multistream CNNs, i.e., using ZY-3 imagery-derived true color, false color, and digital elevation model data to form three CNNs; (2) multisize neighbors, i.e., using different neighbors of optical and topographic data as inputs; and (3) multiscale convolution flows revised from an inception module for optical and topographic data. Results showed that the proposed 3M-CNN model achieved excellent overall accuracies on two different images, and outperformed other comparative models. In particular, the 3M-CNN model yielded obvious better visual performances. In general, the proposed process was beneficial for the FLCC of complex landscape areas.

Facial Beauty Prediction From Facial Parts Using Multi-Task and Multi-Stream Convolutional Neural Networks

Automatic analysis of facial beauty has become an emerging computer vision problem in recent years. Facial beauty prediction (FBP) aims at developing a human-like model that automatically makes facial attractiveness predictions. In this study, we present and evaluate a face attractiveness prediction approach using facial parts as well as a multi-task learning scheme. First, a deep convolutional neural network (CNN) pre-trained on massive face datasets is utilized for face attractiveness prediction, which is capable of automatic learning of high-level face representations. Next, we extend our deep model to other facial attribute recognition tasks. Hence, a multi-task learning scheme is leveraged by our deep model to learn optimal shared features for three correlated tasks (i.e. facial beauty assessment, gender recognition as well as ethnicity identification). To further enhance the attractiveness computation accuracy, specific regions of face images (i.e. left eye, nose and mouth) as well as the whole face are fed into multi-stream CNNs (i.e. three two-stream networks). Each two-stream network adopts a facial part as well as the full face as input. Extensive experiments are conducted on the SCUT-FBP5500 benchmark dataset, where our approach indicates significant improvement in accuracy over the other state-of-the-art methods.

Multi-stream pose convolutional neural networks for human interaction recognition in images

Recognizing human interactions in still images is quite a challenging task since compared to videos, there is only a glimpse of interaction in a single image. This work investigates the role of human poses in recognizing human–human interactions in still images. To this end, a multi-stream convolutional neural network architecture is proposed, which fuses different levels of human pose information to recognize human interactions better. In this context, several pose-based representations are explored. Experimental evaluations in an extended benchmark dataset show that the proposed multi-stream pose Convolutional Neural Network is successful in discriminating a wide range of human–human interactions and human poses when used in conjunction with the overall context provides discriminative cues about human–human interactions.

Spherical-Patches Extraction for Deep-Learning-Based Critical Points Detection in 3D Neuron Microscopy Images.

Digital reconstruction of neuronal structures is very important to neuroscience research. Many existing reconstruction algorithms require a set of good seed points. 3D neuron critical points, including terminations, branch points and cross-over points, are good candidates for such seed points. However, a method that can simultaneously detect all types of critical points has barely been explored. In this work, we present a method to simultaneously detect all 3 types of 3D critical points in neuron microscopy images, based on a spherical-patches extraction (SPE) method and a 2D multi-stream convolutional neural network (CNN). SPE uses a set of concentric spherical surfaces centered at a given critical point candidate to extract intensity distribution features around the point. Then, a group of 2D spherical patches is generated by projecting the surfaces into 2D rectangular image patches according to the orders of the azimuth and the polar angles. Finally, a 2D multi-stream CNN, in which each stream receives one spherical patch as input, is designed to learn the intensity distribution features from those spherical patches and classify the given critical point candidate into one of four classes: termination, branch point, cross-over point or non-critical point. Experimental results confirm that the proposed method outperforms other state-of-the-art critical points detection methods. The critical points based neuron reconstruction results demonstrate the potential of the detected neuron critical points to be good seed points for neuron reconstruction. Additionally, we have established a public dataset dedicated for neuron critical points detection, which has been released along with this article.

VC-Net: Deep Volume-Composition Networks for Segmentation and Visualization of Highly Sparse and Noisy Image Data.

The fundamental motivation of the proposed work is to present a new visualization-guided computing paradigm to combine direct 3D volume processing and volume rendered clues for effective 3D exploration. For example, extracting and visualizing microstructures in-vivo have been a long-standing challenging problem. However, due to the high sparseness and noisiness in cerebrovasculature data as well as highly complex geometry and topology variations of micro vessels, it is still extremely challenging to extract the complete 3D vessel structure and visualize it in 3D with high fidelity. In this paper, we present an end-to-end deep learning method, VC-Net, for robust extraction of 3D microvascular structure through embedding the image composition, generated by maximum intensity projection (MIP), into the 3D volumetric image learning process to enhance the overall performance. The core novelty is to automatically leverage the volume visualization technique (e.g., MIP - a volume rendering scheme for 3D volume images) to enhance the 3D data exploration at the deep learning level. The MIP embedding features can enhance the local vessel signal (through canceling out the noise) and adapt to the geometric variability and scalability of vessels, which is of great importance in microvascular tracking. A multi-stream convolutional neural network (CNN) framework is proposed to effectively learn the 3D volume and 2D MIP feature vectors, respectively, and then explore their inter-dependencies in a joint volume-composition embedding space by unprojecting the 2D feature vectors into the 3D volume embedding space. It is noted that the proposed framework can better capture the small/micro vessels and improve the vessel connectivity. To our knowledge, this is the first time that a deep learning framework is proposed to construct a joint convolutional embedding space, where the computed vessel probabilities from volume rendering based 2D projection and 3D volume can be explored and integrated synergistically. Experimental results are evaluated and compared with the traditional 3D vessel segmentation methods and the state-of-the-art in deep learning, by using extensive public and real patient (micro- )cerebrovascular image datasets. The application of this accurate segmentation and visualization of sparse and complicated 3D microvascular structure facilitated by our method demonstrates the potential in a powerful MR arteriogram and venogram diagnosis of vascular disease.

Blind Image Quality Assessment With Active Inference.

Blind image quality assessment (BIQA) is a useful but challenging task. It is a promising idea to design BIQA methods by mimicking the working mechanism of human visual system (HVS). The internal generative mechanism (IGM) indicates that the HVS actively infers the primary content (i.e., meaningful information) of an image for better understanding. Inspired by that, this paper presents a novel BIQA metric by mimicking the active inference process of IGM. Firstly, an active inference module based on the generative adversarial network (GAN) is established to predict the primary content, in which the semantic similarity and the structural dissimilarity (i.e., semantic consistency and structural completeness) are both considered during the optimization. Then, the image quality is measured on the basis of its primary content. Generally, the image quality is highly related to three aspects, i.e., the scene information (content-dependency), the distortion type (distortion-dependency), and the content degradation (degradation-dependency). According to the correlation between the distorted image and its primary content, the three aspects are analyzed and calculated respectively with a multi-stream convolutional neural network (CNN) based quality evaluator. As a result, with the help of the primary content obtained from the active inference and the comprehensive quality degradation measurement from the multi-stream CNN, our method achieves competitive performance on five popular IQA databases. Especially in cross-database evaluations, our method achieves significant improvements.

Multi-Stream Deep Convolutional Neural Network for PET Preform Surface Defects Detection and Classification

Multi-Stream Deep Convolutional Neural Network for PET Preform Surface Defects Detection and Classification