Multi-branch Convolutional Neural Network Research Articles

Local and Deep Features Based Convolutional Neural Network Frameworks for Brain MRI Anomaly Detection

A brain tumor is an abnormal mass or growth of a cell that leads to certain death, and this is still a challenging task in clinical practice. Early and correct diagnosis of this type of cancer is very important for the treatment process. For this reason, this study aimed to develop computer‐aided systems for the diagnosis of brain tumors. In this research, we proposed three different end‐to‐end deep learning approaches for analyzing effects of local and deep features for brain MRI images anomaly detection. The first proposed system is Directional Bit‐Planes Deep Autoencoder (DBP‐DAE) which extracts and learns local and direction features. The DBP‐DAE by decomposition of a local binary pattern (LBP) into eight bit‐planes extracts are directional and inherent local‐structure features from the input image and learns robust feature for classification purposes. The second one is a Dilated Separable Residual Convolutional Network (DSRCN) which extracts high (deep) and low‐level features. The main advantage of this approach is that it is robust and shows stable results regardless to size of image database and to solve overfitting problems. To explore the effects of mixture of local and deep extracted feature on accuracy of classification of brain anomaly, a multibranch convolutional neural network approach is proposed. This approach is designed according to combination of DBP‐DAE and DSRCN in an end‐to‐end manner. Extensive experiments conducted based on brain tumor in MRI image public access databases and achieves significant results compared to state‐of‐the‐art algorithms. In addition, we discussed the effectiveness and applicability of CNNs with a variety of different features and architectures for brain abnormalities such as Alzheimer’s.

An Improved Multibranch Convolutional Neural Network with a Compensator for Crowd Counting

Image‐based crowd counting has extremely important applications in public safety issues. Most of the previous studies focused on extremely dense crowds. However, as the number of webcams increases, a crowd with extremely high density can obtain less error by summing the images of multiple close‐range webcams, but there are still some problems such as heavy occlusions and large‐scale variation. To solve the above problems, this paper proposes a new type of multibranch neural network with a compensator, in which features are extracted through multibranch subnetworks of different scales. The weights between the branches are adjusted by the compensator, and the captured features are distinguished among different branches. To avoid learning nearly the same features in each branch and reducing the training deviation, the dataset is labeled with head scale, and the adaptive grading loss function is used to calculate the estimated loss of the subregions. The experimental results show that the accuracy of the network proposed in this paper is about 10% higher than that of the comparison network.

Person Identification With Millimeter-Wave Radar in Realistic Smart Home Scenarios

Compared with visual sensors that have light dependence and privacy intrusion issues, non-intrusion millimeter-wave (mmW) radars are more suitable for the daily person identification. In a realistic home scenario, there are new challenges that are not taken into account in the existing research. This letter attempts to address these issues such as multipath interference, complex walking process, and recognition robustness in smart home scenarios and designs a lightweight multi-branch convolutional neural network (CNN) with an Inception-Pool module and a Residual-Pool module to learn and classify gait Doppler features. The experimental results in a home living room scenario indicate that the designed mmW radar person identification system can achieve accurate and robust real-time identification performance.

Few-Shot Learning for Radar Signal Recognition Based on Tensor Imprint and Re-Parameterization Multi-Channel Multi-Branch Model

In an ever-increasingly complex electromagnetic environment, automatic radar signal recognition is becoming vital. Convolutional neural networks have been widely used for radar signal recognition, but deep learning-based algorithms only recognize trained classes. Recognizing novel radar signals with few-shot samples in an open environment is still a challenging research problem. In this letter, a few-shot learning algorithm based on the tensor imprint algorithm and convolutional classification layer is proposed for radar signal recognition, and the proposed convolutional classification layer can avoid spatial information loss caused by the global pooling layer and the fully connected layer. In addition, the lightweight re-parameterization multi-channel multi-branch convolutional neural network (RepMCMBNet) is proposed for feature extraction. The model is trained on a dataset containing 8 types of radar signals, and achieves high recognition accuracy in a test dataset containing 12 types of radar signals. The overall recognition accuracy of the proposed tensor imprint algorithm achieves 93.9% at −6 dB when the number of samples is 5.

A Multibranch Convolutional Neural Network for Hyperspectral Unmixing

Hyperspectral unmixing remains one of the most challenging tasks in the analysis of such data. Deep learning has been blooming in the field and proved to outperform other classic unmixing techniques, and can be effectively deployed onboard Earth observation satellites equipped with hyperspectral imagers. In this letter, we follow this research pathway and propose a multi-branch convolutional neural network that benefits from fusing spectral, spatial, and spectral-spatial features in the unmixing process. The results of our experiments, backed up with the ablation study, revealed that our techniques outperform others from the literature and lead to higher-quality fractional abundance estimation. Also, we investigated the influence of reducing the training sets on the capabilities of all algorithms and their robustness against noise, as capturing large and representative ground-truth sets is time-consuming and costly in practice, especially in emerging Earth observation scenarios.

A multi-branch convolutional neural network with density map for aphid counting

In agriculture, aphids always cause major damage in wheat, corn and rape, which significantly affect the crop yield. Manual aphid counting approaches are often labour-consuming and time-costing for Integrated Pest Management (IPM). In addition, the results of existing aphid counting methods based on computer vision are not satisfactory due to the complex background and the dense distribution. In order to address these problems, a novel multi-branch convolutional neural network (Mb-CNN) with density map for aphid counting is developed in this paper. In this approach, the aphid images are firstly fed into multi-branch convolutional neural networks, which have three branches for extracting the feature maps of different scales. Then, an aphid density map is generated via Mb-CNN, which contains the distribution information of aphids. Finally, the counting of aphids is estimated by using the density map. Experiment results on our dataset demonstrate that our Mb-CNN achieves the performance of 10.22 Mean Absolute Error (MAE) and 12.24 Mean Squared Error (MSE) in the aphid counting, which outweighs the state-of-the-art approaches. • Domain specific dataset for aphid counting in the field. • This dataset may have high application value in practical aphid monitoring. • Propose a multi-branch convolutional neural network with density map for aphid counting.

Anatomically aided PET image reconstruction using deep neural networks.

The developments of PET/CT and PET/MR scanners provide opportunities for improving PET image quality by using anatomical information. In this paper, we propose a novel co-learning three-dimensional (3D) convolutional neural network (CNN) to extract modality-specific features from PET/CT image pairs and integrate complementary features into an iterative reconstruction framework to improve PET image reconstruction. We used a pretrained deep neural network to represent PET images. The network was trained using low-count PET and CT image pairs as inputs and high-count PET images as labels. This network was then incorporated into a constrained maximum likelihood framework to regularize PET image reconstruction. Two different network structures were investigated for the integration of anatomical information from CT images. One was a multichannel CNN, which treated PET and CT volumes as separate channels of the input. The other one was multibranch CNN, which implemented separate encoders for PET and CT images to extract latent features and fed the combined latent features into a decoder. Using computer-based Monte Carlo simulations and two real patient datasets, the proposed method has been compared with existing methods, including the maximum likelihood expectation maximization (MLEM) reconstruction, a kernel-based reconstruction and a CNN-based deep penalty method with and without anatomical guidance. Reconstructed images showed that the proposed constrained ML reconstruction approach produced higher quality images than the competing methods. The tumors in the lung region have higher contrast in the proposed constrained ML reconstruction than in the CNN-based deep penalty reconstruction. The image quality was further improved by incorporating the anatomical information. Moreover, the liver standard deviation was lower in the proposed approach than all the competing methods at a matched lesion contrast. The supervised co-learning strategy can improve the performance of constrained maximum likelihood reconstruction. Compared with existing techniques, the proposed method produced a better lesion contrast versus background standard deviation trade-off curve, which can potentially improve lesion detection.

A multi-branch convolutional transfer learning diagnostic method for bearings under diverse working conditions and devices

Conventional intelligent bearings fault diagnosis methods generally extract fault features with a single channel, which seriously limit the features richness and the diagnostic accuracy. A novel end-to-end diagnosis framework which incorporate multi-scale average processing into the multi-branch convolutional neural network (MBCNN) is established here, thus rich and complementary fault features can be captured from the multi-scale reconstructed signals via multiple parallel residual CNN branches. Besides, with the help of a fine-tuning based transfer learning strategy, the well-trained model on source domain is competent to the related diagnostic tasks of bearings under diverse working conditions and devices. Comparative experiments under 3 cases are conducted, and the results show our approach can be provided with more excellent performances and superior transferability compared with the state-of-the-art ones. For instance, the transfer diagnostic accuracies of MBCNN exceed 98.56% and 98.38% under diverse working conditions and cross-devices, respectively.

Attention Guided Low-Light Image Enhancement with a Large Scale Low-Light Simulation Dataset

Low-light image enhancement is challenging in that it needs to consider not only brightness recovery but also complex issues like color distortion and noise, which usually hide in the dark. Simply adjusting the brightness of a low-light image will inevitably amplify those artifacts. To address this difficult problem, this paper proposes a novel end-to-end attention-guided method based on multi-branch convolutional neural network. To this end, we first construct a synthetic dataset with carefully designed low-light simulation strategies. The dataset is much larger and more diverse than existing ones. With the new dataset for training, our method learns two attention maps to guide the brightness enhancement and denoising tasks respectively. The first attention map distinguishes underexposed regions from well lit regions, and the second attention map distinguishes noises from real textures. With their guidance, the proposed multi-branch decomposition-and-fusion enhancement network works in an input adaptive way. Moreover, a reinforcement-net further enhances color and contrast of the output image. Extensive experiments on multiple datasets demonstrate that our method can produce high fidelity enhancement results for low-light images and outperforms the current state-of-the-art methods by a large margin both quantitatively and visually.

Automated detection of sigmatism using deep learning applied to multichannel speech signal

This paper presents a system for the analysis of acoustic data for the computer-aided diagnosis and therapy of sigmatism in children. The analysis is focused on the detection and recognition of selected articulation disorders in sibilant sounds. The system relies on the dedicated data acquisition device recording the speech signal using 15 microphones spatially arranged around the speaker's mouth. The collected speech corpus contains 923 samples of the /s/ and /ʃ/ consonants from 98 five- and six-year-old children with either normative or pathological pronunciation features. Each recording is supplemented with a detailed speech therapy annotation. A dedicated multibranch convolutional neural network architecture was designed for the speech sample classification. The filter bank energy feature maps are extracted from each channel along with their two derivatives in the time domain. The feature maps are aggregated along different dimensions to constitute a four-dimensional data structure called acoustic volume, being the input data for the deep network. We proposed three ways to aggregate the multichannel data into the acoustic volume and two techniques for the data augmentation to enlarge the available dataset and avoid overfitting. Classification experiments involving different data subsets have proven the system's ability to detect the analyzed pronunciation disorders with reasonable accuracy. The framework with speech data organized spatially in five channels provides the most efficient classification.

A multi-branch deep convolutional fusion architecture for 3D microwave inverse scattering: stored grain application

In this paper, a multi-branch deep convolutional fusion architecture is proposed to solve electromagnetic inverse scattering problems. The conventional methods for solving inverse problems face various challenges, including strong ill-conditioning, expensive computational cost, and unavoidable intrinsic nonlinearity. To overcome these difficulties, we designed a novel multi-branch convolutional neural network (CNN) to reconstruct the 3D images of the moisture distribution in stored grain. Inspired by objective-function techniques for solving the electromagnetic inverse scattering problems, the proposed CNN architecture takes in the scattered-field data and prior information to produce 3D images of the moisture content. With the aim of using inputs of different formats, i.e., a complex-valued vector of scattered-field data and a 3D image of the background moisture distribution as prior information, we propose a multi-branch architecture consisting of decoder-only, and encoder–decoder, convolutional branches. The two branches are later fused to produce the final reconstructed 3D image. To train the CNN, we use the true numerical grain moisture distribution image, which were synthetically generated. The reconstructed moisture distribution images produced by the proposed CNN show that the network is not only able to reconstruct the 3D moisture distribution images directly from measured scattered-field data for high contrast objects-of-interest, but it also achieves a higher imaging quality compared with traditional inversion techniques in microwave imaging. Quantitative evaluations are reported using receiver operating characteristics curves for the hotspot detectability and RMS error. The proposed approach opens a novel path for the deep learning-based real-time quantitative microwave imaging.

Multi-Scale and Multi-Branch Convolutional Neural Network for Retinal Image Segmentation

The accurate segmentation of retinal images is a basic step in screening for retinopathy and glaucoma. Most existing retinal image segmentation methods have insufficient feature information extraction. They are susceptible to the impact of the lesion area and poor image quality, resulting in the poor recovery of contextual information. This also causes the segmentation results of the model to be noisy and low in accuracy. Therefore, this paper proposes a multi-scale and multi-branch convolutional neural network model (multi-scale and multi-branch network (MSMB-Net)) for retinal image segmentation. The model uses atrous convolution with different expansion rates and skip connections to reduce the loss of feature information. Receiving domains of different sizes captures global context information. The model fully integrates shallow and deep semantic information and retains rich spatial information. The network embeds an improved attention mechanism to obtain more detailed information, which can improve the accuracy of segmentation. Finally, the method of this paper was validated on the fundus vascular datasets, DRIVE, STARE and CHASE datasets, with accuracies/F1 of 0.9708/0.8320, 0.9753/0.8469 and 0.9767/0.8190, respectively. The effectiveness of the method in this paper was further validated on the optic disc visual cup DRISHTI-GS1 dataset with an accuracy/F1 of 0.9985/0.9770. Experimental results show that, compared with existing retinal image segmentation methods, our proposed method has good segmentation performance in all four benchmark tests.

Auto-ReID+: Searching for a multi-branch ConvNet for person re-identification

In the field of person re-identification (ReID), multi-branch models are more effective in learning robust features than single-branch models. The current popular multi-branch models are based on ResNet or GoogleNet. These networks are designed initially to solve classification problems. There is an essential difference between ReID and classification problems, so it is particularly important to find a corresponding multi-branch backbone for ReID tasks. We propose to automatically search for a multi-branch convolutional neural network (CNN) for ReID tasks utilizing neural architecture search (NAS). First, we designed a multi-resolution, multi-branch macro search architecture that can extract more abundant scale information. Then in the searching process, the early stopping mechanism is proposed to improve the effectiveness and efficiency of the entire searching process. Finally, we experimentally prove on four mainstream datasets that the searched model can achieve state-of-the-art performance with only 5.7 million parameters.

Multisensor Land Cover Classification With Sparsely Annotated Data Based on Convolutional Neural Networks and Self-Distillation

Extensive research studies have been conductedin recent years to exploit the complementarity among multi-sensor (or multi-modal) remote sensing data for prominentapplications such as land cover mapping. In order to make a step further with respect to previous studies which investigate multi-temporal SAR and optical data or multi-temporal/multi-scale optical combinations, here we propose a deep learning framework that simultaneously combine all these input sources, specifically multi-temporal SAR/optical data and fine scale optical information. Our proposal relies on a patch-based multi-branch convolutional neural network (CNN) that exploits different per source CNN encoders to deal with the specificity of the input signals. Additionally, our framework is equipped with a self-distillation strategy to boost the per source analyses. This new strategy leverages the final prediction of the multi-source framework to guide the learning of the per source CNN encoders supporting the network to learn from itself.Experiments are carried out on two real world benchmarks, namely the Reunion island (a french overseas department) and the Dordogne study site (a southwest department in France)where the annotated reference data was collected under operational constraints (sparsely annotated ground truth data). Obtained results, providing an overall classification accuracyof about 94% (resp. 88%) on theReunion island(resp. theDordogne) study site. These findings highlight the effectivenessof our framework based on Convolutional Neural Networks andself-distillation to combine heterogeneous multi-sensor remotesensing data and confirm the benefit of multi-modal analysisfor downstream tasks such as land cover mapping.

ViT-P: Classification of Genitourinary Syndrome of Menopause From OCT Images Based on Vision Transformer Models

Genitourinary syndrome of menopause (GSM) is a disease caused by a physiological decline in estrogen levels, and it can negatively affect a woman’s overall health and quality of life in terms of sexual function. Real-time optical biopsy images can now be obtained with optical coherence tomography (OCT) systems. In this study, we introduce vision transformer (ViT) to the field of medical OCT images for the first time and propose a deep learning-based approach for GSM lesion screening. Specifically, we first build a GSM dataset to train and evaluate the experimental model performance. The study aims to propose a method that combines null convolution with a deep convolutional adversarial generative network classifier to generate the samples needed for training to alleviate the hindrance of such problems, in response to certain practical problems, such as category imbalance that occur during data collection. Next, the experiments present ViT PLUS (ViT-P) for the vaginal OCT image classification task used, which effectively improves the shortcomings of ViT in extracting Patch Embedding using a multibranch convolutional neural network combined with a channel attention mechanism. The clinical images acquired by the OCT device are then automatically classified on the basis of the OCT device to reduce the medical workload of gynecologists. Experimental results show that the ViT-P model outperforms the CNN model and ViT for case screening in the GSM and UCSD datasets, and the accuracy can reach 99.9% and 99.69%, respectively.

An FSK-MBCNN based method for compound fault diagnosis in wind turbine gearboxes

Accurate identification of compound fault in wind turbine gearbox is very important for condition monitoring of wind turbine systems. This paper proposes a novel compound fault diagnosis method named FSK-MBCNN for wind turbine gearboxes. The proposed method combines the fast spectral kurtosis (FSK) with a multi-branch convolutional neural network (MBCNN). First, FSK transforms the one dimensional vibration signal into the fast kurtogram of a two dimensional image as the input feature map of the convolutional neural network model. Second, a multi-branch module is proposed to provide suitable network architecture adapting to different learning rate and maximum training epoch. The experiment results reveal that the proposed FSK-MBCNN method can diagnose compound fault of wind turbine gearbox with more than 97% accuracy. The fault identification accuracies of noised signal and variable load signal are close to 90%, proving the robustness of the proposed method. Compared with other three methods, the proposed method can recognize 9 different statuses of the wind turbine gearboxes more accurately in the aforementioned cases. This fascinating discovery indicates that the proposed FSK-MBCNN method can be applied to actual condition monitoring of wind turbines to reduce the maintenance cost and the downtime.

Multi-branch convolutional neural network with generalized shaft orbit for fault diagnosis of active magnetic bearing-rotor system

Fault diagnosis based on vibration signals in active magnetic bearing-rotor systems is an important research topic. However, it is difficult to obtain discriminative features to represent faults due to the nonlinear and non-stationary characteristics of the vibration signals and diverse sources of failures. Hence, this paper proposes a novel end-to-end learning mechanism of multi-sensor data fusion to learn fault representation based on the structural characteristics of active magnetic bearings. Taking the five displacement sensors of active magnetic bearing as signal sources, generalized shaft orbits are constructed and converted into discrete 2D images. Based these 2D images, a multi-branch convolutional neural network is designed to achieve high discriminative features and fault types. The experiments are performed on the rig supported by active magnetic bearings, and the effectiveness of the proposed algorithm is verified, proving it suitability in cases with changing rotating speeds and sample lengths.

Copula-Based Data Augmentation on a Deep Learning Architecture for Cardiac Sensor Fusion.

In the wake of Big Data, traditional Machine Learning techniques are now often integrated in the clinical workflow. Despite more capable, Deep Learning methods are not equally accepted given their unsatiated need for great amounts of training data and transversal use of the same architectures in fundamentally different areas with weakly-substantiated adaptations. To address the former, a cardiorespiratory signal synthesizer was designed by conditional sampling from a multimodally trained stochastic system of Gaussian copulas integrated in a Markov chain. With respect to the latter, a multi-branch convolutional neural network architecture was conceived to learn the best cardiac sensor-fusion strategy at every abstraction layer. The network was tailored to the tasks of cycle detection and classification for different cardiac modality combinations by a synthesizer-based data augmentation training framework and Bayesian hyperparameter optimization. The synthesizer yielded highly realistic signals in the time, frequency and phase domains for both healthy and pathological heart cycles as well as artifacts of different modalities. Benchmarking suggested that the network is able to surpass previous architectures and data augmentation provided a performance boost in realistic data availability scenarios. These included insufficient training data volume, as low as 150 cycles long, artifact contamination and absence of a classification data type in training.

Multi-Branch Convolutional Neural Network for Automatic Sleep Stage Classification with Embedded Stage Refinement and Residual Attention Channel Fusion.

Automatic sleep stage classification of multi-channel sleep signals can help clinicians efficiently evaluate an individual’s sleep quality and assist in diagnosing a possible sleep disorder. To obtain accurate sleep classification results, the processing flow of results from signal preprocessing and machine-learning-based classification is typically employed. These classification results are refined based on sleep transition rules. Neural networks—i.e., machine learning algorithms—are powerful at solving classification problems. Some methods apply them to the first two processes above; however, the refinement process continues to be based on traditional methods. In this study, the sleep stage refinement process was incorporated into the neural network model to form real end-to-end processing. In addition, for multi-channel signals, the multi-branch convolutional neural network was combined with a proposed residual attention method. This approach further improved the model classification accuracy. The proposed method was evaluated on the Sleep-EDF Expanded Database (Sleep-EDFx) and University College Dublin Sleep Apnea Database (UCDDB). It achieved respective accuracy rates of 85.7% and 79.4%. The results also showed that sleep stage refinement based on a neural network is more effective than the traditional refinement method. Moreover, the proposed residual attention method was determined to have a more robust channel–information fusion ability than the respective average and concatenation methods.

Scale‐aware limited deformable convolutional neural networks for traffic sign detection and classification

Traffic sign detection and classification is a critical component of intelligent transportation systems, which is applied to inform automatic unmanned driving systems and driving assistance systems about conditions and limits of roads. Although computer vision is widely utilised in traffic sign detection, detection and recognising traffic signs globally remains a great challenge due to the variety of sign types, scale-variance and geometric variations. To address these problems, this study proposes a region-based deep convolutional neural network (CNN) framework for traffic sign detection and classification. Specifically, a multi-branch sample pyramid module is proposed, which is based on multi-branch CNNs for multi-scaled feature exaction. A limited deformable convolutional module is then embedded into the CNN layers to learn the distorted information representation for deformation handing. Moreover, a scale-aware multi-task region proposal network module is applied to detect traffic signs with various scales. The whole network is trained in an end-to-end manner. Finally, experiments are conducted on two public detection data sets to demonstrate the effectiveness of the proposed method.