Multi-branch Convolution Research Articles

A Remote Sensing Image Super-Resolution Reconstruction Model Combining Multiple Attention Mechanisms.

Remote sensing images are characterized by high complexity, significant scale variations, and abundant details, which present challenges for existing deep learning-based super-resolution reconstruction methods. These algorithms often exhibit limited convolutional receptive fields and thus struggle to establish global contextual information, which can lead to an inadequate utilization of both global and local details and limited generalization capabilities. To address these issues, this study introduces a novel multi-branch residual hybrid attention block (MBRHAB). This innovative approach is part of a proposed super-resolution reconstruction model for remote sensing data, which incorporates various attention mechanisms to enhance performance. First, the model employs window-based multi-head self-attention to model long-range dependencies in images. A multi-branch convolution module (MBCM) is then constructed to enhance the convolutional receptive field for improved representation of global information. Convolutional attention is subsequently combined across channels and spatial dimensions to strengthen associations between different features and areas containing crucial details, thereby augmenting local semantic information. Finally, the model adopts a parallel design to enhance computational efficiency. Generalization performance was assessed using a cross-dataset approach involving two training datasets (NWPU-RESISC45 and PatternNet) and a third test dataset (UCMerced-LandUse). Experimental results confirmed that the proposed method surpassed the existing super-resolution algorithms, including Bicubic interpolation, SRCNN, ESRGAN, Real-ESRGAN, IRN, and DSSR in the metrics of PSNR and SSIM across various magnifications scales.

Multi-component quantitative analysis of LIBS using adaptively optimized multi-branch CNN

Multi-component quantitative analysis of laser-induced breakdown spectroscopy (LIBS) is very important for industrial process control, food safety and space exploration. Due to the strong background interference and complex matrix effect, traditional quantitative analysis methods are difficult to realize accurate multi-component regression. Deep learning has the advantages of automatically extracting full-spectrum features and strong nonlinear modeling capabilities, which provides new opportunities for advancing the progress of LIBS quantitative analysis. However, due to the large difference of spectral line intensity and spectral cross-interference, existing deep learning models have limited performance for simultaneous multi-component regression. To this end, a novel adaptively optimized multi-branch convolution neural network (AOMB-CNN) framework is proposed for multi-component quantitative analysis of LIBS. Multi-branch structure is adopted to ensure the accuracy of multi-component content simultaneously. Dynamic weighted loss function and Bayesian optimization algorithm are used to adaptively optimize the deep learning model. The results demonstrate that the AOMB-CNN can effectively improve the performance of multi-component regression on 2 LIBS datasets compared with the traditional quantitative analysis methods. The root mean square error (RMSE) of 8 components of the ChemCam test set are 2.4465, 0.3486, 1.2481, 1.3909, 0.9010, 0.8892, 0.6700, and 0.5498 respectively, and the RMSE of 4 components of the steel test sets are 1.9787, 0.3455, 0.6025 and 1.6631 respectively. The proposed method provides an end-to-end, adaptively optimized, full-spectrum deep learning solution for LIBS multi-component content measurement without human intervention.

YOLOLayout: Multi-Scale Cross Fusion Former for Document Layout Analysis

Document layout analysis (DLA) is a technique used to locate and classify layout elements in a document, such as Table, Figure, List, and Text. While deep-learning-based methods in computer vision have shown excellent performance in detecting Text and Figures, they are still unsatisfactory in accurately recognizing the blocks of List, Title, and Table categories with limited data. To address this issue, we propose a single-stage DLA model that incorporates a Multi-Scale Shallow Visual Feature Enhancement Module (MS-SVFEM) and a Multi-Scale Cross-Feature Fusion Module (MS-CFF). The MS-SVFEM extracts multi-scale spatial information through the channel attention module, spatial attention module, and multi-branch convolution. The MS-CFF fuses different level features through an attention mechanism. The experiments showed that the mAP accuracy of YOLOLayout compared to the baseline model is 2.2% and 1.5% higher on the PubLayNet Dataset and the ISCAS-CLAD dataset.

Enhanced multi-label cardiology diagnosis with channel-wise recurrent fusion

The timely detection of abnormal electrocardiogram (ECG) signals is vital for preventing heart disease. However, traditional automated cardiology diagnostic methods have the limitation of being unable to simultaneously identify multiple diseases in a segment of ECG signals, and do not consider the potential correlations between the 12-lead ECG signals. To address these issues, this paper presents a novel network architecture, denoted as Branched Convolution and Channel Fusion Network (BCCF-Net), designed for the multi-label diagnosis of ECG cardiology to achieve simultaneous identification of multiple diseases. Among them, the BCCF-Net incorporates the Channel-wise Recurrent Fusion (CRF) network, which is designed to enhance the ability to explore potential correlation information between 12 leads. Furthermore, the utilization of the squeeze and excitation (SE) attention mechanism maximizes the potential of the convolutional neural network (CNN). In order to efficiently capture complex patterns in space and time across various scales, the multi branch convolution (MBC) module has been developed. Through extensive experiments on two public datasets with seven subtasks, the efficacy and robustness of the proposed ECG multi-label classification framework have been comprehensively evaluated. The results demonstrate the superior performance of the BCCF-Net compared to other state-of-the-art algorithms. The developed framework holds practical application in clinical settings, allowing for the refined diagnosis of cardiac arrhythmias through ECG signal analysis.

Multi-branch reverse attention semantic segmentation network for building extraction

Extraction of color and texture features of buildings from high-resolution remote sensing images often encounters the problems of interference of background information and varying target scales. In addition, most of the current attention mechanisms focus on building key feature selection for building extraction optimization, but ignore the influence of the complex background. Hence, we propose incorporating a novel reverse attention module into the network. The innovative module enables the model to selectively extract crucial building features while suppressing the impact of intricate background noise. It mitigates the influence of uniform spectral and structurally similar heterogeneous background targets on building segmentation and extraction. As a result, the overall generalizability of the model is improved. The reverse attention can also emphasize and amplify the specific details pertaining to the boundaries of the target. Furthermore, we couple a new multi-branch convolution block into the encoder, integrating dilated convolutions with multiple dilation rates. Compared to other methods that use only one multi-scale module to extract multi-scale information from high-level features, we use different receptive field convolutions to simultaneously capture multi-scale targets from multi-level features, effectively improving the ability of the model to extract multi-scale building features. The experimental findings demonstrate that our proposed multi-branch reverse attention semantic segmentation network achieves IoU of 90.59% and 81.79% on the well-known WHU building and Inria aerial image datasets, respectively.

Intelligent flow field reconstruction based on proper orthogonal decomposition dimensionality reduction and improved multi-branch convolution fusion

The rapid and accurate reconstruction of the supersonic combustor flow field is of great significance for sensing and predicting the combustion state. Existing deep learning methods pay less attention to the convergence speed of flow field reconstruction, which results in longer training and prediction times for the models. This study proposes a method for reconstructing the flow field in supersonic combustor by combining a reduced-order model based on proper orthogonal decomposition (POD) with a multi-branch convolutional neural network. This method first analyzes the effectiveness of POD reconstruction. Then, based on the wall pressure data of the supersonic engine combustor, it performs flow field image reconstruction. Finally, through error calculation and gradient updating with low-resolution principal component flow field shadow images obtained from the POD algorithm, the high-precision and efficient prediction of flow field images is achieved. Different equivalence ratio hydrogen fuel combustion experiments were conducted in a pulsed combustion wind tunnel with an incoming flow Mach number of 2.5. The learning model was trained and tested using the dataset obtained from these experiments. Numerous experiments demonstrated that the model can effectively reconstruct the wave structures of complex flow fields. Multiple evaluation indicators indicated that the reconstructed flow field of the combustor shows good agreement with that obtained from ground wind tunnel testing. Furthermore, after introducing the POD dimensionality reduction model, the training time was reduced by 32.03%, effectively improving the training time complexity of the model.

Classification algorithm for EEG-based motor imagery using hybrid neural network with spatio-temporal convolution and multi-head attention mechanism.

Motor imagery (MI) is a brain-computer interface (BCI) technique in which specific brain regions are activated when people imagine their limbs (or muscles) moving, even without actual movement. The technology converts electroencephalogram (EEG) signals generated by the brain into computer-readable commands by measuring neural activity. Classification of motor imagery is one of the tasks in BCI. Researchers have done a lot of work on motor imagery classification, and the existing literature has relatively mature decoding methods for two-class motor tasks. However, as the categories of EEG-based motor imagery tasks increase, further exploration is needed for decoding research on four-class motor imagery tasks. In this study, we designed a hybrid neural network that combines spatiotemporal convolution and attention mechanisms. Specifically, the data is first processed by spatiotemporal convolution to extract features and then processed by a Multi-branch Convolution block. Finally, the processed data is input into the encoder layer of the Transformer for a self-attention calculation to obtain the classification results. Our approach was tested on the well-known MI datasets BCI Competition IV 2a and 2b, and the results show that the 2a dataset has a global average classification accuracy of 83.3% and a kappa value of 0.78. Experimental results show that the proposed method outperforms most of the existing methods.

A deep learning method for predicting the minimum inhibitory concentration of antimicrobial peptides against Escherichia coli using Multi-Branch-CNN and Attention.

Antimicrobial peptides (AMPs) are a promising alternative to antibiotics to combat drug resistance in pathogenic bacteria. However, the development of AMPs with high potency and specificity remains a challenge, and new tools to evaluate antimicrobial activity are needed to accelerate the discovery process. Therefore, we proposed MBC-Attention, a combination of a multi-branch convolution neural network architecture and attention mechanisms to predict the experimental minimum inhibitory concentration of peptides against Escherichia coli. The optimal MBC-Attention model achieved an average Pearson correlation coefficient (PCC) of 0.775 and a root mean squared error (RMSE) of 0.533 (log μM) in three independent tests of randomly drawn sequences from the data set. This results in a 5-12% improvement in PCC and a 6-13% improvement in RMSE compared to 17 traditional machine learning models and 2 optimally tuned models using random forest and support vector machine. Ablation studies confirmed that the two proposed attention mechanisms, global attention and local attention, contributed largely to performance improvement. IMPORTANCE Antimicrobial peptides (AMPs) are potential candidates for replacing conventional antibiotics to combat drug resistance in pathogenic bacteria. Therefore, it is necessary to evaluate the antimicrobial activity of AMPs quantitatively. However, wet-lab experiments are labor-intensive and time-consuming. To accelerate the evaluation process, we develop a deep learning method called MBC-Attention to regress the experimental minimum inhibitory concentration of AMPs against Escherichia coli. The proposed model outperforms traditional machine learning methods. Data, scripts to reproduce experiments, and the final production models are available on GitHub.

Coal-gangue recognition via multi-branch convolutional neural network based on MFCC in noisy environment

Traditional coal-gangue recognition methods usually do not consider the impact of equipment noise, which severely limits its adaptability and recognition accuracy. This paper mainly studies the more accurate recognition of coal-gangue in the noise site environment with the operation of shearer, conveyor, transfer machine and other device in the process of top coal caving. Mel Frequency Cepstrum Coefficients (MFCC) smoothing method was introduced to express the intrinsic feature of sound pressure more clearly in the coal-gangue recognition site. Then, a multi-branch convolution neural network (MBCNN) model with three branches was developed, and the smoothed MFCC feature was incorporated into this model to realize the recognition of falling coal and gangue in noisy environment. The sound pressure signal datasets under the operation of different device were constructed through a great deal of laboratory and site data acquisition. Comparative experiments were carried out on noiseless dataset, single noise dataset and simulated site dataset, and the results show that our method can provide higher correct recognition accuracy and better robustness. The proposed coal-gangue recognition approach based on MBCNN and MFCC smoothing can not only recognize the state of falling coal or gangue, but also recognize the operational state of site device.

A Pig Mass Estimation Model Based on Deep Learning without Constraint.

The body mass of pigs is an essential indicator of their growth and health. Lately, contactless pig body mass estimation methods based on computer vision technology have gained attention thanks to their potential to improve animal welfare and ensure breeders' safety. Nonetheless, current methods require pigs to be restrained in a confinement pen, and no study has been conducted in an unconstrained environment. In this study, we develop a pig mass estimation model based on deep learning, capable of estimating body mass without constraints. Our model comprises a Mask R-CNN-based pig instance segmentation algorithm, a Keypoint R-CNN-based pig keypoint detection algorithm and an improved ResNet-based pig mass estimation algorithm that includes multi-branch convolution, depthwise convolution, and an inverted bottleneck to improve accuracy. We constructed a dataset for this study using images and body mass data from 117 pigs. Our model achieved an RMSE of 3.52 kg on the test set, which is lower than that of the pig body mass estimation algorithm with ResNet and ConvNeXt as the backbone network, and the average estimation speed was 0.339 s·frame-1 Our model can evaluate the body quality of pigs in real-time to provide data support for grading and adjusting breeding plans, and has broad application prospects.

A Multi-Feature Fusion and Attention Network for Multi-Scale Object Detection in Remote Sensing Images

Accurate multi-scale object detection in remote sensing images poses a challenge due to the complexity of transferring deep features to shallow features among multi-scale objects. Therefore, this study developed a multi-feature fusion and attention network (MFANet) based on YOLOX. By reparameterizing the backbone, fusing multi-branch convolution and attention mechanisms, and optimizing the loss function, the MFANet strengthened the feature extraction of objects at different sizes and increased the detection accuracy. The ablation experiment was carried out on the NWPU VHR-10 dataset. Our results showed that the overall performance of the improved network was around 2.94% higher than the average performance of every single module. Based on the comparison experiments, the improved MFANet demonstrated a high mean average precision of 98.78% for 9 classes of objects in the NWPU VHR-10 10-class detection dataset and 94.91% for 11 classes in the DIOR 20-class detection dataset. Overall, MFANet achieved an mAP of 96.63% and 87.88% acting on the NWPU VHR-10 and DIOR datasets, respectively. This method can promote the development of multi-scale object detection in remote sensing images and has the potential to serve and expand intelligent system research in related fields such as object tracking, semantic segmentation, and scene understanding.

An Efficient Algorithm-Hardware Co-Design for Radar-Based Fall Detection With Multi-Branch Convolutions

In this paper, we propose an efficient algorithm-hardware co-design framework to realize radar-based fall detection with limited resources. We first design a compact neural network model named MB-Net with multi-branch convolutions for feature extraction of radar time series data combined with multi-scale wavelet transform. After that, an FPGA-based neural network (NN) accelerator tailored for the proposed network is designed. The proposed NN accelerator replaces the general multipliers with non-exact multipliers to reduce the hardware cost. For the multi-branch convolution layer, a novel layer computing sequence is introduced to improve the efficiency of the processing element (PE) array and reduce the memory footprint. In addition, the average pooling operation in the proposed network is folded into the quantization factors to reduce hardware cost. The experimental findings show that the MB-Net can maintain competitive performance in comparison to state-of-the-art methods while the hardware cost is significantly lower. The proposed network model is implemented in Zynq ZC702 board using only 3615 LUTs, 1843 FFs, 11.5 BRAMs, and 8 DSPs with 0.234 W power consumption. Through algorithm and hardware co-optimization, the fall detection accelerator can achieve 95% PE efficiency and takes 0.346 ms latency for a radar sample interference with only 80.96 uJ energy consumption.

AGGN: Attention-based glioma grading network with multi-scale feature extraction and multi-modal information fusion

In this paper, a magnetic resonance imaging (MRI) oriented novel attention-based glioma grading network (AGGN) is proposed. By applying the dual-domain attention mechanism, both channel and spatial information can be considered to assign weights, which benefits highlighting the key modalities and locations in the feature maps. Multi-branch convolution and pooling operations are applied in a multi-scale feature extraction module to separately obtain shallow and deep features on each modality, and a multi-modal information fusion module is adopted to sufficiently merge low-level detailed and high-level semantic features, which promotes the synergistic interaction among different modality information. The proposed AGGN is comprehensively evaluated through extensive experiments, and the results have demonstrated the effectiveness and superiority of the proposed AGGN in comparison to other advanced models, which also presents high generalization ability and strong robustness. In addition, even without the manually labeled tumor masks, AGGN can present considerable performance as other state-of-the-art algorithms, which alleviates the excessive reliance on supervised information in the end-to-end learning paradigm.

Edge-enhanced efficient network for remote sensing image super-resolution

ABSTRACT The super-resolution (SR) reconstruction is drawing increasing attention in remote-sensing image processing, owing to improving the spatial resolution and enriching the details of initially obtaining low-resolution (LR) images. Different from natural images, remote sensing images usually have more complex image distribution and degradation processes. Although current deep convolutional neural network (DCNN)-based approaches reach better performance by deepening the network and introducing attention mechanism, this is usually accompanied by more parameters, more computation, and more complex designs. To improve the performance and efficiency of SR reconstruction of remote sensing images, we proposed the edge-enhanced efficient network (EESR). Specifically, we design an inception-like multi-branch convolution block, named edge-enhanced convolution block (EEB), to enrich edge-aware capabilities of the network, which includes multi-order gradient extraction and other feature enhancement branches. Furthermore, re-parameterization is introduced into the inference stage of the network, aiming at boosting efficient inference. We re-parameterize the weights of EEB to the edge-enhanced convolution (EEC) via equivalent transformation to reduce the parameters and computational cost of the inference model. In addition, we construct a paired SR dataset named GF2-Port for remote sensing images with degradation simulation, based on the data of GaoFen-2 port and strait. Extensive experiments on both established and public datasets indicate that the proposed EESR outperforms other comparable approaches in terms of balancing restoration accuracy and inference efficiency.

A multi-feature fusion-based domain adversarial neural network for fault diagnosis of rotating machinery

Deep learning (DL)-based Fault Diagnosis (FD) methods have been wildly used in the industry domain for the guarantee of rotating machinery. Training these models often deserver abundant labeled data from complex or variable working conditions. However, it is knotty to obtain massive data of different types of faults for the working condition of interest in engineering practice which also greatly hinders the improvement of DL-based FD methods. In addition, exiting DL-based method could not achieve satisfactory diagnosis results when the working condition between source-domain (training data) and target-domain (testing data) is different. This paper proposes a novel FD method using multi-feature fusion scheme and an Improved Domain Adversarial Neural Network (IDANN). Firstly, the Fast Fourier Transform (FFT) is utilized for time-to-frequency domain conversion of raw signals. Then, the multi-feature fusion scheme is adopted to fuse the spectral samples with different working conditions, which uses multi-branch convolution layers as feature extractor and fuser. After that, the fused features are fed into IDANN as input, and the adversarial training strategy is used to train the IDANN model until an ideal equilibrium state is achieved. Finally, the feature extractor and label predictor are separated from the trained IDANN model for classification of health conditions. To verify the performance of IDANN, two public bearing datasets from Case Western Reserve University (CWRU) and Paderborn University are utilized, and results show that IDANN achieves superior diagnosis performance by making full use of multi-source of signal data compared with other conventional or DL-based diagnosis methods.

A Lightweight Semantic Segmentation Model of Wucai Seedlings Based on Attention Mechanism

Accurate wucai seedling segmentation is of great significance for growth detection, seedling location, and phenotype detection. To segment wucai seedlings accurately in a natural environment, this paper presents a lightweight segmentation model of wucai seedlings, where U-Net is used as the backbone network. Specifically, to improve the feature extraction ability of the model for wucai seedlings of different sizes, a multi-branch convolution block based on inception structure is proposed and used to design the encoder. In addition, the expectation “maximizationexpectation” maximization attention module is added to enhance the attention of the model to the segmentation object. In addition, because of the problem that a large number of parameters easily increase the difficulty of network training and computational cost, the depth-wise separable convolution is applied to replace the original convolution in the decoding stage to lighten the model. The experimental results show that the precision, recall, MIOU, and F1-score of the proposed model on the self-built wucai seedling dataset are 0.992, 0.973, 0.961, and 0.982, respectively, and the average recognition time of single frame image is 0.0066 s. Compared with several state-of-the-art models, the proposed model achieves better segmentation performance and has the characteristics of smaller-parameter scale and higher real-time performance. Therefore, the proposed model can achieve good segmentation effect for wucai seedlings in natural environment, which can provide important basis for target spraying, growth recognition, and other applications.

Scalable compression for machine and human vision tasks via multi-branch shared module

The existing image compression methods are mainly aimed at human perception tasks instead of machine vision tasks. Rich features learned by the shallow and deep layers of pre-trained visual geometry group (VGG)-net can serve human perception and machine vision tasks, respectively. To improve the machine analysis capabilities of the human-targeted compression method, we propose a scalable image compression framework at low bit-rates. Specifically, the scalable compression framework is composed of a base layer (BL) and an enhancement layer (EL), which utilize the correlation between the above features to perform machine analysis and human perception tasks, respectively. For effectively utilizing the above two types of features, a multi-branch shared module that utilizes the complementarity of multi-branch convolution kernels to retain compact information for the above features to support BL and EL tasks is proposed. In addition, for further improving the accuracy of machine analysis, the machine-vision importance map is introduced in BL; it adaptively utilizes the spatial and channel information from the deepest layer of VGG-net to guide local bit allocation. When the bit-rate is limited to 0.2 bpp, the average recognition accuracy (Top-1) of the BL of proposed method is 5.2%, 13.4%, 6.0%, and 7.4% higher than that of BPG, Webp, Mentzer, and NIC, respectively, on the ILSVRC2012 verification dataset. Meanwhile, the EL provides good visual experience.

Point cloud up-sampling network with multi-level spatial local feature aggregation

Point clouds is one of popular 3D representations in computer vision and computer graphics. However, due to the sparseness and non-uniformity, raw point cloud from scanning devices cannot applied to down-stream geometry analyzing tasks directly. In this paper, we propose an end-to-end point cloud up-sampling network to reconstruct the dense yet uniform-distributed point clouds. Firstly, we utilize the spatial relationship of local regions and capture point-wise features progressively. We then propose a novel network to aggregate those features from different levels. Finally, we design an up-sampling module which consists of multi-branch convolution units to generate the dense point clouds. We conduct sufficient experiments on currently available public benchmarks. Experimental results show that proposed method has achieved 0.103 and 0.010 performance on Hausdorff distance and Chamfer Distance on VisionAir dataset, in comparison with the baseline towards uniformity, proximity-to-surface and mesh reconstruction.

Multi-Vision Network for Accurate and Real-Time Small Object Detection in Optical Remote Sensing Images

Accurate and real-time detection of airplanes, cars, and ships in remote sensing images is an important but challenging task that plays an important role in both military and civilian life. The most challenging issues posed by this task are the intensive and tiny size of objects and the complexity of application scenarios. In this letter, we propose a multi-vision small object detector that can rapidly and accurately detect airplanes, cars, and ships in remote sensing images. We make the following three contributions: a multiscale residual block (MRB) is proposed, whereby dilated convolution is employed in a cascade residual block to capture multiscale context information, thus improving the feature representation ability of convolutional neural networks; a multiscale receptive field enhancement module (MRFEM) is proposed that combines features obtained using dilated convolution at different dilation rates to further enhance the multiscale feature representation of the remote sensing targets; and a multi-vision network (MVNet) is presented that uses multiple low-level feature maps with multi-branch convolution to detect small objects. Experimental results show that the proposed method can achieve a significant mean average precision (mAP) of 94.70% in remote sensing images and can run at 24 FPS on a single NVIDIA 1080Ti GPU.

Bidirectional parallel multi-branch convolution feature pyramid network for target detection in aerial images of swarm UAVs

In this paper, based on a bidirectional parallel multi-branch feature pyramid network (BPMFPN), a novel one-stage object detector called BPMFPN Det is proposed for real-time detection of ground multi-scale targets by swarm unmanned aerial vehicles (UAVs). First, the bidirectional parallel multi-branch convolution modules are used to construct the feature pyramid to enhance the feature expression abilities of different scale feature layers. Next, the feature pyramid is integrated into the single-stage object detection framework to ensure real-time performance. In order to validate the effectiveness of the proposed algorithm, experiments are conducted on four datasets. For the PASCAL VOC dataset, the proposed algorithm achieves the mean average precision (mAP) of 85.4 on the VOC 2007 test set. With regard to the detection in optical remote sensing (DIOR) dataset, the proposed algorithm achieves 73.9 mAP. For vehicle detection in aerial imagery (VEDAI) dataset, the detection accuracy of small land vehicle (slv) targets reaches 97.4 mAP. For unmanned aerial vehicle detection and tracking (UAVDT) dataset, the proposed BPMFPN Det achieves the mAP of 48.75. Compared with the previous state-of-the-art methods, the results obtained by the proposed algorithm are more competitive. The experimental results demonstrate that the proposed algorithm can effectively solve the problem of real-time detection of ground multi-scale targets in aerial images of swarm UAVs.