High-level Fusion Research Articles

Fusion of visible and fluorescence imaging through deep neural network for color value prediction of pelletized red peppers.

A non-destructive method for determining the color value of pelletized red peppers is crucial for pepper processing factories. This study aimed to investigate the potentiality of visible and fluorescence images for the determination of color value of pelletized red pepper. The imaging problem, caused by the cylindrical shape and irregular cross-sectional features of the pelletized red peppers, was reduced through the extraction of an approximate plane region. To integrate the information in the visible and fluorescence images, a baseline convolutional neural network (CNN) architecture was designed and low level, middle level, and high level fusion models (denoted as LL-CNN, ML-CNN, and HL-CNN, respectively) were developed upon the baseline CNN. The effects of input image size and color space were examined. According to the training result, CNN fusion models were developed using visible image in L*a*b* color space and fluorescence image in RGB color space using 56×56 input image size. Among the three types of CNN fusion models, the HL-CNN obtained the best performance, resulting in Rv 2 of 0.828 and RMSEV of 0.351. This study suggests that the fusion of visible and fluorescence image through CNN is a practical approach to save testing time and replace traditional destructive method. The low cost and compact structure of the imaging systems can maintain the commercial appeal of pepper industry.

Information fusion of hyperspectral imaging and self-developed electronic nose for evaluating the degree of black tea fermentation

Fermentation is a crucial step in the processing of black tea. Currently, the evaluation of the fermentation degree of black tea relies mainly on manual experience, which suffers from subjectivity and lacks quantifiability. This study proposed the fusion of hyperspectral imaging and electronic nose to evaluate the degree of black tea fermentation. The self-developed electronic nose was used to acquire aroma information, while hyperspectral imaging was used to obtain color and spectral information during the fermentation process. The samples were collected at different fermentation time including insufficient, moderate, and excessive fermentation. Then, the classification models were constructed based on individual types of information. Subsequently, the electronic nose data were fused with the hyperspectral data using different information fusion strategies (low-level, middle-level, high-level) and the classification models were built accordingly. Six types of classifiers were established. Results showed that with deeper levels of fusion, the accuracy of the model on the test set gradually improved. In the high-level fusion, the accuracy of the test set reached 100 %, with improvements of 10.34 % and 8.62 % compared with that solely based on electronic nose and hyperspectral imaging, respectively. This indicates that the proposed information fusion method is highly effective, providing state-of-the-art insights into and technologies for the evaluation of black tea fermentation degree.

Fully Convolutional Hybrid Fusion Network with Heterogeneous Representations for Identification of S1 and S2 from Phonocardiogram.

Heart auscultation is a simple and inexpensive first-line diagnostic test for the early screening of heart abnormalities. A phonocardiogram (PCG) is a digital recording of an analog heart sound acquired using an electronic stethoscope. A computerized algorithm for PCG analysis can aid in detecting abnormal signal patterns and support the clinical use of auscultation. It is important to detect fundamental components, such as the first and second heart sounds (S1 and S2), to accurately diagnose heart abnormalities. In this study, we developed a fully convolutional hybrid fusion network to identify S1 and S2 locations in PCG. It enables timewise, high-level feature fusion from dimensionally heterogeneous features: 1D envelope and 2D spectral features. For the fusion of heterogeneous features, we proposed a novel convolutional multimodal factorized bilinear pooling approach that enables high-level fusion without temporal distortion. We experimentally demonstrated the benefits of the comprehensive interpretation of heterogeneous features, with the proposed method outperforming other state-of-the-art PCG segmentation methods. To the best of our knowledge, this is the first study to interpret heterogeneous features through a high level of feature fusion in PCG analysis.

TCHFN: Multimodal sentiment analysis based on Text-Centric Hierarchical Fusion Network

Multimodal sentiment analysis (MSA) has become a popular field of research in recent years. The aim is to combine the three modalities of text, video, and audio to obtain comprehensive emotional information. However, current research often treats these three modalities equally, downplaying the crucial role of text modality in MSA and ignoring the redundant information generated during multimodal fusion. To address these problems, we propose the Text-Centric Hierarchical Fusion Network (TCHFN), employing a hierarchical fusion strategy. In this framework, low-level fusion involves cross-modal interactions between pairs of modalities, while high-level fusion extends these interactions to involve all three modalities. Through the design of the Cross-modal Reinforced Transformer (CRT), we achieve cross-modal enhancement of the target modality, facilitating a nuanced fusion process with text serving as its core. Additionally, we design Text-Centric Contrastive Learning (TCCL) to align non-text modalities with the text modality, emphasising the central role of text in the fusion process. After fusion, a multimodal fusion output gate is employed to mitigate redundant information within the multimodal fusion representation, which is subsequently processed by a linear layer for prediction. Simultaneously, to fully leverage limited labelled datasets, we introduced knowledge distillation. This approach involves preserving the model parameters that yield the best performance during training as a teacher model. The teacher model aids in capturing rich emotional information, enabling the model to transcend local optima and discover more optimal parameters, thereby enhancing overall model performance. Extensive experiments on the CMU-MOSI, CMU-MOSEI, and CH-SIMS datasets demonstrate the superiority of our model over state-of-the-art methods in MSA tasks.

Rapid and non-destructive identification of Panax ginseng origins using hyperspectral imaging, visible light imaging, and X-ray imaging combined with multi-source data fusion strategies

The geographical origin of Panax ginseng significantly influences its nutritional value and chemical composition, which in turn affects its market price. Traditional methods for analyzing these differences are often time-consuming and require substantial quantities of reagents, rendering them inefficient. Therefore, hyperspectral imaging (HSI) in conjunction with X-ray technology were used for the swift and non-destructive traceability of Panax ginseng origin. Initially, outlier samples were effectively rejected by employing a combined isolated forest algorithm and density peak clustering (DPC) algorithm. Subsequently, random forest (RF) and support vector machine (SVM) classification models were constructed using hyperspectral spectral data. These models were further optimized through the application of 72 preprocessing methods and their combinations. Additionally, to enhance the model’s performance, four variable screening algorithms were employed: SelectKBest, genetic algorithm (GA), least absolute shrinkage and selection operator (LASSO), and permutation feature importance (PFI). The optimized model, utilizing second derivative, auto scaling, permutation feature importance, and support vector machine (2nd Der-AS-PFI-SVM), achieved a prediction accuracy of 93.4 %, a Kappa value of 0.876, a Brier score of 0.030, an F1 score of 0.932, and an AUC of 0.994 on an independent prediction set. Moreover, the image data (including color information and texture information) extracted from color and X-ray images were used to construct classification models and evaluate their performance. Among them, the SVM model constructed using texture information from X -ray images performed the best, and it achieved a prediction accuracy of 63.0 % on the validation set, with a Brier score of 0.181, an F1 score of 0.518, and an AUC of 0.553. By implementing mid-level fusion and high-level data fusion based on the Stacking strategy, it was found that the model employing a high-level fusion of hyperspectral spectral information and X-ray images texture information significantly outperformed the model using only hyperspectral spectral information. This advanced model attained a prediction accuracy of 95.2 %, a Kappa value of 0.912, a Brier score of 0.027, an F1 score of 0.952, and an AUC of 0.997 on the independent prediction set. In summary, this study not only provides a novel technical path for fast and non-destructive traceability of Panax ginseng origin, but also demonstrates the great potential of the combined application of HSI and X-ray technology in the field of traceability of both medicinal and food products.

Ante- and Post-Mortem Fracture Identification Protocol Based on Low- and High-Level Fusion Using Fourier Transform Infrared Spectroscopy and Raman Spectroscopy Association.

In this study, the application of low-level fusion (LLF) and high-level fusion (HLF) strategies using a combination of Fourier transform infrared spectroscopy (FT-IR) and Raman spectroscopy in the identification of antemortem and postmortem fracture at different postmortem intervals (PMIs) was investigated. On a technical level, the same hard tissue sample can be detected using a mix of FT-IR and Raman techniques. At the method level, two cutting-edge chemometrics approaches (LLF and HLF) combining FT-IR and Raman spectroscopic data are explored. The models were ranked in accordance with their parametric quality as follows: HLF and LLF + HLF models > LLF single model > Raman single model > FT-IR single model. The LLF model performed marginally better than the Raman model, however, when compared to other models, the HLF model performed considerably better. The HLF model achieved the best performance, with both cross-validation accuracy and test data set accuracy of 0.88. The importance of the feature wavelengths in the model construction process was subsequently evaluated by intersection fusion, and it was found that the absorbance bands of amide I, PO43- ν1 ν3, and CH2 in FT-IR and phenylalanine, CO32- ν1- PO43- ν3, and amide III in Raman have outstanding contributions to the construction of antemortem and postmortem fractures identification models. Overall, the combination of FT-IR and Raman with the HLF strategy is a novel and promising approach for developing antemortem and postmortem fracture identification models at different PMIs.

Feature interaction and two-stage cross-modal fusion for RGB-D salient object detection

Most existing RGB-D salient object detection (SOD) methods extract features of both modalities in parallel or adopt depth features as supplementary information for unidirectional interaction from depth modality to RGB modality in the encoder stage. These methods ignore the influence of low-quality depth maps, and there is still room for improvement in effectively fusing RGB features and depth features. To address the above problems, this paper proposes a Feature Interaction Network (FINet), which performs bi-directional interaction through feature interaction module (FIM) in the encoder stage. The feature interaction module is divided into two parts: depth enhancement module (DEM) filters the noise in the depth features through the attention mechanism; and cross enhancement module (CEM) effectively interacts RGB features and depth features. In addition, this paper proposes a two-stage cross-modal fusion strategy: high-level fusion adopts the semantic information of high level for coarse localization of salient regions, and low-level fusion makes full use of the detailed information of low level through boundary fusion, and then we progressively refine high-level and low-level cross-modal features to obtain the final saliency prediction map. Extensive experiments show that the proposed model achieves better performance than eight state-of-the-art models on five standard datasets.

Dilated Spectral–Spatial Gaussian Transformer Net for Hyperspectral Image Classification

In recent years, deep learning-based classification methods for hyperspectral images (HSIs) have gained widespread popularity in fields such as agriculture, environmental monitoring, and geological exploration. This is owing to their ability to automatically extract features and deliver outstanding performance. This study provides a new Dilated Spectral–Spatial Gaussian Transformer Net (DSSGT) model. The DSSGT model incorporates dilated convolutions as shallow feature extraction units, which allows for an expanded receptive field while maintaining computational efficiency. We integrated transformer architecture to effectively capture feature relationships and generate deep fusion features, thereby enhancing classification accuracy. We used consecutive dilated convolutional layers to extract joint low-level spectral–spatial features. We then introduced Gaussian Weighted Pixel Embedding blocks, which leverage Gaussian weight matrices to transform the joint features into pixel-level vectors. By combining the features of each pixel with its neighbouring pixels, we obtained pixel-level representations that are more expressive and context-aware. The transformed vector matrix was fed into the transformer encoder module, enabling the capture of global dependencies within the input data and generating higher-level fusion features with improved expressiveness and discriminability. We evaluated the proposed DSSGT model using five hyperspectral image datasets through comparative experiments. Our results demonstrate the superior performance of our approach compared to those of current state-of-the-art methods, providing compelling evidence of the DSSGT model’s effectiveness.

Non-destructive prediction of the total viable count (TVC) in Fujian oysters (Crassostrea angulata) based on the colorimetric sensor array

The total viable count (TVC) is a traditional method for evaluating seafood freshness. This study aimed to predict the TVC of Fujian oysters stored at 4℃ by using the colorimetric sensing array (CSA) technique combined with imaging and visible near-infrared (Vis-NIR) information channels. An optimal quantitative prediction model based on partial least squares (PLS) was established by combining feature variable selection with a data fusion strategy. All three data-fusion strategies showed better predictive performance than the individual datasets. The high-level fusion strategy performed the best, with a root mean square error of prediction (RMSEP) of 0.17, mean absolute error of prediction (MAEP) of 0.14, coefficient of determination of prediction (Rp2) of 0.9822, and residual prediction deviation (RPD) of 7.37. The experimental results showed that CSA combined with multiple detection channels could be effectively used to predict the TVC of Fujian oysters. In this study, a reliable and comprehensive evaluation model was established to achieve a rapid and accurate prediction of oyster freshness.

Credal Valuation Networks for Machine Reasoning Under Uncertainty

Contemporary undertakings provide limitless opportunities for widespread application of machine reasoning and artificial intelligence in situations characterised by uncertainty, hostility and sheer volume of data. The paper develops a valuation network as a graphical system for higher-level fusion and reasoning under uncertainty in support of the human operators. Valuations, which are mathematical representation of (uncertain) knowledge and collected data, are expressed as credal sets, defined as coherent interval probabilities in the framework of imprecise probability theory. The basic operations with such credal sets, combination and marginalisation, are defined to satisfy the axioms of a valuation algebra. A practical implementation of the credal valuation network is discussed and its utility demonstrated on a small scale example.

Can fusion of vis-NIR and MIR spectra at three levels improve the prediction accuracy of soil nutrients?

Soil nutrients are an important component of soil fertility, and having accurate and timely soil nutrients information is conducive to scientifically guided agricultural fertilization and improved crop yields. Traditional soil nutrient measurement methods are accurate but time-consuming and costly. Visible near-infrared (vis-NIR) and mid-infrared (MIR) as techniques of spectroscopy measurements are cheap, fast, and non-destructive, and, after training, have been used separately to predict soil nutrients. They can give complementary information, so using them separately may not be optimal. This study investigated whether the fusion of vis-NIR and MIR spectra would improve the prediction of six soil nutrients: total nitrogen (TN), total phosphorus (TP), total potassium (TK), alkali-hydrolyzable nitrogen (AN), available phosphorus (AP) and available potassium (AK). The sample set was 501 tillage (0 ∼ 20 cm) soil samples from Guizhou Province (China). Three different sensor fusion techniques were compared: (1) direct fusion of spectra (low-level fusion; SF1); (2) fusion of spectral features selected by the least absolute shrinkage and selection operator (LASSO) algorithm (middle-level fusion; SF2); and (3) fusion using the Granger-Ramanathan averaging (GRA) method of the separate vis-NIR and MIR results (high-level fusion; SF3). Prediction models were built using partial least squares regression (PLSR) and support vector machine (SVM), and evaluated using leave-one-out cross-validation (LOOCV), coefficient of determination (R2), and ratio of performance to interquartile (RPIQ), defined as Rcv2 and RPIQCV. All three sensor fusion methods were more accurate than single-sensor methods in predicting TN, TK and AN (0.60 ≤ Rcv2 ≤ 0.92, 1.86 ≤ RPIQCV ≤ 5.08), but were less accurate for TP, AP and AK (Rcv2 ≤ 0.34, RPIQCV ≤ 1.55). Compared to the best prediction using each single sensor model, low-level fusion improved the prediction of TN and AN; middle-level fusion improved the prediction of TN, TP, AN, AP and AK; and high-level fusion improved the prediction of all soil nutrients. Moreover, the SF2-PLSR model provided the best prediction for TN (Rcv2 = 0.83, RPIQCV = 3.18), the SF2-SVM model for AN (Rcv2 = 0.67, RPIQCV = 2.06), and the SF3-PLSR model for TK (Rcv2 = 0.92, RPIQCV = 5.08). The SF2 and SF3 methods are recommended to predict TN and AN, and MIR spectra are recommended to predict TK (Rcv2 = 0.92, RPIQCV = 5.02) since using only a single sensor is cost-effective. The SF2 and SF3 methods improved the prediction accuracy of AP and AK, but the prediction accuracy was still low. This study only covered one area with its set of soil landscapes, and only used well-established modelling methods. The results can motivate research on new spectra techniques and advanced modelling methods, as well as transferability to other soil landscapes.

Quantitative analysis of moisture content and particle size in a fluidized bed granulation process using near infrared spectroscopy and acoustic emission combined with data fusion strategies

Monitoring granule property is essential for fluidization maintenance and product quality control in fluidized bed granulation (FBG). In this study, two non-invasive techniques, near-infrared (NIR) spectroscopy and acoustic emission (AE), were applied for quantitative analysis of moisture content (MC) and median particle size (D50) in a FBG process, combined with chemometrics and data fusion strategies. Partial least squares (PLS) and support vector machine (SVM) regression models were established based on NIR and AE spectral data. The optimal quantitative models were identified considering the effect of spectra preprocessing and variable selection. In the comparison study, the best separate models for MC and D50 quantification were based on NIR and AE, respectively. The NIR model exhibited the better prediction ability with the determination coefficient of validation set (R2v) of 0.9815, root mean square error of validation set (RMSEv) of 0.2226 %, and residual predictive deviation (RPD) of 7.4674 for MC. Meanwhile, the AE model presented the better prediction performance with R2v of 0.9710, RMSEv of 18.2643 μm, and RPD of 5.9740 for D50. Furthermore, among three data fusion strategies, the high-level fusion model achieved the best overall performance on D50 quantification with R2v of 0.9863, RMSEv of 12.5707 μm, and RPD of 8.6798. The results indicated that both NIR and AE are effective monitoring tools for MC and D50 analysis in fluidized bed granulation process. In addition, a more accurate and reliable analysis of particle size can be achieved by combining NIR and AE technology with high-level data fusion.

Application of visible near-infrared spectroscopy combined with colorimetric sensor array for the aroma quality evaluation in tencha drying process

The drying process is a critical stage in developing the aroma quality of tencha. In our research, visible near infrared (Vis-NIR) and colorimetric sensor array (Vis-NIR-CSA) were used for evaluating the aroma quality of tencha drying process. Vis-NIR recorded the spectral signal of CSA after the reaction in samples. Subsequently, the aroma quality was predicted by a combination of different data fusion strategies and classification and regression tree (CART) in tencha drying process. The high-level fusion strategy showed the best performance, with calibration and prediction set accuracy of 94.68% and 93.48%, respectively. The results indicated that Vis-NIR-CSA combined with high-level data fusion could be applied satisfactorily in the aroma quality evaluation of tencha. Moreover, pentanal was identified to be highly correlated with aroma quality during tencha drying process, which verified the sensor identification results. This study contributed to controlling good manufacturing practices and designing optimal tencha processing systems.

Spectral sensor fusion for prediction of Li and Zr in rocks: Neural network and PLS methods

Integrating information from multiple sensors, known as sensor fusion, is particularly challenging for small datasets where selecting from the plethora of available methods poses a significant challenge in chemometric analysis. This study compares several sensor fusion methods (spanning data-level, feature-level, and decision-level fusion) based on partial least squares (PLS) and convolutional neural network (CNN) models. This study is the first to compare simple sensor fusion methods to the latest multiblock PLS models and parallel-input CNNs, to the best of our knowledge. We demonstrate sensor fusion using a small dataset of 177 rock samples on two prediction tasks: predicting lithium (Li) concentrations and zirconium (Zr) concentrations using three types of spectra, namely X-ray fluorescence (XRF), visible to short-wave infrared (Vis-NIR-SWIR), and laser-induced breakdown spectroscopy (LIBS). The best-performing Li model was PLS using XRF (mean RMSEP: 0.078%) and the best sensor fusion model was ROSA (mean RMSEP: 0.087%). The best Zr model was a high-level fusion of PLS models (mean RMSEP: 0.042%).

TSception: Capturing Temporal Dynamics and Spatial Asymmetry From EEG for Emotion Recognition

The high temporal resolution and the asymmetric spatial activations are essential attributes of electroencephalogram (EEG) underlying emotional processes in the brain. To learn the temporal dynamics and spatial asymmetry of EEG towards accurate and generalized emotion recognition, we propose TSception, a multi-scale convolutional neural network that can classify emotions from EEG. TSception consists of dynamic temporal, asymmetric spatial, and high-level fusion layers, which learn discriminative representations in the time and channel dimensions simultaneously. The dynamic temporal layer consists of multi-scale 1D convolutional kernels whose lengths are related to the sampling rate of EEG, which learns the dynamic temporal and frequency representations of EEG. The asymmetric spatial layer takes advantage of the asymmetric EEG patterns for emotion, learning the discriminative global and hemisphere representations. The learned spatial representations will be fused by a high-level fusion layer. Using more generalized cross-validation settings, the proposed method is evaluated on two publicly available datasets DEAP and MAHNOB-HCI. The performance of the proposed network is compared with prior reported methods such as SVM, KNN, FBFgMDM, FBTSC, Unsupervised learning, DeepConvNet, ShallowConvNet, and EEGNet. TSception achieves higher classification accuracies and F1 scores than other methods in most of the experiments. The codes are available at https://github.com/yi-ding-cs/TSception

A Multi-Step Fusion Network for Semantic Segmentation of High-Resolution Aerial Images.

The demand for semantic segmentation of ultra-high-resolution remote sensing images is becoming increasingly stronger in various fields, posing a great challenge with concern to the accuracy requirement. Most of the existing methods process ultra-high-resolution images using downsampling or cropping, but using this approach could result in a decline in the accuracy of segmenting data, as it may cause the omission of local details or global contextual information. Some scholars have proposed the two-branch structure, but the noise introduced by the global image will interfere with the result of semantic segmentation and reduce the segmentation accuracy. Therefore, we propose a model that can achieve ultra-high-precision semantic segmentation. The model consists of a local branch, a surrounding branch, and a global branch. To achieve high precision, the model is designed with a two-level fusion mechanism. The high-resolution fine structures are captured through the local and surrounding branches in the low-level fusion process, and the global contextual information is captured from downsampled inputs in the high-level fusion process. We conducted extensive experiments and analyses using the Potsdam and Vaihingen datasets of the ISPRS. The results show that our model has extremely high precision.

Multi-block DD-SIMCA as a high-level data fusion tool

Multi-block classification method based on the Data Driven Soft Independent Modeling of Class Analogy (DD-SIMCA) is presented. A high-level data fusion approach is used for the joint analysis of data collected with the help of different analytical instruments. The proposed fusion technique is very simple and straightforward. It uses a Cumulative Analytical Signal which is a combination of outcomes of the individual classification models. Any number of blocks can be combined. Although the high-level fusion eventually leads to a rather complex model, the analysis of partial distances makes it possible to establish a meaningful relationship between the classification results and the influence of individual samples and specific tools.Two real world examples are used to demonstrate the applicability of the multi-block algorithm and the consistency of the multi-block method with its predecessor, a conventional DD-SIMCA.

Fusing one-class and two-class classification – A case study on the detection of pepper fraud

Black pepper is a commercially important commodity, which is susceptible for fraudulent additions. Analytical tools are capable of detection of specific additions, but in most published cases these tools and associated mathematical models are suitable for only one or a few predetermined adulterants. There is a need for methodology that can detect any addition without having to know the type of adulterant a priori . We analyzed a dataset of 200 authentic black pepper samples and a total of 210 adulterated samples consisting of mixtures of black pepper and oil-dress pepper, spent pepper, coffee husk, coffee skin and papaya seeds, respectively. A small, non-destructive spectral tool, a visible-near infrared spectrophotometer, (VIS/NIR) and a slower and more expensive mass spectrometric tool, direct analysis in real time-mass spectrometer (DART-MS), were evaluated according to their performances in terms of adulteration detection for a number of machine learning modes. The often-used approach where an ‘optimal’ model is selected and employed yielded for VIS/NIR very reasonable results for most of the adulterants used, but no single model performed well for all adulterants in the dataset. However, high-level fusion modelling of both one- and two-class models developed for different adulterant types using a penalized excess scoring system led to a performance of typically >75% correct classification, regardless of the nature of the adulterant. DART-MS outperformed VIS/NIR but also led to no single model that was able to detect all adulterants present. From the small number of tested fusion strategies, again the penalized excess score outperformed the other fusion options and yielded perfect classification scores for all but one of the adulterants tested. This shows that this type of modelling for cases where the nature of a target is unknown is a promising approach. It is even speculated that this modelling approach is likely to be suitable for types of adulterant that were not used in the model development phase. • Detection of adulterants in black pepper powder using spectroscopy and DART. • Application of VIS/NIR in a fused prototype device. • Simultaneous detection of multiple adulterants using chemometrics. • Investigation into different fusion approaches of VIS/NIR and DART data. • Fusing of chemometric models enhances and broadens the detection power.

UVA Image Registration Model Based on VGG and Multi-Branch Attention

For UVA images with different resolutions and large areas of weak texture, image feature extraction is insufficient and mis-matching is increased during image registration. To solve these problems, an unsupervised registration model based on VGG feature extraction and multi-branch attention is proposed. First of all, two feature extraction networks with shared weight parameters are used to extract the low and high level fusion features of the moving image and the reference image. The convolution neural network is used to extract the high-dimensional feature map of the image, and the key points are selected according to the conditions that meet both the channel maximum and the local maximum, and the corresponding 512-dimensional descriptor is extracted on the feature map, In the matching stage, add multi-branch attention based on residual block to filter out the wrong features. The algorithm is tested with multiple groups of images and compared with several image matching algorithms. The results show that the algorithm can extract the scale-invariant similar features of images, and has strong adaptability and robustness.

Application of Feature Pyramid Network and Feature Fusion Single Shot Multibox Detector for Real-Time Prostate Capsule Detection

In the process of feature propagation, the low-level convolution layers of the forward feature propagation network lack semantic information, and information loss occurs when fine-grained information is transferred to higher-level convolution; therefore, multi-stage feature fusion networks are needed to solve the interaction between low-level convolution layers and high-level convolution layers. Based on a two-way feature feedback network and feature fusion mechanism, we created a new object detection network called Feature Pyramid Network (FPN)-based Feature Fusion Single Shot Multibox Detector (FFSSD). A bottom-up and top-down architecture with lateral connections enhances the detector’s ability to extract features, then high-level multi-scale semantic feature maps are utilized to generate a feature pyramid network. The results show that the proposed network the mAP for prostate capsule image detection reaches 83.58%, providing real-time detection ability. The context interaction mechanism can transfer high-level semantic information to low-level convolution, and the resulting convolution after low-level and high-level fusion contains richer location and semantic information.