Fusion Mechanism Research Articles

Query-Adaptive Late Fusion for Hierarchical Fine-Grained Video-Text Retrieval.

Recently, a hierarchical fine-grained fusion mechanism has been proved effective in cross-modal retrieval between videos and texts. Generally, the hierarchical fine-grained semantic representations (video-text semantic matching is decomposed into three levels including global-event representation matching, action-relation representation matching, and local-entity representation matching) to be fused can work well by themselves for the query. However, in real-world scenarios and applications, existing methods failed to adaptively estimate the effectiveness of multiple levels of the semantic representations for a given query in advance of multilevel fusion, resulting in a worse performance than expected. As a result, it is extremely essential to identify the effectiveness of hierarchical semantic representations in a query-adaptive manner. To this end, this article proposes an effective query-adaptive multilevel fusion (QAMF) model based on manipulating multiple similarity scores between the hierarchical visual and text representations. First, we decompose video-side and text-side representations into hierarchical semantic representations consisting of global-event level, action-relation level, and local-entity level, respectively. Then, the multilevel representation of the video-text pair is aligned to calculate the similarity score for each level. Meanwhile, the sorted similarity score curves of the good semantic representation are different from the inferior ones, which exhibit a "cliff" shape and gradually decline (see Fig. fig1 as an example). Finally, we leverage the Gaussian decay function to fit the tail of the score curve and calculate the area under the normalized sorted similarity curve as the indicator of semantic representation effectiveness, namely, the area of good semantic representation is small, and vice versa. Extensive experiments on three public benchmark video-text datasets have demonstrated that our method consistently outperforms the state-of-the-art (SoTA). A simple demo of QAMF will soon be publicly available on our homepage: https://github.com/Lab-ANT.

An object detection system using array planar capacitive sensors

This paper presents a target detection system using array-based planar capacitive sensors, demonstrating excellent detection performance in gesture recognition. The system achieves gesture target classification by collecting the capacitance values of the planar capacitive sensor array and optimizing the BP neural network through a genetic algorithm for model training. Additionally, the system employs visual detection for gesture targets, using transfer learning with the MobileNet V2 network in the Keras framework to achieve gesture target classification. The system utilizes a data fusion mechanism to merge visual recognition and planar capacitive sensor array recognition for decision-making. Experimental results demonstrate that the target detection system exhibits outstanding detection performance in gesture recognition, with an average accuracy of up to 98.92%.

Steel surface defect detection algorithm based on ESI-YOLOv8

Abstract To enhance the precision of detecting defects on steel plate surfaces and diminish the incidences of false detection and leakage, the ESI-YOLOv8 algorithm is introduced. This algorithm introduces a novel EP module and integrates the large separation convolutional attention module and the spatial pyramid pooling module to propose the SPPF-LSKA module. Additionally, the original CIOU loss function is replaced with the INNER-CIOU loss function. The EP module minimizes redundant computations and model parameters to optimize efficiency and simultaneously increases the multi-scale fusion mechanism to expand the sensory field. The SPPF-LSKA module reduces computational complexity, accelerates model operation speed, and improves detection accuracy. Additionally, the INNER-CIOU loss function can improve detection speed and model accuracy by controlling the scale size of the auxiliary border.The results of the experiment indicate that, following the improvements made, the algorithm’s detection accuracy has increased to 78%, which is 3.7% higher than the original YOLOv8. Furthermore, the model parameters were reduced, and the verification was conducted using the CoCo dataset, resulting in an average accuracy of 77.8%. In conclusion, the algorithm has demonstrated its ability to perform steel plate surface defect detection with efficiency and accuracy.

A point cloud semantic segmentation method with augmented local geometry

With the development of 3D sensors, complex spatial structures can be established from point cloud data. Accurate semantic segmentation of 3D point clouds is beneficial for tasks such as robotic scene understanding. To extract the geometric features of the point cloud, most prior works use the xyz coordinates to potentially learn the shapes of 3D objects, which is not insufficient to handle complex scenes. In this paper, a Local Geometry Encoding (LGE) module is proposed to provide local normal information and express 3D shapes explicitly. Furthermore, considering that most prior methods use a single decoder to recover from abstract features, a Multi-Decoder (MD) ensemble module is proposed to restore multiple segmentation results from various scale feature maps in the encoder, and then a multi-attention fusion mechanism is utilized to produce single segmentation results. Benefiting from the local geometry encoding module and Multi-Decoder ensemble module, our network performs 6-fold cross-validation on the S3DIS dataset and achieves 75% mIoU with fewer parameters.

A channel-fused gated temporal convolutional network for EMG-based gesture recognition

Background:EMG gesture recognition can be widely applied in many fields, such as prosthetic control and human–computer interaction, to enhance the standard of human life. Purpose:This study aims to develop a deep learning-based model to identify multiple complex gestures from raw EMG signals. Methods:We propose a new channel-fused gated temporal convolutional network. First, a channel fusion and gating mechanism is designed to improve temporal convolutional networks, allowing the model to obtain higher-level features. Second, we improve the channel fusion module by the short-term average energy to fuse the EMG signals of multi-channels more accurately. Results:The model is evaluated on a public dataset of EMG gestures, NinaPro DB5. Results demonstrated that the unbalanced accuracy rate of 53 gesture actions reaches 92.71%, and the balanced accuracy rate 74.79%. Further, ablation experiments validates the effectiveness of each model module. Conclusions:This study demonstrates our proposed approach can improve gesture recognition accuracy for complex gestures and has great potential for practical applications.

Research on Image Classification And Semantic Segmentation Model Based on Convolutional Neural Network

This paper investigates convolutional neural network (CNN)-based approaches for image classification and semantic segmentation, with a focus on addressing spatial detail loss and multi-scale feature integration issues prevalent in semantic segmentation. The introduced EDNET model tackles these challenges through the incorporation of spatial information branches and the design of efficient feature fusion mechanisms. It further enhances performance via the use of global pooling and boundary refinement modules. Evaluations on the PASCAL VOC 2012 dataset reveal an 11.67% increase in mean intersection-over-union (IoU) compared to standard fully convolutional networks, demonstrating substantial improvement over comparable techniques. These results confirm the efficacy and practicality of the EDNET framework.

Sparse graph cascade multi-kernel fusion contrastive learning for microbe–disease association prediction

Predicting microbe–disease associations (MDA) is crucial for proactively demystifying diseases causes and preventing them. Traditional prediction methods endure labor-intensive, time-consuming, and expensive. Therefore, this paper proposes CasMF-GCL, a novel Graph Contrastive Learning model based on sparse relation augmentation and a Cascaded Multi-kernel Fusion Network for MDA prediction. CasMF-GCL maximizes the use of sparse correlation information and eliminates data noise through an association augmentation technique based on low-rank sparse matrix completion. We employ diverse strategies to enhance the features of heterogeneous networks by exploring the similarity between microbes and diseases. Our model employs a new multi-layer graph convolutional network variant with a cascaded multi-kernel fusion mechanism, enabling weighted coding of local and global features for diseases and microbes. To further improve performance, we incorporate a self-supervised contrastive learning schema using multi-grain disease and microbe features and large-scale relation augmentations. Extensive experiments on two renowned datasets demonstrate that CasMF-GCL outperforms current state-of-the-art methods in seven indexes. The AUC values from 5-fold cross-validation are 0.997 and 0.989 for the two datasets, respectively. Ablation studies confirm the effectiveness of graph data augmentation, the power of the contrastive strategy, and the indispensability of the cascaded multi-kernel fusion network. Furthermore, case studies validate the prediction performance of CasMF-GCL.

Nicotinamide Mononucleotide and Nicotinamide Riboside Reverse Ovarian Aging in Rats Via Rebalancing Mitochondrial Fission and Fusion Mechanisms

PurposeThis study examined the effects of nicotinamide mononucleotide (NMN) and nicotinamide riboside (NR) on folliculogenesis and mitochondrial dynamics (fission and fusion mechanisms) in ovaries of middle-aged female rats.MethodsExperimental groups were young, middle-aged (control), middle-aged + NMN and middle-aged + NR. NMN was administered at a concentration of 500 mg/kg intraperitoneally but NR at a concentration of 200 mg/kg by gavage. Follicle stimulating hormone (FSH) and luteinizing hormone (LH) levels were analyzed by ELISA. Hematoxylin-eosin staining sections were used for histopathological examination and follicles-counting. Expression levels of mitochondrial fission (Drp1, Mff and Fis1) and fusion (Mfn1, Mfn2, Opa1, Fam73a and Fam73b) genes as well as Sirt1 gene were analyzed by RT-PCR. Expression levels of fission-related proteins (DRP1, MFF, FIS1 and SIRT1) were analyzed by Western Blot.ResultsHigher ovarian index, more corpus luteum and antral follicles were detected in NMN and NR groups compared to the control. NMN or NR could rebalance LH/FSH ratio. The control group was determined to possess higher expression levels of fission genes and lower expression levels of fusion genes when compared the young group. In comparison with the control group, both NMN and NR group were found to exhibit less mitochondrial fission but more mitochondrial fussion. Higher gene and protein levels for Sirt1 were measured in NMN and NR groups compared to the control group.ConclusionThis study reveals that NMN alone or NR alone can rebalance mitochondrial dynamics by decreasing excessive fission in middle-aged rat ovaries, thus alleviating mitochondrial stress and correcting aging-induced folliculogenesis abnormalities.

PRODUCTION OF DIISOPROPYL ETHER IN THE PRESENCE OF MODIFIED HETEROPOLYACID CATALYSTS CONTAINING TITANIUM OXIDE

The catalytic activity of titanium dioxide and its analogs modified with formoformolybdenum heteropoly acid was studied in the conversion of isopropyl alcohol. It was found that adding 10% phosphoformolybdenum heteropoly acid to titanium dioxide increased the catalyst's stability and its selectivity for diisopropyl ether by more than 1.5 times. It is suggested that the formation of diisopropyl ether in the presence of unmodified titanium dioxide occurs via the fusion mechanism, while in the presence of modified titanium dioxide, it occurs via the shock mechanism. The change in the mechanism of diisopropyl ether formation is associated with a shift in the distribution of acid-base active centers: before modification, strong and weak Lewis acid centers and Bronsted base centers are present on the surface of titanium dioxide, and the introduction of the modifier blocks strong Bronsted base centers, forming a new type—Bronsted basic centers composed of oxygen atoms. Keywords: isopropyl alcohol, catalytic properties, phospho-libdeno heteropoly acid, fusion mechanism, shock mechanism.

Deep multiscale convolutional feature learning for intracranial hemorrhage classification and weakly supervised localization

ObjectiveThis study evaluated the performance of attentional fusion model-based multiscale features in classifying intracerebral hemorrhage and the localization of bleeding focus based on weakly supervised target localization. MethodsA publicly available dataset provided by the American College of Neuroradiology (ASNR) was used, consisting of 750,000 computed tomography (CT) scans of the brain, manually marked by radiologists for intracranial hemorrhage and five hemorrhage subtypes. A multiscale feature classification and weakly supervised localization framework based on an attentional fusion mechanism were applied, which could be annotated at the slice level and provided intracranial hemorrhage classification and hemorrhage focus localization. ResultsThe designed framework achieved excellent performance for classification and localization. The area under the curve (AUC) for predicting bleeding was 0.973. High AUC values were observed for the five hemorrhage subtypes (epidural AUC = 0.891, subdural AUC = 0.991, subarachnoid AUC = 0.983, intraventricular AUC = 0.995, intraparenchymal AUC = 0.990). This model outperformed the average entry-level radiology trainee compared to previously reported data. ConclusionThe designed method quickly and accurately detected intracerebral hemorrhage, classifying hemorrhage subtypes and locating bleeding points with image-level annotation alone. The results indicate that this framework can significantly reduce diagnostic time while improving the detection of intracerebral hemorrhage in emergencies. It can thus be integrated into the diagnostic radiology workflow in the future.

Research on soft sensor modeling method for complex chemical processes based on local semi-supervised selective ensemble learning

To address the adverse effects of nonlinearity and dynamic time-varying in complex chemical processes on the accuracy of the soft sensor model, a local-semi-supervised ensemble learning for soft sensor modeling (local semi-supervised-selective ensemble learning-long short term memory, LS-SEL-LSTM) method is proposed in this article. Firstly, a hierarchical clustering method incorporating spatiotemporal criteria is proposed to reduce the influence of nonlinearity in global model prediction accuracy. The method considers the dynamic time-varying characteristics of temporal data and generates multiple local datasets. Then, to address the issue of multi-rate between auxiliary variables and dominant variables, a semi-supervised weight fusion mechanism based on temporal correlation is proposed, which effectively utilizes auxiliary variables to reconstruct local semi-supervised datasets and establishes local soft sensing models using LSTM. Concurrently, the parameters of the established model were optimized using the flower pollination algorithm. Subsequently, a selective ensemble learning method based on sub-model prediction accuracy and an adaptive combination weight calculation method for sub-models were proposed to improve the prediction accuracy. Finally, the effectiveness of the proposed method was verified through the actual dataset of the sulfur recovery process. The results indicate that LS-SEL-LSTM performs well in handling complex chemical processes with nonlinear and dynamic time-varying characteristics.

Research on the application of English short essay reading emotional analysis in online English teaching under IoT scenario

AbstractSpeech‐emotion analysis plays an important role in English teaching. The existing convolutional neural networks (CNNs) can fully explore the spatial features of speech information, and cannot effectively utilize the temporal dependence of speech signals. In addition, it is difficult to build a more efficient and robust sentiment analysis system by solely utilizing speech information. With the development of the Internet of Things (IoTs), online multimodal information, including speech, video, and text, has become more convenient. To this end, this paper proposes a novel multimodal fusion emotion analysis system. Firstly, by combining convolutional networks with Transformer encoders, the spatiotemporal dependencies of speech information are effectively utilized. To improve multimodal information fusion, we introduce the exchange‐based fusion mechanism. The experimental results on the public dataset indicate that the proposed multimodal fusion model achieves the best performance. In online English teaching, teachers can effectively improve the quality of teaching by leveraging the feedback information of students' emotional states through our proposed deep model.

From easy to hard: Improving personalized response generation of task-oriented dialogue systems by leveraging capacity in open-domain dialogues

A task-oriented dialogue system (TOD) is an important application of artificial intelligence. In the past few years, works on personalized TODs have attracted increased research attention and have seen much progress. The main challenge of such dialogue systems is finding ways to exploit user profiles under conditions of fixed and monolithic dialogue style. However, most of the existing works overlook the observation that the personalization capability of the dialogue system is fundamental and generic, and they treat all attributes of the user profile equally throughout the dialogue flows, which makes them inadequate for developing a well-performing personalized TOD. In this paper, we propose a two-stage learning framework equipped with GPT2 as a backbone to alleviate the above two problems. In the first learning phase, we finetune the GPT2 model on the personalized open-domain dialogues to preliminarily acquire personalization power. Then, we transfer this power to personalized task-oriented dialogues for the second stage of learning. After that, we enable the proposed model to be capable of the desired personalization capacity. Moreover, we present a dynamic profile fusion mechanism and an auxiliary task that detects which attributes contribute to the current utterance to facilitate the model’s performance. Eventually, we rewrite the attribute descriptions of user profiles in sentences to mitigate the consistency gap between the open-domain and task-oriented dialogues. The experimental results show that the proposed model achieves superior results compared to the state-of-the-art models on two versions of the Personalized bAbI dataset.

DL-YOLOX: Real-time object detection via adjustable dilated enhancement for autonomous driving scene

In the domain of autonomous driving, object detection presents several complex challenges, particularly concerning the accurate identification of small and salient objects. This paper introduces DL-YOLOX (Dilated Enhancement YOLOX), which flexibly uses dilated convolution to enhance features to achieve the purpose of improving small objects and silent objects. As we all know, a large receptive field covers a larger area and has greater contextual information, which is more advantageous for detecting large targets. A small receptive field helps capture local details and has better detection capabilities for detecting small targets. To bolster the representation of objects across various scales, we propose the integration of Dilated Adaptive Feature Fusion (DAFF) which has the ability to adaptively fuse features with different receptive fields. This innovative fusion mechanism allows for a more comprehensive understanding of objects, enabling improved detection accuracy even for objects of varying sizes. In addition, we tackle the issue of small object loss during feature propagation by introducing Stack Dilated Module (SDM), a powerful module that mitigates this phenomenon and contributes to better detection performance. Moreover, we endeavor to enhance small object detection further by replacing the conventional Intersection over Union (IoU) metric with Normalized Gaussian Wasserstein Distance (NWD), a novel distance metric that proves to be more effective in accurately gauging small object detection, thus elevating the precision of our algorithm. To thoroughly evaluate the robustness and generalization capabilities of our proposed method, we conduct extensive experiments on two benchmark datasets, namely MS COCO 2017 and BDD100K. The results from our evaluation not only affirm the significant improvements achieved in multi-scale object detection but also highlight the real-time capability of our approach. The impressive performance across these datasets demonstrates the promising potential of DL-YOLOX in revolutionizing object detection techniques in the context of autonomous driving.

APPFNet: Adaptive point-pixel fusion network for 3D semantic segmentation with neighbor feature aggregation

3D semantic segmentation is significant for scene understanding in various domains, such as autonomous driving, mapping, and robotics. Existing research often enhances prediction accuracy by integrating data from camera and LiDAR (light detection and ranging). However, current fusion methods face two primary challenges. Firstly, they frequently project point clouds onto 2D coordinates and employ 2D segmentation networks, resulting in a significant loss of spatial depth information. Secondly, prevailing approaches struggle to establish long-range contextual understanding, limiting comprehensive predictions across multiple objects. To address these challenges, our work proposes the Adaptive Point-Pixel Fusion Network (APPFNet). In the initial stage of modality fusion, this method correlates points with pixels to provide spatial depth information to Red, Green, and Blue (RGB) images. Subsequently, we leverage our adaptive Multi-scale Modal Fusion module (MSMFM) and Dispersed Self-attention (DFSA) mechanism for further feature extraction and fusion. Additionally, we propose a flexible Neighbor Feature Aggregation module (NFAM) that enhances the model’s ability to establish long-range contextual understanding by fusing information from neighboring points. Importantly, this module can be easily transplanted into other networks with simple modifications. Extensive experiments on SemanticKITTI and nuScenes demonstrate the superiority of our proposed approach, where APPFNet achieves a higher mean accuracy (mAcc) in predicting small objects in distant areas compared to state-of-the-art (SOTA) fusion methods.

Investigating the impact of the universal function of the nuclear proximity potential in heavy-ion fusion cross sections

The fusion barriers and cross sections of 15 colliding systems with 320 ≤ Z 1 Z 2 ≤ 1512 are investigated in detail to understand the influence of the universal function of proximity potential formalism in the heavy-ion fusion mechanism. To realize this goal, we select three versions of the phenomenological proximity potentials, including Prox. 77, Zhang 2013, and Guo 2013, to calculate the nucleus–nucleus potential. The experimental fusion cross sections for the selected reactions are analyzed using the standard coupled-channel calculations, including couplings to the low-lying 2+ and 3− states in the target and projectile. The calculated results show that the universal functions of the Guo 2013 and Prox. 77 models provide the lowest and highest fusion barriers, respectively. In addition, it is found that the height of the fusion barriers is enhanced by increasing the mass number of the projectile from light to heavy ones. The highest sensitivity to the mass number of the projectile belongs to the results of Prox. 77. A discussion is also presented on the influence of the universal function on the radial behavior of the interaction potential in the allowed region for overlapping configurations. Our results reveal that the best fit to the experimental data of the fusion cross sections for the reactions involving light and medium nuclei is obtained using the universal function of the Zhang 2013 model. For the heavier systems, the results of the Guo 2013 model at sub-barrier energies provide a good description of the available data.

PermuteDDS: a permutable feature fusion network for drug-drug synergy prediction

MotivationDrug combination therapies have shown promise in clinical cancer treatments. However, it is hard to experimentally identify all drug combinations for synergistic interaction even with high-throughput screening due to the vast space of potential combinations. Although a number of computational methods for drug synergy prediction have proven successful in narrowing down this space, fusing drug pairs and cell line features effectively still lacks study, hindering current algorithms from understanding the complex interaction between drugs and cell lines.ResultsIn this paper, we proposed a Permutable feature fusion network for Drug-Drug Synergy prediction, named PermuteDDS. PermuteDDS takes multiple representations of drugs and cell lines as input and employs a permutable fusion mechanism to combine drug and cell line features. In experiments, PermuteDDS exhibits state-of-the-art performance on two benchmark data sets. Additionally, the results on independent test set grouped by different tissues reveal that PermuteDDS has good generalization performance. We believed that PermuteDDS is an effective and valuable tool for identifying synergistic drug combinations. It is publicly available at https://github.com/littlewei-lazy/PermuteDDS.Scientific contributionFirst, this paper proposes a permutable feature fusion network for predicting drug synergy termed PermuteDDS, which extract diverse information from multiple drug representations and cell line representations. Second, the permutable fusion mechanism combine the drug and cell line features by integrating information of different channels, enabling the utilization of complex relationships between drugs and cell lines. Third, comparative and ablation experiments provide evidence of the efficacy of PermuteDDS in predicting drug-drug synergy.

Dual-aspect attention spatial-spectral transformer and hyperspectral imaging: A novel approach to detecting Aspergillus flavus contamination in peanut kernels

In this study, an innovative dual-aspect attention spatial-spectral transformer (DAASST) was introduced to advance postharvest quality control by the detection of Aspergillus flavus contamination and the accurate identification of contamination times in peanut kernels. The critical importance of maintaining postharvest quality and safety in nuts was recognized, with hyperspectral imaging technology being leveraged due to its great potential in non-destructive testing and quality assessment of nuts. At the heart of DAASST's innovation, an enhanced transformer architecture that incorporated an attention fusion mechanism was employed for the effective integration of the extracted features. This sophisticated integration not only improved the model’s performance but also was significantly surpassed by the capabilities of traditional machine learning methods in the context of postharvest biology and technology. Exceptional accuracy was demonstrated in testing, with 99.40% achieved in detecting Aspergillus flavus contamination and a remarkable 100% in distinguishing between different contamination times. Significant contributions to the field of postharvest biology and technology were made by merging cutting-edge feature extraction techniques, attention mechanisms, and transformer architecture to refine hyperspectral image analysis for postharvest quality control. The proven effectiveness of the DAASST in accurately detecting Aspergillus flavus and determining contamination times in peanut kernels highlighted its potential as a valuable tool for ensuring the safety and quality of postharvest nuts.

Improved Transformer-Based Deblurring of Commodity Videos in Dynamic Visual Cabinets

In the dynamic visual cabinet, the occurrence of motion blur when consumers take out commodities will reduce the accuracy of commodity detection. Recently, although Transformer-based video deblurring networks have achieved results compared to Convolutional Neural Networks in some blurring scenarios, they are still challenging for the non-uniform blurring problem that occurs when consumers pick up the commodities, such as the problem of difficult alignment of blurred video frames of small commodities and the problem of underutilizing the effective information between the video frames of commodities. Therefore, an improved Transformer video deblurring network is proposed. Firstly, a multi-scale Transformer feature extraction method is utilized for non-uniform blurring. Secondly, for the problem of difficult alignment of small-item-blurred video frames, a temporal interactive attention mechanism is designed for video frame alignment. Finally, a feature recurrent fusion mechanism is introduced to supplement the effective information of commodity features. The experimental results show that the proposed method has practical significance in improving the accuracy of commodity detection. Moreover, compared with the recent Transformer deblurring algorithm Video Restoration Transformer, the Peak Signal-to-Noise Ratio of this paper’s algorithm is higher than that of the Deep Video Deblurring dataset and the Fuzzy Commodity Dataset by 0.23 dB and 0.81 dB, respectively.

Complex overlapping pedestrian target detection network based on the yolov3 model

This paper proposes a complex overlapping pedestrian target detection model based on yolov3 model by multi-scale feature fusion and context-aware mechanism. The SONY A7R3a camera shot the model on campus, and the data set was obtained after editing and collating. There were 358 high-definition videos with a resolution of 1920*1080, and the frame rate was 50HZ, about 179,000 frames. Through testing, this paper finds that compared with Single Shot Multibox Detector (SSD), the detection accuracy of the newly proposed model is slightly improved, the detection accuracy is the same as that of Faster R-CNN, and the detection accuracy of the newly proposed model is slightly worse than that of RetinaNet. However, the detection speed of Yolov3 is more than twice that of Single Shot Multibox Detector, RetinaNet and Faster R-CNN. The input size of Yolov3 is 320*320, and the processing of a single image only needs 22ms, so the detection speed of the simplified Yolov3 tiny is faster.