Multi-mode Fusion Research Articles

Phase coherence weighted multi-mode total focusing method imaging of multi-angle defects

ABSTRACT The multi-mode total focusing method (MTFM) enables a region of interest within a specimen to be imaged using different combinations of beam paths and wave-mode. For a crack, the diffraction response of the tip may be evident in certain modes and the specular reflection in others. Accurate characterisation of defects requires combining information from multiple modes. For MTFM images, not all modes have a high signal-to-noise ratio (SNR). In this work, multi-mode phase coherence imaging (MPCI) was proposed by combining MTFM and phase coherence imaging (PCI) method. The better data fusion is mainly the result of the selective use of information from the most appropriate mode(s) and fusion reduces the probability of misjudging defects. In this paper, full matrix data was acquired for aluminium alloy blocks containing notch defects at different angles and imaged using the both multi-mode methods. The signal-to-noise ratio results of the two multimodal methods show that MPCI is better than MTFM for multi-angle defect detection. In addition, the Sobol sensitivity analysis method was used to construct the signal-to-noise ratio sensitivity parameter for quantifying the imaging effect of each mode on defects at different angles, and the results can effectively guide the optimisation of the multi-mode fusion algorithm.

A Composite Recognition Method Based on Multimode Mutual Attention Fusion Network

ABSTRACT To address the problem of single-mode vulnerability to complex environments, a multimode fusion network with mutual attention is proposed. This network combines the use of laser, infrared and millimeter wave modalities to leverage the advantages of each mode in different environments, increasing the network’s resilience to interference. The study begins with the construction of pixel-level fusion networks, feature-weighted fusion networks and the multimode mutual attention fusion network. A comprehensive introduction to the multimode mutual attention fusion network is given, as well as a comparison with the other two networks. The model is then trained and evaluated using data from glide rocket and drone experiments. Finally, an analysis of the anti-outlier interference capability of the multimode fusion network with mutual attention is carried out. The test results show that the multimode mutual attention fusion network containing a feature fusion attention mechanism has the highest detection performance and anti-interference ability. Without interference, the network achieves a remarkable accuracy of 0.98 for multi-target recognition. In addition, with an accuracy of 0.96, it ensures a high level of stability in various interference environments. In addition, the introduction of multi-scale fusion has improved the rocket’s speed adaptability by about 75%.

A Multimode Fusion-Based Aviation Communication System

This paper presents a new design for a multimode fusion communication system, aimed at tackling the complexities of modern aeronautical communication. The system integrates multiple communication technologies, such as ad hoc networking, 5G, BeiDou satellite, RTK positioning, and ADS-B broadcasting. This integration effectively solves the problem of increasing the size and weight of aviation communication equipment while also improving the efficiency and security of data communication. The study demonstrates that the implementation of this fusion communication system can lead to the development of more efficient and intelligent avionics equipment in the future, thereby offering robust technical support for flight safety.

Research on ultrasonic phased array total focusing multi-mode fusion method

Abstract Welded seams are one of the important structural forms in the construction of industrial equipment. But during use, hazardous defects may occur within the weld. Ultrasonic phased-array total focusing method (TFM) is a relatively new post-processing imaging method that can locate defects. In this study, a multi-mode ultrasonic phased-array total focusing fusion method was explored for weld defects to achieve effective quantification and localization of different types of defects. This study explores the application of different fusion methods such as maximum fusion, average fusion and effective region fusion in multi-mode phased array total focusing method. The experimental results show that the fusion methods can improve the accuracy of defect detection compared to single imaging methods, but the effective area fusion method has the best results in detecting weld defects. Taken together, the effective region fusion method can effectively improve the resolution and signal-to-noise ratio of multi-mode imaging methods.

Shape-position perceptive fusion electronic skin with autonomous learning for gesture interaction

Wearable devices, such as data gloves and electronic skins, can perceive human instructions, behaviors and even emotions by tracking a hand's motion, with the help of knowledge learning. The shape or position single-mode sensor in such devices often lacks comprehensive information to perceive interactive gestures. Meanwhile, the limited computing power of wearable applications restricts the multimode fusion of different sensing data and the deployment of deep learning networks. We propose a perceptive fusion electronic skin (PFES) with a bioinspired hierarchical structure that utilizes the magnetization state of a magnetostrictive alloy film to be sensitive to external strain or magnetic field. Installed at the joints of a hand, the PFES realizes perception of curvature (joint shape) and magnetism (joint position) information by mapping corresponding signals to the two-directional continuous distribution such that the two edges represent the contributions of curvature radius and magnetic field, respectively. By autonomously selecting knowledge closer to the user's hand movement characteristics, the reinforced knowledge distillation method is developed to learn and compress a teacher model for rapid deployment on wearable devices. The PFES integrating the autonomous learning algorithm can fuse curvature-magnetism dual information, ultimately achieving human machine interaction with gesture recognition and haptic feedback for cross-space perception and manipulation.

Intelligent Advanced Attack Detection Technology based on Multi-modal Data Fusion

This paper proposes an adaptive Wedman intrusion detection algorithm (AID-DFS) for data fusion. Firstly, feature extraction of abnormal text detection is carried out using a BI-gated loop (Bi-GRU). Multi-branch convolutional recurrent neural network (CNN-RNN) extracts hierarchical features from abnormal images. The multi-mode dynamic fusion uses the intermodal and intramodal attention mechanisms. In this way, a joint representation of multiple modes is obtained. The visual perception mechanism is used to realize multichannel integration and strengthen the function of original information in multichannel. The experimental results show that the proposed method has 99.6% accuracy and 94.9% accuracy. Compared with other algorithms, the proposed method can improve the performance of the intrusion system by about 10.2%.

Time-Frequency Aliased Signal Identification Based on Multimodal Feature Fusion.

The identification of multi-source signals with time-frequency aliasing is a complex problem in wideband signal reception. The traditional method of first separation and identification especially fails due to the significant separation error under underdetermined conditions when the degree of time-frequency aliasing is high. The single-mode recognition method does not need to be separated first. However, the single-mode features contain less signal information, making it challenging to identify time-frequency aliasing signals accurately. To solve the above problems, this article proposes a time-frequency aliasing signal recognition method based on multi-mode fusion (TRMM). This method uses the U-Net network to extract pixel-by-pixel features of the time-frequency and wave-frequency images and then performs weighted fusion. The multimodal feature scores are used as the classification basis to realize the recognition of the time-frequency aliasing signals. When the SNR is 0 dB, the recognition rate of the four-signal aliasing model can reach more than 97.3%.

Humanoid robot action using multi-mode fusion based on frame expert system

Humanoid robot action using multi-mode fusion based on frame expert system

Longitudinal tear detection method for conveyor belt based on multi-mode fusion

Longitudinal tear detection method for conveyor belt based on multi-mode fusion

Method of Cultivating College Students' Independent Learning Ability Based on Integration of Multiple Algorithm

INTRODUCTION: The research on the multi-mode fusion of college students' independent learning ability cultivation method is conducive to college students' change of learning mode and learning thinking, improvement of the utilization rate of educational resources, and the development of the academic environment as well as the reform of the educational concept.
 OBJECTIVES: Aiming at the problems of college students' current independent learning mode, such as the need for more in-depth research and the single study means.
 METHODS: A method for cultivating college students' autonomous learning ability through the integration of intelligent optimization algorithms and multiple modes has been proposed. Firstly, the practices of analyzing the current college students' autonomous learning mode and multiple learning modes are analyzed; then, using the butterfly optimization algorithm, a weight optimization method for the cultivation of college students' independent learning ability based on the fusion of multiple modes is proposed; finally, the validity and robustness of the proposed method are verified through experimental analysis.
 RESULTS: The results show that the proposed method has a high cultivation effect.
 CONCLUSION: Solves the problem of fusion of college students' independent learning ability cultivation modes.

Application of aptamer-functionalized nanomaterials in molecular imaging of tumors

Abstract Cancer is one of the most important causes of human death. Early diagnosis and treatment can make patients live longer. Therefore, there is an urgent need to develop early and accurate diagnosis method for tumors. Molecular imaging technology can be used for qualitative and quantitative analyses at cellular and molecular levels, which provides a new technology for accurate diagnosis of tumors. In recent years, various nanomaterials with unique properties have been used for tumor molecular imaging. Meanwhile, aptamers are becoming an indispensable element in the design of functional nanomaterials because of their small size, high stability, and convenient modification, especially giving nanomaterials the ability to recognize specific targets. Therefore, aptamer-functionalized nanomaterials (AFNs) provide unprecedented opportunities for the field of tumor diagnosis. Here we focus on the latest development of AFNs in the molecular imaging of tumors. First, we introduce the characteristics and advantages of common aptamer-modified organic nanomaterials and inorganic nanomaterials. Then, the applications of AFNs in fluorescence imaging, computed tomography, magnetic resonance imaging, radionuclide imaging, ultrasound imaging, photoacoustic imaging, and multimode fusion imaging are discussed. Finally, we provide some perspectives on the challenges and opportunities that have arisen from this promising area.

Intelligent Fault Diagnosis of Variable-Condition Motors Using a Dual-Mode Fusion Attention Residual

Electric motors play a crucial role in ship systems. Detecting potential issues with electric motors is a critical aspect of ship fault diagnosis. Fault diagnosis in motors is often challenging due to limited and noisy vibration signals. Existing deep learning methods struggle to extract the underlying correlation between samples while being susceptible to noise interference during the feature extraction process. To overcome these issues, this study proposes an intelligent bimodal fusion attention residual model. Firstly, the vibration signal to be encoded undergoes demodulation and is divided into high and low frequencies using the IEEMD (Improved Ensemble Empirical Mode Decomposition) composed of the EEMD (Ensemble Empirical Mode Decomposition) and the MASM (the Mean of the Standardized Accumulated Modes). Subsequently, the high-frequency component is effectively denoised using the wavelet packet threshold method. Secondly, current data and vibration signals are transformed into two-dimensional images using the Gramian Angular Summation Field (GASF) and aggregated into a bimodal Gramian Angle Field diagram. Finally, the proposed model incorporates the Self-Attention Squeeze-and-Excitation Networks (SE) mechanism with the Swish activation function and utilizes the ResNeXt architecture with a Dropout layer to identify and diagnose faults in the multi-mode fusion dataset of motors under various working conditions. Based on the experimental results, a comprehensive discussion and analysis were conducted to evaluate the performance of the proposed intelligent bimodal fusion attention residual model. The results demonstrated that, in comparison to traditional methods and other deep learning models, the proposed model effectively utilized multimodal data, thereby enhancing the accuracy and robustness of fault diagnosis. The introduction of attention mechanisms and residual learning enable the model to focus more effectively on crucial modal data and learn the correlations between modalities, thus improving the overall performance of fault diagnosis.

An online diagnosis method for cancer lesions based on intelligent imaging analysis.

With the popularization and application of artificial intelligence and medical image big data in the field of medical image, the universality of modes and the rapid development of deep learning have endowed multi-mode fusion technology with great development potential. Technologies of 5G and artificial intelligence have rapidly promoted the innovation of online hospitals. To assist doctors in the remote diagnosis of cancer lesions, this article proposes a cancer localization and recognition model based on magnetic resonance images. We combine a convolution neural network with Transformer to achieve local features and global context information, which can suppress the interference of noise and background regions in magnetic resonance imaging. We design a module combining convolutional neural networks and Transformer architecture, which interactively fuses the extracted features to increase the cancer localization accuracy of magnetic resonance imaging (MRI) images. We extract tumor regions and perform feature fusion to further improve the interactive ability of features and achieve cancer recognition. Our model can achieve an accuracy of 88.65%, which means our model can locate cancer regions in MRI images and effectively identify them. Furthermore, our model can be embedded into the online hospital system by 5G technology to provide technical support for the construction of network hospitals.

Deep Transfer Learning-Based Multi-Modal Digital Twins for Enhancement and Diagnostic Analysis of Brain MRI Image.

it aims to adopt deep transfer learning combined with Digital Twins (DTs) in Magnetic Resonance Imaging (MRI) medical image enhancement. MRI image enhancement method based on metamaterial composite technology is proposed by analyzing the application status of DTs in medical direction and the principle of MRI imaging. On the basis of deep transfer learning, MRI super-resolution deep neural network structure is established. To address the problem that different medical imaging methods have advantages and disadvantages, a multi-mode medical image fusion algorithm based on adaptive decomposition is proposed and verified by experiments. the optimal Peak Signal to Noise Ratio (PSNR) of 34.11dB can be obtained by introducing modified linear element and loss function of deep transfer learning neural network structure. The Structural Similarity Coefficient (SSIM) is 85.24%. It indicates that the MRI truthfulness and sharpness obtained by adding composite metasurface are improved greatly. The proposed medical image fusion algorithm has the highest overall score in the subjective evaluation of the six groups of fusion image results. Group III had the highest score in Magnetic Resonance Imaging- Positron Emission Computed Tomography (MRI-PET) image fusion, with a score of 4.67, close to the full score of 5. As for the objective evaluation in group I of Magnetic Resonance Imaging- Single Photon Emission Computed Tomography (MRI-SPECT) images, the Root Mean Square Error (RMSE), Relative Average Spectral Error (RASE) and Spectral Angle Mapper (SAM) are the highest, which are 39.2075, 116.688, and 0.594, respectively. Mutual Information (MI) is 5.8822. the proposed algorithm has better performance than other algorithms in preserving spatial details of MRI images and color information direction of SPECT images, and the other five groups have achieved similar results.

Multi-antenna intelligent spectrum sensing in the presence of non-Gaussian interference

The secondary user (SU) only detects one feature of the primary user (PU) signal in conventional spectrum sensing. However, single feature detection does not fully explore the PU information. Meanwhile, non-Gaussian interference will also affect the spectrum sensing performance. In this paper, we propose a novel spectrum sensing method using multimodal fusion and convolutional neural networks (CNN). First, the received signal at multiple antennas is preprocessed, where the generalized covariance matrix and the generalized Wigner-Ville distribution are used to characterize two different modes of the received signals as the input of CNN. Then, a CNN model using multimodal fusion is constructed. In this model, the attention mechanism is first used to extract features of different modalities, and the multimodal fusion technique fuses the extracted features to obtain the feature vector. Finally, the detection statistics and detection thresholds for spectrum sensing are constructed and compared using the output feature vectors of CNN. Simulation results show that the performance of multi-mode fusion spectrum sensing is better than the single-mode counterpart. This method can effectively realize spectrum sensing with both Gaussian noise and non-Gaussian interference, and it has a high detection probability at low SNR.

Rapid Assessment of Fish Freshness for Multiple Supply-Chain Nodes Using Multi-Mode Spectroscopy and Fusion-Based Artificial Intelligence.

This study is directed towards developing a fast, non-destructive, and easy-to-use handheld multimode spectroscopic system for fish quality assessment. We apply data fusion of visible near infra-red (VIS-NIR) and short wave infra-red (SWIR) reflectance and fluorescence (FL) spectroscopy data features to classify fish from fresh to spoiled condition. Farmed Atlantic and wild coho and chinook salmon and sablefish fillets were measured. Three hundred measurement points on each of four fillets were taken every two days over 14 days for a total of 8400 measurements for each spectral mode. Multiple machine learning techniques including principal component analysis, self-organized maps, linear and quadratic discriminant analyses, k-nearest neighbors, random forest, support vector machine, and linear regression, as well as ensemble and majority voting methods, were used to explore spectroscopy data measured on fillets and to train classification models to predict freshness. Our results show that multi-mode spectroscopy achieves 95% accuracy, improving the accuracies of the FL, VIS-NIR and SWIR single-mode spectroscopies by 26, 10 and 9%, respectively. We conclude that multi-mode spectroscopy and data fusion analysis has the potential to accurately assess freshness and predict shelf life for fish fillets and recommend this study be expanded to a larger number of species in the future.

Multi-mode information fusion navigation system for robot-assisted vascular interventional surgery

BackgroundMinimally invasive vascular intervention (MIVI) is a powerful technique for the treatment of cardiovascular diseases, such as abdominal aortic aneurysm (AAA), thoracic aortic aneurysm (TAA) and aortic dissection (AD). Navigation of traditional MIVI surgery mainly relies only on 2D digital subtraction angiography (DSA) images, which is hard to observe the 3D morphology of blood vessels and position the interventional instruments. The multi-mode information fusion navigation system (MIFNS) proposed in this paper combines preoperative CT images and intraoperative DSA images together to increase the visualization information during operations.ResultsThe main functions of MIFNS were evaluated by real clinical data and a vascular model. The registration accuracy of preoperative CTA images and intraoperative DSA images were less than 1 mm. The positioning accuracy of surgical instruments was quantitatively assessed using a vascular model and was also less than 1 mm. Real clinical data used to assess the navigation results of MIFNS on AAA, TAA and AD.ConclusionsA comprehensive and effective navigation system was developed to facilitate the operation of surgeon during MIVI. The registration accuracy and positioning accuracy of the proposed navigation system were both less than 1 mm, which met the accuracy requirements of robot assisted MIVI.

Action recognition based on multimode fusion for VR online platform.

The current popular online communication platforms can convey information only in the form of text, voice, pictures, and other electronic means. The richness and reliability of information is not comparable to traditional face-to-face communication. The use of virtual reality (VR) technology for online communication is a viable alternative to face-to-face communication. In the current VR online communication platform, users are in a virtual world in the form of avatars, which can achieve "face-to-face" communication to a certain extent. However, the actions of the avatar do not follow the user, which makes the communication process less realistic. Decision-makers need to make decisions based on the behavior of VR users, but there are no effective methods for action data collection in VR environments. In our work, three modalities of nine actions from VR users are collected using a virtual reality head-mounted display (VR HMD) built-in sensors, RGB cameras and human pose estimation. Using these data and advanced multimodal fusion action recognition networks, we obtained a high accuracy action recognition model. In addition, we take advantage of the VR HMD to collect 3D position data and design a 2D key point augmentation scheme for VR users. Using the augmented 2D key point data and VR HMD sensor data, we can train action recognition models with high accuracy and strong stability. In data collection and experimental work, we focus our research on classroom scenes, and the results can be extended to other scenes.

Study on recognition of coal and gangue based on multimode feature and image fusion.

Aiming at the problems of low accuracy of coal gangue recognition and difficult recognition of mixed gangue rate, a coal rock recognition method based on modal fusion of RGB and infrared is proposed. A fully mechanized coal gangue transportation test bed is built, RGB images are obtained by camera, and infrared images are obtained by industrial microwave heating system and infrared thermal imager. the image data of the whole coal, whole gangue, and coal gangue with different gangue mixing as training and test samples, identify the released coal gangue and its mixing rate. The AlexNet, VGG-16, ResNet-18 classification networks and their convolutional neural networks with modal feature fusion are constructed. results: The classification accuracy of ResNet networks on RGB and infrared image data is higher than AlexNet and VGG-16 networks. The early convergence network performance of ResNet is verified through the convergence of different models. The recognition rate of the network is 97.92 the confusion matrix statistics, which verifies the feasibility of the application of modal fusion method in the field of coal gangue recognition. The fusion of modal features and early models of ResNet coal gangue, which is the basic premise for realizing intelligent coal caving.

A feasibility study of multi-mode intelligent fusion medical data transmission technology of industrial Internet of Things combined with medical Internet of Things

As a branch of the Internet of Things (IoT) dedicated to medical care industry, the Internet of Medical Things (IoMT) includes networking devices and applications for medical and health information technology applications. The IoMT still has many shortcomings, such as unstable information transmission, low accuracy of fault diagnosis and classification, and lack of anomaly detection capability. Therefore, this paper first comprehensively describes some relatively new research and popular background technologies of the Industrial Internet of Things (IIoT) and IoMT, and then, according to the functional requirements analysis of the intelligent data transmission system based on IoMT, we discuss in detail the problems of fault diagnosis and resource allocation faced in the IoMT model. An intelligent data transmission model is proposed to apply the wireless communication transmission technology of the industrial Internet of Things to the Internet of Medical Things scene. This model has the ability of high-quality data transmission, high accuracy accident diagnosis classification and real-time anomaly monitoring, which makes up for the shortcomings of traditional IoMT models. In particular, for the accident diagnosis and classification functions in this model, we innovatively adopt the multi-mode data fusion CNN algorithm. The experimental results show that the classification accuracy of the accident diagnosis results is effectively improved. Compared with other algorithms, this model with multi-mode data fusion CNN algorithm improves the data transmission rate, reduces the average data delay, and improves the real-time anomaly monitoring capability of IoMT without increasing the signal leakage rate, thus further improving the overall stability of IoMT.