Multimodal Features Research Articles

Time-segment-wise feature fusion transformer for multi-modal fault diagnosis

Mechanical fault diagnosis is crucial to ensure the safe operations of equipment in intelligent manufacturing systems. Recently, deep learning based fault diagnosis methods have achieved remarkable advancements with monitored data from a single sensor. However, obtaining satisfactory diagnostic results based on a single sensor is often difficult because the complementary information between different sensors is ignored. Extracting comprehensive fault features from multi-modal data is a problem that remains to be solved. To address these challenges, a time-segment-wise feature fusion Transformer (FFTR) is proposed in this paper. First, the signals from various modalities as multiple channels are normalized channel-by-channel and form a multi-modal sample. Second, a time-segment-wise feature learning network is designed to transform a multi-modal sample into several fusion features through the sequential processes of sample segmentation, segment-level feature extraction and time-aligned feature fusion. Finally, a Transformer network is employed for comprehensive multi-modal feature analysis and fault classification. In addition, a joint loss function is designed to comprehensively train the end-to-end FFTR. The comparison experiment with other baseline methods is conducted on two multi-modal datasets. The experimental results show that FFTR achieves 3.69% and 3.93% higher diagnostic accuracy than baseline on two datasets respectively and can address real-world problems effectively.

Cross-Modal Hashing Method With Properties of Hamming Space: A New Perspective.

Cross-modal hashing (CMH) has attracted considerable attention in recent years. Almost all existing CMH methods primarily focus on reducing the modality gap and semantic gap, i.e., aligning multi-modal features and their semantics in Hamming space, without taking into account the space gap, i.e., difference between the real number space and the Hamming space. In fact, the space gap can affect the performance of CMH methods. In this paper, we analyze and demonstrate how the space gap affects the existing CMH methods, which therefore raises two problems: solution space compression and loss function oscillation. These two problems eventually cause the retrieval performance deteriorating. Based on these findings, we propose a novel algorithm, namely Semantic Channel Hashing (SCH). First, we classify sample pairs into fully semantic-similar, partially semantic-similar, and semantic-negative ones based on their similarity and impose different constraints on them, respectively, to ensure that the entire Hamming space is utilized. Then, we introduce a semantic channel to alleviate the issue of loss function oscillation. Experimental results on three public datasets demonstrate that SCH outperforms the state-of-the-art methods. Furthermore, experimental validations are provided to substantiate the conjectures regarding solution space compression and loss function oscillation, offering visual evidence of their impact on the CMH methods.

Semantic-Driven Crossmodal Fusion for Multimodal Sentiment Analysis

In multimodal sentiment analysis (MSA), the fusion strategies of multimodal features significantly influence the performance of MSA models. Previous works frequently face challenges in integrating heterogeneous data without fully leveraging the rich semantic content of text, resulting in poor information association. We propose an MSA model based on Text-Driven Crossmodal Fusion and Mutual Information Estimation, called TeD-MI, which comprises a Stacked Text-Driven Crossmodal Fusion (STDC) module, which efficiently fusions the three modalities driven by the text modality to optimize fusion feature representation and enhance semantic understanding. Furthermore, TeD-MI designed a mutual information estimation module to achieve the best balance between preserving task-related information and filtering out irrelevant noise information as much as possible. Comprehensive experiments conducted on the CMU-MOSI and CMU-MOSEI datasets demonstrate our proposed model achieves varying degrees of improvement across most evaluation metrics.

Detecting functional impairment with the Digital Clock and Recall.

Distinguishing between mild cognitive impairment (MCI) and early dementia requires both neuropsychological and functional assessment that often relies on caregivers' insights. Contacting a patient's caregiver can be time-consuming in a physician's already-filled workday. To assess the utility of a brief, machine learning (ML)-enabled digital cognitive assessment, the Digital Clock and Recall (DCR), for detecting functional dependence. We evaluated whether the DCR can help identify individuals at risk of functional deficits as measured by the informant-rated Functional Activities Questionnaire (FAQ) in older individuals including cognitively unimpaired, MCI, and dementia likely due to Alzheimer's disease. The DCR scaled well with FAQ scores, and ML classifiers trained on multimodal DCR features demonstrated strong performance in predicting functional impairment on a held-out test set. Differences in FAQ scores between DCR-predicted classes were comparable across key demographic groups. The DCR can streamline the clinical decision-making, triage, and intervention planning associated with functional impairment in primary care.

Automatic segmentation-based multi-modal radiomics analysis of US and MRI for predicting disease-free survival of breast cancer: a multicenter study.

Several studies have confirmed the potential value of applying radiomics to predict prognosis of breast cancer. However, the tumor segmentation in these studies depended on delineation or annotation of breast cancer by radiologist, which is often laborious, tedious, and vulnerable to inter- and intra-observer variability. Automatic segmentation is expected to overcome this difficulty. Herein, we aim to investigate the value of automatic segmentation-based multi-modal radiomics signature and magnetic resonance imaging (MRI) features in predicting disease-free survival (DFS) of patients diagnosed with invasive breast cancer. This retrospective multicenter study included a total of 643 female patients with invasive breast cancer who underwent preoperative ultrasound (US) and MRI for prognostic analysis. Data (n = 480) from center 1 were divided into training and internal testing sets, while data (n = 163) from centers 2 and 3 were analyzed as the external testing set. We developed automatic segmentation frameworks for tumor segmentation by deep learning. Then, Least absolute shrinkage and selection operator Cox regression was used to select features to construct radiomics signature, and corresponding radiomics score (Rad-score) was calculated. Finally, six models for predicting DFS were constructed by using Cox regression and assessed in terms of discrimination, calibration, and clinical usefulness. The multi-modal radiomics signature combining intra- and peri-tumoral radiomics signatures of US and MRI achieved a higher C-index in the internal (0.734) and external (0.708) testing sets than most other radiomics signatures in predicting DFS, and successfully stratified patients into low- and high-risk groups. The multi-modal clinical imaging model combining the multi-modal Rad-score and clinical traditional MRI model-score resulted in a higher C-index (0.795) than other models in the external testing set, and it had a better calibration and higher clinical benefit. This study demonstrates that the multi-modal radiomics signature derived from automatic segmentations of US and MRI is a promising risk stratification biomarker for breast cancer, and highlights that the appropriate combination of multi-modal radiomics signature, clinical characteristics, and MRI feature can improve performance of individualized DFS prediction, which might assist in guiding decision-making related to breast cancer.

YOLOv8-seg-CP: a lightweight instance segmentation algorithm for chip pad based on improved YOLOv8-seg model.

Real-time detection and accurate segmentation of chip pads are important tasks to ensure chip alignment and position correction. To address the challenges of small target chip pad detection, segmentation accuracy and model lightweight, this paper proposes a lightweight chip pad instance segmentation algorithm based on an improved YOLOv8-seg, named YOLOv8-seg-CP (chip pad). Firstly, we integrate the next-generation lightweight StarNet into the original backbone network to enhance fine feature capture capabilities while reducing the number of parameters. Then, we construct the C2f-Star module in the neck network, which enhances the feature extraction performance for small chip pad targets. This maintains accuracy, reduces computational load, and improves detection and segmentation speed. On this basis, we introduce a lightweight shared convolution segmentation head (LSCSH), significantly reducing both parameter count and computational load while enhancing segmentation performance. Additionally, we propose a CGCAFusion convolutional attention fusion module. This module uses a content-guided convolutional attention fusion mechanism to dynamically adjust attention weights based on the content of input features, capturing both global and local feature information and enhancing multimodal feature fusion. Experiments on the chip pad dataset demonstrate that our algorithm achieves a detection and segmentation accuracy of 89.8%. The model size, parameters, and FLOPs are 3.7M, 1.7M, and 8.2G respectively, representing reductions of 45.6%, 50%, and 31.7% compared to the baseline model. The FPS is 1399.3, an improvement of 25.8% over the baseline model. The inference time is 0.72ms, which is 0.18ms less than the baseline model. Extensive experimental results on COCO, carparts-seg, and crack-seg datasets further show that the improved YOLOv8n-seg model outperforms many existing advanced methods in terms of performance. This approach holds significant industrial application value for fully automated chip testing and sorting integrated production.

Attention-enhanced multimodal feature fusion network for clothes-changing person re-identification

Clothes-Changing Person Re-Identification is a challenging problem in computer vision, primarily due to the appearance variations caused by clothing changes across different camera views. This poses significant challenges to traditional person re-identification techniques that rely on clothing features. These challenges include the inconsistency of clothing and the difficulty in learning reliable clothing-irrelevant local features. To address this issue, we propose a novel network architecture called the Attention-Enhanced Multimodal Feature Fusion Network (AE-Net). AE-Net effectively mitigates the impact of clothing changes on recognition accuracy by integrating RGB global features, grayscale image features, and clothing-irrelevant features obtained through semantic segmentation. Specifically, global features capture the overall appearance of the person; grayscale image features help eliminate the interference of color in recognition; and clothing-irrelevant features derived from semantic segmentation enforce the model to learn features independent of the person’s clothing. Additionally, we introduce a multi-scale fusion attention mechanism that further enhances the model’s ability to capture both detailed and global structures, thereby improving recognition accuracy and robustness. Extensive experimental results demonstrate that AE-Net outperforms several state-of-the-art methods on the PRCC and LTCC datasets, particularly in scenarios with significant clothing changes. On the PRCC and LTCC datasets, AE-Net achieves Top-1 accuracy rates of 60.4% and 42.9%, respectively.

An explainable longitudinal multi-modal fusion model for predicting neoadjuvant therapy response in women with breast cancer

Multi-modal image analysis using deep learning (DL) lays the foundation for neoadjuvant treatment (NAT) response monitoring. However, existing methods prioritize extracting multi-modal features to enhance predictive performance, with limited consideration on real-world clinical applicability, particularly in longitudinal NAT scenarios with multi-modal data. Here, we propose the Multi-modal Response Prediction (MRP) system, designed to mimic real-world physician assessments of NAT responses in breast cancer. To enhance feasibility, MRP integrates cross-modal knowledge mining and temporal information embedding strategy to handle missing modalities and remain less affected by different NAT settings. We validated MRP through multi-center studies and multinational reader studies. MRP exhibited comparable robustness to breast radiologists, outperforming humans in predicting pathological complete response in the Pre-NAT phase (ΔAUROC 14% and 10% on in-house and external datasets, respectively). Furthermore, we assessed MRP’s clinical utility impact on treatment decision-making. MRP may have profound implications for enrolment into NAT trials and determining surgery extensiveness.

A Lightweight and Efficient Multimodal Feature Fusion Network for Bearing Fault Diagnosis in Industrial Applications

To address the issues of single-structured feature input channels, insufficient feature learning capabilities in noisy environments, and large model parameter sizes in intelligent diagnostic models for mechanical equipment, a lightweight and efficient multimodal feature fusion convolutional neural network (LEMFN) method is proposed. Compared with existing models, LEMFN captures rich fault features at multiple scales by combining time-domain and frequency-domain signals, thereby enhancing the model’s robustness to noise and improving data adaptability under varying operating conditions. Additionally, the convolutional block attention module (CBAM) and random overlapping sampling technology (ROST) are introduced, and through a feature fusion strategy, the accurate diagnosis of mechanical equipment faults is achieved. Experimental results demonstrate that the proposed method not only possesses high diagnostic accuracy and rapid convergence but also exhibits strong robustness in noisy environments. Finally, a graphical user interface (GUI)-based mechanical equipment fault detection system was developed to promote the practical application of intelligent fault diagnosis in mechanical equipment.

Predicting radiation pneumonitis in lung cancer using machine learning and multimodal features: a systematic review and meta-analysis of diagnostic accuracy.

To evaluate the diagnostic accuracy of machine learning models incorporating multimodal features for predicting radiation pneumonitis in lung cancer through a systematic review and meta-analysis. Relevant studies were identified through a systematic search of PubMed, Web of Science, Embase, and the Cochrane Library from October 2003 to December 2023. Additional studies were located by reviewing bibliographies and relevant websites. Two independent researchers screened titles, abstracts, and full-text articles according to predefined inclusion and exclusion criteria. Data extraction was performed using standardized forms, and study quality was assessed using the Quality Assessment of Diagnostic Accuracy Studies-2 tool. The primary outcomes, including combined sensitivity, specificity, positive likelihood ratio (PLR), negative likelihood ratio (NLR), diagnostic odds ratio (DOR), and area under the curve (AUC), were calculated using STATA MP-64 software(Stata Corporation LLC, College Station, USA) with a random-effects model. Meta-analysis was conducted to synthesize diagnostic accuracy measures, and analyses of heterogeneity and publication bias were performed. A total of 1,406 patients with primary lung cancer were included in this systematic review, drawing data from 9 studies. The pooled analysis revealed a sensitivity of 0.74 [0.58-0.85] and a specificity of 0.91 [0.87-0.95] for machine learning models in diagnosing radiation pneumonitis. The positive likelihood ratio (PLR) was 8.69 [5.21-14.50], the negative likelihood ratio (NLR) was 0.28 [0.16-0.49], and the diagnostic odds ratio (DOR) was 30.73 [11.96-78.97]. The area under the curve (AUC) was 0.93 [0.90-0.95], indicating excellent diagnostic performance. Meta-regression analysis identified that the number of machine learning models, year of publication, and study design contributed to heterogeneity among studies. No evidence of publication bias was found. Overall, machine learning models incorporating multimodal characteristics demonstrated 75% accuracy in predicting moderate to severe radiation pneumonitis. In conclusion, by integrating the current machine learning (ML) algorithm's ability in big data mining, a predictive model can be constructed by combining multi-modal features such as genetics, imaging, and cell factors. By selecting multiple machine learning algorithm frameworks and competing for the best combination model based on research goals, the reliability and accuracy of the radiation pneumonitis prediction model can be greatly improved. PROSPERO (CRD42024497599).

Unsupervised Decision Trees for Axis Unimodal Clustering

The use of decision trees for obtaining and representing clustering solutions is advantageous, due to their interpretability property. We propose a method called Decision Trees for Axis Unimodal Clustering (DTAUC), which constructs unsupervised binary decision trees for clustering by exploiting the concept of unimodality. Unimodality is a key property indicating the grouping behavior of data around a single density mode. Our approach is based on the notion of an axis unimodal cluster: a cluster where all features are unimodal, i.e., the set of values of each feature is unimodal as decided by a unimodality test. The proposed method follows the typical top-down splitting paradigm for building axis-aligned decision trees and aims to partition the initial dataset into axis unimodal clusters by applying thresholding on multimodal features. To determine the decision rule at each node, we propose a criterion that combines unimodality and separation. The method automatically terminates when all clusters are axis unimodal. Unlike typical decision tree methods, DTAUC does not require user-defined hyperparameters, such as maximum tree depth or the minimum number of points per leaf, except for the significance level of the unimodality test. Comparative experimental results on various synthetic and real datasets indicate the effectiveness of our method.

Characterizing the role of the microbiota-gut-brain axis in cerebral small vessel disease: An integrative multi‑omics study

BackgroundPrior efforts have revealed changes in gut microbiome, circulating metabolome, and multimodal neuroimaging features in cerebral small vessel disease (CSVD). However, there is a paucity of research integrating the multi-omic information to characterize the role of the microbiota-gut-brain axis in CSVD. MethodsWe collected gut microbiome, fecal and blood metabolome, multimodal magnetic resonance imaging data from 37 CSVD patients with white matter hyperintensities and 46 healthy controls. Between-group comparison was performed to identify the differential gut microbial taxa, followed by performance of multi-stage microbiome-metabolome-neuroimaging-neuropsychology correlation analyses in CSVD patients. ResultsOur data showed both depleted and enriched gut microbes in CSVD patients. Among the differential microbes, Haemophilus and Akkermansia were associated with a range of metabolites enriched for Aminoacyl-tRNA biosynthesis pathway. Furthermore, the affected metabolites were associated with neuroimaging measures involving gray matter morphology, spontaneous intrinsic brain activity, white matter integrity, and global structural network topology, which were in turn related to cognition and emotion in CSVD patients. ConclusionOur findings provide an integrative framework to understand the pathophysiological mechanisms underlying the interplay between gut microbiota dysbiosis and CSVD, highlighting the potential of targeting the microbiota-gut-brain axis as a therapeutic strategy in CSVD patients.

MAMSC: a semantic enhanced representation model for public opinion key node recognition based on multianchor mapping in semantic communities

PurposeInformation is presented in various modalities such as text and images, and it can quickly and widely spread on social networks and among the general public through key communication nodes involved in public opinion events. Therefore, by tracking and identifying key nodes of public opinion, we can determine the direction of public opinion evolution and timely and effectively control public opinion events or curb the spread of false information.Design/methodology/approachThis paper introduces a novel multimodal semantic enhanced representation based on multianchor mapping semantic community (MAMSC) for identifying key nodes in public opinion. MAMSC consists of four core components: multimodal data feature extraction module, feature vector dimensionality reduction module, semantic enhanced representation module and semantic community (SC) recognition module. On this basis, we combine the method of community discovery in complex networks to analyze the aggregation characteristics of different semantic anchors and construct a three-layer network module for public opinion node recognition in the SC with strong, medium and weak associations.FindingsThe experimental results show that compared with its variants and the baseline models, the MAMSC model has better recognition accuracy. This study also provides more systematic, forward-looking and scientific decision-making support for controlling public opinion and curbing the spread of false information.Originality/valueWe creatively combine the construction of variant autoencoder with multianchor mapping to enhance semantic representation and construct a three-layer network module for public opinion node recognition in the SC with strong, medium and weak associations. On this basis, our constructed MAMSC model achieved the best results compared to the baseline models and ablation evaluation models, with a precision of 91.21%.

A Text Generation Method Based on a Multimodal Knowledge Graph for Fault Diagnosis of Consumer Electronics

As consumer electronics evolve towards greater intelligence, their automation and complexity also increase, making it difficult for users to diagnose faults when they occur. To address the problem where users, relying solely on their own knowledge, struggle to diagnose faults in consumer electronics promptly and accurately, we propose a multimodal knowledge graph-based text generation method. Our method begins by using deep learning models like the Residual Network (ResNet) and Bidirectional Encoder Representations from Transformers (BERT) to extract features from user-provided fault information, which can include images, text, audio, and even olfactory data. These multimodal features are then combined to form a comprehensive representation. The fused features are fed into a graph convolutional network (GCN) for fault inference, identifying potential fault nodes in the electronics. These fault nodes are subsequently fed into a pre-constructed knowledge graph to determine the final diagnosis. Finally, this information is processed through the Bias-term Fine-tuning (BitFit) enhanced Chinese Pre-trained Transformer (CPT) model, which generates the final fault diagnosis text for the user. The experimental results show that our proposed method achieves a 4.4% improvement over baseline methods, reaching a fault diagnosis accuracy of 98.4%. Our approach effectively leverages multimodal fault information, addressing the challenges users face in diagnosing faults through the integration of graph convolutional network and knowledge graph technologies.

Classification of Internal and External Distractions in an Educational VR Environment Using Multimodal Features.

Virtual reality (VR) can potentially enhance student engagement and memory retention in the classroom. However, distraction among participants in a VR-based classroom is a significant concern. Several factors, including mind wandering, external noise, stress, etc., can cause students to become internally and/or externally distracted while learning. To detect distractions, single or multi-modal features can be used. A single modality is found to be insufficient to detect both internal and external distractions, mainly because of individual variability. In this work, we investigated multi-modal features: eye tracking and EEG data, to classify the internal and external distractions in an educational VR environment. We set up our educational VR environment and equipped it for multi-modal data collection. We implemented different machine learning (ML) methods, including k-nearest-neighbors (kNN), Random Forest (RF), one-dimensional convolutional neural network - long short-term memory (1 D-CNN-LSTM), and two-dimensional convolutional neural networks (2D-CNN) to classify participants' internal and external distraction states using the multi-modal features. We performed cross-subject, cross-session, and gender-based grouping tests to evaluate our models. We found that the RF classifier achieves the highest accuracy over 83% in the cross-subject test, around 68% to 78% in the cross-session test, and around 90% in the gender-based grouping test compared to other models. SHAP analysis of the extracted features illustrated greater contributions from the occipital and prefrontal regions of the brain, as well as gaze angle, gaze origin, and head rotation features from the eye tracking data.

Development and validation of a cross-modality tensor fusion model using multi-modality MRI radiomics features and clinical radiological characteristics for the prediction of microvascular invasion in hepatocellular carcinoma

ObjectivesTo develop and validate a cross-modality tensor fusion (CMTF) model using multi-modality MRI radiomics features and clinical radiological characteristics for the prediction of microvascular invasion (MVI) in hepatocellular carcinoma (HCC). Materials and methodsThis study included 174 HCC patients (47 MVI-positive and 127 MVI-negative) confirmed by postoperative pathology. The synthetic minority over-sampling technique was used to augment MVI-positive samples. The amplified dataset of 254 samples (127 MVI-positive and 127 MVI-negative) was randomly divided into training and test cohorts in a 7:3 ratio. Radiomics features were respectively extracted from arterial phase, delayed phase, diffusion-weighted imaging, and fat-suppressed T2-weighted imaging. The least absolute shrinkage and selection operator was used for feature selection. Univariate and multivariate logistic regression analyses were employed to identify clinical and radiological independent predictors. The selected multi-modality MRI radiomics features, clinical and radiological characteristics were used to construct the CMTF model, single modality (SM) model, early fusion (EF) model. ResultsThe CMTF model demonstrated superior performance in predicting MVI compared to the SM and EF models. When integrating four MRI modalities, the CMTF model achieved a high area under the curve (AUC) with 95% confidence interval (95% CI) of 0.894 (0.820-0.968). Additionally, incorporating clinical and radiological characteristics further enhanced the predictive performance of CMTF model, the AUC (95% CI) value increased to 0.945 (0.892-0.998). ConclusionThe CMTF model showed promising performance in preoperative MVI prediction, providing a more effective non-invasive detection tool for HCC patients.

CSKINet: A multimodal network model integrating conceptual semantic knowledge injection for relation extraction of Chinese corporate reports

Recognizing the associations among entities in corporate reports accurately is crucial for market regulation and policy development. Nevertheless, confronted with massive corporate information, the traditional manual screening approach is cumbersome, struggling to match the demand. Consequently, we propose a multimodal network model incorporating conceptual semantic knowledge injection, CSKINet, for accurately extracting relations from Chinese corporate reports. The essential highlights in the design of the CSKINet model are the following: (1) Integrate the conceptual descriptions of corporations from external resources to construct the semantic knowledge repository of corporate concepts, which provides a solid semantic foundation for the model. (2) Multimodal features are extracted from the documents by various means and corporate conceptual knowledge is integrated into the model representation to enhance the representation capability of the model. (3) The multimodal self-attention mechanism that captures cross-modal associations and the biaffine classifier with Taylor polynomial loss function that optimizes training iterations further improve the learning efficiency and prediction accuracy. The results on the real corporate report dataset show that our proposed model can more accurately extract the relations from Chinese corporate reports compared to other baseline models, where the F1 score reaches 85.76%.

A novel multimodal image feature fusion mechanism:Application to rabbit liveweight estimation in commercial farms

In the realm of commercial rabbit farming, assessment of liveweight for each rabbit is crucial for production management. Traditional manual weighing methods are labor-intensive, time-consuming, and can induce stress and disease transmission. To solve this problem, this paper introduced a liveweight estimation method for rabbit utilizing near-infrared and depth image, including a segmentation stage and a weight estimation stage. Specifically, a dual-stream feature fusion mechanism was proposed to effectively integrate information from both near-infrared and depth imaging. The network was trained and validated on a dataset of 1,957 overhead images from 300 rabbits collected from a commercial farm, achieving superior performance with an R-Square (R2) of 0.95, Root Mean Square Error (RMSE) of 194.95 g, Mean Absolute Deviation (MAD) of 155.10 g, Mean Absolute Percentage Error (MAPE) of 4.08%, and Mean Coefficient of Variation (MCV) of 3.03%. Ablation experiments revealed that employing the proposed dual-stream feature fusion mechanism led to reductions of 52.87g, 38.19g, 1.01%, and 0.75% in RMSE, MAD, MAPE, and MCV, respectively, compared to not employing the mechanism. Compared to machine learning method, our proposed approach demonstrates superior performance in RMSE, MAD, MAPE, and MCV, achieving more accurate liveweight estimation for large-scale commercial farms.

Stock movement prediction in a hotel with multimodality and spatio-temporal features during the Covid-19 pandemic

Stock movement prediction in a hotel with multimodality and spatio-temporal features during the Covid-19 pandemic

Multimodal invariant feature prompt network for brain tumor segmentation with missing modalities

Multimodal invariant feature prompt network for brain tumor segmentation with missing modalities