Tensor Fusion Research Articles

Development and validation of a cross-modality tensor fusion model using multi-modality MRI radiomics features and clinical radiological characteristics for the prediction of microvascular invasion in hepatocellular carcinoma

ObjectivesTo develop and validate a cross-modality tensor fusion (CMTF) model using multi-modality MRI radiomics features and clinical radiological characteristics for the prediction of microvascular invasion (MVI) in hepatocellular carcinoma (HCC). Materials and methodsThis study included 174 HCC patients (47 MVI-positive and 127 MVI-negative) confirmed by postoperative pathology. The synthetic minority over-sampling technique was used to augment MVI-positive samples. The amplified dataset of 254 samples (127 MVI-positive and 127 MVI-negative) was randomly divided into training and test cohorts in a 7:3 ratio. Radiomics features were respectively extracted from arterial phase, delayed phase, diffusion-weighted imaging, and fat-suppressed T2-weighted imaging. The least absolute shrinkage and selection operator was used for feature selection. Univariate and multivariate logistic regression analyses were employed to identify clinical and radiological independent predictors. The selected multi-modality MRI radiomics features, clinical and radiological characteristics were used to construct the CMTF model, single modality (SM) model, early fusion (EF) model. ResultsThe CMTF model demonstrated superior performance in predicting MVI compared to the SM and EF models. When integrating four MRI modalities, the CMTF model achieved a high area under the curve (AUC) with 95% confidence interval (95% CI) of 0.894 (0.820-0.968). Additionally, incorporating clinical and radiological characteristics further enhanced the predictive performance of CMTF model, the AUC (95% CI) value increased to 0.945 (0.892-0.998). ConclusionThe CMTF model showed promising performance in preoperative MVI prediction, providing a more effective non-invasive detection tool for HCC patients.

Graph-based stock prediction with multisource information and relational data fusion

With the application of multisource information in different fields, the combination of different types of information, such as numerical data and text information, has become a favourable choice for performing stock market analyses. Despite the rich information provided by multisource data, building structured relationships remains challenging. In addition, some market relationship-based analysis methods use a predefined graph structure as a stock relationship graph, which makes it impossible to sensitively aggregate attribute features, and these methods cannot dynamically update market relationships or relationship strengths. In this paper, we propose a novel dynamic attribute-driven graph attention network incorporating sentiment (AGATS) information, transaction data, and text data. Inspired by behavioural finance, we separately extract sentiment information as a factor of technical indicators, and further realize the early fusion of technical indicators and textual data through tensor fusion. In particular, real-time intramarket dependencies and key attribute information are captured with graph networks, enabling dynamic relationship and relationship strength updates. Experiments conducted on real datasets show that our model is capable of ourperforming previously developed methods in prediction and trading.

Multi-kernel clustering with tensor fusion on Grassmann manifold for high-dimensional genomic data

The high dimensionality and noise challenges in genomic data make it difficult for traditional clustering methods. Existing multi-kernel clustering methods aim to improve the quality of the affinity matrix by learning a set of base kernels, thereby enhancing clustering performance. However, directly learning from the original base kernels presents challenges in handling errors and redundancies when dealing with high-dimensional data, and there is still a lack of feasible multi-kernel fusion strategies. To address these issues, we propose a Multi-Kernel Clustering method with Tensor fusion on Grassmann manifolds, called MKCTM. Specifically, we maximize the clustering consensus among base kernels by imposing tensor low-rank constraints to eliminate noise and redundancy. Unlike traditional kernel fusion approaches, our method fuses learned base kernels on the Grassmann manifold, resulting in a final consensus matrix for clustering. We integrate tensor learning and fusion processes into a unified optimization model and propose an effective iterative optimization algorithm for solving it. Experimental results on ten datasets, comparing against 12 popular baseline clustering methods, confirm the superiority of our approach. Our code is available at https://github.com/foureverfei/MKCTM.git.

Unsupervised Hyperspectral and Multispectral Image Blind Fusion Based on Deep Tucker Decomposition Network With Spatial-Spectral Manifold Learning.

Hyperspectral image (HSI) and multispectral image (MSI) fusion aims to generate high spectral and spatial resolution hyperspectral image (HR-HSI) by fusing high-resolution multispectral image (HR-MSI) and low-resolution hyperspectral image (LR-HSI). However, existing fusion methods encounter challenges such as unknown degradation parameters, and incomplete exploitation of the correlation between high-dimensional structures and deep image features. To overcome these issues, in this article, an unsupervised blind fusion method for LR-HSI and HR-MSI based on Tucker decomposition and spatial-spectral manifold learning (DTDNML) is proposed. We design a novel deep Tucker decomposition network that maps LR-HSI and HR-MSI into a consistent feature space, achieving reconstruction through decoders with shared parameters. To better exploit and fuse spatial-spectral features in the data, we design a core tensor fusion network (CTFN) that incorporates a spatial-spectral attention mechanism for aligning and fusing features at different scales. Furthermore, to enhance the capacity to capture global information, a Laplacian-based spatial-spectral manifold constraint is introduced in shared-decoders. Sufficient experiments have validated that this method enhances the accuracy and efficiency of hyperspectral and multispectral fusion on different remote sensing datasets. The source code is available at https://github.com/Shawn-H-Wang/DTDNML.

Improving compound-protein interaction prediction by focusing on intra-modality and inter-modality dynamics with a multimodal tensor fusion strategy

Identifying novel compound–protein interactions (CPIs) plays a pivotal role in target identification and drug discovery. Although the recent multimodal methods have achieved outstanding advances in CPI prediction, they fail to effectively learn both intra-modality and inter-modality dynamics, which limits their prediction performance. To address the limitation, we propose a novel multimodal tensor fusion CPI prediction framework, named MMTF-CPI, which contains three unimodal learning modules for structure, heterogeneous network and transcriptional profiling modalities, a tensor fusion module and a prediction module. MMTF-CPI is capable of focusing on both intra-modality and inter-modality dynamics with the tensor fusion module. We demonstrated that MMTF-CPI is superior to multiple state-of-the-art multimodal methods across seven datasets. The prediction performance of MMTF-CPI is significantly improved with the tensor fusion module compared to other fusion methods. Moreover, our case studies confirmed the practical value of MMTF-CPI in target identification. Via MMTF-CPI, we also discovered several candidate compounds for the therapy of breast cancer and non-small cell lung cancer.

Bearing Fault Diagnosis Method Based on Multiple-Level Feature Tensor Fusion

Bearing Fault Diagnosis Method Based on Multiple-Level Feature Tensor Fusion

Learning Spatiotemporal Brain Dynamics in Adolescents via Multimodal MEG and fMRI Data Fusion Using Joint Tensor/Matrix Decomposition.

Brain function is understood to be regulated by complex spatiotemporal dynamics, and can be characterized by a combination of observed brain response patterns in time and space. Magnetoencephalography (MEG), with its high temporal resolution, and functional magnetic resonance imaging (fMRI), with its high spatial resolution, are complementary imaging techniques with great potential to reveal information about spatiotemporal brain dynamics. Hence, the complementary nature of these imaging techniques holds much promise to study brain function in time and space, especially when the two data types are allowed to fully interact. We employed coupled tensor/matrix factorization (CMTF) to extract joint latent components in the form of unique spatiotemporal brain patterns that can be used to study brain development and function on a millisecond scale. Using the CMTF model, we extracted distinct brain patterns that revealed fine-grained spatiotemporal brain dynamics and typical sensory processing pathways informative of high-level cognitive functions in healthy adolescents. The components extracted from multimodal tensor fusion possessed better discriminative ability between high- and low-performance subjects than single-modality data-driven models. Multimodal tensor fusion successfully identified spatiotemporal brain dynamics of brain function and produced unique components with high discriminatory power. The CMTF model is a promising tool for high-order, multimodal data fusion that exploits the functional resolution of MEG and fMRI, and provides a comprehensive picture of the developing brain in time and space.

Joint low-rank tensor fusion and cross-modal attention for multimodal physiological signals based emotion recognition

Objective. Physiological signals based emotion recognition is a prominent research domain in the field of human-computer interaction. Previous studies predominantly focused on unimodal data, giving limited attention to the interplay among multiple modalities. Within the scope of multimodal emotion recognition, integrating the information from diverse modalities and leveraging the complementary information are the two essential issues to obtain the robust representations. Approach. Thus, we propose a intermediate fusion strategy for combining low-rank tensor fusion with the cross-modal attention to enhance the fusion of electroencephalogram, electrooculogram, electromyography, and galvanic skin response. Firstly, handcrafted features from distinct modalities are individually fed to corresponding feature extractors to obtain latent features. Subsequently, low-rank tensor is fused to integrate the information by the modality interaction representation. Finally, a cross-modal attention module is employed to explore the potential relationships between the distinct latent features and modality interaction representation, and recalibrate the weights of different modalities. And the resultant representation is adopted for emotion recognition. Main results. Furthermore, to validate the effectiveness of the proposed method, we execute subject-independent experiments within the DEAP dataset. The proposed method has achieved the accuracies of 73.82% and 74.55% for valence and arousal classification. Significance. The results of extensive experiments verify the outstanding performance of the proposed method.

Optimizing Emotional Insight through Unimodal and Multimodal Long Short-term Memory Models

The field of multimodal emotion recognition is increasingly gaining popularity as a research area. It involves analyzing human emotions across multiple modalities, such as acoustic, visual, and language. Emotion recognition is more effective as a multimodal learning task than relying on a single modality. In this paper, we present an unimodal and multimodal long short-term memory model with a class weight parameter technique for emotion recognition on the CMU-Multimodal Opinion Sentiment and Emotion Intensity dataset. In addition, a critical challenge lies in selecting the most effective fusion method for integrating multiple modalities. To address this, we applied four different fusion techniques: Early fusion, late fusion, deep fusion, and tensor fusion. These fusion methods improved the performance of multimodal emotion recognition compared to unimodal approaches. With the highly imbalanced number of samples per emotion class in the MOSEI dataset, adding a class weight parameter technique leads our model to outperform the state of the art on all three modalities — acoustic, visual, and language — as well as on all the fusion models. The challenges of class imbalance, which can lead to biased model performance, and using an effective fusion method for integrating multiple modalities often result in decreased accuracy in recognizing less frequent emotion classes. Our proposed model shows 2–3% performance improvement in the unimodal and 2% in the multimodal over the state-of-the-art achieved results.

Prediction model of early recurrence of multimodal hepatocellular carcinoma with tensor fusion

Objective. In oncology, clinical decision-making relies on a multitude of data modalities, including histopathological, radiological, and clinical factors. Despite the emergence of computer-aided multimodal decision-making systems for predicting hepatocellular carcinoma (HCC) recurrence post-hepatectomy, existing models often employ simplistic feature-level concatenation, leading to redundancy and suboptimal performance. Moreover, these models frequently lack effective integration with clinically relevant data and encounter challenges in integrating diverse scales and dimensions, as well as incorporating the liver background, which holds clinical significance but has been previously overlooked. Approach. To address these limitations, we propose two approaches. Firstly, we introduce the tensor fusion method to our model, which offers distinct advantages in handling multi-scale and multi-dimensional data fusion, potentially enhancing overall performance. Secondly, we pioneer the consideration of the liver background’s impact, integrating it into the feature extraction process using a deep learning segmentation-based algorithm. This innovative inclusion aligns the model more closely with real-world clinical scenarios, as the liver background may contain crucial information related to postoperative recurrence. Main results. We collected radiomics (MRI) and histopathological images from 176 cases diagnosed by experienced clinicians across two independent centers. Our proposed network underwent training and 5-fold cross-validation on this dataset before validation on an external test dataset comprising 40 cases. Ultimately, our model demonstrated outstanding performance in predicting early recurrence of HCC postoperatively, achieving an AUC of 0.883. Significance. These findings signify significant progress in addressing challenges related to multimodal data fusion and hold promise for more accurate clinical outcome predictions. In this study, we exploited global 3D liver background into modelling which is crucial to to the prognosis assessment and analyzed the whole liver background in addition to the tumor region. Both MRI images and histopathological images of HCC were fused at high-dimensional feature space using tensor techniques to solve cross-scale data integration issue.

Abstract 890: Multimodal modeling of digitized histopathology slides improves risk stratification in hormone receptor-positive breast cancer patients

Abstract In early-stage hormone receptor-positive breast cancer, genomic risk scores identify patients who stand to benefit from up-front chemotherapy but introduce financial and logistical hurdles to care. We assembled a cohort of 5,244 patients with 11,671 corresponding whole-side images of breast tumors stained with hematoxylin and eosin. We developed a multimodal machine learning model to infer risk of distal metastatic recurrence from routine clinical data. Specifically, the model interprets text from the pathologist’s report using a large language model and uses self-supervised vision transformers to interpret the corresponding whole-slide image. Tensor fusion joins the modalities to infer Genomic Health’s Oncotype DX recurrence score. Inferred recurrence score from the multimodal model correlated with measured score with a concordance correlation coefficient of 0.64 (95% C.I. 0.59 - 0.69) in the withheld test set, compared to 0.55 (95% C.I. 0.49 - 0.61) and 0.56 (95% C.I. 0.52 - 0.60) for the linguistic and visual unimodal models, respectively. The multimodal model attains an area under the precision-recall curve (AUPRC) of 0.69 (AUROC=0.88) for identifying high-risk disease in the full-information setting (when images and pathology reports with quantitative hormone receptor status and grade are available) in a withheld test set, compared to AUPRC of 0.61 and 0.66 for the linguistic and visual models, respectively. By comparison, in the same full-information setting, the clinical nomogram introduced by Orucevic et al. in 2019 achieves an AUPRC of 0.48. We suggest the operating point at which precision is 94.4% and recall is 33.3%. Digitized whole-slide images of routine breast biopsies and their associated synoptic pathology reports contain much of the information necessary to stratify patients by risk of distal metastatic recurrence, when modeled appropriately. Our model could enable hospitals to rapidly triage the need for genomic risk testing, possibly precluding one third of orders without loss of accuracy. This helps allocate scarce resources for genomic tests and valuable weeks prior to beginning therapy while maintaining the standard of precision oncology. Citation Format: Kevin M. Boehm, Antonio Marra, Jorge S. Reis-Filho, Sarat Chandarlapaty, Fresia Pareja, Sohrab P. Shah. Multimodal modeling of digitized histopathology slides improves risk stratification in hormone receptor-positive breast cancer patients [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl):Abstract nr 890.

A fusion TFDAN-Based framework for rotating machinery fault diagnosis under noisy labels

Traditional fault diagnosis (FD) for rotating machinery solely based on vibration signals has problems such as inconvenient collection, low accuracy, and poor robustness. This article proposes a fusion framework based on tensor fusion and dual attention network (TFDAN), utilizing acoustic and vibration signals from two datasets of centrifugal pumps and cylindrical roller bearings. Firstly, continuous wavelet transform (CWT) is used to transform two original signals into two-dimensional time–frequency maps to highlight time–frequency features. Then, the images are fed into the fusion framework, where tensor fusion can construct the corresponding time–frequency maps of the working conditions into multi-channel datasets, enhancing the feature connections between acoustic and vibration signals. The dual attention network takes on the fused samples, extracts local features of the image using its positional attention module, and further aggregates feature correlations between channels using its channel attention mechanism. Finally, in order to simulate the actual production situation, different proportions of noisy labels are added to the dataset. In response to the impact of noisy labels, we incorporate an improved contrastive regularization function (ICRF) into the model, fully utilizing its advantage of preventing overfitting of noisy labels. The effectiveness of our proposed method has been demonstrated through two experimental cases. Compared with other methods, our method has better performance in terms of diagnostic accuracy and robustness to noisy labels.

Joint Entity and Relation Extraction Model Based on Inner and Outer Tensor Dot Product and Single-Table Filling

Joint relational triple extraction is a crucial step in constructing a knowledge graph from unstructured text. Recently, multiple methods have been proposed for extracting relationship triplets. Notably, end-to-end table-filling methods have garnered significant research interest due to their efficient extraction capabilities. However, existing approaches usually generate separate tables for each relationship, which neglects the global correlation between relationships and context, producing a large number of useless blank tables. This problem results in issues of redundant information and sample imbalance. To address these challenges, we propose a novel framework for joint entity and relation extraction based on a single-table filling method. This method incorporates all relationships as prompts within the text sequence and associates entity span information with relationship labels. This approach reduces the generation of redundant information and enhances the extraction capability for overlapping triplets. We utilize the internal and external multi-head tensor fusion approach to generate two sets of table feature vectors. These vectors are subsequently merged to capture a wider range of global information. Experimental results on the NYT and WebNLG datasets demonstrate the effectiveness of our proposed model, which maintains excellent performance, even in complex scenarios involving overlapping triplets.

Evaluation method for insulation degradation of power transformer windings based on incomplete internet of things sensing data

AbstractThis paper proposes a novel evaluation method to address the challenge of evaluating insulation degradation in power transformer windings based on incomplete online Internet of Things (IoT) sensing data. The method leverages the Wasserstein Slim Generative Adversarial Imputation Network with Gradient Penalty algorithm to fill the irregularly missing power transformer IoT perception data, including voltage, current, temperature, and partial discharge. Subsequently, electrical, thermal, and mechanical performance degradation damage indicators for transformer winding insulation are constructed using the filled and complete IoT perception data. By applying the tensor fusion algorithm, the characteristics of these degradation damage indicators are fused, leading to the development of a comprehensive degradation evaluation index for the winding insulation. The evaluation of the winding insulation degradation state is achieved through the minimum quantization error method. The proposed method is validated using the real‐world transformer IoT perception data, and the experimental results demonstrate its ability to accurately assess the degree of winding insulation degradation, regardless of the presence of random or continuous irregularities in IoT sensing data.

Low-rank tensor fusion and self-supervised multi-task multimodal sentiment analysis

Low-rank tensor fusion and self-supervised multi-task multimodal sentiment analysis

Multilinear subspace learning for Person Re-Identification based fusion of high order tensor features

Video surveillance image analysis and processing is an important field in computer vision and among the challenging tasks for Person Re-Identification (PRe-ID). The latter aims at finding a target person who has already been identified and appeared on a camera network using a powerful description of their pedestrian images. The success of recent research on person PRe-ID is largely backed into the effective features extraction and representation with a powerful learning of these features to correctly discriminate pedestrian images. To this end, two powerful features, Convolutional Neural Network (CNN) and Local Maximal Occurrence (LOMO) are modeled on a multidimensional data in the proposed method, High-Dimensional Feature Fusion (HDFF). Specifically, a new tensor fusion scheme is introduced to take advantage and combine two types of features in the same tensor data even if its dimensions are not the same. To improve the accuracy, we use Tensor Cross-View Quadratic Analysis (TXQDA) to perform multilinear subspace learning followed by the Cosine similarity for matching. TXQDA efficiently ensures the learning ability and reduces the high dimensionality resulting from high-order tensor data. The effectiveness of the proposed method is verified through experiments on three challenging widely-used PRe-ID datasets namely, VIPeR, GRID, and PRID450S. Extensive experiments show that the proposed method performs very well when compared with recent state-of-the-art methods.

Advancing classroom fatigue recognition: A multimodal fusion approach using self-attention mechanism

This paper presents a novel multimodal tensor fusion network based on the self-attention mechanism (MTFN-SAM) for classroom fatigue recognition. The LSTM network is first employed to learn the fatigue information of different modalities over time. Then, a self-attention mechanism is utilized to capture the connections within the modal features. Finally, a multimodal tensor fusion layer is employed to fuse features from multiple modalities, including electroencephalogram (EEG), heart rate (HR), galvanic skin response (GSR), and acceleration (ACC). The classification accuracy of the experimental results was compared with TFN, LWF, LSTM, and other fusion algorithms, confirming that the proposed MTFN-SAM network had better performance in multimodal feature fusion. Additionally, we found that as the number of modalities increased, the information contained in the multimodal fusion became more abundant, as confirmed by the experimental results (94.11% accuracy after EEG+HR+GSR+ACC fusion). Thus, the method could efficiently identify students’ classroom fatigue.

OF-WFBP: A near-optimal communication mechanism for tensor fusion in distributed deep learning

The communication bottleneck has severely restricted the scalability of distributed deep learning. Tensor fusion improves the scalability of data parallelism by overlapping computation and communication tasks. However, existing tensor fusion schemes only result in suboptimal training performance. In this paper, we propose an efficient communication mechanism (OF-WFBP) to find the optimal tensor fusion scheme for synchronous data parallelism. We present the mathematical model of OF-WFBP and prove it is an NP-hard problem. We mathematically solve the mathematical model of OF-WFBP in two cases. We propose an improved sparrow search algorithm (GradSSA) to find the near-optimal tensor fusion scheme efficiently in other cases. Experimental results on two different GPU clusters show that OF-WFBP achieves up to 1.43x speedup compared to the state-of-the-art tensor fusion mechanisms.

A Dynamic Attributes-driven Graph Attention Network Modeling on Behavioral Finance for Stock Prediction

Stock prediction is a challenging task due to multiple influencing factors and complex market dependencies. Traditional solutions are based on a single type of information. With the success of multi-source information in different fields, the combination of different types of information such as numerical and textual information has become a promising option. Although multi-source information provides rich multi-view information, how to mine and construct structured relationships from them is a difficult problem. Specifically, most existing methods usually extract features from commonly used multi-source information as predictive information sources, without further pre-constructing stock relationship graphs with dependencies using broader information. More importantly, they typically treat each stock as an isolated forecasting, or employ stock market correlations based on a fixed predefined graph structure, but current methods are not sensitive enough to aggregate the attribute features extracted from multi-source information and stock relationship graph, to obtain the dynamic update of market relations and relationship strength. The stock market is highly temporally, and the attributes of nodes are affected by the time perception of other attributes, which is not fully considered. To address these problems, we propose a novel dynamic attributes-driven graph attention networks incorporating sentiment (DGATS) information, transaction data, and text data. Inspired by behavioral finance, we separately extract sentiment information as a factor of technical indicators, and further realize the early fusion of technical indicators and textual data through Kronecker product-based tensor fusion. In particular, by LSTM and temporal attention network, the short-term and long-term transition features are gradually grasped from the local composition of the fused stock trading sequence. Furthermore, real-time intra-market dependencies and key attributes information are captured with graph networks, enabling dynamic updates of relationships and relationship strengths in predefined graphs. Experiments on the real datasets show that the architecture can outperform the previous methods in prediction performance.

Knowledge Tensor-Aided Breast Ultrasound Image Assistant Inference Framework.

Breast cancer is one of the most prevalent cancers in women nowadays, and medical intervention at an early stage of cancer can significantly improve the prognosis of patients. Breast ultrasound (BUS) is a widely used tool for the early screening of breast cancer in primary care hospitals but it relies heavily on the ability and experience of physicians. Accordingly, we propose a knowledge tensor-based Breast Imaging Reporting and Data System (BI-RADS)-score-assisted generalized inference model, which uses the BI-RADS score of senior physicians as the gold standard to construct a knowledge tensor model to infer the benignity and malignancy of breast tumors and axes the diagnostic results against those of junior physicians to provide an aid for breast ultrasound diagnosis. The experimental results showed that the diagnostic AUC of the knowledge tensor constructed using the BI-RADS characteristics labeled by senior radiologists achieved 0.983 (95% confidential interval (CI) = 0.975-0.992) for benign and malignant breast cancer, while the diagnostic performance of the knowledge tensor constructed using the BI-RADS characteristics labeled by junior radiologists was only 0.849 (95% CI = 0.823-0.876). With the knowledge tensor fusion, the AUC is improved to 0.887 (95% CI = 0.864-0.909). Therefore, our proposed knowledge tensor can effectively help reduce the misclassification of BI-RADS characteristics by senior radiologists and, thus, improve the diagnostic performance of breast-ultrasound-assisted diagnosis.