Tensor Factorization Model Research Articles

Bayesian Allocation Model: Marginal Likelihood-Based Model Selection for Count Tensors

In this article, we introduce a dynamic generative model, the Bayesian allocation model (BAM), for modeling count data. BAM covers various probabilistic nonnegative tensor factorization (NTF) and topic models under one general framework. In BAM, allocations are made using a Bayesian network, whose conditional probability tables can be integrated out analytically. We show that, when allocations are viewed as sequential, the resulting marginal process is a special type of Polya urn process, which we name as Polya-Bayes process, an integer valued self-reinforcing process. Exploiting the Polya urn construction, we develop a novel sequential Monte Carlo (SMC) algorithm for marginal likelihood estimation in BAM, leading to a unified scoring method for discrete variable Bayesian networks with hidden nodes, including various NTF and topic models. The SMC estimator for marginal likelihood has the remarkable property of being unbiased in contrast to variational algorithms which are generally biased. We also demonstrate how our novel SMC-based likelihood estimation can be integrated within a Markov chain Monte Carlo algorithm for a principled and correct (in terms of respecting the true posterior distribution) Bayesian model selection and hyperparameter estimation for BAM. We provide several numerical examples, both on artificial and real datasets, that demonstrate the performance of the algorithms for various data regimes.

Neural Graph for Personalized Tag Recommendation

Traditional personalized tag recommendation methods cannot guarantee that the coupling relationships hidden in the interactions among entities are effectively encoded in the process of learning the representations of entities, resulting in insufficient expressive capacity for characterizing the preferences or attributes of entities. In this article, we first propose a graph neural networks boosted personalized tag recommendation model, namely NGTR, which integrates the graph neural networks into the pairwise interaction tensor factorization model. Specifically, we exploit the graph neural networks to capture the coupling relationships and integrate the coupling relationships into the learning of representations of entities by transmitting and assembling the representations of neighbors along the interaction graphs. In addition, we also propose a light graph neural networks boosted personalized tag recommendation model, namely LNGTR. Different from NGTR, our proposed LNGTR model removes feature transformation and nonlinear activation components as well as adopts the weighted sum of the embeddings learned at all layers as the final embedding. Experimental results on real world datasets show that our proposed personalized tag recommendation models outperform the traditional tag recommendation methods.

Variational auto-encoder based Bayesian Poisson tensor factorization for sparse and imbalanced count data

Non-negative tensor factorization models enable predictive analysis on count data. Among them, Bayesian Poisson-Gamma models can derive full posterior distributions of latent factors and are less sensitive to sparse count data. However, current inference methods for these Bayesian models adopt restricted update rules for the posterior parameters. They also fail to share the update information to better cope with the data sparsity. Moreover, these models are not endowed with a component that handles the imbalance in count data values. In this paper, we propose a novel variational auto-encoder framework called VAE-BPTF which addresses the above issues. It uses multi-layer perceptron networks to encode and share complex update information. The encoded information is then reweighted per data instance to penalize common data values before aggregated to compute the posterior parameters for the latent factors. Under synthetic data evaluation, VAE-BPTF tended to recover the right number of latent factors and posterior parameter values. It also outperformed current models in both reconstruction errors and latent factor (semantic) coherence across five real-world datasets. Furthermore, the latent factors inferred by VAE-BPTF are perceived to be meaningful and coherent under a qualitative analysis.

Nonrecurrent traffic congestion detection with a coupled scalable Bayesian robust tensor factorization model

Nonrecurrent traffic congestion (NRTC) usually brings unexpected delays to commuters. Hence, it is critical to accurately detect and recognize the NRTC in a real-time manner. The advancement of road traffic detectors provides researchers with a large-scale multivariable temporal-spatial traffic data, which allows the deep research on NRTC to be conducted. However, it remains a challenging task to construct an analytical framework through which the natural temporal-spatial structural properties of multivariable traffic information can be effectively represented and exploited to better understand and detect NRTC. In this paper, we present a novel analytical training-free framework based on the coupled scalable Bayesian robust tensor factorization (Coupled SBRTF). The framework can couple multivariable traffic variables including traffic flow, road speed, and occupancy through sharing the same sparse structure. Moreover, it naturally captures the high-dimensional temporal-spatial patterns of the traffic data by tensor factorization. With its entries revealing the distribution and magnitude of NRTC, the shared sparse structure of the framework compasses sufficiently abundant information about NRTC. While the low-rank part of the framework, expresses the distribution of general expected traffic conditions as an auxiliary product. Experimental results on real-world traffic data show that the proposed method outperforms the NRTC detection models based on the coupled Bayesian robust principal component analysis (coupled BRPCA), the rank sparsity tensor decomposition (RSTD), and standard normal deviates (SND). The proposed method performs even better when only traffic data in weekdays are utilized, and hence can provide more precise estimations of NRTC for daily commuters.

Understanding Urban Dynamics via Context-Aware Tensor Factorization with Neighboring Regularization

Recent years have witnessed the world-wide emergence of mega-metropolises with incredibly huge populations. Understanding residents mobility patterns, or urban dynamics, thus becomes crucial for building modern smart cities. In this paper, we propose a Neighbor-Regularized and context-aware Non-negative Tensor Factorization model (NR-cNTF) to discover interpretable urban dynamics from urban heterogeneous data. Different from many existing studies concerned with prediction tasks via tensor completion, NR-cNTF focuses on gaining urban managerial insights from spatial, temporal, and spatio-temporal patterns. This is enabled by high-quality Tucker factorizations regularized by both POI-based urban contexts and geographically neighboring relations. NR-cNTF is also capable of unveiling long-term evolutions of urban dynamics via a pipeline initialization approach. We apply NR-cNTF to a real-life data set containing rich taxi GPS trajectories and POI records of Beijing. The results indicate: 1) NR-cNTF accurately captures four kinds of city rhythms and seventeen spatial communities; 2) the rapid development of Beijing, epitomized by the CBD area, indeed intensifies the job-housing imbalance; 3) the southern areas with recent government investments have shown more healthy development tendency. Finally, NR-cNTF is compared with some baselines on traffic prediction, which further justifies the importance of urban contexts awareness and neighboring regulations.

KGEL: A novel end-to-end embedding learning framework for knowledge graph completion

Knowledge graphs (KGs) have recently become increasingly popular due to the broad range of essential applications in various downstream tasks including intelligent search, personalized recommendations, intelligent financial data analytics, etc. During an automated construction of a KG, the knowledge facts from multiple knowledge sources are automatically extracted in the form of triples, and these observed triples are used to derive new unobserved triples for KG completion (also known as link prediction). State-of-the-art link prediction methods are known to be primarily KG embedding models, among which tensor factorization models have recently drawn much attention due to their scalability and expressive feature embeddings, and hence, perform well for link prediction. However, these embedding models consider each KG triple individually and fail to capture the useful information present in the neighborhood of a node. To this end, we propose a novel end-to-end KG embedding learning framework that consists of an encoder of a dual weighted graph convolutional network, and a decoder of a novel fully expressive tensor factorization model. The proposed encoder extends weighted graph convolutional network to generate two rich and high quality embedding vectors for each node by aggregating information from the neighboring nodes. The proposed decoder has a flexible and powerful tensor representation form of the Tensor Train decomposition that takes benefit of the two representations of each node in its embedding space to accurately model the KG triples. We also derive a bound on the size of the embeddings for full expressivity and show that our proposed tensor factorization model is fully expressive. Additionally, we show the relationship of our tensor factorization model to previous tensor factorization models. The experimental results show the effectiveness of the proposed framework that consistently marks performance gains over several previous models on recent standard link prediction datasets.

A High-Quality Photon-Counting CT Technique Based on Weight Adaptive Total-Variation and Image-Spectral Tensor Factorization for Small Animals Imaging

Photon-counting X-ray computed tomography (CT) has been attracting great attention in tissue characterization, material discrimination, and so on. The emitting X-ray energy spectrum cutting into several energy bins that can result in only a part of X-ray photons can be collected within each narrow bin. This can compromise the image quality. In this case, how to obtain high-quality tomography is a big challenge. In this study, to overcome these issues, we mainly focus on developing an advanced imaging software based on the latest photon-counting CT system (MARS scanner). Specifically, we first design a weight adaptive total variation (TV) using compressed sensing theory. Then, combining the weight adaptive TV and nonlocal low-rank tensor factorization to formulate a new weight adaptive total-variation and image-spectral tensor factorization (WATITF) model for high-quality imaging. Finally, the optimization model is performed to obtain its solution. The studies including the numerical and preclinical mice are performed to validate and evaluate its outperformance.

TrustTF: A tensor factorization model using user trust and implicit feedback for context-aware recommender systems

In recent years, context information has been widely used in recommender systems. Tensor factorization is an effective method to process high-dimensional information. However, data sparsity is more serious in tensor factorization, and it is difficult to build a more accurate recommender system only based on user–item–context interaction information. Making full use of user’s social information and implicit feedback can alleviate this problem. In this paper, we propose a new tensor factorization model named TrustTF, which mainly works as follows: (1) Using user’s social trust information and implicit feedback to extend the bias tensor factorization (BiasTF), effectively alleviate data sparsity problem and improve the recommendation accuracy; (2) Dividing user’s trust relationship into unilateral trust and mutual trust, which makes better use of user’s social information. To our knowledge, this is the first work to consider the effects of both user trust and implicit feedback on the basis of the BiasTF model. The experimental results in two real-world data sets demonstrate that the TrustTF proposed in this paper can achieve higher accuracy than BiasTF and other social recommendation methods.

A Methodology to Attain Public Transit Origin–Destination Mobility Patterns Using Multi-Layered Mesoscopic Analysis

Knowledge about mobility patterns has become increasingly important to urban development. In this work, public transit origin-destination (OD) mobility patterns are undergoing meso-level analysis in using the advantages of big data and for the creation of a new planning and decision-based tool. An ensemble clustering method is proposed to abstract the common OD pairs by fully considering link-based information, and the nonnegative tensor factorization model is adopted to effectively extract and visualize quantitatively the mobility patterns of OD pairs. This is attained by using multi-layered analysis such as of traffic demand, traffic accessibility and traffic congestion to enable different visual and quantitative mobility patterns. In the case study of Beijing, these patterns were analyzed and discussed by temporal and spatial factors. The results of the various patterns show explicitly when and where to provide remedies to traffic problems, by time and space. This is analogue, to some extent, to detecting and treating black spots of road accidents. The new developed multi-layered mesoscopic analysis could, therefore, be an important tool for improving urban planning, public transit planning, traffic management, and emergency intervention.

DTF: Deep Tensor Factorization for predicting anticancer drug synergy.

Combination therapies have been widely used to treat cancers. However, it is cost and time consuming to experimentally screen synergistic drug pairs due to the enormous number of possible drug combinations. Thus, computational methods have become an important way to predict and prioritize synergistic drug pairs. We proposed a Deep Tensor Factorization (DTF) model, which integrated a tensor factorization method and a deep neural network (DNN), to predict drug synergy. The former extracts latent features from drug synergy information while the latter constructs a binary classifier to predict the drug synergy status. Compared to the tensor-based method, the DTF model performed better in predicting drug synergy. The area under precision-recall curve (PR AUC) was 0.58 for DTF and 0.24 for the tensor method. We also compared the DTF model with DeepSynergy and logistic regression models, and found that the DTF outperformed the logistic regression model and achieved similar performance as DeepSynergy using several performance metrics for classification task. Applying the DTF model to predict missing entries in our drug-cell-line tensor, we identified novel synergistic drug combinations for 10 cell lines from the 5 cancer types. A literature survey showed that some of these predicted drug synergies have been identified in vivo or in vitro. Thus, the DTF model could be a valuable in silico tool for prioritizing novel synergistic drug combinations. Source code and data are available at https://github.com/ZexuanSun/DTF-Drug-Synergy. Supplementary data are available at Bioinformatics online.

An online and generalized non-negativity constrained model for large-scale sparse tensor estimation on multi-GPU

Non-negative Tensor Factorization (NTF) models are effective and efficient in extracting useful knowledge from various types of probabilistic distribution with multi-way information. Current NTF models are mostly designed for problems in computer vision which involve the whole Matricized Tensor TimesKhatri−RaoProduct (MTTKRP). Meanwhile, a Sparse NTF (SNTF) proposed to solve the problem of sparse Tensor Factorization (TF) can result in large-scale intermediate data. A Single-thread-based SNTF (SSNTF) model is proposed to solve the problem of non-linear computing and memory overhead caused by large-scale intermediate data. However, the SSNTF is not a generalized model. Furthermore, the above methods cannot describe the stream-like data from industrial applications in mainstream processors, e.g, Graphics Processing Unit (GPU) and multi-GPU in an online way. To address these two issues, a Generalized SSNTF (GSSNTF) is proposed, which extends the works of SSNTF to the Euclidean distance, KullbackLeibler (KL)-divergence, and ItakuraSaito (IS)-divergence. The GSSNTF only involves the feature elements instead of the entire factor matrices during its update process, which can avoid the formation of large-scale intermediate matrices with convergence and accuracy promises. Furthermore, GSSNTF can merge the new data into the state-of-the-art built tree dataset for sparse tensor, and then online learning has the promise of the correct data format. At last, a model of Compute Unified Device Architecture (CUDA) parallelizing GSSNTF (CUGSSNTF) is proposed on GPU and Multi-GPU (MCUGSSNTF). Thus, CUGSSNTF has linear computing complexity and space requirement, and linear communication overhead on multi-GPU. CUGSSNTF and MCUGSSNTF are implemented on 8 P100 GPUs in this work, and the experimental results from real-world industrial data sets indicate the linear scalability and 40X speedup performances of CUGSSNTF than the state-of-the-art parallelized approachs.

Robust Tensor Factorization for Color Image and Grayscale Video Recovery

Low-rank tensor completion (LRTC) plays an important role in many fields, such as machine learning, computer vision, image processing, and mathematical theory. Since rank minimization is an NP-hard problem, one strategy is that it is converted into a convex relaxation tensor nuclear norm (TNN) that requires the repeated calculation of time-consuming SVD, and the other is to convert it into some product of two smaller tensors that are easy to fall into the local minimum. In order to overcome the above shortcomings, we propose a robust tensor factorization (RTF) model for solving LRTC. In RTF, the noisy tensor data with missing entries is decomposed into low-rank tensor and noisy tensor, and then the low-rank tensor is equivalently decomposed into t-products (essentially vectors convolution) of two smaller tensors: orthogonal dictionary tensor and low-rank representation tensor. Meanwhile, the TNN of low-rank representation tensor is adopted to characterize the low-rank structure of the tensor data for preserving global information. Then, an effective iterative update algorithm based on the alternating direction method of multipliers (ADMM) is proposed to solve RTF. Finally, numerical experiments on image recovering and video completion tasks show the effectiveness of the proposed RTF model compared with several state-of-the-art tensor completion models.

Ontology Guided Sparse Tensor Factorization for Joint Recommendation with Hierarchical Relationships

Abstract Although recommender systems enjoy widespread adoption in numerous different production settings, standard methods draw only on previous purchases or ratings, and optionally simple customer or product features. In many domains, however, the purchase or rating history is very sparse. Standard approaches suffer from such data sparsity and neglect to account for important additional dependencies that can be taken into consideration. This motivates us to design a recommendation model with the ability to exploit hierarchical relationships such as product series, manufacturers, or even suppliers. To this end, we propose our Ontology Guided Multi-Relational Tensor Factorization model, which models such connections via a multilevel tree structure. To solve the challenging optimization problem, we develop an efficient iterative algorithm relying on Moreau-Yosida regularization. On real-world data crawled from automobile-related websites, we find that the proposed model outperforms state-of-the art methods.

Non-local blind hyperspectral image super-resolution via 4d sparse tensor factorization and low-rank

Hyperspectral image (HSI) super-resolution is a technique to improve the spatial resolution of a HSI for better visual perception and down stream applications. This is a very ill-posed inverse problem and is often solved by fusing the low-resolution (LR) HSI with a high-resolution (HR) multispectral image (MSI). It is more challenging for blind HSI super-resolution, i.e., when the spatial degradation operators are completely unknown. In this paper, we propose a novel sparse tensor factorization model for the task of blind HSI super-resolution using the spatial non-local self-similarity and spectral global correlation of HSIs. Image clustering method is employed to collect some similar 3D cubes of HSIs which can be formed as some 4D image clusters with high correlation. We conduct cluster wise computation to not only save computation time but also to introduce a non-local regularity originated from the redundancy of cubes. By using the sparsity of tensor decomposition and the low-rank in non-local self-similarity direction underlying 4D similar clusters, we design a sparse tensor regularization term, which preserves the spatial-spectral structural correlation of HSIs. In addition, we present a proximal alternating direction method of multipliers (ADMM) based algorithm to efficiently solve the proposed model. Numerical experiments demonstrate that the proposed model outperforms many state-of-the-art HSI super-resolution methods.

A Tensor Factorization on Rating Prediction for Recommendation by Feature Extraction from Reviews

In many online review sites or social media, each user is encouraged to assign a numeric rating and write a textual review as a feedback to each item that he had gotten, e.g., a product that he had bought, a place that he had visited, a service that he had received. Sometimes, feedbacks by some users would be affected by some contextual factors such as weather, distance, time, and season. Therefore, the context-aware approach is being developed by utilizing the user's contextual information to produce more precise recommendations than traditional approaches. Furthermore, previous works have already approved the drawback of the ignorance of textual reviews would bring mediocre performance for rating prediction. In this work, we propose a framework TF+ for rating prediction models based on Tensor Factorization (TF) which is an extended version of Matrix Factorization (MF) by adding another dimension. We consider seasonal context as the additional dimension. Firstly, in our framework, each of the reviews is characterized by a numeric feature vector. Secondly, it uses TF which is trained by the proposed first-order gradient descent method for TF named Feature Vector Gradient Descent (FVGD). For the training of TF in TF+, FVGD decides the learning rates based on the feature vectors of reviews. In our evaluation, we use pre-processed data of five cities in YELP challenge dataset, and apply one of LDA, Doc2Vec and SCDV to get numeric feature vectors of reviews. We conduct experimental comparisons, and the results show that methods by TF+ improve the performance significantly as compared to the basic TF model.

Tensor Factorization with Total Variation and Tikhonov Regularization for Low-Rank Tensor Completion in Imaging Data

The main aim of this paper is to study tensor factorization for low-rank tensor completion in imaging data. Due to the underlying redundancy of real-world imaging data, the low-tubal-rank tensor factorization (the tensor–tensor product of two factor tensors) can be used to approximate such tensor very well. Motivated by the spatial/temporal smoothness of factor tensors in real-world imaging data, we propose to incorporate a hybrid regularization combining total variation and Tikhonov regularization into low-tubal-rank tensor factorization model for low-rank tensor completion problem. We also develop an efficient proximal alternating minimization (PAM) algorithm to tackle the corresponding minimization problem and establish a global convergence of the PAM algorithm. Numerical results on color images, color videos, and multispectral images are reported to illustrate the superiority of the proposed method over competing methods.

Removing the Effect of Hemodynamic Response Function in Joint Factorization of EEG and fMRI Datasets

Purpose: One of the most well-known multimodality techniques is the integration of EEG and fMRI datasets. Convolution of EEG signals with hemodynamic response function is one of the most important methods to consider the effect of HRF in the fusion of EEG and fMRI data. However, the latencies and amplitudes of ERPs and fMRI spatial components are affected by the low pass filtering effect of HRF in each trial. Materials and Methods: In this paper, we have proposed a new method based on Advanced Coupled Matrix Tensor Factorization model to jointly factorize the EEG tensor and fMRI matrix while we simultaneously remove the effect of HRF through decomposition of fMRI dataset. Results: Applying the proposed method to an auditory oddball paradigm of simultaneous EEG-fMRI recording, the well-known ERP of oddball paradigm and the corresponding fMRI spatial maps are estimated. Conclusion: The results demonstrate that our proposed approach is strongly capable of extracting the ERPs and their corresponding fMRI spatial components, while simultaneously estimates the trial to trial variations of these factors with accurate amplitude and latency in each trial.

Coupled Higher-Order Tensor Factorization for Hyperspectral and LiDAR Data Fusion and Classification

Hyperspectral and light detection and ranging (LiDAR) data fusion and classification has been an active research topic, and intensive studies have been made based on mathematical morphology. However, matrix-based concatenation of morphological features may not be so distinctive, compact, and optimal for classification. In this work, we propose a novel Coupled Higher-Order Tensor Factorization (CHOTF) model for hyperspectral and LiDAR data classification. The innovative contributions of our work are that we model different features as multiple third-order tensors, and we formulate a CHOTF model to jointly factorize those tensors. Firstly, third-order tensors are built based on spectral-spatial features extracted via attribute profiles (APs). Secondly, the CHOTF model is defined to jointly factorize the multiple higher-order tensors. Then, the latent features are generated by mode-n tensor-matrix product based on the shared and unshared factors. Lastly, classification is conducted by using sparse multinomial logistic regression (SMLR). Experimental results, conducted with two popular hyperspectral and LiDAR data sets collected over the University of Houston and the city of Trento, respectively, indicate that the proposed framework outperforms the other methods, i.e., different dimensionality-reduction-based methods, independent third-order tensor factorization based methods, and some recently proposed hyperspectral and LiDAR data fusion and classification methods.

Improved Knowledge Graph Embedding Using Background Taxonomic Information

Knowledge graphs are used to represent relational information in terms of triples. To enable learning about domains, embedding models, such as tensor factorization models, can be used to make predictions of new triples. Often there is background taxonomic information (in terms of subclasses and subproperties) that should also be taken into account. We show that existing fully expressive (a.k.a. universal) models cannot provably respect subclass and subproperty information. We show that minimal modifications to an existing knowledge graph completion method enables injection of taxonomic information. Moreover, we prove that our model is fully expressive, assuming a lower-bound on the size of the embeddings. Experimental results on public knowledge graphs show that despite its simplicity our approach is surprisingly effective.

Learning Phenotypes and Dynamic Patient Representations via RNN Regularized Collective Non-Negative Tensor Factorization

Non-negative Tensor Factorization (NTF) has been shown effective to discover clinically relevant and interpretable phenotypes from Electronic Health Records (EHR). Existing NTF based computational phenotyping models aggregate data over the observation window, resulting in the learned phenotypes being mixtures of disease states appearing at different times. We argue that by separating the clinical events happening at different times in the input tensor, the temporal dynamics and the disease progression within the observation window could be modeled and the learned phenotypes will correspond to more specific disease states. Yet how to construct the tensor for data samples with different temporal lengths and properly capture the temporal relationship specific to each individual data sample remains an open challenge. In this paper, we propose a novel Collective Non-negative Tensor Factorization (CNTF) model where each patient is represented by a temporal tensor, and all of the temporal tensors are factorized collectively with the phenotype definitions being shared across all patients. The proposed CNTF model is also flexible to incorporate non-temporal data modality and RNN-based temporal regularization. We validate the proposed model using MIMIC-III dataset, and the empirical results show that the learned phenotypes are clinically interpretable. Moreover, the proposed CNTF model outperforms the state-of-the-art computational phenotyping models for the mortality prediction task.