Factorization Machines Research Articles

Memory-aware gated factorization machine for top-N recommendation

Factorization machine (FM) has recently become one of the most popular methods in collaborative filtering due to its flexibility of incorporating auxiliary information, e.g., user demographics and item genres. However, standard FM method and its deep variants (e.g., NFM and DeepFM) suffer from two key issues: (1) failing to effectively leverage user historical records, i.e., all historical records are treated equally without considering the relevance to the targeted user–item pair and (2) failing to adaptively weigh the importance of auxiliary information, i.e., auxiliary information may have negative effects on the accuracy in certain cases but existing methods cannot effectively detect and eliminate the negative effects. To this end, this paper proposes a memory-aware gated factorization machine (MAGFM), which improves the FM method by introducing two new components: (1) an external user memory matrix is introduced to each user, which can enrich the representation capacity by leveraging user historical items and the auxiliary information associated with the historical items and (2) gated filtering units are applied on top of the embedding of user/item auxiliary information, which can adaptively filter out the features with negative effects to achieve higher accuracy. Experimental studies on real-world datasets demonstrate that MAGFM can substantially outperform FM, NFM and DeepFM methods by 0.31% – 12.77% relatively in top-N recommendation.

EEG Signal and Feature Interaction Modeling-Based Eye Behavior Prediction Research.

In recent years, with the development of brain science and biomedical engineering, as well as the rapid development of electroencephalogram (EEG) signal analysis methods, using EEG signals to monitor human health has become a very popular research field. The innovation of this paper is to analyze the EEG signal for the first time by building a depth factorization machine model, so that on the basis of analyzing the characteristics of user interaction, we can use EEG data to predict the binomial state of eyes (open eyes and closed eyes). The significance of the research is that we can diagnose the fatigue and the health of the human body by detecting the state of eyes for a long time. On the basis of this inference, the proposed method can make a further useful auxiliary support for improving the accuracy of the recommendation system recommendation results. In this paper, we first extract the features of EEG data by wavelet transform technology and then build a depth factorization machine model (FM+LSTM) which combines factorization machine (FM) and Long Short-Term Memory (LSTM) in parallel. Through the test of real data set, the proposed model gets more efficient prediction results than other classifier models. In addition, the model proposed in this paper is suitable not only for the determination of eye features but also for the acquisition of interactive features (user fatigue) in the recommendation system. The conclusion obtained in this paper will be an important factor in the determination of user preferences in the recommendation system, which will be used in the analysis of interactive features by the graph neural network in the future work.

Attention-over-Attention Field-Aware Factorization Machine

Factorization Machine (FM) has been a popular approach in supervised predictive tasks, such as click-through rate prediction and recommender systems, due to its great performance and efficiency. Recently, several variants of FM have been proposed to improve its performance. However, most of the state-of-the-art prediction algorithms neglected the field information of features, and they also failed to discriminate the importance of feature interactions due to the problem of redundant features. In this paper, we present a novel algorithm called Attention-over-Attention Field-aware Factorization Machine (AoAFFM) for better capturing the characteristics of feature interactions. Specifically, we propose the field-aware embedding layer to exploit the field information of features, and combine it with the attention-over-attention mechanism to learn both feature-level and interaction-level attention to estimate the weight of feature interactions. Experimental results show that the proposed AoAFFM improves FM and FFM with large margin, and outperforms state-of-the-art algorithms on three public benchmark datasets.

A personalized point-of-interest recommendation system for O2O commerce

Online-to-offline (O2O) commerce, e.g., the internet celebrity economy, provides a seamless service experience between online commerce and offline bricks-and-mortar commerce. This type of commerce model is closely related to location-based social networks (LBSNs), which incorporate mobility patterns and human social ties. Personalized point-of-interest (POI) recommendations are crucial for O2O commerce in LBSNs; such recommendations not only help users explore new venues but also enable many location-based services, e.g., the targeting of mobile advertisements to users. However, producing personalized POI recommendations for O2O commerce is highly challenging, since LBSNs involve heterogeneous types of data and the user-POI matrix is very sparse. LBSNs have substantially altered how people interact by sharing a wide range of user information, such as the products and services that users use and the places and events that users visit. To address these challenges in O2O commerce LBSNs, we analyze users’ check-in behaviors in detail and introduce the concept of a heterogeneous information network (HIN). Then, we propose a HIN-based POI recommendation system, which consists of two components: an improved singular value decomposition (SVD++) and factorization machines (FMs). The results of experiments on two real-world O2O commerce websites, namely, Gowalla and Foursquare, demonstrate that our method is more accurate than baseline methods. Additionally, a case study of the bricks-and-mortar brand of internet celebrity indicates that our proposed POI recommendation system can be used to conduct online promotion and purchasing to drive offline marketing and consumption.

Transient Stability Assessment of Power System Based on XGBoost and Factorization Machine

As the deployment of wide-area measurement systems (WAMS) expands, data-driven methods are playing an increasingly important role in transient stability assessment (TSA). However, if the measured data is disturbed by noise or the topological structure of the power system changes, the performance of the model based on data-mining would decline so that it could not meet the needs of real-world scenarios. In this paper, we develop a TSA methodology based on extreme gradient boosting (XGBoost) and factorization machine (FM). Utilizing XGBoost to complete automatically the feature construction and transform the power flow into a sparse matrix. The influence of noise can be reduce due to the sparsity of the new feature set. On this basis, FM algorithm, which has advantage in processing the large sparse matrix, is introduced into the model to complete fault classification. Furthermore, the feature crossing function of FM further mines the interactive information of the spatio-temporal features. For changed topology scenarios, we propose a extended-training set scheme. Adding some pivotal data of topology changes to the training set improves the robustness of the model. Compared with existing studies, the proposed assessment model based on XGBoost-FM not only has the better generalization performance in the case of noise interference or changed topology, but also has the least time consumption and complexity.

Low-Rate DDoS Attack Detection Based on Factorization Machine in Software Defined Network

As the Software Define Network (SDN) adopts centralized control logic, it is vulnerable to various types of Distributed Denial of Service (DDoS) attacks. At present, almost all the research work focuses on high-rate DDoS attack against the SDN control layer. Moreover, most of the existing detection methods are effective for high-rate DDoS attack detection of the control layer, while a low-rate DDoS attack against the SDN data layer is highly concealed, and the detection accuracy against this kind of attack is low. In order to improve the detection accuracy of the low-rate DDoS attack against the SDN data layer, this paper studies the mechanism of such attacks, and then proposes a multi-feature DDoS attack detection method based on Factorization Machine (FM). The features extracted from the flow rules are used to detect low-rate DDoS attacks, and the detection of low-rate DDoS attacks based on FM machine learning algorithms is implemented. The experimental results show that the method can effectively detect the low-rate DDoS attack against the SDN data layer, and the detection accuracy reaches 95.80 percent. Because FM algorithm can achieve fine-grained detection for low-rate DDoS attack, which provides a reliable condition for defending against such attacks. Finally, this paper proposes a defense method based on dynamic deletion of flow rules, and carries out experimental simulation and analysis to prove the effectiveness of the defense method, and the success rate of forwarding normal packets reached 97.85 percent.

Extended Factorization Machines for Sequential Recommendation

Users' historical activities are usually contained in real-life sequential recommendation systems to predict their future behaviors. In this situation, traditional Factorization Machines (FMs) approaches may be not suitable. Recently, a new surge of interest aims to use recurrent neural networks(RNN) to encode users' dynamic features with temporal characteristics. However, most of these works fail to reproduce computational simplicity of FMs. In this paper, we propose an architecture of extended-FM for sequential recommendation, which presents temporal feature interactions in an explicit way as traditional FM's formula. Our approach also allows us to accomplish computation of the model in linear time. Furthermore, we merge extended-FM into higher-order interaction framework without significant changes to the deeper models themselves. We conduct comprehensive experiments on two real-world datasets. The results demonstrate that extended-FM outperforms traditional FMs as well as deep learning feature combination models on sequential recommendation tasks.

Hierarchical Attentional Factorization Machines for Expert Recommendation in Community Question Answering

The most challenging task of Community Question Answering (CQA) is to provide high-quality answers to users’ questions. Currently, a variety of expert recommendation methods have been proposed and greatly improved the effective matching between questions and potential good answerers. However, the performance of existing methods can be adversely affected by many common factors such as data sparsity and noise problem, which cause less precise user modeling. Moreover, existing methods often model user-question interactions through simple ways, failing to capture the multiple scale interactions of question and answerers, which make it difficult to find answerers who are able to provide the best answers. In this paper, we propose an attention-based variant of Factorization Machines (FM) called Hierarchical Attentional Factorization Machines (HaFMRank) for answerer recommendation in CQA, which not only models the interactions between pairs of individual features but emphasizes the roles of crucial features and pairwise interactions. Specifically, we introduce the within-field attention layer to capture the inner structure of features belonging to the same field, while a feature-interaction attention layer is adopted to examine the importance of each pairwise interaction. A pre-training procedure is designed to generate latent FM feature embedding that encode question context and user history into the training process of HaFMRank. The performance of the proposed HaFMRank is evaluated by using real-world datasets of Stack Exchange and experimental results demonstrate that it outperforms several state-of-the-art methods in best answerer recommendation.

Anomalous Telecom Customer Behavior Detection and Clustering Analysis Based on ISP’s Operating Data

Mobile networks and smart phones have become ubiquitous in our daily life. Large amount of customer related telecom data from various sources are generated every day, from which diversified behavior patterns can be revealed, including some anomalous behaviors that are vicious. It becomes increasingly important to achieve both efficient and effective customer behavior analysis based on the telecom big data. In this paper, the Multi-faceted Telecom Customer Behavior Analysis (MTCBA) framework for anomalous telecom customer behavior detection and clustering analysis is proposed. In this framework, we further design the hierarchical Locality Sensitive Hashing-Local Outlier Factor (hierarchical LSH-LOF) scheme for suspicious customer detection, and the Autoencoders with Factorization Machines (FM-AE) structure for dimension reduction to achieve more efficient clustering. Hierarchical LSH-LOF is an improved algorithm of LOF, in which we design a hierarchical LSH process that selects the approximate k nearest neighbors from coarse to fine by gradually narrowing down the scope. Experiments show its superiority over KD-tree w.r.t searching speed. FM-AE exploits Factorization Machines for learning second order feature interactions, which we prove to be useful by designing comparative experiments with five dimension reduction algorithms. With the proposed MTCBA framework, efficient and effective telecom customer behavior analysis including anomalous customer behavior detection and clustering analysis is performed on the real world telecom operating data provided by one of the major Internet service providers (ISPs) in China. Meanwhile, interpretable clustering results of six clusters are obtained to provide valuable information for the precision marketing of telecom operators, criminal combating, and social credit system construction.

HoAFM: A High-order Attentive Factorization Machine for CTR Prediction

Modeling feature interactions is of crucial importance to predict click-through rate (CTR) in industrial recommender systems. However, manually crafting cross features usually requires extensive domain knowledge and labor-intensive feature engineering to obtain the desired cross features. To alleviate this problem, the factorization machine (FM) is proposed to model feature interactions from raw features automatically. In particular, it embeds each feature in a vector representation and discovers second-order interactions as the product of two feature representations. In order to learn nonlinear and complex patterns, recent works, such as NFM, PIN, and DeepFM, exploited deep learning techniques to capture higher-order feature interactions. These approaches lack guarantees about the effectiveness of high-order pattern as they model feature interactions in a rather implicit way. To address this limitation, xDeepFM is recently proposed to generate high-order interactions of features in an explicit fashion, where multiple interaction networks are stacked. Nevertheless, xDeepFM suffers from rather high complexity which easily leads to overfitting.In this paper, we develop a more expressive but lightweight solution based on FM, named High-order Attentive Factorization Machine (HoAFM), by accounting for the higher-order sparse feature interactions in an explicit manner. Beyond the linearity of FM, we devise a cross interaction layer, which updates a feature’s representation by aggregating the representations of other co-occurred features. In addition, we perform a bit-wise attention mechanism to determine the different importance of co-occurred features on the granularity of dimensions. By stacking multiple cross interaction layers, we can inject high-order feature interactions into feature representation learning, in order to establish expressive and informative cross features. Extensive experiments are performed on two benchmark datasets, Criteo and Avazu, to demonstrate the rationality and effectiveness of HoAFM. Empirical results suggest that HoAFM achieves significant improvement over other state-of-the-art methods, such as NFM and xDeepFM. https://github.com/zltao/HoAFM.

Holographic Factorization Machines for Recommendation

Factorization Machines (FMs) are a class of popular algorithms that have been widely adopted for collaborative filtering and recommendation tasks. FMs are characterized by its usage of the inner product of factorized parameters to model pairwise feature interactions, making it highly expressive and powerful. This paper proposes Holographic Factorization Machines (HFM), a new novel method of enhancing the representation capability of FMs without increasing its parameter size. Our approach replaces the inner product in FMs with holographic reduced representations (HRRs), which are theoretically motivated by associative retrieval and compressed outer products. Empirically, we found that this leads to consistent improvements over vanilla FMs by up to 4% improvement in terms of mean squared error, with improvements larger at smaller parameterization. Additionally, we propose a neural adaptation of HFM which enhances its capability to handle nonlinear structures. We conduct extensive experiments on nine publicly available datasets for collaborative filtering with explicit feedback. HFM achieves state-of-theart performance on all nine, outperforming strong competitors such as Attentional Factorization Machines (AFM) and Neural Matrix Factorization (NeuMF).

Accurate and Interpretable Factorization Machines

Factorization Machines (FMs), a general predictor that can efficiently model high-order feature interactions, have been widely used for regression, classification and ranking problems. However, despite many successful applications of FMs, there are two main limitations of FMs: (1) FMs consider feature interactions among input features by using only polynomial expansion which fail to capture complex nonlinear patterns in data. (2) Existing FMs do not provide interpretable prediction to users. In this paper, we present a novel method named Subspace Encoding Factorization Machines (SEFM) to overcome these two limitations by using non-parametric subspace feature mapping. Due to the high sparsity of new feature representation, our proposed method achieves the same time complexity as the standard FMs but can capture more complex nonlinear patterns. Moreover, since the prediction score of our proposed model for a sample is a sum of contribution scores of the bins and grid cells that this sample lies in low-dimensional subspaces, it works similar like a scoring system which only involves data binning and score addition. Therefore, our proposed method naturally provides interpretable prediction. Our experimental results demonstrate that our proposed method efficiently provides accurate and interpretable prediction.

Which Factorization Machine Modeling Is Better: A Theoretical Answer with Optimal Guarantee

Factorization machine (FM) is a popular machine learning model to capture the second order feature interactions. The optimal learning guarantee of FM and its generalized version is not yet developed. For a rank k generalized FM of d dimensional input, the previous best known sampling complexity is O[k3d · polylog(kd)] under Gaussian distribution. This bound is sub-optimal comparing to the information theoretical lower bound O(kd). In this work, we aim to tighten this bound towards optimal and generalize the analysis to sub-gaussian distribution. We prove that when the input data satisfies the so-called τ-Moment Invertible Property, the sampling complexity of generalized FM can be improved to O[k2d · polylog(kd)/τ2]. When the second order self-interaction terms are excluded in the generalized FM, the bound can be improved to the optimal O[kd · polylog(kd)] up to the logarithmic factors. Our analysis also suggests that the positive semi-definite constraint in the conventional FM is redundant as it does not improve the sampling complexity while making the model difficult to optimize. We evaluate our improved FM model in real-time high precision GPS signal calibration task to validate its superiority.

Knowledge Tracing Machines: Factorization Machines for Knowledge Tracing

Knowledge tracing is a sequence prediction problem where the goal is to predict the outcomes of students over questions as they are interacting with a learning platform. By tracking the evolution of the knowledge of some student, one can optimize instruction. Existing methods are either based on temporal latent variable models, or factor analysis with temporal features. We here show that factorization machines (FMs), a model for regression or classification, encompasses several existing models in the educational literature as special cases, notably additive factor model, performance factor model, and multidimensional item response theory. We show, using several real datasets of tens of thousands of users and items, that FMs can estimate student knowledge accurately and fast even when student data is sparsely observed, and handle side information such as multiple knowledge components and number of attempts at item or skill level. Our approach allows to fit student models of higher dimension than existing models, and provides a testbed to try new combinations of features in order to improve existing models.

Random Feature Maps for the Itemset Kernel

Although kernel methods efficiently use feature combinations without computing them directly, they do not scale well with the size of the training dataset. Factorization machines (FMs) and related models, on the other hand, enable feature combinations efficiently, but their optimization generally requires solving a non-convex problem. We present random feature maps for the itemset kernel, which uses feature combinations, and includes the ANOVA kernel, the all-subsets kernel, and the standard dot product. Linear models using one of our proposed maps can be used as an alternative to kernel methods and FMs, resulting in better scalability during both training and evaluation. We also present theoretical results for a proposed map, discuss the relationship between factorization machines and linear models using a proposed map for the ANOVA kernel, and relate the proposed feature maps to prior work. Furthermore, we show that the maps can be calculated more efficiently by using a signed circulant matrix projection technique. Finally, we demonstrate the effectiveness of using the proposed maps for real-world datasets..

Interaction-Aware Factorization Machines for Recommender Systems

Factorization Machine (FM) is a widely used supervised learning approach by effectively modeling of feature interactions. Despite the successful application of FM and its many deep learning variants, treating every feature interaction fairly may degrade the performance. For example, the interactions of a useless feature may introduce noises; the importance of a feature may also differ when interacting with different features. In this work, we propose a novel model named Interaction-aware Factorization Machine (IFM) by introducing Interaction-Aware Mechanism (IAM), which comprises the feature aspect and the field aspect, to learn flexible interactions on two levels. The feature aspect learns feature interaction importance via an attention network while the field aspect learns the feature interaction effect as a parametric similarity of the feature interaction vector and the corresponding field interaction prototype. IFM introduces more structured control and learns feature interaction importance in a stratified manner, which allows for more leverage in tweaking the interactions on both feature-wise and field-wise levels. Besides, we give a more generalized architecture and propose Interaction-aware Neural Network (INN) and DeepIFM to capture higher-order interactions. To further improve both the performance and efficiency of IFM, a sampling scheme is developed to select interactions based on the field aspect importance. The experimental results from two well-known datasets show the superiority of the proposed models over the state-of-the-art methods.

Top-N Recommendation with Multi-Channel Positive Feedback using Factorization Machines

User interactions can be considered to constitute different feedback channels, for example, view, click, like or follow, that provide implicit information on users’ preferences. Each implicit feedback channel typically carries a unary, positive-only signal that can be exploited by collaborative filtering models to generate lists of personalized recommendations. This article investigates how a learning-to-rank recommender system can best take advantage of implicit feedback signals from multiple channels. We focus on Factorization Machines (FMs) with Bayesian Personalized Ranking (BPR), a pairwise learning-to-rank method, that allows us to experiment with different forms of exploitation. We perform extensive experiments on three datasets with multiple types of feedback to arrive at a series of insights. We compare conventional, direct integration of feedback types with our proposed method, which exploits multiple feedback channels during the sampling process of training. We refer to our method as multi-channel sampling. Our results show that multi-channel sampling outperforms conventional integration, and that sampling with the relative “level” of feedback is always superior to a level-blind sampling approach. We evaluate our method experimentally on three datasets in different domains and observe that with our multi-channel sampler the accuracy of recommendations can be improved considerably compared to the state-of-the-art models. Further experiments reveal that the appropriate sampling method depends on particular properties of datasets such as popularity skewness.

Joint Deep Model with Multi-Level Attention and Hybrid-Prediction for Recommendation.

The Recommender System (RS) has obtained a pivotal role in e-commerce. To improve the performance of RS, review text information has been extensively utilized. However, it is still a challenge for RS to extract the most informative feature from a tremendous amount of reviews. Another significant issue is the modeling of user–item interaction, which is rarely considered to capture high- and low-order interactions simultaneously. In this paper, we design a multi-level attention mechanism to learn the usefulness of reviews and the significance of words by Deep Neural Networks (DNN). In addition, we develop a hybrid prediction structure that integrates Factorization Machine (FM) and DNN to model low-order user–item interactions as in FM and capture the high-order interactions as in DNN. Based on these two designs, we build a Multi-level Attentional and Hybrid-prediction-based Recommender (MAHR) model for recommendation. Extensive experiments on Amazon and Yelp datasets showed that our approach provides more accurate recommendations than the state-of-the-art recommendation approaches. Furthermore, the verification experiments and explainability study, including the visualization of attention modules and the review-usefulness prediction test, also validated the reasonability of our multi-level attention mechanism and hybrid prediction.

Vibration Test of Single Coal Gangue Particle Directly Impacting the Metal Plate and the Study of Coal Gangue Recognition Based on Vibration Signal and Stacking Integration

In order to realize the recognition of coal gangue in the top coal caving process, a scheme of the coal gangue recognition based on the collision vibration signal between coal gangue and the metal plate is proposed in this paper, a systematic and standardized impacting test between coal gangue particles and the metal plate is designed for the first time, the vibration signal standardized processing method by the signal intercepting and the coal gangue impact vibration signal recognition algorithm by stacking integration are innovatively proposed. First, a single particle impact on the metal plate test-bed was designed and constructed. Then 1,000 groups coal and 1,000 groups gangue impact on the metal plate tests were carried out respectively, and the vibration acceleration signals of the metal plate were collected. After that, through the signal intercepting, calculating the time-domain characteristics and HHT processing of the vibration signal, 10 time-frequency characteristics, such as the variance of the intercepted signal and the Hilbert marginal spectrum energy value, are determined to form the feature vector. Finally, based on the two different type of the signal samples, the intercepted signal feature vector, and the original intercepted signal, coal gangue recognition by the seven machine learning algorithms, including the decision tree (DT), random forest (RF), XGBoost, long short-term memory (LSTM), support vector machine (SVM), factorization machine (FM), and stacking integration is carried out respectively, and the basis for selecting recognition schemes is discussed. The results show that the coal gangue recognition rate with the same recognition algorithm by using the intercepted signal samples is higher than that of the feature vector samples, the Staking integration algorithm based on the same sample has the highest recognition rate, and the Staking integration algorithm based on the feature vector has the most significant comprehensive advantage in top coal caving process.

An Efficient Alternating Newton Method for Learning Factorization Machines

To date, factorization machines (FMs) have emerged as a powerful model in many applications. In this work, we study the training of FM with the logistic loss for binary classification, which is a nonlinear extension of the linear model with the logistic loss (i.e., logistic regression). For the training of large-scale logistic regression, Newton methods have been shown to be an effective approach, but it is difficult to apply such methods to FM because of the nonconvexity. We consider a modification of FM that is multiblock convex and propose an alternating minimization algorithm based on Newton methods. Some novel optimization techniques are introduced to reduce the running time. Our experiments demonstrate that the proposed algorithm is more efficient than stochastic gradient algorithms and coordinate descent methods. The parallelism of our method is also investigated for the acceleration in multithreading environments.