Factorization Machines Research Articles

FM-GRU: A Time Series Prediction Method for Water Quality Based on seq2seq Framework

Water quality prediction plays a crucial role in both enterprise management and government environmental management. However, due to the variety in water quality data, inconsistent frequency of data acquisition, inconsistency in data organization, and volatility and sparsity of data, predicting water quality accurately and efficiently has become a key problem. This paper presents a recurrent neural network water quality prediction method based on a sequence-to-sequence (seq2seq) framework. The gate recurrent unit (GRU) model is used as an encoder and decoder, and a factorization machine (FM) is integrated into the model to solve the problem of high sparsity and high dimensional feature interaction in the data, which was not addressed by the water quality prediction models in prior research. Moreover, due to the long period and timespan of water quality data, we add a dual attention mechanism to the seq2seq framework to address memory failures in deep learning. We conducted a series of experiments, and the results show that our proposed method is more accurate than several typical water quality prediction methods.

A joint learning model for click-through prediction in display advertising

Click-through rate (CTR) prediction is essential for targeted advertising and recommendation. At present, machine learning models are widely used to build CTR estimators, including logistic regression (LR), factorization machine (FM), and deep neural network (DNN). Unfortunately, these models adopt the single structure that only considers either low-order feature interactions (such as LR and FM) or high-order feature interactions (such as DNN), not sufficient for CTR prediction. Therefore, the joint learning models are proposed, such as Wide & Deep and DeepFM, which can exploit both high- and low-order feature interactions to predict CTR by combining two different models. In this paper, we first analyze the typical CTR prediction models’ structures and performance, and then summarize the general form and design rules of CTR estimators. Based on the general form, we further design a new joint learning model that combines two different residual networks to explore the feature interactions automatically. Compared with the widely adopted feed-forward neural network, the residual network is more capable of exploring complex feature interactions at different layers. Additionally, we introduce a neural attention network to learn the importance of each second-order interaction of features from various fields. Finally, we evaluate the prediction performance of the proposed model based on two real-world datasets (i.e., Criteo and Avazu) in terms of LogLoss and AUC metrics. The extensive experimental results demonstrate our model performs the best compared to the state-of-the-art baselines.

AutoFM: an efficient factorization machine model via probabilistic auto-encoders

Studies show that conventional factorization machines (FMs) have low performance in capturing both local and global structures of user–item correlation simultaneously. Recently, deep neural networks (DNNs) have been applied to improve FMs. However, DNNs increase the complexity of the training process. Moreover, DNN-based FMs ignore the integration of neighborhood-based approaches. An efficient method called factorization machine model via probabilistic auto-encoders (AutoFM) is proposed to resolve this issue in the present study. The proposed AutoFM can extract non-trivial and local structures characteristics from user–user/item–item co-occurrence pairs by integrating a low-complexity probabilistic auto-encoder. Furthermore, it supports both explicit and implicit feedback datasets. Extensive experiments on four real-world datasets demonstrate the effectiveness of the proposed method. The results show that the AutoFM outperforms the current state-of-the-art methods in rating prediction tasks. Compared with the DNN-based FM models, the proposed AutoFM model improves the item ranking at least 1.16% $$\sim$$ 4.37%.

Solving two-stage stochastic route-planning problem in milliseconds via end-to-end deep learning

With the rapid development of e-economy, ordering via online food delivery platforms has become prevalent in recent years. Nevertheless, the platforms are facing lots of challenges such as time-limitation and uncertainty. This paper addresses a complex stochastic online route-planning problem (SORPP) which is mathematically formulated as a two-stage stochastic programming model. To meet the immediacy requirement of online fashion, an end-to-end deep learning model is designed which is composed of an encoder and a decoder. To embed different problem-specific features, different network layers are adopted in the encoder; to extract the implicit relationship, the probability mass functions of stochastic food preparation time is processed by a convolution neural network layer; to provide global information, the location map and rider features are handled by the factorization-machine (FM) and deep FM layers, respectively; to screen out valuable information, the order features are embedded by attention layers. In the decoder, the permutation sequence is predicted by long-short term memory cells with attention and masking mechanism. To learn the policy for finding optimal permutation under complex constraints of the SORPP, the model is trained in a supervised learning way with the labels obtained by iterated greedy search algorithm. Extensive experiments are conducted based on real-world data sets. The comparative results show that the proposed model is more efficient than meta-heuristics and is able to yield higher quality solutions than heuristics. This work provides an intelligent optimization technique for complex online food delivery system.

XGB+FM for Severe Convection Forecast and Factor Selection

In the field of meteorology, radiosonde data and observation data are critical for analyzing regional meteorological characteristics. Because of the high false alarm rate, severe convection forecasting is still challenging. In addition, the existing methods are difficult to use to capture the interaction of meteorological factors at the same time. In this research, a cascade of extreme gradient boosting (XGBoost) for feature transformation and a factorization machine (FM) for second-order feature interaction to capture the nonlinear interaction—XGB+FM—is proposed. An attention-based bidirectional long short-term memory (Att-Bi-LSTM) network is proposed to impute the missing data of meteorological observation stations. The problem of class imbalance is resolved by the support vector machines–synthetic minority oversampling technique (SVM-SMOTE), in which two oversampling strategies based on the support vector discrimination mechanism are proposed. It is proven that the method is effective, and the threat score (TS) is 7.27~14.28% higher than other methods. Moreover, we propose the meteorological factor selection method based on XGB+FM and improve the forecast accuracy, which is one of our contributions, as well as the forecast system.

Multi-level attentive deep user-item representation learning for recommendation system

With the development of e-commerce platforms, user reviews have become a vital source of information to address the sparsity problems for enhancing the predictive performance of the recommendation systems (RSs). However, the traditional methods of the RSs used to model user/item latent features based on static vectors in an independent manner without considering the dynamic nature of the user-item interactions which potentially affect the accuracy of the recommendation process. Thus, this paper proposes a RS model that exploits neural attention techniques to learn user/item representations by jointly considering the fine-grained semantic information for the user-item pairs. The proposed model utilizes both review-based and interaction-specific features for the user/item reviews to learn heterogeneous user/item representations. First, a BiLSTM sequence encoder is used to learn the contextual information of words, and a Co-attention network is then designed to jointly capture the most relevant semantic information of reviews for the user-item pair. To better capture user/item latent factors comprehensively, interaction-specific features based on the rating scores are further integrated with the review-specific latent features via a shared hidden layer. Finally, an attentive factorization machine (FM) is then applied on the shared hidden layer of the integrated user/item features for the final prediction. We carry out a series of experiments using real-world datasets and the results demonstrate that our proposed method is better than the baseline approaches in terms of both rating prediction and ranking performance.

An Exponential Factorization Machine with Percentage Error Minimization to Retail Sales Forecasting

This article proposes a new approach to sales forecasting for new products (stock-keeping units [SKUs]) with long lead time but short product life cycle. These SKUs are usually sold for one season only, without any replenishments. An exponential factorization machine (EFM) sales forecast model is developed to solve this problem which not only takes into account SKU attributes, but also pairwise interactions. The EFM model is significantly different from the original Factorization Machines (FM) from two fold: (1) the attribute-level formulation for explanatory/input variables; and (2) exponential formulation for the positive response/output/target variable. The attribute-level formation excludes infeasible intra-attribute interactions and results in more efficient feature engineering comparing with the conventional one-hot encoding, while the exponential formulation is demonstrated more effective than the log-transformation for the positive but not skewed distributed responses. In order to estimate the parameters, percentage error squares (PES) and error squares (ES) are minimized by a proposed adaptive batch gradient descent method over the training set. To overcome the over-fitting problem, a greedy forward stepwise feature selection method is proposed to select the most useful attributes and interactions. Real-world data provided by a footwear retailer in Singapore are used for testing the proposed approach. The forecasting performance in terms of both mean absolute percentage error (MAPE) and mean absolute error (MAE) compares favorably with not only off-the-shelf models but also results reported by extant sales and demand forecasting studies. The effectiveness of the proposed approach is also demonstrated by two external public datasets. Moreover, we prove the theoretical relationships between PES and ES minimization, and present an important property of the PES minimization for regression models; that it trains models to underestimate data. This property fits the situation of sales forecasting where unit-holding cost is much greater than the unit-shortage cost (e.g., perishable products).

Research on Disease Prediction Based on Improved DeepFM and IoMT

In recent years, with the increase of computer computing power, Deep Learning has begun to be favored. Its learning of non-linear feature combinations has played a role that traditional machine learning cannot reach in almost every field. The application of Deep Learning has also driven the advancement of Factorization Machine (FM) in the field of recommendation systems, because Deep Learning and FM can learn high-order and low-order features combinations respectively, and FM's hidden vector system enables it to learn information from sparse data. The integration of them has attracted the attention of many scholars. They have researched many classic models such as Factorization-supported Neural Network (FNN), Product-based Neural Networks (PNN), Inner PNN (IPNN), Wide&Deep, Deep&Cross, DeepFM, etc. for the Click-Through-Rate (CTR) problem, and their performance is getting better and better. This kind of model is also suitable for agriculture, meteorology, disease prediction and other fields due to the above advantages. Based on the DeepFM model, we predicts the incidence of hepatitis in each sample in the structured disease prediction data of the 2020 Artificial Intelligence Challenge Preliminary Competition, and make minor improvements and parameter adjustments to DeepFM. Compared with other models, the improved DeepFM has excellent performance in AUC. This research can be applied to electronic medical records to reduce the workload of doctors and make doctors focus on the samples with higher predicted incidence rates. For some changing data, such as blood pressure, height, weight, cholesterol, etc., we can introduce the Internet of Medical Things (IoMT). IoMT's sensors can be used to conduct transmission to ensure that the disease can be predicted in time, just in case. After joining IoMT, a healthcare system is formed, which is superior in forecasting and time performance.

Retracted] Using Factor Decomposition Machine Learning Method to Music Recommendation

The user data mining was introduced into the model construction process, and the user behavior was decomposed by analyzing various influencing factors through the factorization machine (FM) learning method. In the recommendation screening stage, the collaborative filtering recommendation is combined to screen the recommendation candidate set. The idea of user‐based collaborative filtering (CF) is used for reference to obtain music works favored by similar users. On the other hand, we learn from item‐based CF, which ensures that the candidate set covers user preference. Firstly, the user’s interest value is predicted by using dynamic interest model. Then, the common problems such as cold start and hot items processing are fully considered. The frequent pattern growth algorithm is compared with the association rule algorithm based on the collaborative filtering recommendation algorithm and the content‐based recommendation algorithm, which proves the superiority of the algorithm and its role in solving the recommendation problem after applying the recommendation. The music data in the database data conversion effectively improve the efficiency and accuracy of mining. According to the implementation of the algorithm described in this article, the accuracy of the music recommendation results used to recommend user satisfaction is proved. And the recommended music is indeed feasible and practical.

Edge-Cloud Collaboration Enabled Video Service Enhancement: A Hybrid Human-Artificial Intelligence Scheme

In this paper, a video service enhancement strategy is investigated under an edge-cloud collaboration framework, where video caching and delivery decisions are made at the cloud and edge respectively. We aim to guarantee the user fairness in terms of video coding rate under statistical delay constraint and edge caching capacity constraint. A hybrid human-artificial intelligence approach is developed to improve the user hit rate for video caching. Specifically, individual user interest is first characterized by merging factorization machine (FM) model and multi-layer perceptron (MLP) model, where both low-order and high-order features can be well learned simultaneously. Thereafter, a social aware similarity model is constructed to transfer individual user interest to group interest, based on which, videos can be selected to cache at the network edge. Furthermore, a dual bisection exploration scheme is proposed to optimize wireless resource allocation and video coding rate. The effectiveness of the proposed video caching and delivery scheme is finally validated by extensive experiments with a real-world dataset.

Graph-Convolved Factorization Machines for Personalized Recommendation

Factorization machines (FMs) and their neural network variants (neural FMs) for modeling second-order feature interactions are effective in building modern recommendation systems. However, feature interactions are based upon pairs of features, whereas multi-features correlations commonly arise in real-world financial product recommendation scenarios. We propose an effective neural recommender system, graph-convolved factorization machine (GCFM), with the spirit of the symbolic graph reasoning principle that provides lightweight and interpretable recommendation suggestions. Given a sample for the recommendation, GCFM constructs the corresponding dense feature embeddings and computes the sample-specific feature relationship graph. Then, a multi-filter graph-convolved feature crossing (GCFC) layer for feature embeddings establishes cross features with their neighboring embeddings. GCFM thus extends the feature interactions from pairs to neighbors to capture more comprehensive and explainable information while simultaneously reaping the advantages of representation learning. To exploit these capabilities, we apply a Graph Bayesian Optimization (GBO). During training, our GBO automatically optimizes our GCFM, including training hyperparameters and architecture hyperparameters. Besides, we conduct extensive experiments on two public financial applications benchmarks, USCFC and OTC, and two real-world datasets that we collect offline. Our GCFM significantly outperforms state-of-the-art algorithms and shows its interpretability in recommendation tasks.

Robust Detection of Malicious URLs with Self-Paced Wide & Deep Learning

As cybercrimes grow in scale with devastating economic costs, it is important to protect potential victims against diverse attacks. It is the uniform resource locators (URLs) that connect vulnerable users with potential attacks. Although numerous solutions (e.g., rule-based solutions and machine learning-based methods) are proposed for malicious URL detection, they can not provide robust performance due to the diversity of cybercrimes and can not cope with the explosive growth of malicious URLs with the evolution of obfuscation strategies. In this paper, we propose a deep learning-based system, dubbed as CyberLen, to detect malicious URLs robustly and effectively. Specifically, we use factorization machine (FM) to learn the latent interaction among lexical features. For the deep structural features, position embedding is introduced for token vectorization to reduce the ambiguity of URL tokens. Meanwhile, temporal convolution network (TCN) is utilized to learn the long-distance dependency among URL tokens. To fuse heterogeneous features, self-paced wide & deep learning strategy is proposed to train a robust model effectively. The proposed solution is evaluated on a large-scale URL dataset. Our experimental results show that position embedding is constructive to reducing the ambiguity of URL tokens, and the self-paced wide & deep learning strategy shows superior performance in terms of F1 score and convergence speed.

Modeling low- and high-order feature interactions with FM and self-attention network

Click-Through Rate (CTR) prediction has always been a very popular topic. In many online applications, such as online advertising and product recommendation, a small increase in CTR will bring great returns. However, CTR prediction has always faced several challenges. A large number of users and items and the different sizes of the feature space of different data types lead to high-dimensional and sparse input, and high-order feature interactions rely too much on expert knowledge and are very time-consuming. In this paper, we build a novel model called multi-order interactive features aware factorization machine (MoFM) for CTR prediction. To effectively capturing both low-order and high-order interactive features, three different types of prediction models are integrated, of which logistic regression (LR) and factorization machine (FM) model the original features and 2-order interactive features respectively, and a multi-head self-attention network with residual connections is used to automatically identify high-value high-order feature combinations. There is also an embedding layer in the model to realize a unified embedding processing of different data types, avoiding diversification, sparsity, and high dimensionality of features. Since, feature engineering is not required, we can carry out end-to-end model learning. Experiments on three public datasets show the superiority of the proposed model over the state-of-the-art models, and the flexibility and scalability of the model structure have also been verified.

Enhancing Factorization Machines With Generalized Metric Learning

Factorization Machines (FMs) are effective in incorporating side information to overcome the cold-start and data sparsity problems in recommender systems. Traditional FMs adopt the inner product to model the second-order interactions between different attributes, which are represented via feature vectors. The problem is that the inner product violates the triangle inequality property of feature vectors. As a result, it cannot well capture fine-grained attribute interactions, resulting in sub-optimal performance. Recently, the euclidean distance is exploited in FMs to replace the inner product and has delivered better performance. However, previous FM methods including the ones equipped with the euclidean distance all focus on the attribute-level interaction modeling, ignoring the critical intrinsic feature correlations inside attributes. Thereby, they fail to model the complex and rich interactions exhibited in the real-world data. To tackle this problem, in this paper, we propose a FM framework equipped with generalized metric learning techniques to better capture these feature correlations. In particular, based on this framework, we present a Mahalanobis distance and a deep neural network (DNN) methods, which can effectively model the linear and non-linear correlations between features, respectively. Besides, we design an efficient approach for simplifying the model functions. Experiments on several benchmark datasets demonstrate that our proposed framework outperforms several state-of-the-art baselines by a large margin. Moreover, we collect a new large-scale dataset on second-hand trading to justify the effectiveness of our method over cold-start and data sparsity problems in recommender systems.

Neural attention model for recommendation based on factorization machines

In recommendation systems, it is of vital importance to comprehensively consider various aspects of information to make accurate recommendations for users. When the low-order feature interactions between items are insufficient, it is necessary to mine information to learn higher-order feature interactions. In addition, to distinguish the different importance levels of feature interactions, larger weights should be assigned to features with larger contributions to predictions, and smaller weights to those with smaller contributions. Therefore, this paper proposes a neural attention model for recommendation (NAM), which deepens factorization machines (FMs) by adding an attention mechanism and fully connected layers. Through the attention mechanism, NAM can learn the different importance levels of low-order feature interactions. By adding fully connected layers on top of the attention component, NAM can model high-order feature interactions in a nonlinear way. Experiments on two real-world datasets demonstrate that NAM has excellent performance and is superior to FM and other state-of-the-art models. The results demonstrate the effectiveness of the proposed model and the potential of using neural networks for prediction under sparse data.

AIRec: Attentive intersection model for tag-aware recommendation

Tag-aware recommender systems (TRS) utilize rich tagging information to better depict user portraits and item features. Recently, many efforts have been done to improve TRS with neural networks. However, existing methods construct user representations through either explicit tagging behaviors or implicit interacted items, which is inadequate to capture multi-aspect user preferences. Besides, there are still lacks of investigation about the intersection between user and item tags, which is crucial for better recommendation.In this paper, we propose AIRec, an attentive intersection model for TRS, to address the above issues. More precisely, we first project the sparse tag vectors into a latent space through multi-layer perceptron (MLP). Then, the user representations are constructed with a hierarchical attention network, where the item-level attention differentiates the contributions of interacted items and the preference-level attention discriminates the saliencies between explicit and implicit preferences. After that, the intersection between user and item tags is exploited to enhance the learning of conjunct features. Finally, the user and item representations are concatenated and fed to factorization machines (FM) for score prediction. We conduct extensive experiments on two real-world datasets, demonstrating significant improvements of AIRec over state-of-the-art methods for tag-aware top-n recommendation.

Toward better prediction of recurrence for Cushing’s disease: a factorization-machine based neural approach

Cushing's disease (CD) is a rare disease that occurs in 1.2–1.4 persons per million population per year. Recurrence prediction after transsphenoidal surgery (TSS) is important for determining individual treatment and follow-up strategies. Between 2000 and 2017, 354 CD patients with initial postoperative remission and long-term follow-up data were enrolled from Peking union medical college hospital (PUMCH) to predict recurrence, and PUMCH is one of the largest CD treatment centers in the world. We first investigated the effect of a factorization machine (FM)-based neural network to predict recurrence after TSS for CD. This method could automatically reduce a portion of the cross-feature selection work with acceptable parameters. We conducted a performance comparison of various algorithms on the collected dataset. To address the lack of interpretability of neural network models, we also used the local interpretable model-agnostic explanations approach, which provides an explanation in the form of relevant features of the predicted results by approximating the model behavior of the variables in a local manner. Compared with existing methods, the DeepFM model obtained the highest AUC value (0.869) and the lowest log loss value (0.256). According to the importance of each feature, three top features for the DeepFM model were postoperative morning adrenocorticotropic hormone level, age, and postoperative morning serum cortisol nadir. In the post hoc explanation phase, the above-mentioned importance-leading features made a great contribution to the prediction probability. The results showed that deep learning-based models could better aid neurosurgeons in recurrence prediction after TTS for patients with CD, and could contribute to determining individual treatment strategies.

Agricultural product E‐commerce recommendation system based on neural factorization machine in smart cities

Recently, e‐commerce becomes a new way to sell agricultural products, such as orange, apple, tea, rice. For the farmers, they can obtain more earnings by avoid intermediate links. For the customers, they have more choices and can buy what they want. The e‐commerce directly links farmers and customers together and recommends personalized agricultural products to customers. In order to recommend proper agricultural products to customers, we propose an agricultural product e‐commerce recommendation system in which neural factorization Machine (NFM) is used as recommendation algorithm. NFM combines factorization machine (FM) and neural network (NN) and captures high‐order information between features. Therefore, the generalization of NFM is better than previous recommendation algorithms. Experimental results show that NFM can improve the recommendation accuracy of the agricultural product e‐commerce system about 3%‐4%.

Practical Privacy Preserving POI Recommendation

Point-of-Interest (POI) recommendation has been extensively studied and successfully applied in industry recently. However, most existing approaches build centralized models on the basis of collecting users’ data. Both private data and models are held by the recommender, which causes serious privacy concerns. In this article, we propose a novel Privacy preserving POI Recommendation (PriRec) framework. First, to protect data privacy, users’ private data (features and actions) are kept on their own side, e.g., Cellphone or Pad. Meanwhile, the public data that need to be accessed by all the users are kept by the recommender to reduce the storage costs of users’ devices. Those public data include: (1) static data only related to the status of POI, such as POI categories, and (2) dynamic data dependent on user-POI actions such as visited counts. The dynamic data could be sensitive, and we develop local differential privacy techniques to release such data to the public with privacy guarantees. Second, PriRec follows the representations of Factorization Machine (FM) that consists of a linear model and the feature interaction model. To protect the model privacy, the linear models are saved on the users’ side, and we propose a secure decentralized gradient descent protocol for users to learn it collaboratively. The feature interaction model is kept by the recommender since there is no privacy risk, and we adopt a secure aggregation strategy in a federated learning paradigm to learn it. To this end, PriRec keeps users’ private raw data and models in users’ own hands, and protects user privacy to a large extent. We apply PriRec in real-world datasets, and comprehensive experiments demonstrate that, compared with FM, PriRec achieves comparable or even better recommendation accuracy.

A Generalized Locally Linear Factorization Machine with Supervised Variational Encoding

Factorization Machines (FMs) learn weights for feature interactions, and achieve great success in many data mining tasks. Recently, Locally Linear Factorization Machines (LLFMs) have been proposed to capture the underlying structures of data for better performance. However, one obvious drawback of LLFM is that the local coding is only operated in the original feature space, which limits the model to be applied to high-dimensional and sparse data. In this work, we present a generalized LLFM (GLLFM) which overcomes this limitation by modeling the local coding procedure in a latent space. Moreover, a novel Supervised Variational Encoding (SVE) technique is proposed such that the distance can effectively describe the similarity between data points. Specifically, the proposed GLLFM-SVE trains several local FMs in the original space to model the higher order feature interactions effectively, where each FM associates to an anchor point in the latent space induced by SVE. The prediction for a data point is computed by a weighted sum of several local FMs, where the weights are determined by local coding coordinates with anchor points. Actually, GLLFM-SVE is quite flexible and other Neural Network (NN) based FMs can be easily embedded into this framework. Experimental results show that GLLFM-SVE significantly improves the performance of LLFM. By using NN-based FMs as local predictors, our model outperforms all the state-of-the-art methods on large-scale real-world benchmarks with similar number of parameters and comparable training time.