High-order Feature Interactions Research Articles

Measuring and classifying IP usage scenarios: a continuous neural trees approach

Understanding user behavior via IP addresses is a crucial measure towards numerous pragmatic IP-based applications, including online content delivery, fraud prevention, marketing intelligence, and others. While profiling IP addresses through methods like IP geolocation and anomaly detection has been thoroughly studied, the function of an IP address—e.g., whether it pertains to a private enterprise network or a home broadband—remains underexplored. In this work, we initiate the first attempt to address the IP usage scenario classification problem. We collect data consisting of IP addresses from four large-scale regions. A novel continuous neural tree-based ensemble model is proposed to learn IP assignment rules and complex feature interactions. We conduct extensive experiments to evaluate our model in terms of classification accuracy and generalizability. Our results demonstrate that the proposed model is capable of efficiently uncovering significant higher-order feature interactions that enhance IP usage scenario classification, while also possessing the ability to generalize from the source region to the target one.

Location-Aware Deep Interaction Forest for Web Service QoS Prediction

With the rapid development of the web service market, the number of web services shows explosive growth. QoS is an important factor in the recommendation scene; how to accurately recommend a high-quality service for users among the massive number of web services has become a tough problem. Previous methods usually acquired feature interaction information by network structures like DNN to improve the QoS prediction accuracy, but this generates unnecessary computations. Aiming at addressing the above problem, inspired by the multigrained scanning mechanism in a deep forest, we propose a location-aware deep interaction forest approach for web service QoS prediction (LDIF). This approach offers the following innovations: The model fuses the location similarity of users and services as a latent feature representation of them. In addition, we designed a scanning interaction structure (SIS), which obtains multiple local feature combinations from the interaction between user and service features, uses interactive computing to extract feature interaction information, and concatenates the feature interaction information with original features, which aims to enhance the dimension of the features. Equipped with these, we compose a layer-by-layer cascade by using SIS to fuse low- and high-order feature interaction information, and the early-stop mechanism controls the cascade depth to avoid unnecessary computation. The experiments demonstrate that our model outperforms eight other state-of-the-art methods on MAE and RMSE common metrics on real public datasets.

IeMTLF: Interaction-enhanced Multi-Task Learning Framework for next location prediction

Location-Based Services (LBSs) provide spatial and semantic information of various locations, thanks to the advances in mobile devices and commercial map services. However, most existing studies on next location prediction neglect the semantic information and do not investigate the effects of different levels of semantic granularity. To address these issues, we propose an Interaction-enhanced Multi-Task Learning Framework (IeMTLF) that jointly predicts the next location and its semantic category and leverages higher-order feature interactions to enrich the semantic learning process. Moreover, we examine the effect of two different levels of semantic granularity, namely, specific and abstract, on prediction performance. We conducted extensive experiments on two real-world datasets and demonstrate that IeMTLF outperforms seven baselines on all evaluation metrics, with a maximum gain of more than 13% for Acc@5 compared to the state-of-the-art method. Our implementation code is available at https://github.com/mrc-wyh/IeMTLF.

Feature space reduction method for ultrahigh-dimensional, multiclass data: random forest-based multiround screening (RFMS)

In recent years, several screening methods have been published for ultrahigh-dimensional data that contain hundreds of thousands of features, many of which are irrelevant or redundant. However, most of these methods cannot handle data with thousands of classes. Prediction models built to authenticate users based on multichannel biometric data result in this type of problem. In this study, we present a novel method known as random forest-based multiround screening (RFMS) that can be effectively applied under such circumstances. The proposed algorithm divides the feature space into small subsets and executes a series of partial model builds. These partial models are used to implement tournament-based sorting and the selection of features based on their importance. This algorithm successfully filters irrelevant features and also discovers binary and higher-order feature interactions. To benchmark RFMS, a synthetic biometric feature space generator known as BiometricBlender is employed. Based on the results, the RFMS is on par with industry-standard feature screening methods, while simultaneously possessing many advantages over them.

AutoEIS: Automatic feature embedding, interaction and selection on default prediction

Deep models have shown the effectiveness in various areas, e.g., finance, healthcare and recommendation system. Among them, default prediction is a major application in the financial field. However, there are still problems remained, like insufficient learning for numerical feature encoding and difficulty of explicitly modelling high-order feature interactions. To address these issues, we propose an automatic end-to-end deep learning framework named AutoEIS. In this framework, we embed categorical and numerical features with distinct strategies. In particular, we design a Multi-field-aware Mixture-of-Experts (MfMoE) structure for numerical value embedding, which can simultaneously learn the single-field and global-field information. Then, by organically integrating attention mechanism, weighted-average aggregation and bilinear interaction, we can effectively generate significant high-order explicit interactions. Moreover, we integrate a DNN block to further capture the complex relationships among different variables. Comprehensive experiments on two real-world datasets of about 30,000 samples show the superiority of AutoEIS on default prediction, boosting average AUC and KS metrics over the best classical baseline by 0.49% and 3.5%, and the best of deep baselines by 0.48% and 2.55%. Furthermore, as a model-agnostic strategy, we can generalize MfMoE to other deep models like DeepFM, thereby boosting their performance.

Service Recommendation of Industrial Software Components Based on Explicit and Implicit Higher-Order Feature Interactions and Attentional Factorization Machines

In the context of the rapid advancement of the Industrial Internet and Urban Internet, a crucial trend is emerging in the realization of unified, service-oriented, and componentized encapsulation of IT and OT heterogeneous entities underpinned by a service-oriented architecture. This is pivotal for achieving componentized construction and development of extensive industrial software systems. In addressing the diverse demands of application tasks, the efficient and precise recommendation of service components has emerged as a pivotal concern. Existing recommendation models either focus solely on low-order interactions or emphasize high-order interactions, disregarding the distinction between implicit and explicit aspects within high-order interactions as well as the integration of high-order and low-order interactions. This oversight leads to subpar accuracy in recommendations. Real-world data exhibit intricate structures and nonlinearity. In practical applications, different interaction components exhibit varying predictive capabilities. Therefore, in this paper we propose an EIAFM model that fuses explicit and implicit higher-order feature interactions and introduce an attention mechanism to identify which low-level feature interactions contribute more significantly to the prediction results. This approach leads to increased interpretability, combining both generalization and memory capabilities. Through comprehensive experiments on authentic datasets that align with the characteristics of the Service Recommendation of Industrial Software Components problem, we demonstrate that the EIAFM model excels compared to other cutting-edge models in terms of recommendation effectiveness, with the evaluation metrics for the AUC and log-loss reaching values of 0.9281 and 0.3476, respectively.

BaGFN: Broad Attentive Graph Fusion Network for High-Order Feature Interactions.

Modeling feature interactions is of crucial significance to high-quality feature engineering on multifiled sparse data. At present, a series of state-of-the-art methods extract cross features in a rather implicit bitwise fashion and lack enough comprehensive and flexible competence of learning sophisticated interactions among different feature fields. In this article, we propose a new broad attentive graph fusion network (BaGFN) to better model high-order feature interactions in a flexible and explicit manner. On the one hand, we design an attentive graph fusion module to strengthen high-order feature representation under graph structure. The graph-based module develops a new bilinear-cross aggregation function to aggregate the graph node information, employs the self-attention mechanism to learn the impact of neighborhood nodes, and updates the high-order representation of features by multihop fusion steps. On the other hand, we further construct a broad attentive cross module to refine high-order feature interactions at a bitwise level. The optimized module designs a new broad attention mechanism to dynamically learn the importance weights of cross features and efficiently conduct the sophisticated high-order feature interactions at the granularity of feature dimensions. The final experimental results demonstrate the effectiveness of our proposed model.

Gated Multi-task Learning for QoS-aware Service Recommendation

In the ever-increasing diversity of cloud environment, the recommendation of web service-related applications based on service quality is a basic tasks that both service providers and web developers are very interested in. A key challenge is how to get personalized and accurate QoS values of web services from the multi-source information of users/services. In addition, complex contexts and multi-tasking attributes also make the collaborative services quality prediction difficult to meet. Inspired by the progress of multi-task deep learning, this paper proposes a new framework of service quality prediction called multi-task gated expert network (MGEN), which is based on gating mechanism and multi-task learning. The MGEN consists of three parts: expert-based context-source feature extraction, gated interaction learning and information fusion. First, multiple expert networks are used to capture the latent representation in raw data of web services, including both implicit features in the QoS matrix and explicit features of service descriptions. In the process, several independent experts extract features from different dimensions. Second, output embedding of different expert networks is transmitted to each task layer through different weights, learning both global and local high-order feature interactions. Third, the interactive information of different tasks is fused by embedding splicing, and then fed back to the prediction module for final service recommendation. Results of the comparative experiment demonstrate that the proposed MGEN significantly outperforms the state-of-the-art models of service recommendation, where both the implicit and explicit information of various multi-source features and interaction sequences are effectively extracted. In addition, the gating mechanism performs well in the QoS prediction by independently modeling each task.

A knowledge distillation-based deep interaction compressed network for CTR prediction

Click-through rate (CTR) prediction is an essential task in online advertising recommender systems. As a hot research frontier driven by academic and industrial needs, CTR prediction has observed remarkable progress in recent years. How to effectively learn sophisticated feature interactions hidden behind user behaviors is critical in maximizing CTR. Targeted at the above issue, this paper first constructs a deep interaction compressed network model (abbreviated as DICN) that aims to generate both low- and high-order feature interactions with different weights in both explicit and implicit fashions. Then, a KD-DICN model is created using a knowledge distillation framework to reduce the complexity of DICN. Finally, comprehensive experiments are conducted on two public datasets to demonstrate the effectiveness and efficiency of our models, which outperform the existing state-of-the-art methods. Therefore, the research presented in this paper can not only enrich and develop the theories related to CTR prediction, but it can also assist some advertising industries in better defining their service audience and recommending more popular contents to consumers more accurately.

AAIN: Attentional aggregative interaction network for deep learning based recommender systems

Feature engineering is a classical problem in recommender systems, and feature interactions is one of the most important parts of feature engineering. Factorization based models are widely used for explicit feature interactions. However, most current works utilize separate features to model cross features. Such a pattern limits the significance of cross features, since realistic recommendation scenarios are rich in associations between features.In this paper, we classify the basic feature interactions into sum-interaction and product-interaction, and improve the current general strategy of explicit feature interactions. Based on these theoretical studies, we propose a novel explicit feature interactions model Attentional Aggregative Interaction Network (AAIN), which models higher-order features using a cyclic explicit module. Specifically, we introduce attention mechanism for the reorganization of separate features, followed by product-interaction and higher-order features’ compression and output. The model is efficient since: 1) AAIN automatically learns high-order feature interactions and filters them with different weights. 2) AAIN optimizes the interaction between features into the interaction between feature groups, which allows for other relevant information to be considered when performing interactions. Furthermore, we integrate AAIN model with the classical deep neural network (DNN) model into a new model Deep Attentional Aggregative Interaction Network (DAAIN). Experiments on real-world datasets show that our models achieve state-of-the-art results.

Biomarker Identification via a Factorization Machine-Based Neural Network With Binary Pairwise Encoding.

Biomolecules, microRNAs (miRNAs) and long non-coding RNAs (lncRNAs), play critical roles in diverse fundamental and vital biological processes. They can serve as disease biomarkers as their dysregulations could cause complex human diseases. Identifying those biomarkers is helpful with the diagnosis, treatment, prognosis, and prevention of diseases. In this study, we propose a factorization machine-based deep neural network with binary pairwise encoding, DFMbpe, to identify the disease-related biomarkers. First, to comprehensively consider the interdependence of features, a binary pairwise encoding method is designed to obtain the raw feature representations for each biomarker-disease pair. Second, the raw features are mapped into their corresponding embedding vectors. Then, the factorization machine is conducted to get the wide low-order feature interdependence, while the deep neural network is applied to obtain the deep high-order feature interdependence. Finally, two kinds of features are combined to get the final prediction results. Unlike other biomarker identification models, the binary pairwise encoding considers the interdependence of features even though they never appear in the same sample, and the DFMbpe architecture emphasizes both low-order and high-order feature interactions simultaneously. The experimental results show that DFMbpe greatly outperforms the state-of-the-art identification models on both cross-validation and independent dataset evaluation. Besides, three types of case studies further demonstrate the effectiveness of this model.

LFDNN: A Novel Hybrid Recommendation Model Based on DeepFM and LightGBM

Hybrid recommendation algorithms perform well in improving the accuracy of recommendation systems. However, in specific applications, they still cannot reach the requirements of the recommendation target due to the gap between the design of the algorithms and data characteristics. In this paper, in order to learn higher-order feature interactions more efficiently and to distinguish the importance of different feature interactions better on the prediction results of recommendation algorithms, we propose a light and FM deep neural network (LFDNN), a hybrid recommendation model including four modules. The LightGBM module applies gradient boosting decision trees for feature processing, which improves LFDNN's ability to handle dense numerical features; the shallow model introduces the FM model for explicitly modeling the finite-order feature crosses, which strengthens the expressive ability of the model; the deep neural network module uses a fully connected feedforward neural network to allow the model to obtain more high-order feature crosses information and mine more data patterns in the features; finally, the Fusion module allows the shallow model and the deep model to obtain a better fusion effect. The results of comparison, parameter influence and ablation experiments on two real advertisement datasets shows that the LFDNN reaches better performance than the representative recommendation models.

FMGNN: A Method to Predict Compound-Protein Interaction With Pharmacophore Features and Physicochemical Properties of Amino Acids.

Identifying interactions between compounds and proteins is an essential task in drug discovery. To recommend compounds as new drug candidates, applying the computational approaches has a lower cost than conducting the wet-lab experiments. Machine learning-based methods, especially deep learning-based methods, have advantages in learning complex feature interactions between compounds and proteins. However, deep learning models will over-generalize and lead to the problem of predicting less relevant compound-protein pairs when the compound-protein feature interactions are high-dimensional sparse. This problem can be overcome by learning both low-order and high-order feature interactions. In this paper, we propose a novel hybrid model with Factorization Machines and Graph Neural Network called FMGNN to extract the low-order and high-order features, respectively. Then, we design a compound-protein interactions (CPIs) prediction method with pharmacophore features of compound and physicochemical properties of amino acids. The pharmacophore features can ensure that the prediction results much more fit the expectation of biological experiment and the physicochemical properties of amino acids are loaded into the embedding layer to improve the convergence speed and accuracy of protein feature learning. The experimental results on several datasets, especially on an imbalanced large-scale dataset, showed that our proposed method outperforms other existing methods for CPI prediction. The western blot experiment results on wogonin and its candidate target proteins also showed that our proposed method is effective and accurate for finding target proteins. The computer program of implementing the model FMGNN is available at https://github.com/tcygxu2021/FMGNN.

Nearest Neighbor-Based Contrastive Learning for Hyperspectral and LiDAR Data Classification

The joint hyperspectral image (HSI) and light detection and ranging (LiDAR) data classification aims to interpret ground objects at more detailed and precise level. Although deep learning methods have shown remarkable success in the multisource data classification task, self-supervised learning has rarely been explored. It is commonly nontrivial to build a robust self-supervised learning model for multisource data classification, due to the fact that the semantic similarities of neighborhood regions are not exploited in the existing contrastive learning framework. Furthermore, the heterogeneous gap induced by the inconsistent distribution of multisource data impedes the classification performance. To overcome these disadvantages, we propose a nearest neighbor-based contrastive learning network (NNCNet), which takes full advantage of large amounts of unlabeled data to learn discriminative feature representations. Specifically, we propose a nearest neighbor-based data augmentation scheme to use enhanced semantic relationships among nearby regions. The intermodal semantic alignments can be captured more accurately. In addition, we design a bilinear attention module to exploit the second-order and even high-order feature interactions between the HSI and LiDAR data. Extensive experiments on four public datasets demonstrate the superiority of our NNCNet over state-of-the-art methods. The source codes are available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/summitgao/NNCNet</uri> .

A Dual Adaptive Interaction Click-Through Rate Prediction Based on Attention Logarithmic Interaction Network.

Click-through rate (CTR) prediction is crucial for computing advertisement and recommender systems. The key challenge of CTR prediction is to accurately capture user interests and deliver suitable advertisements to the right people. However, there are an immense number of features in CTR prediction datasets, which hardly fit when only using an individual feature. To solve this problem, feature interaction that combines several features via an operation is introduced to enhance prediction performance. Many factorizations machine-based models and deep learning methods have been proposed to capture feature interaction for CTR prediction. They follow an enumeration-filter pattern that could not determine the appropriate order of feature interaction and useful feature interaction. The attention logarithmic network (ALN) is presented in this paper, which uses logarithmic neural networks (LNN) to model feature interactions, and the squeeze excitation (SE) mechanism to adaptively model the importance of higher-order feature interactions. At first, the embedding vector of the input was absolutized and a very small positive number was added to the zeros of the embedding vector, which made the LNN input positive. Then, the adaptive-order feature interactions were learned by logarithmic transformation and exponential transformation in the LNN. Finally, SE was applied to model the importance of high-order feature interactions adaptively for enhancing CTR performance. Based on this, the attention logarithmic interaction network (ALIN) was proposed for the effectiveness and accuracy of CTR, which integrated Newton's identity into ALN. ALIN supplements the loss of information, which is caused by the operation becoming positive and by adding a small positive value to the embedding vector. Experiments are conducted on two datasets, and the results prove that ALIN is efficient and effective.

FIRE: knowledge-enhanced recommendation with feature interaction and intent-aware attention networks.

To solve the information overload issue and enhance the user experience of various web applications, recommender systems aim to better model user interests and preferences. Knowledge Graphs (KGs), consisting of real-world objective facts and fruitful entities, play a vital role in recommender systems. Recently, a technological trend has been to develop end-to-end Graph Neural Networks (GNNs)-based knowledge-aware recommendation (a.k.a., Knowledge Graph Recommendation, KGR) models. Unfortunately, current GNNs-based KGR approaches focus on how to capture high-order feature information on KGs while neglecting the following two crucial limitations: 1) The explicitly high-order feature interaction and fusion mechanism and 2) The valid user intent modelling mechanism. As such, these issues lead to insufficient user/item representation learning capability and unsatisfactory KGR performance. In this work, we present a novel Knowledge-enhanced Re commendation with F eature I nteraction and Intent-aware Attention Networks (FIRE) to address the latent intent modelling and high-order feature interaction deficiencies ignored by existing KGR methods. Based on the prototype user/item representation learning leveraging the GNNs-based approach, our model offers the following major improvements: One is the innovative use of Convolutional Neural Networks (CNNs) that perform vertical convolutional (a.k.a., bit-level convolutional) and horizontal convolutional (a.k.a., vector-level convolutional) processes to model multi-granular high-order feature interactions to enhance item-side representation learning. Another is to model users' latent intent factors by utilizing a two-level attention mechanism (i.e., node- and intent-level attention mechanism) to enhance user-side representation learning. Extensive experiments on three KGs domain public datasets demonstrate that our method outperforms the existing state-of-the-art baseline. Last but not least, numerous ablation- and model studies demystify the working mechanism and elucidate the plausibility of the proposed model.

A Ranking Recommendation Algorithm Based on Dynamic User Preference.

In recent years, hybrid recommendation techniques based on feature fusion have gained extensive attention in the field of list ranking. Most of them fuse linear and nonlinear models to simultaneously learn the linear and nonlinear features of entities and jointly fit user-item interactions. These methods are based on implicit feedback, which can reduce the difficulty of data collection and the time of data preprocessing, but will lead to the lack of entity interaction depth information due to the lack of user satisfaction. This is equivalent to artificially reducing the entity interaction features, limiting the overall performance of the model. To address this problem, we propose a two-stage recommendation model named A-DNR, short for Attention-based Deep Neural Ranking. In the first stage, user short-term preferences are modeled through an attention mechanism network. Then the user short-term preferences and user long-term preferences are fused into dynamic user preferences. In the second stage, the high-order and low-order feature interactions are modeled by a matrix factorization (MF) model and a multi-layer perceptron (MLP) model, respectively. Then, the features are fused through a fully connected layer, and the vectors are mapped to scores. Finally, a ranking list is output through the scores. Experiments on three real-world datasets (Movielens100K, Movielens1M and Yahoo Movies) show that our proposed model achieves significant improvements compared to existing methods.

A Multi-modal Feature Fusion-based Approach for Mobile Application Classification and Recommendation

&lt;p&gt;With the rapid growth of the number and type of mobile applications, it becomes challenging to accurately classify and recommend mobile applications according to users&amp;rsquo; individual requirements. The existing mobile application classification and recommendation methods, for one thing, do not take into account the correlation between large-scale data and model. For another, they also do not fully exploit the multi-modal, fine-grained interaction features with high-order and low-order in mobile application. To tackle this problem, we propose a mobile application classification and recommendation method based on multi-modal feature fusion. The method firstly extracts the image and description features of the mobile application using an &amp;ldquo;involution residual network + pre-trained language representation&amp;rdquo; model (i.e. the TRedBert model). Afterwards, these features are fused by using the attention mechanism in the transformer model. Then, the method classifies the mobile applications based on the fused features through a softmax classifier. Finally, the method extracts the high-order and low-order embedding features of the mobile app with a bi-linear feature interaction model (FiBiNET) based on the classification results of the mobile app, by combining the Hadamard product and inner product to achieve fine-grained high-order and low-order feature interaction, to update the mobile app representation and complete the recommendation task. The multiple sets of comparison experiments are performed on Kaggle&amp;rsquo;s real dataset, i.e., 365K IOS Apps Dataset. And the experimental results demonstrated that the proposed approach outperforms other methods in terms of Macro F1, Accuracy, AUC and Logloss.&lt;/p&gt; &lt;p&gt;&amp;nbsp;&lt;/p&gt;

GAIN: A Gated Adaptive Feature Interaction Network for Click-Through Rate Prediction.

CTR (Click-Through Rate) prediction has attracted more and more attention from academia and industry for its significant contribution to revenue. In the last decade, learning feature interactions have become a mainstream research direction, and dozens of feature interaction-based models have been proposed for the CTR prediction task. The most common approach for existing models is to enumerate all possible feature interactions or to learn higher-order feature interactions by designing complex models. However, a simple enumeration will introduce meaningless and harmful interactions, and a complex model structure will bring a higher complexity. In this work, we propose a lightweight, yet effective model called the Gated Adaptive feature Interaction Network (GAIN). We devise a novel cross module to drop meaningless feature interactions and preserve informative ones. Our cross module consists of multiple gated units, each of which can independently learn an arbitrary-order feature interaction. We combine the cross module with a deep module into GAIN and conduct comparative experiments with state-of-the-art models on two public datasets to verify its validity. Our experimental results show that GAIN can achieve a comparable or even better performance compared to its competitors. Furthermore, in order to verify the effectiveness of the feature interactions learned by GAIN, we transfer learned interactions to other models, such as Logistic Regression (LR) and Factorization Machines (FM), and find out that their performance can be significantly improved.

Kernelized convolutional transformer network based driver behavior estimation for conflict resolution at unsignalized roundabout

The modeling of driver behavior plays an essential role in developing Advanced Driver Assistance Systems (ADAS) to support the driver in various complex driving scenarios. The behavior estimation of surrounding vehicles is crucial for an autonomous vehicle to safely navigate through an unsignalized intersection. This work proposes a novel kernelized convolutional transformer network (KCTN) with multi-head attention (MHA) mechanism to estimate driver behavior at a challenging unsignalized three-way roundabout. More emphasis has been placed on creating convolution in non-linear space by introducing a kervolution operation into the proposed network. It generalizes convolution, improves model capacity, and captures higher-order feature interactions by using Gaussian kernel function. The proposed model is validated using the real-world ACFR dataset, where it outperforms current state-of-the-art in terms of behavior prediction accuracy and provides a significant lead time before potential conflict situations.