High-order Feature Interactions Research Articles

Interweaving Insights: High-Order Feature Interaction for Fine-Grained Visual Recognition

AbstractThis paper presents a novel approach for Fine-Grained Visual Classification (FGVC) by exploring Graph Neural Networks (GNNs) to facilitate high-order feature interactions, with a specific focus on constructing both inter- and intra-region graphs. Unlike previous FGVC techniques that often isolate global and local features, our method combines both features seamlessly during learning via graphs. Inter-region graphs capture long-range dependencies to recognize global patterns, while intra-region graphs delve into finer details within specific regions of an object by exploring high-dimensional convolutional features. A key innovation is the use of shared GNNs with an attention mechanism coupled with the Approximate Personalized Propagation of Neural Predictions (APPNP) message-passing algorithm, enhancing information propagation efficiency for better discriminability and simplifying the model architecture for computational efficiency. Additionally, the introduction of residual connections improves performance and training stability. Comprehensive experiments showcase state-of-the-art results on benchmark FGVC datasets, affirming the efficacy of our approach. This work underscores the potential of GNN in modeling high-level feature interactions, distinguishing it from previous FGVC methods that typically focus on singular aspects of feature representation. Our source code is available at https://github.com/Arindam-1991/I2-HOFI.

Research on the Wild Mushroom Recognition Method Based on Transformer and the Multi-Scale Feature Fusion Compact Bilinear Neural Network

Wild mushrooms are popular for their taste and nutritional value; however, non-experts often struggle to distinguish between toxic and non-toxic species when foraging in the wild, potentially leading to poisoning incidents. To address this issue, this study proposes a compact bilinear neural network method based on Transformer and multi-scale feature fusion. The method utilizes a dual-stream structure that integrates multiple feature extractors, enhancing the comprehensiveness of image information capture. Additionally, bottleneck attention and efficient multi-scale attention modules are embedded to effectively capture multi-scale features while maintaining low computational costs. By employing a compact bilinear pooling module, the model achieves high-order feature interactions, reducing the number of parameters without compromising performance. Experimental results demonstrate that the proposed method achieves an accuracy of 98.03%, outperforming existing comparative methods. This proves the superior recognition performance of the model, making it more reliable in distinguishing wild mushrooms while capturing key information from multiple dimensions, enabling it to better handle complex scenarios. Furthermore, the development of public-facing identification tools based on this method could help reduce the risk of poisoning incidents. Building on these findings, the study suggests strengthening the research and development of digital agricultural technologies, promoting the application of intelligent recognition technologies in agriculture, and providing technical support for agricultural production and resource management through digital platforms. This would provide a theoretical foundation for the innovation of digital agriculture and promote its sustainable development.

Multi-order feature interaction-aware intrusion detection scheme for ensuring cyber security of intelligent connected vehicles

The evolution of technology has raised concerns regarding cybersecurity for intelligent connected vehicles (ICVs). In-vehicle network in ICVs lacks robust protection mechanisms, making it vulnerable to cyber threats. In response, intrusion detection systems (IDSs) for ICVs have been developed to protect vehicles from malicious cyber attacks. However, current IDS methods solely rely on independent features, limiting their learning capabilities and increasing the number of false detections. Moreover, many IDSs require the knowledge of mapping between network messages and contents, which restricts their application. To address these limitations, we propose the Multi-order Feature Interaction-aware Intrusion Detection (MIFI) scheme for ICVs. Feature attention cross network is designed to address higher-order feature interactions, while factorization machine is used for second-order interactions. Then a discriminator is utilized to detect the attacks. MIFI expands the feature space through features interaction, thereby enhancing its ability to detect attacks. Moreover, it perceives the relationships of vehicle messages, facilitating intrusion detection without knowing the corresponding rules of vehicle messages. The performance of the proposed method is evaluated on two real-vehicle datasets, affirming its effectiveness and robustness. MIFI achieves an accuracy of over 99% in detecting different attacks. The proposed method can improve the accuracy of traditional IDS to a maximum of 99.99%, and increase the highest F1-score to 97.18%, demonstrating the model’s ability of achieving multi-order feature interactions. Ultimately, MIFI is suitable for intrusion detection in different types of ICV networks, significantly contributing to the cybersecurity of ICVs.

APPLYING HYPERGRAPHS TO STUDIES IN QUANTITATIVE BIOLOGY

The objective of this research is to demonstrate hypergraph versatility and applicability for modeling diverse biological systems. The inherent structure of hypergraphs allows for encoding of higher-order feature interactions, providing a flexible framework for efficient models that can enhance our understanding of physical phenomena and one that can be generalized across various datasets. By adopting innovative methods including centrality measure and populations of models rather than singular instances, biases and overfitting tendencies are mitigated, again presenting promise for application across a broad spectrum of biological systems. Furthermore, emphasis is placed on the significance of probabilistic distribution analysis in elucidating threshold selection and feature relevance while maintaining high levels of accuracy. Our results demonstrate the advantages of hypergraph models on two different datasets; with the first on gene expression and the identification of outlier genes and the second on classifying starch grains. There is significant scope in the application of the hypergraph to a wider class of biological systems, with the potential to improve understanding of the biological processes.

Measuring and classifying IP usage scenarios: a continuous neural trees approach

Understanding user behavior via IP addresses is a crucial measure towards numerous pragmatic IP-based applications, including online content delivery, fraud prevention, marketing intelligence, and others. While profiling IP addresses through methods like IP geolocation and anomaly detection has been thoroughly studied, the function of an IP address—e.g., whether it pertains to a private enterprise network or a home broadband—remains underexplored. In this work, we initiate the first attempt to address the IP usage scenario classification problem. We collect data consisting of IP addresses from four large-scale regions. A novel continuous neural tree-based ensemble model is proposed to learn IP assignment rules and complex feature interactions. We conduct extensive experiments to evaluate our model in terms of classification accuracy and generalizability. Our results demonstrate that the proposed model is capable of efficiently uncovering significant higher-order feature interactions that enhance IP usage scenario classification, while also possessing the ability to generalize from the source region to the target one.

Location-Aware Deep Interaction Forest for Web Service QoS Prediction

With the rapid development of the web service market, the number of web services shows explosive growth. QoS is an important factor in the recommendation scene; how to accurately recommend a high-quality service for users among the massive number of web services has become a tough problem. Previous methods usually acquired feature interaction information by network structures like DNN to improve the QoS prediction accuracy, but this generates unnecessary computations. Aiming at addressing the above problem, inspired by the multigrained scanning mechanism in a deep forest, we propose a location-aware deep interaction forest approach for web service QoS prediction (LDIF). This approach offers the following innovations: The model fuses the location similarity of users and services as a latent feature representation of them. In addition, we designed a scanning interaction structure (SIS), which obtains multiple local feature combinations from the interaction between user and service features, uses interactive computing to extract feature interaction information, and concatenates the feature interaction information with original features, which aims to enhance the dimension of the features. Equipped with these, we compose a layer-by-layer cascade by using SIS to fuse low- and high-order feature interaction information, and the early-stop mechanism controls the cascade depth to avoid unnecessary computation. The experiments demonstrate that our model outperforms eight other state-of-the-art methods on MAE and RMSE common metrics on real public datasets.

IeMTLF: Interaction-enhanced Multi-Task Learning Framework for next location prediction

Location-Based Services (LBSs) provide spatial and semantic information of various locations, thanks to the advances in mobile devices and commercial map services. However, most existing studies on next location prediction neglect the semantic information and do not investigate the effects of different levels of semantic granularity. To address these issues, we propose an Interaction-enhanced Multi-Task Learning Framework (IeMTLF) that jointly predicts the next location and its semantic category and leverages higher-order feature interactions to enrich the semantic learning process. Moreover, we examine the effect of two different levels of semantic granularity, namely, specific and abstract, on prediction performance. We conducted extensive experiments on two real-world datasets and demonstrate that IeMTLF outperforms seven baselines on all evaluation metrics, with a maximum gain of more than 13% for Acc@5 compared to the state-of-the-art method. Our implementation code is available at https://github.com/mrc-wyh/IeMTLF.

Feature space reduction method for ultrahigh-dimensional, multiclass data: random forest-based multiround screening (RFMS)

In recent years, several screening methods have been published for ultrahigh-dimensional data that contain hundreds of thousands of features, many of which are irrelevant or redundant. However, most of these methods cannot handle data with thousands of classes. Prediction models built to authenticate users based on multichannel biometric data result in this type of problem. In this study, we present a novel method known as random forest-based multiround screening (RFMS) that can be effectively applied under such circumstances. The proposed algorithm divides the feature space into small subsets and executes a series of partial model builds. These partial models are used to implement tournament-based sorting and the selection of features based on their importance. This algorithm successfully filters irrelevant features and also discovers binary and higher-order feature interactions. To benchmark RFMS, a synthetic biometric feature space generator known as BiometricBlender is employed. Based on the results, the RFMS is on par with industry-standard feature screening methods, while simultaneously possessing many advantages over them.

AutoEIS: Automatic feature embedding, interaction and selection on default prediction

Deep models have shown the effectiveness in various areas, e.g., finance, healthcare and recommendation system. Among them, default prediction is a major application in the financial field. However, there are still problems remained, like insufficient learning for numerical feature encoding and difficulty of explicitly modelling high-order feature interactions. To address these issues, we propose an automatic end-to-end deep learning framework named AutoEIS. In this framework, we embed categorical and numerical features with distinct strategies. In particular, we design a Multi-field-aware Mixture-of-Experts (MfMoE) structure for numerical value embedding, which can simultaneously learn the single-field and global-field information. Then, by organically integrating attention mechanism, weighted-average aggregation and bilinear interaction, we can effectively generate significant high-order explicit interactions. Moreover, we integrate a DNN block to further capture the complex relationships among different variables. Comprehensive experiments on two real-world datasets of about 30,000 samples show the superiority of AutoEIS on default prediction, boosting average AUC and KS metrics over the best classical baseline by 0.49% and 3.5%, and the best of deep baselines by 0.48% and 2.55%. Furthermore, as a model-agnostic strategy, we can generalize MfMoE to other deep models like DeepFM, thereby boosting their performance.

Service Recommendation of Industrial Software Components Based on Explicit and Implicit Higher-Order Feature Interactions and Attentional Factorization Machines

In the context of the rapid advancement of the Industrial Internet and Urban Internet, a crucial trend is emerging in the realization of unified, service-oriented, and componentized encapsulation of IT and OT heterogeneous entities underpinned by a service-oriented architecture. This is pivotal for achieving componentized construction and development of extensive industrial software systems. In addressing the diverse demands of application tasks, the efficient and precise recommendation of service components has emerged as a pivotal concern. Existing recommendation models either focus solely on low-order interactions or emphasize high-order interactions, disregarding the distinction between implicit and explicit aspects within high-order interactions as well as the integration of high-order and low-order interactions. This oversight leads to subpar accuracy in recommendations. Real-world data exhibit intricate structures and nonlinearity. In practical applications, different interaction components exhibit varying predictive capabilities. Therefore, in this paper we propose an EIAFM model that fuses explicit and implicit higher-order feature interactions and introduce an attention mechanism to identify which low-level feature interactions contribute more significantly to the prediction results. This approach leads to increased interpretability, combining both generalization and memory capabilities. Through comprehensive experiments on authentic datasets that align with the characteristics of the Service Recommendation of Industrial Software Components problem, we demonstrate that the EIAFM model excels compared to other cutting-edge models in terms of recommendation effectiveness, with the evaluation metrics for the AUC and log-loss reaching values of 0.9281 and 0.3476, respectively.

BaGFN: Broad Attentive Graph Fusion Network for High-Order Feature Interactions.

Modeling feature interactions is of crucial significance to high-quality feature engineering on multifiled sparse data. At present, a series of state-of-the-art methods extract cross features in a rather implicit bitwise fashion and lack enough comprehensive and flexible competence of learning sophisticated interactions among different feature fields. In this article, we propose a new broad attentive graph fusion network (BaGFN) to better model high-order feature interactions in a flexible and explicit manner. On the one hand, we design an attentive graph fusion module to strengthen high-order feature representation under graph structure. The graph-based module develops a new bilinear-cross aggregation function to aggregate the graph node information, employs the self-attention mechanism to learn the impact of neighborhood nodes, and updates the high-order representation of features by multihop fusion steps. On the other hand, we further construct a broad attentive cross module to refine high-order feature interactions at a bitwise level. The optimized module designs a new broad attention mechanism to dynamically learn the importance weights of cross features and efficiently conduct the sophisticated high-order feature interactions at the granularity of feature dimensions. The final experimental results demonstrate the effectiveness of our proposed model.

Gated Multi-task Learning for QoS-aware Service Recommendation

In the ever-increasing diversity of cloud environment, the recommendation of web service-related applications based on service quality is a basic tasks that both service providers and web developers are very interested in. A key challenge is how to get personalized and accurate QoS values of web services from the multi-source information of users/services. In addition, complex contexts and multi-tasking attributes also make the collaborative services quality prediction difficult to meet. Inspired by the progress of multi-task deep learning, this paper proposes a new framework of service quality prediction called multi-task gated expert network (MGEN), which is based on gating mechanism and multi-task learning. The MGEN consists of three parts: expert-based context-source feature extraction, gated interaction learning and information fusion. First, multiple expert networks are used to capture the latent representation in raw data of web services, including both implicit features in the QoS matrix and explicit features of service descriptions. In the process, several independent experts extract features from different dimensions. Second, output embedding of different expert networks is transmitted to each task layer through different weights, learning both global and local high-order feature interactions. Third, the interactive information of different tasks is fused by embedding splicing, and then fed back to the prediction module for final service recommendation. Results of the comparative experiment demonstrate that the proposed MGEN significantly outperforms the state-of-the-art models of service recommendation, where both the implicit and explicit information of various multi-source features and interaction sequences are effectively extracted. In addition, the gating mechanism performs well in the QoS prediction by independently modeling each task.

A knowledge distillation-based deep interaction compressed network for CTR prediction

Click-through rate (CTR) prediction is an essential task in online advertising recommender systems. As a hot research frontier driven by academic and industrial needs, CTR prediction has observed remarkable progress in recent years. How to effectively learn sophisticated feature interactions hidden behind user behaviors is critical in maximizing CTR. Targeted at the above issue, this paper first constructs a deep interaction compressed network model (abbreviated as DICN) that aims to generate both low- and high-order feature interactions with different weights in both explicit and implicit fashions. Then, a KD-DICN model is created using a knowledge distillation framework to reduce the complexity of DICN. Finally, comprehensive experiments are conducted on two public datasets to demonstrate the effectiveness and efficiency of our models, which outperform the existing state-of-the-art methods. Therefore, the research presented in this paper can not only enrich and develop the theories related to CTR prediction, but it can also assist some advertising industries in better defining their service audience and recommending more popular contents to consumers more accurately.

AAIN: Attentional aggregative interaction network for deep learning based recommender systems

Feature engineering is a classical problem in recommender systems, and feature interactions is one of the most important parts of feature engineering. Factorization based models are widely used for explicit feature interactions. However, most current works utilize separate features to model cross features. Such a pattern limits the significance of cross features, since realistic recommendation scenarios are rich in associations between features.In this paper, we classify the basic feature interactions into sum-interaction and product-interaction, and improve the current general strategy of explicit feature interactions. Based on these theoretical studies, we propose a novel explicit feature interactions model Attentional Aggregative Interaction Network (AAIN), which models higher-order features using a cyclic explicit module. Specifically, we introduce attention mechanism for the reorganization of separate features, followed by product-interaction and higher-order features’ compression and output. The model is efficient since: 1) AAIN automatically learns high-order feature interactions and filters them with different weights. 2) AAIN optimizes the interaction between features into the interaction between feature groups, which allows for other relevant information to be considered when performing interactions. Furthermore, we integrate AAIN model with the classical deep neural network (DNN) model into a new model Deep Attentional Aggregative Interaction Network (DAAIN). Experiments on real-world datasets show that our models achieve state-of-the-art results.

Biomarker Identification via a Factorization Machine-Based Neural Network With Binary Pairwise Encoding.

Biomolecules, microRNAs (miRNAs) and long non-coding RNAs (lncRNAs), play critical roles in diverse fundamental and vital biological processes. They can serve as disease biomarkers as their dysregulations could cause complex human diseases. Identifying those biomarkers is helpful with the diagnosis, treatment, prognosis, and prevention of diseases. In this study, we propose a factorization machine-based deep neural network with binary pairwise encoding, DFMbpe, to identify the disease-related biomarkers. First, to comprehensively consider the interdependence of features, a binary pairwise encoding method is designed to obtain the raw feature representations for each biomarker-disease pair. Second, the raw features are mapped into their corresponding embedding vectors. Then, the factorization machine is conducted to get the wide low-order feature interdependence, while the deep neural network is applied to obtain the deep high-order feature interdependence. Finally, two kinds of features are combined to get the final prediction results. Unlike other biomarker identification models, the binary pairwise encoding considers the interdependence of features even though they never appear in the same sample, and the DFMbpe architecture emphasizes both low-order and high-order feature interactions simultaneously. The experimental results show that DFMbpe greatly outperforms the state-of-the-art identification models on both cross-validation and independent dataset evaluation. Besides, three types of case studies further demonstrate the effectiveness of this model.

LFDNN: A Novel Hybrid Recommendation Model Based on DeepFM and LightGBM

Hybrid recommendation algorithms perform well in improving the accuracy of recommendation systems. However, in specific applications, they still cannot reach the requirements of the recommendation target due to the gap between the design of the algorithms and data characteristics. In this paper, in order to learn higher-order feature interactions more efficiently and to distinguish the importance of different feature interactions better on the prediction results of recommendation algorithms, we propose a light and FM deep neural network (LFDNN), a hybrid recommendation model including four modules. The LightGBM module applies gradient boosting decision trees for feature processing, which improves LFDNN's ability to handle dense numerical features; the shallow model introduces the FM model for explicitly modeling the finite-order feature crosses, which strengthens the expressive ability of the model; the deep neural network module uses a fully connected feedforward neural network to allow the model to obtain more high-order feature crosses information and mine more data patterns in the features; finally, the Fusion module allows the shallow model and the deep model to obtain a better fusion effect. The results of comparison, parameter influence and ablation experiments on two real advertisement datasets shows that the LFDNN reaches better performance than the representative recommendation models.

FMGNN: A Method to Predict Compound-Protein Interaction With Pharmacophore Features and Physicochemical Properties of Amino Acids.

Identifying interactions between compounds and proteins is an essential task in drug discovery. To recommend compounds as new drug candidates, applying the computational approaches has a lower cost than conducting the wet-lab experiments. Machine learning-based methods, especially deep learning-based methods, have advantages in learning complex feature interactions between compounds and proteins. However, deep learning models will over-generalize and lead to the problem of predicting less relevant compound-protein pairs when the compound-protein feature interactions are high-dimensional sparse. This problem can be overcome by learning both low-order and high-order feature interactions. In this paper, we propose a novel hybrid model with Factorization Machines and Graph Neural Network called FMGNN to extract the low-order and high-order features, respectively. Then, we design a compound-protein interactions (CPIs) prediction method with pharmacophore features of compound and physicochemical properties of amino acids. The pharmacophore features can ensure that the prediction results much more fit the expectation of biological experiment and the physicochemical properties of amino acids are loaded into the embedding layer to improve the convergence speed and accuracy of protein feature learning. The experimental results on several datasets, especially on an imbalanced large-scale dataset, showed that our proposed method outperforms other existing methods for CPI prediction. The western blot experiment results on wogonin and its candidate target proteins also showed that our proposed method is effective and accurate for finding target proteins. The computer program of implementing the model FMGNN is available at https://github.com/tcygxu2021/FMGNN.

Nearest Neighbor-Based Contrastive Learning for Hyperspectral and LiDAR Data Classification

The joint hyperspectral image (HSI) and light detection and ranging (LiDAR) data classification aims to interpret ground objects at more detailed and precise level. Although deep learning methods have shown remarkable success in the multisource data classification task, self-supervised learning has rarely been explored. It is commonly nontrivial to build a robust self-supervised learning model for multisource data classification, due to the fact that the semantic similarities of neighborhood regions are not exploited in the existing contrastive learning framework. Furthermore, the heterogeneous gap induced by the inconsistent distribution of multisource data impedes the classification performance. To overcome these disadvantages, we propose a nearest neighbor-based contrastive learning network (NNCNet), which takes full advantage of large amounts of unlabeled data to learn discriminative feature representations. Specifically, we propose a nearest neighbor-based data augmentation scheme to use enhanced semantic relationships among nearby regions. The intermodal semantic alignments can be captured more accurately. In addition, we design a bilinear attention module to exploit the second-order and even high-order feature interactions between the HSI and LiDAR data. Extensive experiments on four public datasets demonstrate the superiority of our NNCNet over state-of-the-art methods. The source codes are available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/summitgao/NNCNet</uri> .

A Dual Adaptive Interaction Click-Through Rate Prediction Based on Attention Logarithmic Interaction Network.

Click-through rate (CTR) prediction is crucial for computing advertisement and recommender systems. The key challenge of CTR prediction is to accurately capture user interests and deliver suitable advertisements to the right people. However, there are an immense number of features in CTR prediction datasets, which hardly fit when only using an individual feature. To solve this problem, feature interaction that combines several features via an operation is introduced to enhance prediction performance. Many factorizations machine-based models and deep learning methods have been proposed to capture feature interaction for CTR prediction. They follow an enumeration-filter pattern that could not determine the appropriate order of feature interaction and useful feature interaction. The attention logarithmic network (ALN) is presented in this paper, which uses logarithmic neural networks (LNN) to model feature interactions, and the squeeze excitation (SE) mechanism to adaptively model the importance of higher-order feature interactions. At first, the embedding vector of the input was absolutized and a very small positive number was added to the zeros of the embedding vector, which made the LNN input positive. Then, the adaptive-order feature interactions were learned by logarithmic transformation and exponential transformation in the LNN. Finally, SE was applied to model the importance of high-order feature interactions adaptively for enhancing CTR performance. Based on this, the attention logarithmic interaction network (ALIN) was proposed for the effectiveness and accuracy of CTR, which integrated Newton's identity into ALN. ALIN supplements the loss of information, which is caused by the operation becoming positive and by adding a small positive value to the embedding vector. Experiments are conducted on two datasets, and the results prove that ALIN is efficient and effective.

FIRE: knowledge-enhanced recommendation with feature interaction and intent-aware attention networks.

To solve the information overload issue and enhance the user experience of various web applications, recommender systems aim to better model user interests and preferences. Knowledge Graphs (KGs), consisting of real-world objective facts and fruitful entities, play a vital role in recommender systems. Recently, a technological trend has been to develop end-to-end Graph Neural Networks (GNNs)-based knowledge-aware recommendation (a.k.a., Knowledge Graph Recommendation, KGR) models. Unfortunately, current GNNs-based KGR approaches focus on how to capture high-order feature information on KGs while neglecting the following two crucial limitations: 1) The explicitly high-order feature interaction and fusion mechanism and 2) The valid user intent modelling mechanism. As such, these issues lead to insufficient user/item representation learning capability and unsatisfactory KGR performance. In this work, we present a novel Knowledge-enhanced Re commendation with F eature I nteraction and Intent-aware Attention Networks (FIRE) to address the latent intent modelling and high-order feature interaction deficiencies ignored by existing KGR methods. Based on the prototype user/item representation learning leveraging the GNNs-based approach, our model offers the following major improvements: One is the innovative use of Convolutional Neural Networks (CNNs) that perform vertical convolutional (a.k.a., bit-level convolutional) and horizontal convolutional (a.k.a., vector-level convolutional) processes to model multi-granular high-order feature interactions to enhance item-side representation learning. Another is to model users' latent intent factors by utilizing a two-level attention mechanism (i.e., node- and intent-level attention mechanism) to enhance user-side representation learning. Extensive experiments on three KGs domain public datasets demonstrate that our method outperforms the existing state-of-the-art baseline. Last but not least, numerous ablation- and model studies demystify the working mechanism and elucidate the plausibility of the proposed model.