Complex Feature Interactions Research Articles

A framework for feature selection through boosting

As dimensions of datasets in predictive modelling continue to grow, feature selection becomes increasingly practical. Datasets with complex feature interactions and high levels of redundancy still present a challenge to existing feature selection methods. We propose a novel framework for feature selection that relies on boosting, or sample re-weighting, to select sets of informative features in classification problems. The method uses as its basis the feature rankings derived from fast and scalable tree-boosting models, such as XGBoost. We compare the proposed method to standard feature selection algorithms on 9 benchmark datasets. We show that the proposed approach reaches higher accuracies with fewer features on most of the tested datasets, and that the selected features have lower redundancy.

A joint learning model for click-through prediction in display advertising

Click-through rate (CTR) prediction is essential for targeted advertising and recommendation. At present, machine learning models are widely used to build CTR estimators, including logistic regression (LR), factorization machine (FM), and deep neural network (DNN). Unfortunately, these models adopt the single structure that only considers either low-order feature interactions (such as LR and FM) or high-order feature interactions (such as DNN), not sufficient for CTR prediction. Therefore, the joint learning models are proposed, such as Wide & Deep and DeepFM, which can exploit both high- and low-order feature interactions to predict CTR by combining two different models. In this paper, we first analyze the typical CTR prediction models’ structures and performance, and then summarize the general form and design rules of CTR estimators. Based on the general form, we further design a new joint learning model that combines two different residual networks to explore the feature interactions automatically. Compared with the widely adopted feed-forward neural network, the residual network is more capable of exploring complex feature interactions at different layers. Additionally, we introduce a neural attention network to learn the importance of each second-order interaction of features from various fields. Finally, we evaluate the prediction performance of the proposed model based on two real-world datasets (i.e., Criteo and Avazu) in terms of LogLoss and AUC metrics. The extensive experimental results demonstrate our model performs the best compared to the state-of-the-art baselines.

Unsupervised Anomaly Detection with Distillated Teacher-Student Network Ensemble.

We address the problem of unsupervised anomaly detection for multivariate data. Traditional machine learning based anomaly detection algorithms rely on specific assumptions of normal patterns and fail to model complex feature interactions and relations. Recently, existing deep learning based methods are promising for extracting representations from complex features. These methods train an auxiliary task, e.g., reconstruction and prediction, on normal samples. They further assume that anomalies fail to perform well on the auxiliary task since they are never trained during the model optimization. However, the assumption does not always hold in practice. Deep models may also perform the auxiliary task well on anomalous samples, leading to the failure detection of anomalies. To effectively detect anomalies for multivariate data, this paper introduces a teacher-student distillation based framework Distillated Teacher-Student Network Ensemble (DTSNE). The paradigm of the teacher-student distillation is able to deal with high-dimensional complex features. In addition, an ensemble of student networks provides a better capability to avoid generalizing the auxiliary task performance on anomalous samples. To validate the effectiveness of our model, we conduct extensive experiments on real-world datasets. Experimental results show superior performance of DTSNE over competing methods. Analysis and discussion towards the behavior of our model are also provided in the experiment section.

Visualizing complex feature interactions and feature sharing in genomic deep neural networks

BackgroundVisualization tools for deep learning models typically focus on discovering key input features without considering how such low level features are combined in intermediate layers to make decisions. Moreover, many of these methods examine a network’s response to specific input examples that may be insufficient to reveal the complexity of model decision making.ResultsWe present DeepResolve, an analysis framework for deep convolutional models of genome function that visualizes how input features contribute individually and combinatorially to network decisions. Unlike other methods, DeepResolve does not depend upon the analysis of a predefined set of inputs. Rather, it uses gradient ascent to stochastically explore intermediate feature maps to 1) discover important features, 2) visualize their contribution and interaction patterns, and 3) analyze feature sharing across tasks that suggests shared biological mechanism. We demonstrate the visualization of decision making using our proposed method on deep neural networks trained on both experimental and synthetic data. DeepResolve is competitive with existing visualization tools in discovering key sequence features, and identifies certain negative features and non-additive feature interactions that are not easily observed with existing tools. It also recovers similarities between poorly correlated classes which are not observed by traditional methods. DeepResolve reveals that DeepSEA’s learned decision structure is shared across genome annotations including histone marks, DNase hypersensitivity, and transcription factor binding. We identify groups of TFs that suggest known shared biological mechanism, and recover correlation between DNA hypersensitivities and TF/Chromatin marks.ConclusionsDeepResolve is capable of visualizing complex feature contribution patterns and feature interactions that contribute to decision making in genomic deep convolutional networks. It also recovers feature sharing and class similarities which suggest interesting biological mechanisms. DeepResolve is compatible with existing visualization tools and provides complementary insights.

Estimating latent feature-feature interactions in large feature-rich graphs

Real-world complex networks describe connections between objects; in reality, those objects are often endowed with some kind of features. How does the presence or absence of such features interplay with the network link structure? Although the situation here described is truly ubiquitous, there is a limited body of research dealing with large graphs of this kind. Many previous works considered homophily as the only possible transmission mechanism translating node features into links. Other authors, instead, developed more sophisticated models, that are able to handle complex feature interactions, but are unfit to scale to very large networks. We expand on the MGJ model, where interactions between pairs of features can foster or discourage link formation. In this work, we will investigate how to estimate the latent feature-feature interactions in this model. We shall propose two solutions: the first one assumes feature independence and it is essentially based on Naive Bayes; the second one, which relaxes the independence assumption assumption, is based on perceptrons. In fact, we show it is possible to cast the model equation in order to see it as the prediction rule of a perceptron. We analyze how classical results for the perceptrons can be interpreted in this context; then, we define a fast and simple perceptron-like algorithm for this task, which can process $10^8$ links in minutes. We then compare these two techniques, first with synthetic datasets that follows our model, gaining evidence that the Naive independence assumptions are detrimental in practice. Secondly, we consider a real, large-scale citation network where each node (i.e., paper) can be described by different types of characteristics; there, our algorithm can assess how well each set of features can explain the links, and thus finding meaningful latent feature-feature interactions.

A scaffolding approach to coreference resolution integrating statistical and rule-based models

AbstractWe describe a scaffolding approach to the task of coreference resolution that incrementally combines statistical classifiers, each designed for a particular mention type, with rule-based models (for sub-tasks well-matched to determinism). We motivate our design by an oracle-based analysis of errors in a rule-based coreference resolution system, showing that rule-based approaches are poorly suited to tasks that require a large lexical feature space, such as resolving pronominal and common-noun mentions. Our approach combines many advantages: it incrementally builds clusters integrating joint information about entities, uses rules for deterministic phenomena, and integrates rich lexical, syntactic, and semantic features with random forest classifiers well-suited to modeling the complex feature interactions that are known to characterize the coreference task. We demonstrate that all these decisions are important. The resulting system achieves 63.2 F1 on the CoNLL-2012 shared task dataset, outperforming the rule-based starting point by over seven F1 points. Similarly, our system outperforms an equivalent sieve-based approach that relies on logistic regression classifiers instead of random forests by over four F1 points. Lastly, we show that by changing the coreference resolution system from relying on constituent-based syntax to using dependency syntax, which can be generated in linear time, we achieve a runtime speedup of 550 per cent without considerable loss of accuracy.

Devising an interpretable calibrated scale to quantitatively assess the dementia stage of subjects with alzheimer's disease: A machine learning approach

Abstract Background: Machine learning and data mining techniques have been successfully applied on MRI images for detecting Alzheimer's disease (AD). But only a few studies have explored the possibility of AD detection from non-image data. These studies have applied traditional data visualization and classification algorithms. There is a need for new sophisticated learning algorithms, for detecting and quantifying the severity of AD by exploring the complex interactions between the features in AD subjects. Method: In this work, a supervised learning model to effectively capture the complex feature interactions, in the sample space of AD data, is presented for knowledge discovery. The discovered knowledge is further used to quantify the similarity of a test subject to the demented class. Results: Evaluation of the proposed model, on OASIS database of Alzheimer's subjects, validates the well established risk factors and identifiers for AD: Age, Socio-Economic Status, MMSE Score and Whole Brain Volume. The Test subjects are affiliated to either non-demented (ND) or AD class, with non-overlapping and measurable similarity indices: Female ND (CDR=0) [0.48–2.90], Female AD (CDR=0.5) [90.16–774.51], Female AD (CDR=1) [1633.90–7182.23], Female AD (CDR=2) [55258.51–66382.44], Male ND (CDR=0)[0.69–3.66], Male AD (CDR=0.5) [99.18–647.51] and Male AD (CDR=1) [3880.16–6519.40]. Conclusion: The outcome of the work clearly demonstrates that, supervised learning model can be used effectively to quantify the severity of AD on a standard measurable scale. This scale of distance can be used as a supplement for clinical dementia rating.

STPA-based Method to Identify and Control Feature Interactions in Large Complex Systems

This research develops and evaluates a new approach that can be applied during STPA Step 1 (identify Unsafe Control Actions) to efficiently identify complex feature interactions among multiple controllers that can result in dysfunctional system behavior. The new approach is applied and evaluated using several automated automotive systems with an emphasis on controllers that may interact or interfere with each other directly or indirectly. The approach is shown to analyze hundreds of interactions with an order of magnitude less effort than has been possible previously. In addition, formal methods are applied to support reasoning about completeness and to enable tool assistance during the search for dysfunctional interactions. Humans are explicitly included as controllers that may interact with automated systems, and accident scenarios involving complex human interactions such as software-induced human errors can be identified.

Race detection for web applications

Modern web pages are becoming increasingly full-featured, and this additional functionality often requires greater use of asynchrony. Unfortunately, this asynchrony can trigger unexpected concurrency errors, even though web page scripts are executed sequentially. We present the first formulation of a happens-before relation for common web platform features. Developing this relation was a non-trivial task, due to complex feature interactions and browser differences. We also present a logical memory access model for web applications that abstracts away browser implementation details. Based on the above, we implemented WebRacer, the first dynamic race detector for web applications. WebRacer is implemented atop the production-quality WebKit engine, enabling testing of full-featured web sites. WebRacer can also simulate certain user actions, exposing more races. We evaluated WebRacer by testing a large set of Fortune 100 company web sites. We discovered many harmful races, and also gained insights into how developers handle asynchrony in practice.

The virtual customer

Communication and information technologies are adding new capabilities for rapid and inexpensive customer input to all stages of the product development (PD) process. In this article we review six web-based methods of customer input as examples of the improved Internet capabilities of communication, conceptualization, and computation. For each method we give examples of user-interfaces, initial applications, and validity tests. We critique the applicability of the methods for use in the various stages of PD and discuss how they complement existing methods. For example, during the fuzzy front end of PD the information pump enables customers to interact with each other in a web-based game that provides incentives for truth-telling and thinking hard, thus providing new ways for customers to verbalize the product features that are important to them. Fast polyhedral adaptive conjoint estimation enables PD teams to screen larger numbers of product features inexpensively to identify and measure the importance of the most promising features for further development. Meanwhile, interactive web-based conjoint analysis interfaces are moving this proven set of methods to the web while exploiting new capabilities to present products, features, product use, and marketing elements in streaming multimedia representations. User design exploits the interactivity of the web to enable users to design their own virtual products thus enabling the PD team to understand complex feature interactions and enabling customers to learn their own preferences for new products. These methods can be valuable for identifying opportunities, improving the design and engineering of products, and testing ideas and concepts much earlier in the process when less time and money is at risk. As products move toward pretesting and testing, virtual concept testing on the web enables PD teams to test concepts without actually building the product. Further, by combining virtual concepts and the ability of customers to interact with one another in a stock-market-like game, securities trading of concepts provides a novel way to identify winning concepts. Prototypes of all six methods are available and have been tested with real products and real customers. These tests demonstrate reliability for web-based conjoint analysis, polyhedral methods, virtual concept testing, and stock-market-like trading; external validity for web-based conjoint analysis and polyhedral methods; and consistency for web-based conjoint analysis versus user design. We report on these tests, commercial applications, and other evaluations.

The virtual customer

Communication and information technologies are adding new capabilities for rapid and inexpensive customer input to all stages of the product development (PD) process. In this article we review six web‐based methods of customer input as examples of the improved Internet capabilities of communication, conceptualization, and computation. For each method we give examples of user‐interfaces, initial applications, and validity tests. We critique the applicability of the methods for use in the various stages of PD and discuss how they complement existing methods.For example, during the fuzzy front end of PD the information pump enables customers to interact with each other in a web‐based game that provides incentives for truth‐telling and thinking hard, thus providing new ways for customers to verbalize the product features that are important to them. Fast polyhedral adaptive conjoint estimation enables PD teams to screen larger numbers of product features inexpensively to identify and measure the importance of the most promising features for further development. Meanwhile, interactive web‐based conjoint analysis interfaces are moving this proven set of methods to the web while exploiting new capabilities to present products, features, product use, and marketing elements in streaming multimedia representations. User design exploits the interactivity of the web to enable users to design their own virtual products thus enabling the PD team to understand complex feature interactions and enabling customers to learn their own preferences for new products. These methods can be valuable for identifying opportunities, improving the design and engineering of products, and testing ideas and concepts much earlier in the process when less time and money is at risk. As products move toward pretesting and testing, virtual concept testing on the web enables PD teams to test concepts without actually building the product. Further, by combining virtual concepts and the ability of customers to interact with one another in a stock‐market‐like game, securities trading of concepts provides a novel way to identify winning concepts.Prototypes of all six methods are available and have been tested with real products and real customers. These tests demonstrate reliability for web‐based conjoint analysis, polyhedral methods, virtual concept testing, and stock‐market‐like trading; external validity for web‐based conjoint analysis and polyhedral methods; and consistency for web‐based conjoint analysis versus user design. We report on these tests, commercial applications, and other evaluations. © 2002 Elsevier Science Inc. All rights reserved.

Verifying software requirements: a requirement tracing methodology and its software tool-RADIX

The waterfall-based software development methodology usually requires significant efforts and lengthy intervals to develop, review, inspect, and test very large and complex real-time software systems. Although the methodology itself is stable and mature, the quality of software products still largely depends on development efforts expended in testing, which is usually very costly and time consuming. The author describes the implementation of an integrated approach of software development, review, inspection, and testing based on well-identified software requirements. Due to the huge size and the complex feature interactions of the 5ESS Switch software, the process of review, inspection, and testing has been very important to the project. To enhance the quality of the products, it is necessary to ensure the feature requirements requested by customers, external and internal, are developed correctly. The requirement traceability procedure described in the paper is a systematic method to assist 5ESS Switch scientists and engineers to achieve the goal. The results of the implementation of the requirement traceability procedure and its RADIX tool are significant. >