Consensus Inference Research Articles

BenchMFC: A benchmark dataset for trustworthy malware family classification under concept drift

Concept drift poses a critical challenge in deploying machine learning models to mitigate practical malware threats. It refers to the phenomenon that the distribution of test data changes over time, gradually deviating from the original training data and degrading model performance. A promising direction for addressing concept drift is to detect drift samples and then retrain the model. However, this field currently lacks a unified, well-curated, and comprehensive benchmark, which often leads to unfair comparisons and inconclusive outcomes. To improve the evaluation and advance further, this paper presents a new Benchmark dataset for trustworthy Malware Family Classification (BenchMFC), which includes 223 K samples of 526 families that evolve over years. BenchMFC provides clear family, packer, and timestamp tags for each sample, it thus can support research on three types of malware concept drift: 1) unseen families, 2) packed families, and 3) evolved families. To collect unpacked family samples from large-scale candidates, we introduce a novel crowdsourcing malware annotation pipeline, which unifies packing detection and family annotation as a consensus inference problem to prevent costly packing detection. Moreover, we provide two case studies to illustrate the application of BenchMFC in 1) concept drift detection and 2) model retraining. The first case demonstrates the impact of three types of malware concept drift and compares nine notable concept drift detectors. The results show that existing detectors have their own advantages in dealing with different types of malware concept drift, and there is still room for improvement in malware concept drift detection. The second case explores how static feature-based machine learning operates on packed samples when retraining a model. The experiments illustrate that packers do preserve some kind of signals that appear to be “effective” for machine learning models, but the robustness of these signals requires further research. BenchMFC has been released to the community at https://github.com/crowdma/benchmfc.

GENECI: A novel evolutionary machine learning consensus-based approach for the inference of gene regulatory networks

Gene regulatory networks define the interactions between DNA products and other substances in cells. Increasing knowledge of these networks improves the level of detail with which the processes that trigger different diseases are described and fosters the development of new therapeutic targets. These networks are usually represented by graphs, and the primary sources for their correct construction are usually time series from differential expression data. The inference of networks from this data type has been approached differently in the literature. Mostly, computational learning techniques have been implemented, which have finally shown some specialization in specific datasets. For this reason, the need arises to create new and more robust strategies for reaching a consensus based on previous results to gain a particular capacity for generalization. This paper presents GENECI (GEne NEtwork Consensus Inference), an evolutionary machine learning approach that acts as an organizer for constructing ensembles to process the results of the main inference techniques reported in the literature and to optimize the consensus network derived from them, according to their confidence levels and topological characteristics. After its design, the proposal was confronted with datasets collected from academic benchmarks (DREAM challenges and IRMA network) to quantify its accuracy. Subsequently, it was applied to a real-world biological network of melanoma patients whose results could be contrasted with medical research collected in the literature. Finally, it has been proved that its ability to optimize the consensus of several networks leads to outstanding robustness and accuracy, gaining a certain generalization capacity after facing the inference of multiple datasets. The source code is hosted in a public repository at GitHub under MIT license: https://github.com/AdrianSeguraOrtiz/GENECI. Moreover, to facilitate its installation and use, the software associated with this implementation has been encapsulated in a python package available at PyPI: https://pypi.org/project/geneci/.

From Many to One: Consensus Inference in a MIP

AbstractA Model Intercomparison Project (MIP) consists of teams who estimate the same underlying quantity (e.g., temperature projections to the year 2070). A simple average of the ensemble of the teams' outputs gives a consensus estimate, but it does not recognize that some outputs are more variable than others. Statistical analysis of variance (ANOVA) models offer a way to obtain a weighted frequentist consensus estimate of outputs with a variance that is the smallest possible. Modulo dependence between MIP outputs, the ANOVA approach weights a team's output inversely proportional to its variance, from which optimally weighted estimates follow. ANOVA weights can also provide a prior distribution for Bayesian Model Averaging of the MIP outputs when external evaluation data are available. We use a MIP of carbon‐dioxide‐flux inversions to illustrate the ANOVA‐based weighting and subsequent frequentist consensus inferences.

Reexamining the Outcome Effect: Are Auditors Penalized for Exercising Professional Skepticism?

Prior research has documented an outcome effect such that appropriately skeptical auditors receive higher performance evaluations when they identify a misstatement than when they do not. In Experiment 1, we find differences in performance evaluations (as identified in prior research and that we empirically confirm) are driven by a positive reaction to auditors who identify a misstatement rather than a negative reaction to auditors who do not. In Experiment 2, we use the evaluation fact pattern from Experiment 1 and find outcome effects do not differentially impact auditors’ subsequent professional skepticism, so long as the auditor is aware of their initial task performance evaluation. Taken together, results from our experiments contradict the consensus inference in prior research that the audit evaluation system inadvertently disincentivizes profession skepticism. Rather, our results suggest the evaluation system not only does not deter professional skepticism, but also is critical in facilitating auditors’ professional skepticism.

COSIFER: a Python package for the consensus inference of molecular interaction networks.

SummaryThe advent of high-throughput technologies has provided researchers with measurements of thousands of molecular entities and enable the investigation of the internal regulatory apparatus of the cell. However, network inference from high-throughput data is far from being a solved problem. While a plethora of different inference methods have been proposed, they often lead to non-overlapping predictions, and many of them lack user-friendly implementations to enable their broad utilization. Here, we present Consensus Interaction Network Inference Service (COSIFER), a package and a companion web-based platform to infer molecular networks from expression data using state-of-the-art consensus approaches. COSIFER includes a selection of state-of-the-art methodologies for network inference and different consensus strategies to integrate the predictions of individual methods and generate robust networks.Availability and implementationCOSIFER Python source code is available at https://github.com/PhosphorylatedRabbits/cosifer. The web service is accessible at https://ibm.biz/cosifer-aas.Supplementary information Supplementary data are available at Bioinformatics online.

A Parts Database with Consensus Parameter Estimation for Synthetic Circuit Design.

Mathematical modeling and numerical simulation are crucial to support design decisions in synthetic biology. Accurate estimation of parameter values is key, as direct experimental measurements are difficult and time-consuming. Insufficient data, incompatible measurements, and specialized models that lack universal parameters make this task challenging. Here, we have created a database (PAMDB) that integrates data from 135 publications that contain 118 circuits and 165 genetic parts of the bacterium Escherichia coli. We used a succinct, universal model formulation to describe the part behavior in each circuit. We introduce a constrained consensus inference method that was used to infer the value of the model parameters and evaluated its performance through cross-validation in a benchmark of 23 circuits. We discuss these results and summarize the challenges in data integration and parameter inference. This work provides a resource and a methodology that can be used as a point of reference for synthetic circuit modeling.

Harnessing Diversity towards the Reconstructing of Large Scale Gene Regulatory Networks

Elucidating gene regulatory network (GRN) from large scale experimental data remains a central challenge in systems biology. Recently, numerous techniques, particularly consensus driven approaches combining different algorithms, have become a potentially promising strategy to infer accurate GRNs. Here, we develop a novel consensus inference algorithm, TopkNet that can integrate multiple algorithms to infer GRNs. Comprehensive performance benchmarking on a cloud computing framework demonstrated that (i) a simple strategy to combine many algorithms does not always lead to performance improvement compared to the cost of consensus and (ii) TopkNet integrating only high-performance algorithms provide significant performance improvement compared to the best individual algorithms and community prediction. These results suggest that a priori determination of high-performance algorithms is a key to reconstruct an unknown regulatory network. Similarity among gene-expression datasets can be useful to determine potential optimal algorithms for reconstruction of unknown regulatory networks, i.e., if expression-data associated with known regulatory network is similar to that with unknown regulatory network, optimal algorithms determined for the known regulatory network can be repurposed to infer the unknown regulatory network. Based on this observation, we developed a quantitative measure of similarity among gene-expression datasets and demonstrated that, if similarity between the two expression datasets is high, TopkNet integrating algorithms that are optimal for known dataset perform well on the unknown dataset. The consensus framework, TopkNet, together with the similarity measure proposed in this study provides a powerful strategy towards harnessing the wisdom of the crowds in reconstruction of unknown regulatory networks.

Bayesian Calibration of a Stochastic Kinetic Computer Model Using Multiple Data Sources

In this article, we describe a Bayesian approach to the calibration of a stochastic computer model of chemical kinetics. As with many applications in the biological sciences, the data available to calibrate the model come from different sources. Furthermore, these data appear to provide somewhat conflicting information about the model parameters. We describe a modeling framework that allows us to synthesize this conflicting information and arrive at a consensus inference. In particular, we show how random effects can be incorporated into the model to account for between-individual heterogeneity that may be the source of the apparent conflict.

Consensus Inference in Neuroimaging

We introduce model averaging in neuroimaging. We show that model summary images can be directly compared and averaged after histogram equalization. We demonstrate that averaging enhances the ROC curve in a simulation study. The averaging procedure is applied to a fMRI study of motor cortex.