Aggregating expert predictions in a networked environment

Abstract

This paper describes an approach for combining the classifications or predictions of n local experts into a single composite prediction. We describe a Java-based application that allows a user to select up to n prediction experts that provide information for assigning an object to one of two predetermined groups. An advantage of this type of application is that it is capable of interacting with the Internet in a relatively seamless way. We examine the accuracy and robustness of our technique by comparing the classification accuracy of our technique, a maximum entropy-based aggregation technique, and four classification methods on a real-world, two-group data-set concerned with bank failure prediction. The classificaiton methods studied in this work include Quinlan's C4.5 decision-tree classifier, logistic regression, mahalanobis distance measures, and a neural network classifier. Our model includes a fundamental component (i.e., a transaction manager) that helps improve the general performance of applications that perform network-based classification. This component is found to provide reliable and secure connections along with ways to direct traffic across the Internet. Our results suggest three major contributions: (1) a transaction manager increases the flexibility of a network-based classifier since it is capable of transacting with one or more specific types of prediction expert(s) over the Internet; (2) our approach tends to be more accurate than the individual classification methods we examined; and, (3) our approach can outperform a recently introduced statistically based aggregation technique. Scope and purpose The emergence of the Internet has produced a need for employing new types of programming and research tools that are capable of accessing information resources located throughout the world. There is only a limited amount of research available in this area and this work describes a network-based tool that solves a two-group classification problem. The two-group classification problem in discriminant analysis is concerned with developing a rule for predicting to which of k=2 mutually exclusive groups an observation of unknown origin belongs. This problem commonly occurs in business and other areas, and a plethora of statistical and artificial intelligence (AI) techniques exist to help decision-makers effectively analyze their data. A number of recent studies have compared the classificatory performance of various AI techniques to the more traditional statistical techniques, however, decision makers are left in somewhat of a quandary about which of the many available classification techniques to use to solve a specific classification problem. This paper proposes a new aggregation technique that focuses on combining or aggregating the predictions from multiple classification techniques into a single composite prediction. Our approach provides a simple method for aggregating expert predictions coming from remote locations by combining Java and Common Object Request Broker Architecture (CORBA) into a general classification tool. Object-oriented models developed using Java are platform independent and can be easily modified. CORBA provides the services necessary to establish and manage network connections. Computational results show that our technique outperforms a recently introduced maximum entropy-based aggregation technique using a real-world data set.