THE APPLICATION OF DECISION THEORY AND SCALING METHODS TO SELECTION TEST EVALUATION

Abstract

Abstract : The correlational approach to selection test evaluation was examined and found to have serious limitations. An approach based on statistical decision theory was developed. Two new methods were presented, one called the utility function method and the other the decision-theoretic method. The former involves the comparison of criterion groups in terms of their utility to the institution using the selection test. The decision-theoretic method is based on statistical decision theory and involves the construction of a payoff matrix corresponding to the contingency table relating the test to the criterion. The cell frequencies are weighted in a utility equation by the payoff values in the corresponding cells of the payoff matrix. This utility equation represents a new test evaluation index which directly expresses the utility of the test to the institution using it. Both of these new methods require the measurement of values peculiar to the institution using the test. The utility function method requires that the performance criterion be translated to a utility function; while the decision-theoretic method requires that a payoff matrix be developed which reflects the gains and losses each cell observation represents to the institution. The three methods (correlational, utility function, and decision- theoretic) were compared with tests used to select students for A-Schools in the U.S. Navy. The three methods led to quite different indications regarding the utility of the selection tests evaluated. The two new methods agreed in terms of the proportion improvement over chance prediction provided by the tests while the correlational method tended to underestimate this proportion.