Class Of Utility Models Research Articles

A Coverage-Based Utility Model for Identifying Unknown Unknowns

A classifier’s low confidence in prediction is often indicative of whether its prediction will be wrong; in this case, inputs are called known unknowns. In contrast, unknown unknowns (UUs) are inputs on which a classifier makes a high confidence mistake. Identifying UUs is especially important in safety-critical domains like medicine (diagnosis) and law (recidivism prediction). Previous work by Lakkaraju et al. (2017) on identifying unknown unknowns assumes that the utility of each revealed UU is independent of the others, rather than considering the set holistically. While this assumption yields an efficient discovery algorithm, we argue that it produces an incomplete understanding of the classifier’s limitations. In response, this paper proposes a new class of utility models that rewards how well the discovered UUs cover (or "explain") a sample distribution of expected queries. Although choosing an optimal cover is intractable, even if the UUs were known, our utility model is monotone submodular, affording a greedy discovery strategy. Experimental results on four datasets show that our method outperforms bandit-based approaches and achieves within 60.9% utility of an omniscient, tractable upper bound.

Reanalysis of the Chechile-Cooke Experiment: Correcting for Mismatched Gambles

Chechile and Cooke (1997) experimentally tested a broad class of utility models subsumed under the Miyamoto (1988, 1992) generic utility theory. The Chechile and Cooke study required participants to match on each trial, a fully specified reference gamble to a partially specified comparison gamble by adjusting the probability of a win on the comparison gamble. The Chechile and Cooke experiment, however, contained a subset of trials which were intrinsically unmatchable. In such cases, the participants could only give an extreme probability (either 0 or 1). In this paper, those extreme trials were omitted and the results from the experiment reanalyzed. Despite the mismatch problem, the conclusions of the Chechile and Cooke experiment were again supported. For nine implementations of generic utility there is model failure due to the systematic variation of a parameter that should be a constant.

An Experimental Test of a General Class of Utility Models: Evidence for Context Dependency

Generic utility theory, a general axiomatization of utility principles developed by Miyamoto (1988, 1992), is discussed as a formulation that captures a large class of utility theories. Several general mathematical functions were used to specify further the scaling of utility within this class of models. The scaling parameters in the generic utility representation should remain invariant across gambling contexts, and this predicted invariance provided a means for testing the theory. Evidence is presented that the prediction of scaling-parameter invariance is violated. This failure is interpreted as a consequence of employing an absolute reference system for a problem that is context-sensitive.