Testing the impartiality of surveys to measure differential risk perception

Abstract

The paper addresses the validation of opinion measurement by survey, specifically how the “readings” or views are consolidated for policy use into a single figure held to be representative of the population as a whole. While the sample mean might seem the obvious choice, a number of influential safety studies have employed a more complicated metric, the Valuation Index, as discussed within, to assess whether more should be spent to protect against some hazards than others. The question arises as to whether the Valuation Index treats the views of the different people in the survey sample impartially or not.The Valuation Index possesses the property of reciprocality, in the sense that the Index of the reciprocal is equal to the reciprocal of the Index. Jensen’s inequality reveals that this reciprocality comes at the price of an inconsistent treatment of the individual’s view whenever there is any difference of opinion in the sample. The Valuation Index is found to be non-unique, in that a Second Valuation Index can be constructed using equivalent arguments. Thus if the Valuation Index is valid, then so must be the Second Valuation Index, but the latter returns a different value, implying a contradiction that can be resolved only by regarding neither as valid.Crucially, both the Valuation Index and the Second Valuation Index fail the test of structural view independence. Both indices are biased and thus neither is a valid measure for consolidating human opinions.The finding has particular importance for the UK because opinion surveys interpreted using the Valuation Index have been influential in the changes to safety policy brought in for UK rail transport, where the amount recommended to avert death in multiple fatality accidents since 2003 has been reduced to about a third of what it used to be.