Sensitivity of benefit-cost analysis of data programs to monotone misspecification

Abstract

Abstract The sensitivity of a benefit-cost analysis of a data program to incorrect specifications of the cost function C and benefit function B is an important consideration for producers and consumers of the analyses. We consider the effects of misspecification of B or C on the optimal data expenditure and optimal data quality. Geometric and analytical results show when the analysis will be sensitive and when order of magnitude estimates for costs and benefits will suffice in practice. A benefit-cost analysis of the 1970 census illustrates these results. We also show why misspecification of benefits typically has a larger effect than comparable misspecification of costs. If the benefits and the costs are both large but their difference small, the optimal expenditure for data (or optimal data quality) will be highly sensitive to changes in specifications of costs, benefits, data use, etc. In other cases, the sensitivity of the prescribed optimal expenditure to scalemisspecification of costs or benefits depends on the sharpness of the bend of the graph of B versus C near the optimum. Although the prescribed optimal expenditure is the same when B is misspecified as when C is comparably misspecified, the levels of data quality attained can differ. When B is misspecified, the data quality attained decreases if benefits are understated. When C is misspecified, the data quality attained usually decreases if costs are overstated, but for some unusual decision problems it increases and in rare, ‘self-correcting’ problems, it remains unchanged.