Benchmark Estimate Research Articles

How to Efficiently Measure the Intelligibility of People With Parkinson's Disease.

The purpose of this study was to determine the most efficient approaches to measuring the intelligibility of people with Parkinson's disease (PD) when considering the estimation method, listener experience, number of listeners, number of sentences, and the ways these factors may interact. Speech-language pathologists (SLPs) and inexperienced listeners estimated the intelligibility of people with and without PD using orthographic transcription or a visual analog scale (VAS). Intelligibility estimates were based on 11 Speech Intelligibility Test sentences. We simulated all combinations of listeners and sentences to compare intelligibility estimates based on fewer listeners and sentences to a speaker-specific benchmark estimate based on the mean intelligibility across all sentences and listeners. Intelligibility estimates were closer to the benchmark (i.e., more accurate) when more listeners and sentences were included in the estimation process for transcription- and VAS-based estimates and for SLPs and inexperienced listeners. Differences between the benchmark and subset-based intelligibility estimates were, in some cases, smaller than the minimally detectable change in intelligibility for people with PD. The intelligibility of people with PD can be measured more efficiently by reducing the number of listeners and/or sentences, up to a point, while maintaining the ability to detect change in this outcome. Clinicians and researchers may prioritize either fewer listeners or fewer sentences, depending on the specific constraints of their work setting. However, consideration must be given to listener experience and estimation method, as the effect of reducing the number of listeners and sentences varied with these factors.

The Effect of Lead Exposure on Children’s Learning in the Developing World: A Meta-Analysis

Abstract Around half of children in low-income countries have elevated blood-lead levels. What role does lead play in explaining low educational outcomes in these settings? We conduct a new systematic review and meta-analysis of observational studies on the relationship between lead exposure and learning outcomes. Adjusting for observable confounds and publication bias yields a benchmark estimate of a $-0.12$ standard-deviation reduction in learning per natural log unit of blood lead. As all estimates are non-experimental, we present evidence on the likely magnitude of unobserved confounding, and summarize results from a smaller set of natural experiments. Our benchmark estimate accounts for over a fifth of the gap in learning outcomes between rich and poor countries, and implies moderate learning gains from targeted interventions for highly exposed groups ($\approx$ 0.1 standard deviations) and modest learning gains ($\lt 0.05$ standard deviations) from broader public health campaigns.

National expenditures on local amenities

We develop a framework for estimating Americans' implicit expenditures on spatially varying nonmarket amenities. We focus on location-specific factors that affect the quality of life but are not formally traded. Examples include climate, geography, pollution, local public goods, and transportation infrastructure. Households pay for residential access to these amenities indirectly, through housing prices, wages and property taxes. We construct a database of 75 amenities, match it to 5 million households' location choices, and use hedonic methods to estimate their total amenity expenditures. Our benchmark estimate for the year 2000 is $562 billion--equivalent to 8% of Americans’ personal consumption expenditures.

Raking of data from a large Australian cohort study improves generalisability of estimates of prevalence of health and behaviour characteristics and cancer incidence

BackgroundHealth surveys are commonly somewhat non-representative of their target population, potentially limiting the generalisability of prevalence estimates for health/behaviour characteristics and disease to the population. To reduce bias, weighting methods have been developed, though few studies have validated weighted survey estimates against generally accepted high-quality independent population benchmark estimates.MethodsWe applied post-stratification and raking methods to the Australian 45 and Up Study using Census data and compared the resulting prevalence of characteristics to accepted population benchmark estimates and separately, the incidence rates of lung, colorectal, breast and prostate cancer to whole-of-population estimates using Standardised Incidence Ratios (SIRs).ResultsThe differences between 45 and Up Study and population benchmark estimates narrowed following sufficiently-informed raking, e.g. 13.6% unweighted prevalence of self-reported fair/poor overall health, compared to 17.0% after raking and 17.9% from a population benchmark estimate. Raking also improved generalisability of cancer incidence estimates. For example, unweighted 45 and Up Study versus whole-of-population SIRs were 0.700 (95%CI:0.574–0.848) for male lung cancer and 1.098 (95%CI:1.002–1.204) for prostate cancer, while estimated SIRs after sufficiently-informed raking were 0.828 (95%CI:0.684–0.998) and 1.019 (95%CI:0.926–1.121), respectively.ConclusionRaking may be a useful tool for improving the generalisability of exposure prevalence and disease incidence from surveys to the population.

Understanding the Forecast Skill of Rapid Arctic Sea Ice Loss on Subseasonal Time Scales

Abstract The predictability of sea ice during extreme sea ice loss events on subseasonal (daily to weekly) time scales is explored in dynamical forecast models. These extreme sea ice loss events (defined as the 5th percentile of the 5-day change in sea ice extent) exhibit substantial regional and seasonal variability; in the central Arctic Ocean basin, most subseasonal rapid ice loss occurs in the summer, but in the marginal seas rapid sea ice loss occurs year-round. Dynamical forecast models are largely able to capture the seasonality of these extreme sea ice loss events. In most regions in the summertime, sea ice forecast skill is lower on extreme sea ice loss days than on nonextreme days, despite evidence that links these extreme events to large-scale atmospheric patterns; in the wintertime, the difference between extreme and nonextreme days is less pronounced. In a damped anomaly forecast benchmark estimate, the forecast error remains high following extreme sea ice loss events and does not return to typical error levels for many weeks; this signal is less robust in the dynamical forecast models but still present. Overall, these results suggest that sea ice forecast skill is generally lower during and after extreme sea ice loss events and also that, while dynamical forecast models are capable of simulating extreme sea ice loss events with similar characteristics to what we observe, forecast skill from dynamical models is limited by biases in mean state and variability and errors in the initialization. Significance Statement We studied weather model forecasts of changes in Arctic sea ice extent on day-to-day time scales in different regions and seasons. We were especially interested in extreme sea ice loss days, or days in which sea ice melts very quickly or is reduced due to diverging forces such as winds, ocean currents, and waves. We find that forecast models generally capture the observed timing of extreme sea ice loss days. We also find that forecasts of sea ice extent are worse on extreme sea ice loss days compared to typical days, and that forecast errors remain elevated following extreme sea ice loss events.

A Simple Question Goes a Long Way: A Wording Experiment on Bank Account Ownership.

Ownership of a bank account is an objective measure and should be relatively easy to elicit via survey questions. Yet, depending on the interview mode, the wording of the question and its placement within the survey may influence respondents' answers. The Health and Retirement Study (HRS) asset module, as administered online to members of the Understanding America Study (UAS), yielded substantially lower rates of reported bank account ownership than either a single question on ownership in the Current Population Survey (CPS) or the full asset module administered to HRS panelists (both interviewer-administered surveys). We designed and implemented an experiment in the UAS comparing the original HRS question eliciting bank account ownership with two alternative versions that were progressively simplified. We document strong evidence that the original question leads to systematic underestimation of bank account ownership. In contrast, the proportion of bank account owners obtained from the simplest alternative version of the question is very similar to the population benchmark estimate. We investigate treatment effect heterogeneity by cognitive ability and financial literacy. We find that questionnaire simplification affects responses of individuals with higher cognitive ability substantially less than those with lower cognitive ability. Our results suggest that high-quality data from surveys start from asking the right questions, which should be as simple and precise as possible and carefully adapted to the mode of interview.

Analysis of Methodology for Scaling up Building Retrofits: Is There a Role for Virtual Energy Audits?—A First Step in Hawai’i, USA

Energy audits are a time-consuming and expensive initial step in the building retrofit process. Virtual energy audits purport to be an alternative that remotely identifies energy efficiency measures (EEMs) that may reduce electricity consumption and offset operational costs to businesses operating during and after the COVID-19 pandemic. This case study reviews virtual energy audits as a means to benchmark energy use and estimate cost savings from future EEMs. A novel feature was the estimation of energy costs associated with increasing ventilation to improve indoor air quality. The authors analyzed ten virtual energy audits performed in Honolulu, Hawai’i, over a two-week period that used existing building information and electricity use data to estimate a potential 9% to 41% annual electricity use reduction per building and a 24 MWh to 1195 MWh reduction, respectively. This paper makes a significant contribution through its assessment of virtual energy audits as a step beyond benchmarking, which has merit as an educational tool to motivate business owners to reduce energy use and improve indoor air quality. Further evaluation and improvements are suggested to study how often the virtual energy audits instigate action, how they compare with in-person audits, and their potential for use at a large scale.

The Reform of Housing Policies in Urban China and Household Consumption

Since 1998, housing policies in urban China have gone through a series of reforms that transformed housing allocation from a centralized welfare housing system to a market-oriented housing system. Through the process welfare dwelling residents are eligible to purchase their existing dwelling at a subsidized price. This access to welfare housing drives a wedge in the cost of acquiring property and subsequently can imply a substantial difference of net housing wealth and housing debts. This paper investigates the impact of access to welfare housing on household consumption by using a large household-level survey data from 2010 to 2018. Our results indicate that households owned welfare housing in the past had a higher level of consumption than households without access to welfare housing, and the benchmark estimate shows owning welfare housing can increase consumption by 11.5 per cent. Our results also suggest that this positive impact on consumption decays over time, and is particularly significant for low-consumption households and for those residing in first-tier cities.

Are we ready for a National Forest Information System? State of the art of forest maps and airborne laser scanning data availability in Italy

Forest planning, forest management, and forest policy require updated, reliable, and harmonized spatial datasets. In Italy a national geographic Forest Information System (FIS) designed to store and facilitate the access and analysis of spatial datasets is still missing. Among the different information layers which are useful to start populating a FIS, two are essential for their multiple use in the assessment of forest resources: (i) forest mapping, and (ii) data from Airborne Laser Scanning (ALS). Both layers are not available wall-to-wall for Italy, though different local sources of information potentially useful for their implementation already exist. The objectives of this work were to: (i) review forest maps and ALS data availability in Italy; (ii) develop for the first time a high resolution forest mask of Italy which was validated against the official statistics of the Italian National Forest Inventory; (iii) develop the first mosaic of all the main ALS data available in Italy producing a consistent Canopy Height Model (CHM). An on-line geographic FIS with free access to both layers from (ii) and (iii) was developed for demonstration purposes. The total area of forest and other wooded lands computed from the forest mask was 102.608.82 km2 (34% of the Italian territory), i.e., 1.9% less than the NFI benchmark estimate. This map is currently the best wall-to-wall forest mask available for Italy. We showed that only the 63% of the Italian territory (the 60% of the forest area) is covered by ALS data. These results highlight the urgent need for a national strategy to complete the availability of forest data in Italy.

Ambiguous protection

Import regulations are globally the most prevalent form of intervention in international trade. The regulations should, under rules of the WTO, protect consumers and the environment but can be used to protect producers. We investigate the ambiguity of intent of the regulations. We set out a model that when applied empirically suggests, as a benchmark estimate, equal divide between social benefit and producer protection. Inefficiency and distributional effects of the regulations are consistent with producer-protecting trade policy. Country diversity in use of regulations supplements our estimates in suggesting presence of producer-protecting intent. We look at how WTO procedures have allowed producer protection in the guise of social benefit.

Elucidation of the Effects of Bisphenol A and Structural Analogs on Germ and Steroidogenic Cells Using Single Cell High-Content Imaging.

Concerns about the potential adverse effects of bisphenol A (BPA) have led to an increase in the use of replacements, yet the toxicity data for several of these chemicals are limited. Using high-content imaging, we compared the effects of BPA, BPAF, BPF, BPS, BPM, and BPTMC in germ (C18-4 spermatogonial) and steroidogenic (MA-10 Leydig and KGN granulosa) cell lines. Effects on cell viability and phenotypic markers were analyzed to determine benchmark concentrations (BMCs) and estimate administered equivalent doses (AEDs). In all 3 cell lines, BPA was one of the least cytotoxic bisphenol compounds tested, whereas BPM and BPTMC were the most cytotoxic. Interestingly, BPF and BPS were cytotoxic only in MA-10 cells. Effects on phenotypic parameters, including mitochondria, lysosomes, lipid droplets, and oxidative stress, were both bisphenol- and cell-line specific. BPA exposure affected mitochondria (BMC: 1.2 μM; AED: 0.09 mg/kg/day) in C18-4 cells. Lysosome numbers were increased in MA-10 cells exposed to BPA or BPAF but decreased in KGN cells exposed to BPAF or BPM. Lipid droplets were decreased in C18-4 cells exposed to BPF and in MA-10 cells exposed to BPTMC but increased in BPF, BPM, and BPTMC-exposed KGN cells. BPA and BPM exposure induced oxidative stress in MA-10 and KGN cells, respectively. In summary, structurally similar bisphenols displayed clear cell-line-specific differences in BMC and AED values for effects on cell viability and phenotypic endpoints. This approach, together with additional data on human exposure, may aid in the selection and prioritization of responsible replacements for BPA. .

Benchmark estimate of the effect of anthropogenic emissions on the ocean surface

AbstractInvestigations into the role of anthropogenic emissions in the occurrence of extreme weather often use a method that compares simulations of atmospheric climate models run under a factual scenario of historical boundary conditions observed during the period of the event against simulations run under a counterfactual scenario of what those boundary conditions might naturally have been over that same period in the absence of anthropogenic emissions. A particular requirement for this experiment design is an accurate estimation of ocean surface boundary conditions for use by the counterfactual natural simulations. Here we use output from the CMIP5 multi‐climate‐model archive to develop a robust estimate of sea surface temperatures and sea ice conditions for use in counterfactual natural simulations, intended as a benchmark estimate to facilitate comparison across climate models and across studies. This development includes tests to ensure that the final estimate is stable from year‐to‐year and stable against other perturbations to the methodology, as well as consideration of the strengths and weaknesses in comparison to other available attributable warming estimates. While this estimate is tailored specifically for the International CLIVAR C20C+ Detection and Attribution Project, it can be used by related projects as well.

Endogeneity in Okun’s law

ABSTRACTThis article contributes to the literature of Okun’s law by addressing an under-noticed but important problem, i.e. endogeneity. By using annual variation in the international oil price weighted with countries’ average oil net-export GDP ratios as a plausibly exogenous instrumental variable for economic growth, the two-stage least squares estimates indicate that faster real GDP growth is associated with lower overall, male and female unemployment rates, confirming the validity of Okun’s law. According to our benchmark estimate, 1 percentage point increase in economic growth leads to a 0.252 percentage point decrease in civil unemployment rate.

Within study comparisons and risk of bias in international development: Systematic review and critical appraisal.

Many systematic reviews incorporate nonrandomised studies of effects, sometimes called quasi-experiments or natural experiments. However, the extent to which nonrandomised studies produce unbiased effect estimates is unclear in expectation or in practice. The usual way that systematic reviews quantify bias is through "risk of bias assessment" and indirect comparison of findings across studies using meta-analysis. A more direct, practical way to quantify the bias in nonrandomised studies is through "internal replication research", which compares the findings from nonrandomised studies with estimates from a benchmark randomised controlled trial conducted in the same population. Despite the existence of many risks of bias tools, none are conceptualised to assess comprehensively nonrandomised approaches with selection on unobservables, such as regression discontinuity designs (RDDs). The few that are conceptualised with these studies in mind do not draw on the extensive literature on internal replications (within-study comparisons) of randomised trials. Our research objectives were as follows:Objective 1: to undertake a systematic review of nonrandomised internal study replications of international development interventions.Objective 2: to develop a risk of bias tool for RDDs, an increasingly common method used in social and economic programme evaluation. We used the following methods to achieve our objectives.Objective 1: we searched systematically for nonrandomised internal study replications of benchmark randomised experiments of social and economic interventions in low- and middle-income countries (L&MICs). We assessed the risk of bias in benchmark randomised experiments and synthesised evidence on the relative bias effect sizes produced by benchmark and nonrandomised comparison arms.Objective 2: We used document review and expert consultation to develop further a risk of bias tool for quasi-experimental studies of interventions (ROBINS-I) for RDDs. Objective 1: we located 10 nonrandomised internal study replications of randomised trials in L&MICs, six of which are of RDDs and the remaining use a combination of statistical matching and regression techniques. We found that benchmark experiments used in internal replications in international development are in the main well-conducted but have "some concerns" about threats to validity, usually arising due to the methods of outcomes data collection. Most internal replication studies report on a range of different specifications for both the benchmark estimate and the nonrandomised replication estimate. We extracted and standardised 604 bias coefficient effect sizes from these studies, and present average results narratively.Objective 2: RDDs are characterised by prospective assignment of participants based on a threshold variable. Our review of the literature indicated there are two main types of RDD. The most common type of RDD is designed retrospectively in which the researcher identifies post-hoc the relationship between outcomes and a threshold variable which determines assignment to intervention at pretest. These designs usually draw on routine data collection such as administrative records or household surveys. The other, less common, type is a prospective design where the researcher is also involved in allocating participants to treatment groups from the outset. We developed a risk of bias tool for RDDs. Internal study replications provide the grounds on which bias assessment tools can be evidenced. We conclude that existing risk of bias tools needs to be further developed for use by Campbell collaboration authors, and there is a wide range of risk of bias tools and internal study replications to draw on in better designing these tools. We have suggested the development of a promising approach for RDD. Further work is needed on common methodologies in programme evaluation, for example on statistical matching approaches. We also highlight that broader efforts to identify all existing internal replication studies should consider more specialised systematic search strategies within particular literatures; so as to overcome a lack of systematic indexing of this evidence.

Benchmark validation of statistical models: Application to mediation analysis of imagery and memory.

This article describes benchmark validation, an approach to validating a statistical model. According to benchmark validation, a valid model generates estimates and research conclusions consistent with a known substantive effect. Three types of benchmark validation-(a) benchmark value, (b) benchmark estimate, and (c) benchmark effect-are described and illustrated with examples. Benchmark validation methods are especially useful for statistical models with assumptions that are untestable or very difficult to test. Benchmark effect validation methods were applied to evaluate statistical mediation analysis in eight studies using the established effect that increasing mental imagery improves recall of words. Statistical mediation analysis led to conclusions about mediation that were consistent with established theory that increased imagery leads to increased word recall. Benchmark validation based on established substantive theory is discussed as a general way to investigate characteristics of statistical models and a complement to mathematical proof and statistical simulation. (PsycINFO Database Record (c) 2018 APA, all rights reserved).

Population relatedness and cross-country idea flows: evidence from book translations

This paper establishes a robust relationship between idea flows across countries, as captured by book translations, and two measures of population relatedness. I argue that linguistic distance imposes a cost on idea flows, whereas genetic distance captures an incentive to communicate when dissimilar countries have more to learn from each other. Consistent with this hypothesis, I find that linguistic distance is negatively associated with book translations, whereas genetic distance is positively associated with book translations after conditioning on linguistic and geographic distance. In particular, the benchmark estimate indicates that a one standard deviation increase in linguistic distance reduces book translations by 12%, while a one standard deviation increase in genetic distance increases book translations by 10%.

Weights to Address Non-parallel Trends in Panel Difference-in-differences Models

Causal inference using the difference‐in‐differences (DD) method relies on the untestable assumption of parallel counterfactual trends across units that are assigned to different treatments. To facilitate the application of the method in settings where the parallel‐trends assumption is seemingly violated, I suggest weighting observations such that the conditional correlations between treatments and pre‐treatment outcome trends are minimised, i.e. weighted trends are parallel. I evaluate the performance of a weighted parallel trends (WPT) DD estimator in a Monte Carlo study and provide an application to a case‐study context in which a benchmark estimate exists. The WPT DD approach can be applied in settings with multiple continuous treatment variables as well as to estimating time‐varying treatment effects.

Designs of Empirical Evaluations of Nonexperimental Methods in Field Settings.

Over the last three decades, a research design has emerged to evaluate the performance of nonexperimental (NE) designs and design features in field settings. It is called the within-study comparison (WSC) approach or the design replication study. In the traditional WSC design, treatment effects from a randomized experiment are compared to those produced by an NE approach that shares the same target population. The nonexperiment may be a quasi-experimental design, such as a regression-discontinuity or an interrupted time-series design, or an observational study approach that includes matching methods, standard regression adjustments, and difference-in-differences methods. The goals of the WSC are to determine whether the nonexperiment can replicate results from a randomized experiment (which provides the causal benchmark estimate), and the contexts and conditions under which these methods work in practice. This article presents a coherent theory of the design and implementation of WSCs for evaluating NE methods. It introduces and identifies the multiple purposes of WSCs, required design components, common threats to validity, design variants, and causal estimands of interest in WSCs. It highlights two general approaches for empirical evaluations of methods in field settings, WSC designs with independent and dependent benchmark and NE arms. This article highlights advantages and disadvantages for each approach, and conditions and contexts under which each approach is optimal for addressing methodological questions.

The economic cost of subway congestion: Estimates from Paris

Related to the increased encouragement of public transport (PT) by policy-makers, over-crowding in PT has become a major issue worldwide. Whilst the impact of in-vehicle crowding on individuals' travel costs has been considered, we focus on aggregate welfare losses. We apply a Pigouvian framework to the case of subways and compute the economic cost of congestion (ECC). We combine data of the 14 metro services of the Paris network with survey data from a contingent valuation study of in-vehicle congestion. The gap between current and optimal PT patronage is 9%, and ECC is moderate. For the entire Paris subway network our benchmark estimate of 64.6 million euros per year amounts to 0.9% of total users' costs. We also propose marginal congestion costs relevant for socioeconomic appraisals of transport projects.

Dimension reduction and shrinkage methods for high dimensional disease risk scores in historical data.

BackgroundMultivariable confounder adjustment in comparative studies of newly marketed drugs can be limited by small numbers of exposed patients and even fewer outcomes. Disease risk scores (DRSs) developed in historical comparator drug users before the new drug entered the market may improve adjustment. However, in a high dimensional data setting, empirical selection of hundreds of potential confounders and modeling of DRS even in the historical cohort can lead to over-fitting and reduced predictive performance in the study cohort. We propose the use of combinations of dimension reduction and shrinkage methods to overcome this problem, and compared the performances of these modeling strategies for implementing high dimensional (hd) DRSs from historical data in two empirical study examples of newly marketed drugs versus comparator drugs after the new drugs’ market entry—dabigatran versus warfarin for the outcome of major hemorrhagic events and cyclooxygenase-2 inhibitor (coxibs) versus nonselective non-steroidal anti-inflammatory drugs (nsNSAIDs) for gastrointestinal bleeds.ResultsHistorical hdDRSs that included predefined and empirical outcome predictors with dimension reduction (principal component analysis; PCA) and shrinkage (lasso and ridge regression) approaches had higher c-statistics (0.66 for the PCA model, 0.64 for the PCA + ridge and 0.65 for the PCA + lasso models in the warfarin users) than an unreduced model (c-statistic, 0.54) in the dabigatran example. The odds ratio (OR) from PCA + lasso hdDRS-stratification [OR, 0.64; 95 % confidence interval (CI) 0.46–0.90] was closer to the benchmark estimate (0.93) from a randomized trial than the model without empirical predictors (OR, 0.58; 95 % CI 0.41–0.81). In the coxibs example, c-statistics of the hdDRSs in the nsNSAID initiators were 0.66 for the PCA model, 0.67 for the PCA + ridge model, and 0.67 for the PCA + lasso model; these were higher than for the unreduced model (c-statistic, 0.45), and comparable to the demographics + risk score model (c-statistic, 0.67).ConclusionshdDRSs using historical data with dimension reduction and shrinkage was feasible, and improved confounding adjustment in two studies of newly marketed medications.Electronic supplementary materialThe online version of this article (doi:10.1186/s12982-016-0047-x) contains supplementary material, which is available to authorized users.