Important Source Of Measurement Error Research Articles

Integrating sphere port error in diffuse reflectance measurements.

The impact of a finite thickness integrating sphere port on the measurement of diffuse reflectance is addressed in a combined numerical and experimental study. It is shown that for a finite thickness port, additional light losses occur due to scattering between the sphere port wall and the test sample, causing the sample reflectance to be underestimated. Monte Carlo ray tracing is applied to obtain quantitative estimates of the resulting measurement error for the case of a diffusely reflecting sample. The effects of sample reflectance, port geometry, and illumination beam size on the measurement error are explored. Experimental data collected with a pair of integrating sphere reflectometers with different port geometries support the validity of the numerical results. It is argued that finite port thickness may be an important source of measurement error, even for a well-designed integrating sphere port, and a strategy for minimizing this error is discussed.

Reassessing the missing link in general deterrence research: A behavioral economic approach

PurposeGiven the increasing application of behavioral economics within criminology, the present investigation seeks to reassess the ‘missing link’ in general deterrence research from the angle of coherent arbitrariness. Such an effort seeks to bridge the divide between two competing positions within the literature: one that views the lack of correspondence between actual and perceived indicators of punishment as a challenge to the theoretical construct, while the other views this lack of correspondence as both misleading and uninformative for deterrence research. MethodsDrawing on a dataset of certainty, severity, and celerity estimates from 1500 respondents nested in 54 urban counties, the transitivity of individual-level punishment perceptions were mapped against similarly rank-ordered, county-specific punishment risk levels. Clustered Wilcoxon signed-rank tests were estimated to compare these deterrence constructs. ResultsAfter adjusting for an important source of measurement error in elicited risk perceptions, results indicated significant differences, and thus a lack of correspondence, between the rank-orderings of actual and perceived levels of punishment certainty, severity, and celerity. ConclusionsThe continuing lack of calibration between perceptions of punishment risk and objective levels of the certainty, severity, and celerity of punishment requires additional theorization and analysis within criminology, broadly, and deterrence scholarship, specifically.

Patterns of Missing Data With Ecological Momentary Assessment Among People Who Use Drugs: Feasibility Study Using Pilot Study Data.

BackgroundEcological momentary assessment (EMA) is a set of research methods that capture events, feelings, and behaviors as they unfold in their real-world setting. Capturing data in the moment reduces important sources of measurement error but also generates challenges for noncompliance (ie, missing data). To date, EMA research has only examined the overall rates of noncompliance.ObjectiveIn this study, we identify four types of noncompliance among people who use drugs and aim to examine the factors associated with the most common types.MethodsData were obtained from a recent pilot study of 28 Nebraskan people who use drugs who answered EMA questions for 2 weeks. We examined questions that were not answered because they were skipped, they expired, the phone was switched off, or the phone died after receiving them.ResultsWe found that the phone being switched off and questions expiring comprised 93.34% (1739/1863 missing question-instances) of our missing data. Generalized structural equation model results show that participant-level factors, including age (relative risk ratio [RRR]=0.93; P=.005), gender (RRR=0.08; P=.006), homelessness (RRR=3.80; P=.04), personal device ownership (RRR=0.14; P=.008), and network size (RRR=0.57; P=.001), are important for predicting off missingness, whereas only question-level factors, including time of day (ie, morning compared with afternoon, RRR=0.55; P<.001) and day of week (ie, Tuesday-Saturday compared with Sunday, RRR=0.70, P=.02; RRR=0.64, P=.005; RRR=0.58, P=.001; RRR=0.55, P<.001; and RRR=0.66, P=.008, respectively) are important for predicting expired missingness. The week of study is important for both (ie, week 2 compared with week 1, RRR=1.21, P=.03, for off missingness and RRR=1.98, P<.001, for expired missingness).ConclusionsWe suggest a three-pronged strategy to preempt missing EMA data with high-risk populations: first, provide additional resources for participants likely to experience phone charging problems (eg, people experiencing homelessness); second, ask questions when participants are not likely to experience competing demands (eg, morning); and third, incentivize continued compliance as the study progresses. Attending to these issues can help researchers ensure maximal data quality.

Pre-analytical Factors Influence Accuracy of Urine Spot Iodine Assessment in Epidemiological Surveys.

Urinary iodine concentration (UIC) is commonly used to assess iodine status of subjects in epidemiological surveys. As pre-analytical factors are an important source of measurement error and studies about this phase are scarce, our objective was to assess the influence of urine sampling conditions on UIC, i.e., whether the child ate breakfast or not, urine void rank of the day, and time span between last meal and urine collection. A nationwide, two-stage, stratified, cross-sectional study including 1560 children (6-12years) was performed in 2012. UIC was determined by the Sandell-Kolthoff method. Pre-analytical factors were assessed from children's mothers by using a questionnaire. Association between iodine status and pre-analytical factors were adjusted for one another and socio-economic characteristics by multivariate linear and multinomial regression models (RPR: relative prevalence ratios). Skipping breakfast prior to morning urine sampling decreased UIC by 40 to 50μg/L and the proportion of UIC < 100μg/L was higher among children having those skipped breakfast (RPR = 3.2[1.0-10.4]). In unadjusted analyses, UIC was less among children sampled more than 5h from their last meal. UIC decreased with rank of urine void (e.g., first vs. second, P < 0.001); also, the proportion of UIC < 100μg/L was greater among 4th rank samples (vs. second RPR = 2.1[1.1-4.0]). Subjects' breakfast status and urine void rank should be accounted for when assessing iodine status. Providing recommendations to standardize pre-analytical factors is a key step toward improving accuracy and comparability of survey results for assessing iodine status from spot urine samples. These recommendations have to be evaluated by future research.

Investigating the validity of oral assessment rater training program: A mixed-methods study of raters’ perceptions and attitudes before and after training

Rater variability has always been identified as an important source of measurement error in performance assessment, especially for oral proficiency tests. Rater training is commonly used as a means for compensating various sources of rater variability and adjusting their assessment quality. However, there is little research regarding the nature of training programs and raters’ perception using both a qualitative and a quantitative research design. Despite previous data on test takers’ reactions to oral test performance, there is little research regarding the application of test feedback and raters’ perceptions of the given feedback on their scoring performance and its probable usefulness on raters’ ratings. Twenty raters rated 300 test takers’ oral performances before and after a training program and their perceptions, attitudes, expectation, and evaluations were identified via questionnaires, interviews, and observations. The findings of qualitative and quantitative data analyses demonstrated that training programs are quite useful in satisfying their attitudes, perceptions, and evaluations about it. This will definitely result in the reduction of their severity and biases and increase in their consistency level. Besides, informing raters of the goals of performance assessment in training programs will result in less halo effect. Finally, those having positive attitudes toward rating feedback were able to incorporate it more successfully in their rating and thus achieved more consistency and less biasedness in their subsequent ratings. Consequently, decision-makers should not be concerned about raters’ expertise levels, but they should establish rater training programs to increase rater consistency and reduce their biases in measurement.

Policy and evidence: a critical analysis of the Year 1 Phonics Screening Check in England

Abstract“This paper aims to present a critical analysis of the Year 1 Phonics Screening Check (PSC), with special focus on the relationship between the UK Department for Education's policy‐making and the evidence considered in the process of developing and evaluating the PSC. The reports from the in‐house Standards and Testing Agency and from commissioned external organisations with limited remits have been analysed, with emphasis on how the screening check findings have been used in DfE decision‐making. My conclusion is that the department has overestimated both the validity and the reliability of the check: partly because important sources of measurement error have not been explored, and partly because the available evidence has not been analysed in sufficient depth — or perhaps simply been ignored by the department. The whole phonics screening check seems to be the product of policy‐based evidence, rather than evidence‐based policy‐making. This has, in my view, seriously undermined the value of the PSC as a screening instrument, and one must wonder whether it is ‘fit for purpose’.

Analgesia use during pregnancy and risk of cryptorchidism: a systematic review and meta-analysis.

Are boys who are born to mothers who use analgesics during pregnancy at increased risk of cryptorchidism compared to those born to mothers who do not take analgesia? In this systematic review and meta-analysis of 10 published studies, we observed only weak evidence of an association between analgesia use during pregnancy and risk of cryptorchidism in the son. Concentrations of analgesia relevant to human exposure have been implicated as causing endocrine disturbances in the developing foetal testis. However, when viewed collectively there appears to be conflicting evidence regarding an association between maternal use of analgesics and development of cryptorchidism. A systematic review and meta-analysis of studies on analgesia use during pregnancy and risk of cryptorchidism was performed. The search terms used were (analges* OR paracetamol OR acetaminophen) AND (cryptorchidism OR cryptorchism OR undescended test* OR non-descended test* OR non descended test*) for the databases Ovid Medline, Embase, Scopus and Web of Science. The search included all published articles up until23 May 2016 and no limits were set in terms of language. Abstracts were screened by one reviewer to remove irrelevant studies, with a 10% random sample of these verified by a second reviewer. The full text of all remaining papers was assessed by two reviewers. Abstracts included in the final analysis were studies which reported associations between the exposure (analgesia) and the outcome (cryptorchidism). Studies were only included if data were provided from which summary associations (odds ratios (ORs) or relative risks) and their 95%CIs could be calculated, or if summary associations were provided by the authors themselves. For each included study, two reviewers independently extracted study meta-data in line with PRISMA recommendations. We assessed study quality and potential for bias using the criteria outlined in the Newcastle-Ottawa Quality Assessment Scale, but did not determine a quality score. Two reviewers independently assessed study quality against these criteria. After screening 350 manuscripts, 10 were included in our review (5 case-control studies, 5 cohort studies). We observed weak evidence of an association between ever use of analgesia and risk of cryptorchidism (pooled crude OR: 1.11, 95% CI: 1.00-1.23), with case-control studies revealing a marginally stronger association (1.23, 95% CI: 0.85-1.78) than cohort studies (1.09, 95% CI: 0.97-1.22). We observed weak evidence of a dose-response relationship between increasing weeks of analgesia exposure and risk of cryptorchidism, as well as weak evidence of an effect of timing on analgesia exposure and risk of cryptorchidism. Assessment of study quality via the Newcastle-Ottawa criteria revealed little (if any) evidence of substantial bias that may have meaningfully affected a given study's results. While confounding does not appear to be important, misclassification of the exposure is possibly an important source of measurement error in this context. The systematic review is open to reporting bias. Owing to scant data, no meta-analyses for two key questions (relating to dose-response and timing of exposure) could be performed. Medications were grouped based on their common effect and this offers little insight into the relation between specific types of analgesia and cryptorchidism. Finally, there are limitations in assuming that analgesia use reported by mothers is synonymous with actual intrauterine exposure. The ubiquity of analgesia use during pregnancy makes this exposure particularly important from a population health perspective. About 9 of the 10 studies were conducted in Europe or USA, limiting generalizability of our observations. While the observations from our systematic review and meta-analysis suggest that analgesia use during pregnancy is not strongly associated with cryptorchidism development in the son, they also highlight the need for further detailed assessments of this relationship. This study was funded by the Health Research Council of New Zealand (reference #: 14/052). The authors have no conflict of interest to declare. CRD42016041414.

Moving towards a reliable HIV incidence test - current status, resources available, future directions and challenges ahead.

In 2011 the Incidence Assay Critical Path Working Group reviewed the current state of HIV incidence assays and helped to determine a critical path to the introduction of an HIV incidence assay. At that time the Consortium for Evaluation and Performance of HIV Incidence Assays (CEPHIA) was formed to spur progress and raise standards among assay developers, scientists and laboratories involved in HIV incidence measurement and to structure and conduct a direct independent comparative evaluation of the performance of 10 existing HIV incidence assays, to be considered singly and in combinations as recent infection test algorithms. In this paper we report on a new framework for HIV incidence assay evaluation that has emerged from this effort over the past 5 years, which includes a preliminary target product profile for an incidence assay, a consensus around key performance metrics along with analytical tools and deployment of a standardized approach for incidence assay evaluation. The specimen panels for this evaluation have been collected in large volumes, characterized using a novel approach for infection dating rules and assembled into panels designed to assess the impact of important sources of measurement error with incidence assays such as viral subtype, elite host control of viraemia and antiretroviral treatment. We present the specific rationale for several of these innovations, and discuss important resources for assay developers and researchers that have recently become available. Finally, we summarize the key remaining steps on the path to development and implementation of reliable assays for monitoring HIV incidence at a population level.

Observing instructional quality in the context of school evaluation

Observational measures of instructional quality are used in both research and evaluation contexts. In European school evaluation systems, one main intended use of classroom observation results is to inform schools' instructional improvement efforts. This study examined the quality of such diagnostic information in the context of a school evaluation system in Germany. The factor analytic results indicate that the empirical structure of the observation instrument lacks correspondence with its original normative model but does mirror a five-factor model based on recent literature, including classroom management, two different aspects of student orientation, cognitive activation, and classroom assessment. The generalizability analyses show that both indicators and observers represent important sources of measurement error. Additionally, their magnitude is larger for shorter observations (lesson segments). At least two, sometimes three, observers and between five and ten indicators per dimension are needed to achieve sufficiently reliable results, but results importantly depend on which dimension of instructional quality is being assessed.

The Predictive Validity of Subjective Mortality Expectations: Evidence From the Health and Retirement Study

Several recent studies suggest that individual subjective survival forecasts are powerful predictors of both mortality and behavior. Using 15 years of longitudinal data from the Health and Retirement Study, I present an alternative view. Across a wide range of ages, predictions of in-sample mortality rates based on subjective forecasts are substantially less accurate than predictions based on population life tables. Subjective forecasts also fail to capture fundamental properties of senescence, including increases in yearly mortality rates with age. To shed light on the mechanisms underlying these biases, I develop and estimate a latent-factor model of how individuals form subjective forecasts. The estimates of this model's parameters imply that these forecasts incorporate several important sources of measurement error that arguably swamp the useful information they convey.

Vergleich verschiedener DON-Standards

DON is the only mycotoxin of the trichothecene group that has been regulated in food so far by the German authorities. The quantitative determination of DON in food and feed therefore is of great importance regarding compliance with legal limits. An important source of measurement error is the bias introduced by different absolute concentration of calibration standards. To make an estimation of the contribution of calibration standards to total error six different DON-standards provided by the participants of the project “Analysis and occurence of important Fusarium toxins (DON and ZEA) and uptake of these toxins by the German consumer” sponsored by the German Ministry of Consumer Protection, Nutrition and Agriculture (BMVEL, Projekt BLE 00HS055) were analyzed and compared to three batches of commercial standard solutions. The range of the results was between 29.2 and 26.3% for the laboratory standards and 11.3 respective 3% for the commercial standards depending on different calculation modes (UV height and area and fluorescence height). Therefore a substantial reduction of the measurement error by using commercial standards instead of laboratory-prepared standards seems possible.

The reliability of the time to execute various forms of the Finger-to-Nose Test in healthy subjects

The objective of this study was to establish reliability estimates of different versions of the Finger-to-Nose test (FNT) using the generalisability theory. Generalizability theory was used to estimate the relative contribution of trials and occasions to the total measurement error and to generate reliability coefficients. Forty-two healthy adults (aged 20–63 yrs old) free of arm pain or other pathology participated. Participants performed 10 versions of the FNT during two evaluation sessions separated by a 24-hour interval. The time (s) to perform five trials of each version was recorded. The mean reliability coefficient for the one-occasion/one-trial testing situation among the test versions was 0.79 (range = 0.77–0.82). For all versions, the principal source of variance of the test performances came from the participants; minimally important sources of measurement error were also associated with occasion and the interaction effects between participants and occasion. Number of trials was not found to be an important source of error. The 10 versions of the FNT under investigation provide reliable measures of upper limb coordination among healthy adults when assessed by the one evaluator generalised across two days. These results are important to the development of norms for this measure.

Triphasic MRI of pelvic organ descent: sources of measurement error

Purpose: To identify sources of error when measuring pelvic organ displacement during straining using triphasic dynamic magnetic resonance imaging (MRI). Materials and methods: Ten healthy nulliparous woman underwent triphasic dynamic 1.5 T pelvic MRI twice with 1 week between studies. The bladder was filled with 200 ml of a saline solution, the vagina and rectum were opacified with ultrasound gel. T2 weighted images in the sagittal plane were analysed twice by each of the two observers in a blinded fashion. Horizontal and vertical displacement of the bladder neck, bladder base, introitus vaginae, posterior fornix, cul-de sac, pouch of Douglas, anterior rectal wall, anorectal junction and change of the vaginal axis were measured eight times in each volunteer (two images, each read twice by two observers). Variance components were calculated for subject, observer, week, interactions of these three factors, and pure error. An overall standard error of measurement was calculated for a single observation by one observer on a film from one woman at one visit. Results: For the majority of anatomical reference points, the range of displacements measured was wide and the overall measurement error was large. Intra-observer error and week-to-week variation within a subject were important sources of measurement error. Conclusion: Important sources of measurement error when using triphasic dynamic MRI to measure pelvic organ displacement during straining were identified. Recommendations to minimize those errors are made.

Rounding Error: A Distorting Influence on Index Data

Rounding error is an important source of measurement error that is common in index data. The problem can be traced to rounding that occurs to limit the number of digits after the decimal place to be reported in rebased index data. Rounding error introduces distortions that affect variance properties, alter the lag distributions of time series models, and cause a systematic bias in estimated coefficients. For instance, spuriously choppy inflation rates are obtained when constructed using the official CPI, rebased with 1982-84 = 100. Fortunately, the distortions can be generally avoided by using versions of data that have greater precision.

MEASUREMENT ERROR IN RESEARCH ON HUMAN RESOURCES and FIRM PERFORMANCE: HOW MUCH ERROR IS THERE AND HOW DOES IT INFLUENCE EFFECT SIZE ESTIMATES?

Studies of the relationship between human resource (HR) practices and firm performance typically use a single respondent to assess firm level HR practices or HR effectiveness. However, previous research in other substantive areas suggests that rater differences are a potentially important source of measurement error. We demonstrate analytically the potential consequences of both random and systematic measurement error in research on HR and firm performance. However, our main focus is on random error and we show how generalizability theory can be applied to obtain better estimates of reliability by simultaneously recognizing multiple sources (e.g., items, raters) of random measurement error. These more inclusive reliability estimates, in turn, offer the possibility of more precisely quantifying substantive relationships in the HR and firm performance literature. In our sample, reliabilities (as estimated by generalizability coefficients) for single‐rater assessments of HR variables were generally below .50. This degree of measurement error, if present in substantive studies on HR and firm performance, could lead to considerable bias, given that an unstandardized regression coefficient is corrected for measurement error in the independent variable by dividing by its reliability coefficient (not its square root). We also found only limited convergent validity between HR and line managers ratings of a second type of HR measure, HR effectiveness. In general, our findings suggest that future researchers need to devote greater attention to measurement error and construct validity issues. Our study provides an example of how generalizability theory can be useful in this pursuit.

The application of generalizability theory to reliability assessment: an illustration using isometric force measurements.

The objective of this study was to examine the potential use of the generalizability theory for assessing reliability of muscle force measurements in clinical applications. Reliability is expressed in terms of standard error of measurement (SEM) and the indexes derived from the SEM. Using generalizability theory, potential sources of measurement error can be recognized and estimated. Ten healthy women, aged 23 to 47 years (means = 29.5, SD = 7.1), participated in the study. The method is illustrated by presenting a pilot study. Repeated measurements of maximal isometric knee extension force were carried out by two therapists. The time interval between measurement occasions was 1 week. A functional prototype of a hand-held dynamometer was used, and measurements were carried out according to standardized test protocols. Relatively important sources of measurement error were associated with interaction effects between subject and therapist as well as with interactions including the factor occasion. The SEM of the net knee moment was estimated for various hypothetical applications of the dynamometer. It is concluded that the SEM, the corresponding confidence interval, and the smallest detectable difference are practical measures for expressing reliability of measurements on an individual. Generalizability theory is a powerful tool for estimating the magnitude of multiple sources of measurement error and for assessing the reliability of measurements tailored to specific clinical applications.

Estimating Measurement Error on Highly Speeded Tests

Despite many advances in test theory, estimating measurement error which arises from temporary changes in the person or test situation has re mained relatively unchanged. Unfortunately, not only are short-term instabilities the most important source of measurement error for many traits, especially those measured from highly speeded tests; but the classical test-retest formula for es timating error is based on untenable assumptions with respect to practice effects. The current paper presents a method which gives maximum likelihood estimates of measurement error within the context of a simplex model for practice effects. The appropri ateness of the model is tested for five traits, and error estimates are compared to the classical for mula estimates.

A review of statistical methods in the analysis of data arising from observer reliability studies (Part I)*

Summary This paper reviews research situations in medicine, epidemiology and psychiatry, in psychological measurement and testing, and in sample surveys in which the observer(rater or interviewer) can be an important source of measurement error. Moreover, most of the statistical literature in observer variability is surveyed with attention given to a notational unification of the various models proposed. In the continuous data case, the usual analysis of variance (ANOVA) components of variance models are presented with an emphasis on the intraclass correlation coefficient as a measure of reliability. Other modified ANOVA models, response error models in sample surveys, and related multivariate extensions are also discussed. For the categorical data case, special attention is given to measures of agreement and tests of hypotheses when the data consist of dichotomous responses. In addition, similarities between the dichotomous and continous cases are illustrated in terms of intraclass correlation coefficients. Finally, measures of agreement, such as kappa and weighted‐kappa, are discussed in the context of nominal and ordinal data. A proposed unifying framework for the categorical data case is given in the form of concluding remarks.