Rater Estimates Research Articles

Multispecies initial numerical validation of an efficient algorithm prototype for auditory brainstem response hearing threshold estimation.

The auditory brainstem response (ABR) can be used to evaluate hearing sensitivity of animals. However, typical measurement protocols are time-consuming. Here, an adaptive algorithm is proposed for efficient ABR threshold estimation. The algorithm relies on the update of the predicted hearing threshold from a Gaussian process model as ABR data are collected using iteratively optimized stimuli. To validate the algorithm, ABR threshold estimation is simulated by adaptively subsampling pre-collected ABR datasets. The simulated experiment is performed on 5 datasets of mouse, budgerigar, gerbil, and guinea pig ABRs (27 ears). The datasets contain 68-106 stimuli conditions, and the adaptive algorithm is configured to terminate after 20 stimuli conditions. The algorithm threshold estimate is compared against human rater estimates who visually inspected the full waveform stacks. The algorithm threshold matches the human estimates within 10 dB, averaged over frequency, for 15 of the 27 ears while reducing the number of stimuli conditions by a factor of 3-5 compared to standard practice. The intraclass correlation coefficient is 0.81 with 95% upper and lower bounds at 0.74 and 0.86, indicating moderate to good reliability between human and algorithm threshold estimates. The results demonstrate the feasibility of a Bayesian adaptive procedure for rapid ABR threshold estimation.

Visual rating and the use of image analysis for assessing canopy density in a pecan provenance collection during leaf fall

A collection representing the native range of pecan was planted at the USDA − ARS Southeastern Fruit and Tree Nut Research Station, Byron, GA. The collection (867 trees) is a valuable genetic resource for characterizing important horticultural traits. Canopy density during leaf fall is important as the seasonal canopy dynamics provides insights to environmental cues and breeding potential of germplasm. The ability of visual raters to estimate canopy density on a subset of the provenance collection (76 trees) as an indicator of leaf shed during autumn along with image analysis values was explored. Mean canopy density using the image analysis software was less compared to visual estimates (11.9% vs 18.4%, respectively). At higher canopy densities, the raters overestimated foliage density, but overall agreement between raters and measured values was good (ρc = 0.849 to 0.915), and inter-rater reliability was high (R2 = 0.910 to 0.953). The provenance from Missouri (MO-L), the northernmost provenance, had the lowest canopy density in November, and results show that the higher the latitude of the provenance, the lower the canopy density. Based on regression, the source provenance latitude explained 0.609 of the variation using image analysis, and 0.551 to 0.640 when based on the rater estimates of canopy density. Visual assessment of pecan canopy density due to late season leaf fall for comparing pecan genotypes provides accurate and reliable estimates and could be used in future studies of the whole provenance collection.

The Effects of Incomplete Rating Designs in Combination With Rater Effects

AbstractResearchers have explored a variety of topics related to identifying and distinguishing among specific types of rater effects, as well as the implications of different types of incomplete data collection designs for rater‐mediated assessments. In this study, we used simulated data to examine the sensitivity of latent trait model indicators of three rater effects (leniency, central tendency, and severity) in combination with different types of incomplete rating designs (systematic links, anchor performances, and spiral). We used the rating scale model and the partial credit model to calculate rater location estimates, standard errors of rater estimates, model–data fit statistics, and the standard deviation of rating scale category thresholds as indicators of rater effects and we explored the sensitivity of these indicators to rater effects under different conditions. Our results suggest that it is possible to detect rater effects when each of the three types of rating designs is used. However, there are differences in the sensitivity of each indicator related to type of rater effect, type of rating design, and the overall proportion of effect raters. We discuss implications for research and practice related to rater‐mediated assessments.

A discussion on disease severity index values. Part I: warning on inherent errors and suggestions to maximise accuracy

AbstractA special type of ordinal scale comprising a number of intervals of known numeric ranges can be used when estimating severity of a plant disease. The interval ranges are most often based on the percent area with symptoms [e.g. the Horsfall–Barratt (H–B) scale]. Studies in plant pathology and plant breeding often use this type of ordinal scale. The disease severity is estimated by a rater as a value on the scale and has been used to determine a disease severity index (DSI) on a percentage basis, where DSI (%) = [sum (class frequency × score of rating class)]/[(total number of plants) × (maximal disease index)] × 100. However, very few studies have investigated the effects of different scales on accuracy of the DSI. Therefore, the objectives of this study were to investigate the process of calculating a DSI on a percentage basis from ordinal scale data, and to use simulation approaches to explore the effect of using different methods for calculation of the interval range and the nature of the ordinal scales used on the DSI estimates (%). We found that the DSI is particularly prone to overestimation when using the above formula if the midpoint values of the rating class are not considered. Moreover, the results of the simulation studies show that, if rater estimates are unbiased, compared with other methods tested in this study, the most accurate method for estimation of a DSI is to use the midpoint of the severity range for each class with an amended 10% ordinal scale (an ordinal scale based on a 10% linear scale emphasising severities ≤50% disease, with additional grades at low severities). As for biased conditions, the accuracy for calculating DSI estimates (%) will depend mainly on the degree and direction of the rater bias relative to the actual mean value.

Disease Severity Estimates—Effects of Rater Accuracy and Assessment Methods for Comparing Treatments

Assessment of disease severity is required for several purposes in plant pathology; most often, the estimates are made visually. It is established that visual estimates can be inaccurate and unreliable. The ramifications of biased or imprecise estimates by raters have not been fully explored using empirical data, partly because of the logistical difficulties involved in different raters assessing the same leaves for which actual disease has been measured in a replicated experiment with multiple treatments. In this study, nearest percent estimates (NPEs) of Septoria leaf blotch (SLB) on leaves of winter wheat from nontreated and fungicide-treated plots were assessed in both 2006 and 2007 by four raters and compared with assumed actual values measured using image analysis. Lin's concordance correlation (LCC, ρc) was used to assess agreement between the two approaches. NPEs were converted to Horsfall-Barratt (HB) midpoints and were compared with actual values. The estimates of SLB severity from fungicide-treated and nontreated plots were analyzed using generalized linear mixed modeling to ascertain effects of rater using both the NPE and HB values. Rater 1 showed good accuracy (ρc = 0.986 to 0.999), while raters 3 and 4 were less accurate (ρc = 0.205 to 0.936). Conversion to the HB scale had little effect on bias but reduced numerically both precision and accuracy for most raters on most assessment dates (precision, r = -0.001 to -0.132; and accuracy, ρc = -0.003 to -0.468). Interrater reliability was also reduced slightly by conversion of estimates to HB midpoint values. Estimates of mean SLB severity were significantly different between image analysis and raters 2, 3, and 4, and there were frequently significant differences among raters (F = 151 to 1,260, P = 0.001 to P < 0.0001). Only on 26 June 2007 did conversion to the HB scale change the means separation ranking of rater estimates. Nonetheless, image analysis and all raters were able to differentiate control and treated-plot treatments (F = 116 to 1,952, P = 0.002 to P < 0.0001, depending on date and rater). Conversion of NPEs to the HB scale tended to reduce F values slightly (2006: NPEs, F = 116 to 276, P = 0.002 to 0.0005; and, for the HB-converted values, F = 101 to 270, P = 0.002 to 0.0005; 2007: NPEs, F = 164 to 1,952, P = 0.001 to P < 0.0001; and, for HB-converted values, F = 126 to 1,633, P = 0.002 to P < 0.0001). The results reaffirm the need for accurate and reliable disease assessment to minimize over- or underestimates compared with actual disease, and the data we present support the view that, where multiple raters are deployed, they should be assigned in a manner to reduce any potential effect of rater differences on the analysis.

Assessing disease severity: accuracy and reliability of rater estimates in relation to number of diagrams in a standard area diagram set

Error in estimates of plant disease severity occur and standard area diagrams (SADs) help improve accuracy and reliability. The effects of diagram number in SADs are unknown. The objective of this study was to compare estimates of pecan scab severity made without SADs, and using three‐, five‐, seven‐ or 10‐diagram SADs. Disease severity was estimated to the nearest percent (NPE), or classified to the closest reference diagram value using a scale. Twelve raters assessed 20 images of scabbed pecan valves with and without the SADs using NPEs and the scale method (values were converted to midpoints prior to analysis). Increases in diagram number using NPEs did not necessarily result in more accurate or reliable estimates. Inter‐rater reliability was positively correlated with number of diagrams using NPEs (r = 0·3288 (P &lt; 0·0001)) or a scale (r = 0·2803 (P &lt; 0·0001)). The least accurate estimates improved the most using SADs with NPEs, but the gain did not relate to number of diagrams; as few as three diagrams reduced error of inaccurate estimates as much as five, seven or 10 diagrams. When used as a scale, only estimates made with 10‐diagram SADs had similar accuracy and inter‐rater reliability to estimates made using SADs with NPEs. Maximum disease severity and the disease severity range for the pathosystem and the SADs are important factors and will probably influence resulting accuracy and reliability. The ramifications of diagram numbers in SADs and assessment methods on accuracy and reliability of disease estimates are discussed.

A standard area diagram set to aid estimation of the severity of Asiatic citrus canker on ripe sweet orange fruit

Asiatic citrus canker (ACC) is a major disease of citrus in many tropical and subtropical citrus-growing regions. Severe disease can develop on the fruit, making it unmarketable. Research activities need accurate and reliable methods to estimate the severity of citrus canker, most often done visually by raters. The objective was to validate and compare two standard area diagram sets (SADs) as aids for raters to improve the accuracy and reliability of visual estimates of canker severity on ripe fruit of sweet orange (unripe fruit are green, and have slightly different symptoms compared to ripe fruit, warranting different SADs). A 5-diagram SAD set (0.7, 2.0, 7.0, 21.0 and 39.0 %) and a 6-diagram SAD set (0.7, 2.0, 4.0, 10.0, 21.0 and 39.0 %) were compared; the 6-diagram set had an additional diagram in the most common severity range. Fifteen raters evaluated 40 images of cankered, unripe fruit. Both the 5- and 6-diagram SADs improved the accuracy and reliability of rater estimates. Agreement, measured by Lin’s concordance correlation coefficient (ρ c ) was 0.735 to 0.906 when not using SADs, 0.931 to 0.985 when using 5-diagram SADs, and 0.879 to 0.979 when using 6-diagram SADs. Mean inter-rater reliability (R2) was 0.638 when not using SADs, 0.889 when using 5-diagram SADs, and 0.834 when using 6-diagram SADs. The 5-diagram SADs resulted in significantly more accurate and reliable estimates compared with the 6-diagram set, and is thus recommended for use. The SADs should have application for estimating ACC severity on ripe fruit of other citrus, including grapefruit.

Development and validation of standard area diagrams to aid assessment of pecan scab symptoms on fruit

Pecan scab (Fusicladium effusum) causes losses of pecan nutmeat yield and quality in the southeastern United States. Disease assessment relies on visual rating, which can be inaccurate and imprecise, with poor inter‐rater reliability. A standard area diagram (SAD) set for pecan scab on fruit valves was developed. A set of 40 images of diseased fruit valves with known severity was assessed twice by 23 raters. The first assessment was conducted without SADs, and the second assessment was made using the SADs as an aid. SADs improved rater accuracy (correction factor, Cb = 0·86 and 0·97, without and with SADs, respectively) and agreement (Lin’s concordance correlation coefficient, ρc = 0·79 and 0·89 without and with SADs, respectively) with true values. SADs improved inter‐rater reliability (intra‐class correlation coefficient, ρ = 0·77 and 0·96 without and with SADs, respectively). The least accurate and precise raters without SADs improved more using SADs compared to the most accurate and precise raters. Experienced raters had significantly higher accuracy and precision compared to inexperienced raters, but only when unaided by the SAD set. There was no significant difference in time to assess images without SADs, but experienced raters using SADs were faster compared to inexperienced raters. There was a slight tendency for faster raters to assess more slowly, and slower raters to assess faster when using SADs. SADs improve rater estimates of pecan scab severity on fruit, and this SAD set should be useful for assessment where greater precision, accuracy and inter‐rater reliability are required.

C09 Age at onset of Huntington's disease: rater estimates versus data based extrapolation

BackgroundThe accurate determination of age at onset (AAO) of Huntington's disease is critical to develop and evaluate therapies that aim to delay it.Aim(1) To compare the raters' estimates of AAO...

Comparison of Assessment of Citrus Canker Foliar Symptoms by Experienced and Inexperienced Raters.

Citrus canker (Xanthomonas citri subsp. citri) is destructive in many citrus production regions in tropical and subtropical parts of the world. Assessment of canker symptoms is required for diverse reasons, including monitoring epidemics, evaluating the efficacy of control strategies, and disease response in breeding material. The objectives were to compare the ability of experienced and inexperienced raters at assessing citrus canker, to identify factors that affect the quality of the assessment, to determine common sources of error, and to discern how error is related to actual disease magnitude. Two-hundred digital leaf images (0 to 37% area infected) were assessed once by 28 raters, five of whom were experienced plant pathologists (PPs), and 23 who had no experience in disease severity assessment (NPPs). True disease (lesion number [LN], % necrotic area [%N], and % chlorotic+necrotic area [%CN]) was measured using image analysis on a leaf-by-leaf basis, and each parameter was estimated by the 28 raters. LN was neither severely over- nor underestimated, while %N was greatly overestimated, with a lesser tendency to overestimate %CN over the true severity range of these two symptom types. A linear relationship existed between estimate of the disease and true disease for all measures of severity. Data were heteroscedastic and error was not constant with increasing true disease. Agreement between rater estimates and true disease was measured with Lin's concordance correlation coefficient (ρc). LN showed greatest agreement (ρc = 0.88 to 0.99), followed by %CN (ρc = 0.80 to 0.95) and %N (ρc = 0.19 to 0.84). Greater lesion number resulted in overestimation of area infected for both %N and %CN. Overestimation was particularly noticeable at low disease severities. There was a linear relationship between log variance and log true disease for LN (r2 = 0.71), %N (r2 = 0.85), and %CN (r2 = 0.88), and raters tended to estimate disease above 10% to the nearest 5 or 10%. GLM analysis showed differences between PP and NPP groups in assessing disease. For LN, precision of assessment for both groups was similar (r2 > 0.92 and 0.94, respectively), but for estimates of %N and %CN, the PPs were more precise (%N and %CN, r2 = 0.61 and 0.73, respectively) compared to NPPs (%N and %CN, r2 = 0.45 and 0.58, respectively). Absolute error for mean LN was low. The absolute error of %N and %CN showed overestimation to approximately 8% area infected. Above 8%, absolute error increased, but comprised both over- and underestimation. For %N and %CN, relative error was almost exclusively positive and dramatic at severity <8% (up to approximately 600%), but at severity >10% it was relatively small. Error in rater estimates of canker severity is ubiquitous. Understanding these sources of error will aid in the development of both appropriate training and relevant rating aids.

The Horsfall-Barratt scale and severity estimates of citrus canker

Citrus canker assessment data were used to investigate effects of using the Horsfall-Barratt (H-B) scale to estimate disease compared to direct estimation to the nearest percent. Twenty-eight raters assessed each of two-hundred infected leaves (0–38% true diseased area). The data were converted to the H-B scale. Correlation (r) showed that direct estimates had higher inter-rater reliability compared to H-B scaled data (r = 0.75 and 0.71 for direct estimates and H-B scaled data, respectively). Lin’s concordance correlation (LCC, ρ c ) analysis showed individual rater estimates by direct estimation had better agreement with true values compared to H-B scaled data. The direct estimates were more precise compared to H-B scaled data (r = 0.80–0.95 and 0.61–0.90, respectively), but measures of generalised bias or accuracy (C b ) were similar for both methods (0.38–1.00). Cumulative mean disease and cumulative variance of the means were calculated for each rater on a leaf-by-leaf basis. Direct estimates were closer to the true severity 59.5% of the time, and to the cumulative true sample mean 53.7% of the time, and to the cumulative true sample mean variance 63.6% of the time. Estimates of mean severity for each leaf based on estimates by 3, 5, 10, 20 and 28 raters were compared to true disease severity. LCC showed that rater-means based on more raters had better agreement with true values compared to individual estimates, but H-B scale data were less precise, although with means based on ≥ 10 raters, agreement was the same for both assessment methods. Magnitude and dispersion of the variance of the means based on H-B scaled data was greater than that by direct estimates. H-B scaling did not improve reliability, accuracy or precision of the estimate of citrus canker severity compared to direct visual estimation.

The effects of treatment compliance on outcome in cognitive-behavioral therapy for panic disorder: quality versus quantity.

Cognitive-behavioral therapy (CBT) is skill based and assumes active patient participation in regard to treatment-related assignments. The effects of patient compliance in CBT outcome studies are equivocal, however, and 1 gap in the literature concerns the need to account for the quality versus the quantity of assigned work. In this study, both quality and quantity of home-based practice were assessed to better evaluate the effects of treatment compliance in patients with panic disorder (N = 48) who participated in a 12-session CBT protocol. Patient estimates of compliance were not significantly associated with most outcome measures. On the other hand, therapist ratings of compliance significantly predicted positive changes on most outcome measures. Moreover, therapist and independent rater estimates of the quality of the participant's work, relative to the quantity of the work, were relatively better predictors of outcome.

Is the clinical concept of spinal stiffness multidimensional?

This study investigated whether the poor reliability of judgments of posteroanterior (PA) spinal stiffness is due to rater bias or is a consequence of raters each having individual concepts of PA stiffness. Three pairs of manipulative physical therapists with a minimum of 5 years of experience took part in the study. The raters were required to make stiffness judgments of a series of metal springs, and their performance at this task was compared with that obtained when they rated the PA stiffness of patients with low back pain. A range of reliability indices were calculated and evaluated to establish whether rater bias contributed to poor reliability in either task. The relationship between each rater's estimates of the magnitude of the stimuli and the measured stiffness of the springs was also assessed using the Pearson Product-Moment Correlation Coefficient. The average intraclass correlation coefficient (2, 1) for rating spring stiffness was found to be .60, whereas for human spines it was found to be .19. There was no evidence of rater bias contributing to poor reliability for rating stiffness of human spines. The average correlation between the rater's estimates of the magnitude of the stimuli and the measured stiffness of the stimuli was .80. Physical therapists demonstrated much better ability to judge spring stiffness than the PA stiffness of human spines. This difference in performance implies that mechanical stiffness is not equivalent to the clinical concept of PA stiffness. Posteroanterior stiffness may have more than one dimension, and individual interpretation of stiffness as a construct may lead to rater disagreement in the clinic. The reliability of judgments of PA spinal stiffness may be enhanced in the future if its dimensions can be identified, defined, and taken into account during clinical procedures.

Correspondence of teacher ratings and direct observations of classroom behavior of psychiatric inpatient children.

The present investigation examined the correspondence of teacher ratings and direct observations of classroom behavior. Teachers, extraclass raters, and observers completed standard rating scales and/or measures of overt classroom behaviors of psychiatric inpatient children (N = 32). The study assessed if the correspondence between ratings and direct observations was influenced by who evaluates the child (teachers, raters) and the assessment format (general ratings, discrete behaviors). The results indicated that (1) measures from different assessors correlated in the low to moderate range, (2) data from extraclass raters corresponded more closely with direct observations than with data from teachers, (3) teacher and rater estimates of overt child behavior did not correlate more highly with direct observations than did standard rating scales, and (4) teachers and raters viewed child behavior as more appropriate than direct observations indicated. Measures from teachers, raters, and observers readily distinguished attention deficit disorder children with hyperactivity from their peers. However, teacher evaluations delineated these children more sharply than other assessors.