Within-observer Variability Research Articles

Color perception and compensation in color deficiencies assessed with hue scaling

Anomalous trichromats have three classes of cone receptors but with smaller separation in the spectral sensitivities of their longer-wave (L or M) cones compared to normal trichromats. As a result, the differences in the responses of the longer-wave cones are smaller, resulting in a weaker input to opponent mechanisms that compare the LvsM responses. Despite this, previous studies have found that their color percepts are more similar to normal trichromats than the smaller LvsM differences predict, suggesting that post-receptoral processes might amplify their responses to compensate for the weaker opponent inputs. We evaluated the degree and form of compensation using a hue-scaling task, in which the appearance of different hues is described by the perceived proportions of red-green or blue-yellow primary colors. The scaling functions were modeled to estimate the relative salience of the red-green to blue-yellow components. The red-green amplitudes of the 10 anomalous observers were 1.5 times weaker than for a group of 26 normal controls. However, their relative sensitivity at threshold for detecting LvsM chromatic contrast was on average 6 times higher, consistent with a 4-fold gain in the suprathreshold hue-scaling responses. Within-observer variability in the settings was similar for the two groups, suggesting that the suprathreshold gain did not similarly amplify the noise, at least for the dimension of hue. While the compensation was pronounced it was nevertheless partial, and anomalous observers differed systematically from the controls in the shapes of the hue-scaling functions and the corresponding loci of their color categories. Factor analyses further revealed different patterns of individual differences between the groups. We discuss the implications of these results for understanding both the processes of compensation for a color deficiency and the limits of these processes.

Cystic Fibrosis Specific Computed Tomography Scoring

This article presents a review and discussion of the current knowledge regarding cystic fibrosis (CF)-specific scoring of chest computed tomography (CT) scans. First, the basic principles of CT scoring systems in CF are described. Second, between- and within-observer variability of a composite CT score and of component CT scores are reviewed, and issues regarding training of CT scan readers discussed. Third, arguments regarding whether CT scoring systems are ready to be used in clinical studies as a surrogate endpoint are summarized. The between- and within-observer variability of the present CT composite scoring systems is low enough to be useful for clinical studies, although the variability for some of the component scores is larger than for others. Scoring systems fulfill the requirements for surrogate endpoints for CF lung disease, but this role could be further strengthened by including CT scans in large trials and demonstrating the correlation with true endpoints. The conclusion presented is that, given the experience of the variety of published scoring systems, it is important to develop a consensus CT scoring system for future studies in CF. Such a scoring system should evaluate all lung lobes individually and include all relevant CT findings in CF. Development of reference images for the components of this system will be important in reducing the variability between observers and to train new readers.

Inappropriate left ventricular mass: reliability and limitations of echocardiographic measurement for risk stratification and follow-up in single patients

The appropriateness of left ventricular (LV) mass to cardiac workload may be calculated by the ratio of observed LV mass to the value predicted for an individual's sex, height, and stroke work at rest. To investigate test-retest reproducibility of observed/predicted LV mass in a single patient. We measured intraclass correlation and interval of agreement between two M-mode tracings, recorded both at the same session and 3-10 days apart in 200 participants (45 +/- 13 years, body mass index 25 +/- 4 kg/m; 99 hypertensive and 101 normotensive; 50% of each group were women) in 16 centres in Italy. Tracings were read by two observers in each centre. The values of observed/predicted LV mass value ranged from 40.78 to 215.50%. On the same day, the within-observer 90% interval of agreement between tracings 1 and 2 was -23 to +20%. For day-to-day test-retest within-observer variability (average three cycles), the 90% interval of agreement of the observed/predicted LV mass was -30 to +32%. The 90% interval of agreement of test-retest between-observer variability was -25 to +25%. The categorical consistency of retest in the identification of subjects with appropriate LV mass, classified in the first study (i.e. > 73% and < 128%), was 90% (k = 0.87). Measurement of the appropriateness of LV mass in single patients allows acceptable risk stratification, with a coefficient of consistency similar to that reported for LV mass. The probability of a true change (90% chance) in observed/predicted LV mass over time is maximized for a single-reader difference greater than 22%, although lesser differences might also have clinical relevance.

EFFECT OF OBSERVER VARIABILITY ON GLOMERULAR FILTRATION RATE MEASUREMENT BY RENAL SCINTIGRAPHY IN DOGS

Observer variation in kidney depth measurement for correction of soft-tissue attenuation and kidney region of interest (ROI) drawing was evaluated using 60 clinical dogs with a wide range of glomerular filtration rate (GFR) for their effect on the calculated percentage uptake of 99mTc-diethylenetriamine pentaacetic acid (DTPA) and individual kidney GFR by scintigraphy. Kidney depth was measured separately on the lateral image using two color tables: a threshold and a continuous red-green-blue. Within-observer variability of the semi-automatic ROI drawing of the estimated total GFR was up to 10% for the right kidney (RK) and 9% for the left kidney (LK). The variability was lower between observers, 6% for RK and 8% for LK. Manual ROI drawing caused more within observer variation than semi-automatic: up to 14% for RK and 11% for LK. Continuous red-green-blue table caused more variation within and between observers than threshold table. Average within-observer variability from both observers of kidney depth measurement on different color tables could vary up to 5.5% and 6.5% variation of the GFR of RK and LK, respectively. Most variation affecting the DTPA percentage uptake came from the ROI drawing technique. Variations of the method because of the effects of both kidney depth and kidney ROI drawing were up to 8% and 10% for RK and LK, respectively. To minimize these variations a threshold scale should be used for the kidney depth measurement and an automatic or semi-automatic ROI should be used whenever possible. In sequential examinations the same person should make all the measurements.

Observer reliability of industrial activity analysis based on video recordings

The aim of this study was to assess the agreement between observers analyzing activity patterns during truck engine assembly work based on video recordings. Two observers observed the recordings of nine workers, on the average 2.2 h long, assigning activities to four activity categories. For each activity category data were obtained on the mean duration of uninterrupted sequences of activities and their relative time proportion in the job. This data was analyzed with 2-way crossed ANOVA algorithms to derive the components of variance attributed to disagreement between observers, to differences between filmed subjects, and to residual “unexplained” variance. The latter was interpreted as an estimate of within-observer variability and possible interactions between subject and observer. While the observers disagreed about the overall time proportions for the four activity categories by no more than 3.7% of time, their second-to-second classification disagreed for 13% of the total analysis time. The between-observer variance was small as compared to within-observer variance and the variance between subjects performing the same job. Simulations based on the variance components showed that a group mean of the proportion of direct work could be determined with a standard deviation within 5% of the mean by having two observers analyzing one 2-h video recording once, each. Relevance to industry The results of this study may support decision making when designing a reliable video-based analysis of industrial work. Thus, the study helps production engineers, ergonomics practitioners and researchers allocate resources between data collection and data analysis, based on their preferences for precision and power of a particular study.

Estimating cover of benthic organisms from underwater video images: variability associated with multiple observers

Using trained observers and video images of reef transects from many parts of the Great Barrier Reef, we investigated (1) accuracy of classification of benthos and (2) variability contributed by observers to the precision of estimates of benthic cover obtained from video tapes. In order to esti- mate accuracy of identification, benthic organisms were identified twice, first in the field and later from video images. These identifications were then compared. The effect of observer error on preci- sion of benthic cover estimates was examined by having 2 observers sample the same video images on 3 separate occasions. These estimates were then compared at the level of different benthic groups (hard coral, soft coral and algae) and for different hierarchical levels of classification of hard corals (life form, family, genus and species). 'Benthic groups' (mean accuracy of 90 ± 8%) and 'families of hard coral' (91 ± 7%) were identified most accurately and least variably from video images, although many genera and some distinctive species were also identified reliably. Life forms of hard corals proved to be the least accurate and most variable level of classification, with a mean accuracy rating of 74 ± 16%. There was little additional variation in estimates of cover when 2 trained observers sampled images, compared with variation in estimates of cover from repeated samples of images by a single observer. At 10% cover, variability in estimates made by a single observer resulted in mean CIs of 7.9 to 12.1%. Inclusion of variation between observers expanded CIs by only ±0.22%. Furthermore, total observer error was small relative to estimates of cover. For example, at 30% cover, the mean CI due to both between- and within-observer variability was 27.2 to 32.8%.

Reliability and limitations of echocardiographic measurement of left ventricular mass for risk stratification and follow-up in single patients

To investigate the clinical reliability of repeated measurements of left ventricular mass in a single patient. We used test-retest reliability analysis, within-class correlation and interval of agreement measures. Two M-mode tracings (three consecutive cycles) were recorded in the same session and 3-10 days apart (5+/-2 days; mean +/- SD) in 261 participants (age 45+/-13 years, body mass index 24.7+/-3.6 kg/m2; 131 hypertensive and 130 normotensive; 50% of each group women) in 16 centres in Italy. The two tracings were read by two observers in each centre, after classification by a three-order quality score (1 = poor, 2 = sufficient, 3 = optimal). The average quality score was 2.11+/-0.71 (21% poor, 50% sufficient, 29% optimal). Left ventricular mass values ranged from 56 to 419 g (170+/-61 g). On the same day, within-observer 90% interval of agreement between tracing 1 and tracing 2 was -28 to +22 g (-17 to +11% of tracing 1). For day-to-day test-retest within-observer variability (average three cycles), the 90% interval of agreement was -30 to +35 g (-18 to +18%). This variability decreased to -13 to +12% at the 80% interval of agreement and -12 to +11% at the 75% interval of agreement. The 90% interval of agreement of test- retest between-observer variability was -26 to 30 g (-19 to +15%). A negligible regression toward the mean was identified. Categorical consistency of retest in the identification of hypertensive patients with left ventricular hypertrophy, classified in the first study, was 87% (k = 0.87). Measurement of left ventricular mass in single patients allows reliable risk stratification on the basis of the presence of left ventricular hypertrophy. The probability of a true change in left ventricular mass over time is maximized for a single-reader difference greater than 18% of the initial value, although differences of 10-13% might also have clinical relevance.

Suprasternal Doppler ultrasound for assessment of stroke distance

An assessment of a non-invasive technique for measurement of stroke distance was made using a portable Doppler ultrasound machine. The aim was to determine the measurement error of repeated stroke...

Prevalence and control of arterial hypertension in the south-east of Spain: a radical but still insufficient improvement.

Stroke mortality rates in Spain are one of the highest in all of Europe. At the same time, the Murcia region (south-east Spain) shows, for both genders, the highest age-adjusted stroke mortality rates in all of Spain. The earliest available hypertension figure estimations for this area go back to 1981, when a high prevalence combined with an almost nonexistent control was detected. One decade later, updated prevalence estimations of hypertension are presented jointly with their degree of control and their association with other risk factors based on the results of a prevalence survey in a random population sample (n = 3,091). Arterial blood pressure was measured following the MONICA protocol, maintaining a tight quality control on between and within-observer variability. As hypertensive was considered any person with systolic blood pressure (SBP) > or = 140 mmHg or with diastolic blood pressure (DBP) > or = 90 mmHg or with antihypertensive pharmacological treatment. Detected prevalence rises to 32.3% (CI 95%: 29.1%-35.5%) among males and to 23.7% (CI 95%: 21.4%-26%) among females, maintaining its level regarding figures observed back in 1981. However, its control has been increased, especially among hypertensive women [from less than 5% in 1981 to 35% (95% CI: 32%-37.8%) at present]. Hypertension is strongly associated to hipercholesterolemia, overweight, obesity and diabetes (p < 0.01). On the contrary, it shows an opposite association with current smoking, higher educational level and leisure time physical activity (p < 0.01). The highest educational level was associated with better hypertension treatment and control. In summary, while hypertension prevalence is stabilized in our population, its control has improved in a measurable but still insufficient way. These results are in accordance with a decreasing trend in stroke mortality registered in the Murcia Region along the last decade.

Quality control in the measurement of blood pressure

A method for evaluating within-observer variability when measuring blood pressure was tested. Six observers recorded blood pressure levels from a video including several duplicate readings. Within-observer variability was assessed by calculating the differences between the readings of each observer and those which were accepted as correct. This value was contrasted with the maximum acceptable deviation based on a repeated measurement ANOVA with a grouping factor (observer) and a within-group repetition factor obtained by 2 observers. Three follow-up evaluations were performed. The deviation in systolic blood pressure was maintained below the accepted limit from the beginning, while diastollc blood pressure deviation in all the observers was reater than the acceptable limit in the first evaluation, but decreased thereafter.

Reproducibility of ultrasonic measurement of fetal cardiac haemodynamics

Two observers made independent estimates of volume flow across all four heart valves in 27 fetuses using a combination of cross-sectional and Doppler echocardiography. There were considerable discrepancies between observers in volume flow estimation arising from inconsistency both in cross-sectional and in Doppler measurements. One observer also produced a second estimate by making new measurements from the original recordings. There was considerable within-observer variability which was mainly due to errors in cross-sectional measurements. The problems of applying this technique to fetal cardiac haemodynamics are discussed.

Measurement of intima-media thickness of common carotid arteries with high-resolution B-mode ultrasonography: Inter- and intra-observer variability

High-resolution B-mode ultrasonography enables quantitative measurement of the thickness of the intima-media layer of superficial large arteries noninvasively. We investigated the inter- and intra-observer variability of this measurement in the common carotid arteries in 10 randomly selected men. The maximal right and left carotid intima-media thickness (IMT) was measured with calipers during the scanning from frozen images by four observers in a blinded fashion. Three observers also repeated the scanning and the measurements twice with a week's intervals, with no knowledge of the previous readings. The inter-observer coefficient of variation (CV) was 10.5%. The intra-observer CV (mean of right and left CCA) was 5.4–5.8% for the three observers who carried out the measurements three times. The mean absolute difference between the first and third measurement was 0.087 mm. The intra-observer variation accounted for only 4% of the total measurement variability, 96% being attributable to inter-observer variation. These data show that most of the measurement variability in ultrasonographic B-mode IMT measurements is due to differences between observers, whereas the within-observer variability over time appears proportionately very small.

Excerpts from a letter by C. Tsokos to D. Perkel

Anomalous trichromats have three classes of cone receptors but with smaller separation in the spectral sensitivities of their longer-wave (L or M) cones compared to normal trichromats. As a result, the differences in the responses of the longer-wave cones are smaller, resulting in a weaker input to opponent mechanisms that compare the LvsM responses. Despite this, previous studies have found that their color percepts are more similar to normal trichromats than the smaller LvsM differences predict, suggesting that post-receptoral processes might amplify their responses to compensate for the weaker opponent inputs. We evaluated the degree and form of compensation using a hue-scaling task, in which the appearance of different hues is described by the perceived proportions of red-green or blue-yellow primary colors. The scaling functions were modeled to estimate the relative salience of the red-green to blue-yellow components. The red-green amplitudes of the 10 anomalous observers were 1.5 times weaker than for a group of 26 normal controls. However, their relative sensitivity at threshold for detecting LvsM chromatic contrast was on average 6 times higher, consistent with a 4-fold gain in the suprathreshold hue-scaling responses. Within-observer variability in the settings was similar for the two groups, suggesting that the suprathreshold gain did not similarly amplify the noise, at least for the dimension of hue. While the compensation was pronounced it was nevertheless partial, and anomalous observers differed systematically from the controls in the shapes of the hue-scaling functions and the corresponding loci of their color categories. Factor analyses further revealed different patterns of individual differences between the groups. We discuss the implications of these results for understanding both the processes of compensation for a color deficiency and the limits of these processes.