Between-observer Differences Research Articles

Validation of a novel semi-automated method for three-dimensional surface rendering of condyles using cone beam computed tomography data

Morphological changes of the condyles are often observed following orthognathic surgery. In addition to clinical assessment, radiographic evaluation of the condyles is required to distinguish the physiological condylar remodelling from pathological condylar resorption. The low contrast resolution and distortion of greyscale values in cone beam computed tomography (CBCT) scans have impeded an accurate three-dimensional (3D) rendering of the condyles. The current study proposes a novel semi-automated method for 3D rendering of condyles using CBCT datasets, and provides a clinical validation of this method. Ten patients were scanned using a standard CBCT scanning protocol. After defining the volume of interest, a greyscale cut-off value was selected to allow an automatic reconstruction of the condylar outline. The condylar contour was further enhanced manually by two independent observers to correct for the under- and over-contoured voxels. Volumetric measurements and surface distance maps of the condyles were computed. The mean within-observer and between-observer differences in condylar volume were 8.62mm3 and 6.13mm3, respectively. The mean discrepancy between intra- and inter-observer distance maps of the condylar surface was 0.22mm and 0.13mm, respectively. This novel method provides a reproducible tool for the 3D rendering of condyles, allowing longitudinal follow-up and quantitative analysis of condylar changes following orthognathic surgery.

Reproducible Analysis of Rat Brain PET Studies Using an Additional [18F]NaF Scan and an MR-Based ROI Template

Background. An important step in the analysis of positron emission tomography (PET) studies of the brain is the definition of regions of interest (ROI). Image coregistration, ROI analysis, and quantification of brain PET data in small animals can be observer dependent. The purpose of this study was to investigate the feasibility of ROI analysis based on a standard MR template and an additional [18F]NaF scan. Methods. [18F]NaF scans of 10 Wistar rats were coregistered with a standard MR template by 3 observers and derived transformation matrices were applied to corresponding [11C]AF150(S) images. Uptake measures were derived for several brain regions delineated using the MR template. Overall agreement between the 3 observers was assessed by interclass correlation coefficients (ICC) of uptake data. In addition, [11C]AF150(S) ROI data were compared with ex vivo biodistribution data. Results. For all brain regions, ICC analysis showed excellent agreement between observers. Reproducibility, estimated by calculation of standard deviation of the between-observer differences, was demonstrated by an average of 17% expressed as coefficient of variation. Uptake of [11C]AF150(S) derived from ROI analysis closely matched ex vivo biodistribution data. Conclusions. The proposed method provides a reproducible and tracer-independent method for ROI analysis of rat brain PET data.

Reliability of indicators of sheep welfare assessed by a group observation method

Assessments of animal-based outcomes form the core of routine veterinary clinical examinations and are being increasingly used as indicators of animal welfare. A method of group observation that did not require gathering and handling of individual sheep, was used to assess eight animal-based indicators of sheep welfare: demeanour, skin irritation, wool loss, excessive panting, coughing, lameness, and cleanliness of the ventral abdominal and 'breech' (perineum/gluteal/caudal hindlimb) areas. The inter-observer reliability of two or three observers who independently assessed these indicators was tested on 2406 adult sheep and growing lambs across 36 farms and the intra-observer reliability of an experienced, veterinary assessor--the 'test standard observer'--was assessed on 88 adult sheep during four on-farm assessments. Observer reliability was evaluated using Cronbach's alpha (α) comparison of the recorded proportion (%) of sheep affected by each welfare condition and by binomial logistic regression modelling. High levels of inter-observer reliability were identified for the assessment of group lameness (α 0.76-1.00) and cleanliness of the breech area (α 0.97-1.00). Excellent intra-observer reliability was determined for lameness (α 0.99), cleanliness of the breech area (α 0.97), demeanour (α 1.00) and wool loss (α 1.00). In addition, proportion data and logistic regression models identified few between-observer differences. The results suggest that welfare outcomes based on observations of the behaviour and physical appearance of individual animals within a group may offer a reliable and feasible measurement tool for the on-farm assessment of sheep welfare.

Reproducibility of Regional Pulse Wave Velocity in Healthy Subjects

Background/AimsDespite the clinical importance and widespread use of pulse wave velocity (PWV), there are no standards for pulse sensors or for system requirements to ensure accurate pulse wave measurement. We assessed the reproducibility of PWV values using a newly developed PWV measurement system.MethodsThe system used in this study was the PP-1000, which simultaneously provides regional PWV values from arteries at four different sites (carotid, femoral, radial, and dorsalis pedis). Seventeen healthy male subjects without any cardiovascular disease participated in this study. Two observers performed two consecutive measurements in the same subject in random order. To evaluate the reproducibility of the system, two sets of analyses (within-observer and between-observer) were performed.ResultsThe means±SD of PWV for the aorta, arm, and leg were 7.0±1.48, 8.43±1.14, and 8.09±0.98 m/s as measured by observer A and 6.76±1.00, 7.97±0.80, and 7.97±0.72 m/s by observer B, respectively. Between-observer differences for the aorta, arm, and leg were 0.14±0.62, 0.18±0.84, and 0.07±0.86 m/s, respectively, and the correlation coefficients were high, especially for aortic PWV (r=0.93). All the measurements showed significant correlation coefficients, ranging from 0.94 to 0.99.ConclusionsThe PWV measurement system used in this study provides accurate analysis results with high reproducibility. It is necessary to provide an accurate algorithm for the detection of additional features such as flow wave, reflection wave, and dicrotic notch from a pulse waveform.

Indices of cardiovascular function derived from peripheral pulse wave analysis using radial applanation tonometry: a measurement repeatability study

Pulse wave analysis (PWA) using applanation tonometry is a non-invasive technique for assessing cardiovascular function. It produces three important indices: ejection duration index (ED%), augmentation index adjusted for heart rate (AIX@75), and subendocardial viability ratio (SEVR%). The aim of this study was to assess within- and between-observer repeatability of these measurements. After resting supine for 15 minutes, 20 ambulant patients (16 male) in sinus rhythm underwent four PWA measurements on a single occasion. Two nurses (A & B) independently and alternately undertook PWA measurements using the same equipment (Omron HEM-757; SphygmoCor with Millar hand-held tonometer) blind to the other nurse's PWA measurements. Within- and between-observer differences were analysed using the Bland-Altman ;limits of agreement' approach (mean difference +/- 2 standard deviations, 2SD). Mean age was 56 (blood pressure, BP 136/79; pulse rate 64). BP/PWA measurements remained stable during assessment. Based on the average of two PWA measurements the mean +/- 2SD between-observer difference in ED% was 0.3 +/- 2.0; AIX@75 1.0 +/- 3.9; and SEVR% 1.7 +/- 14.2. Based on a single PWA measurement the between-observer difference was ED% 0.3 +/- 3.3; AIX@75 1.7 +/- 6.9; and SEVR% 0.6 +/- 22.6. Within-observer differences for nurse-A were ED% 0.0 +/- 5.4; AIX@75 1.5 +/- 7.0; and SEVR% 1.7 +/- 39.0 (nurse-B: 0.1 +/- 3.8; 0.1 +/- 8.0; and 0.6 +/- 23.3, respectively). PWA demonstrates high levels of repeatability even when used by relatively inexperienced staff and has the potential to be included in the routine cardiovascular assessment of ambulant patients.

Between-Observer Differences in Relative Corn Yield vs. Rated Weed Control

Crop yield and weed control rating have been used to measure weed and crop response to weed management treatments, eliminate unacceptable weed management treatments, and select “best” treatments for recommendation to farmers. However, the mathematical relationship between crop yield and rated weed control has not been reported before from such treated screening experiments. Likewise, differences have not been reported before in rated weed control among experienced observers (i.e., reliability) when rating the same experiments and for an experienced observer over time (i.e., repeatability). Data from published experiments on zone herbicide application in field corn in which weeds reduced yield to various amounts were reanalyzed to examine these issues. For this study, relative corn yield was calculated as a percentage of the 1× broadcast herbicide rate for two observers and either three experimental site-years or their average. For observer A, relative corn yield (%) increased linearly as rated total weed control (%) increased for all 3 site-yr and their average. For observer B, equations were curvilinear in 2 of 3 site-yr. For both observers, equations accounted for little data variability in relative corn yield (r2= 0.25 and 0.25 in site-year 1, respectively, 0.38 and 0.36 in site-year 2, 0.58 and 0.57 in site-year 3, and 0.43 and 0.42 for their average). When rated total weed control by observer A was graphed against that of observer B, the relationship was a nearly ideal 1:1 linear relationship in only 1 of 3 site-yr. In two other site-years, equations were nonlinear, indicating that one observer distinguished smaller differences between treatments at lower rated control than the other observer. Between-row total weed cover and in-row total weed height influenced observer weed control rating.

Validation of grading scales for contact lens complications

The validity of use of two artist-rendered and two photographic sets of grading scales (grading 'systems') designed for gauging the severity of contact lens-related ocular pathology was assessed in terms of precision and reliability. Thirteen observers each graded 30 images--by interpolation or extrapolation to the nearest 0.1 increment--of each of the three contact lens complications (corneal staining, conjunctival redness and papillary conjunctivitis) that were common to all four grading systems. This entire procedure was repeated approximately two weeks later, yielding a total data base comprising of 9360 individual grading estimates. Analysis of variance revealed statistically significant differences in both precision and reliability between systems, observers and conditions (p < 0.03 for system reliability; p = 0.0001 for all other combinations). The artist-rendered systems generally afforded lower grading estimates and better grading reliability than the photographic systems. Corneal staining could be graded less reliably than conjunctival redness and papillary conjunctivitis. Grading reliability was generally unaffected by the severity of the condition being assessed. Notwithstanding the above differences, all four grading systems are validated for clinical use and practitioners can initially expect to use these systems with average 95% confidence limits of +/- 1.2 grading scale units (observer range +/- 0.7 to +/- 2.5 grading scale units). In view of the significant between-system differences revealed in this study, it is advisable to consistently use the same grading system. It may be possible to reduce between-observer differences by applying personalised correction factors to normalise grading estimates.

Validation of grading scales for contact lens complications

The validity of use of two artist-rendered and two photographic sets of grading scales (grading ‘systems’) designed for gauging the severity of contact lens-related ocular pathology was assessed in terms of precision and reliability. Thirteen observers each graded 30 images — by interpolation or extrapolation to the nearest 0.1 increment — of each of the three contact lens complications (corneal staining, conjunctival redness and papillary conjunctivitis) that were common to all four grading systems. This entire procedure was repeated approximately two weeks later, yielding a total data base comprising of 9360 individual grading estimates. Analysis of variance revealed statistically significant differences in both precision and reliability between systems, observers and conditions (p<0.03 for system reliability; p=0.0001 for all other combinations). The artist-rendered systems generally afforded lower grading estimates and better grading reliability than the photographic systems. Corneal staining could be graded less reliably than conjunctival redness and papillary conjunctivitis. Grading reliability was generally unaffected by the severity of the condition being assessed. Notwithstanding the above differences, all four grading systems are validated for clinical use and practitioners can initially expect to use these systems with average 95% confidence limits of ±1.2 grading scale units (observer range ±0.7 to ±2.5 grading scale units). In view of the significant between-system differences revealed in this study, it is advisable to consistently use the same grading system. It may be possible to reduce between-observer differences by applying personalised correction factors to normalise grading estimates.

Measuring Total Catch at Sea: Use of a Motion-Compensated Flow Scale to Evaluate Observer Volumetric Methods

We evaluated procedures used by North Pacific groundfish observers to estimate total catch aboard a factory trawler in the Bering Sea during 1997. A motion-compensated flow scale was tested for precision and bias by using known weights of fish. The flow scale performed within an error limit of 3% in daily tests, although a positive bias of +1% was detected. We used flow scale weights to evaluate volumetric methods of haul weight estimation: codend volume and bin volume measurements by observers and bin volume measurements by electronic bin sensors. Codend volume estimates were most variable; slight volume overestimation was detected for codends larger than the trawl alley, and significant between-observer differences were detected. Bin volume estimates were more precise than codend volume estimates and showed no differences between observers. Bin sensor volume estimates agreed closely with observer visual estimates when fish level was at least 1 m below the transducers but gave inaccurate readings when bins were full or nearly full. All volumetric methods depend on the correct volume-to-weight conversion (density) factor. Several statistical approaches to estimating density indicated that the mean density of catch in the fishery for walleye pollock Theragra chalcogramma is 1.02 metric tons (mt)/m3 in codends, while the mean density in bins is 0.98 mt/m3. Evaluation of density estimates made with sampling baskets, the current procedure used in many North Pacific groundfish fisheries, revealed several problems, especially a tendency to underestimate density. Density estimates obtained by using a specially designed density sampler were consistent between observers and similar to the density of catch in bins. We recommend further research to obtain prescribed densities for volumetric haul weight estimation in other groundfish fisheries.

Anthropometric measurement error and the assessment of nutritional status.

Anthropometry involves the external measurement of morphological traits of human beings. It has a widespread and important place in nutritional assessment, and while the literature on anthropometric measurement and its interpretation is enormous, the extent to which measurement error can influence both measurement and interpretation of nutritional status is little considered. In this article, different types of anthropometric measurement error are reviewed, ways of estimating measurement error are critically evaluated, guidelines for acceptable error presented, and ways in which measures of error can be used to improve the interpretation of anthropometric nutritional status discussed. Possible errors are of two sorts; those that are associated with: (1) repeated measures giving the same value (unreliability, imprecision, undependability); and (2) measurements departing from true values (inaccuracy, bias). Imprecision is due largely to observer error, and is the most commonly used measure of anthropometric measurement error. This can be estimated by carrying out repeated anthropometric measures on the same subjects and calculating one or more of the following: technical error of measurement (TEM); percentage TEM, coefficient of reliability (R), and intraclass correlation coefficient. The first three of these measures are mathematically interrelated. Targets for training in anthropometry are at present far from perfect, and further work is needed in developing appropriate protocols for nutritional anthropometry training. Acceptable levels of measurement error are difficult to ascertain because TEM is age dependent, and the value is also related to the anthropometric characteristics of the group of population under investigation. R > 0.95 should be sought where possible, and reference values of maximum acceptable TEM at set levels of R using published data from the combined National Health and Nutrition Examination Surveys I and II (Frisancho, 1990) are given. There is a clear hierarchy in the precision of different nutritional anthropometric measures, with weight and height being most precise. Waist and hip circumference show strong between-observer differences, and should, where possible, be carried out by one observer. Skinfolds can be associated with such large measurement error that interpretation is problematic. Ways are described in which measurement error can be used to assess the probability that differences in anthropometric measures across time within individuals are due to factors other than imprecision. Anthropometry is an important tool for nutritional assessment, and the techniques reported here should allow increased precision of measurement, and improved interpretation of anthropometric data.

Evaluation of the IITC tail cuff blood pressure recorder in the rat against intraarterial pressure according to criteria for human devices

National standards govern the manufacture and marketing of medical devices in the United States, including those for indirect blood pressure measurement in man. There are no comparable standards for devices for recording in laboratory animals. Noninvasive tail cuff blood pressure (BP) recording in the rat is widely accepted, but beset by methodologic difficulties. Intraarterial recording is regarded as the “gold standard” but is invasive and also susceptible to methodologic error. We compared the IITC Mark 12 photoelectric/oscillometric tail cuff system (IITC Life Sciences, Woodland Hills, CA) versus simultaneous femoral intraarterial recordings in spontaneously hypertensive rats, during anesthesia and 1 to 2 days after recovery (150 recordings under each condition), according to the guidelines for human data collection and analysis suggested by the American National Standard for automated sphygmomanometers. Within- and between-observer disagreements in estimates made by two observers from 40 anesthetized recordings were less for intraarterial measurements than for the tail cuff method. Within-observer differences (mean ± SD of differences [SDD]) for systolic, diastolic, and mean pressure were 0 ± 1, 0 ± 1, and 0 ± 1 mm Hg for intraarterial versus −1 ± 3, 0 ± 8, and 0 ± 5 mm Hg for tail cuff. Between-observer differences were 0 ± 2, 0 ± 1, and 6 ± 2 mm Hg versus 5 ± 4, 13 ± 7, and 0 ± 5 mm Hg, respectively. Differences between tail cuff and intraarterial methods were 16 ± 13, −5 ± 11, and 2 ± 8 mm Hg in anesthetized animals and 8 ± 14, −5 ± 9, and 0 ± 9 mm Hg in conscious animals (39% to 82% of differences exceeded 5 mm Hg). The upper limits of clinically acceptable disagreement in the American National Standard are: mean of 5 mm Hg, SDD of 8 mm Hg. The disagreement between tail cuff and intraarterial recordings cannot be ascribed to either method with certainty. These findings do not support the manufacturer's guarantee of tail-cuff readings within “5 mm Hg of intraarterial.” Inaccuracy and unreliability of devices intended for laboratory animal use have considerable scientific, fiscal, and ethical implications. Marketing of these devices should also be governed by rigorous standards.

Observer Differences in the North American Breeding Bird Survey

Because count data collected in many bird surveys are only an index to population size, factors that can influence the counts must be identified and incorporated into analyses. Observer quality is often ignored in analyses of population changes from survey data, but observers differ in methods and capabilities and, hence, tend to count different numbers of birds. We assess the consequences of between-observer differences in counts for estimation of population trends in the North American Breeding Bird Survey. Observer differences in numbers of birds counted were found in 50% of the 369 species we examined. For many species, observers in later years tended to count more birds than observers in earlier years, suggesting an increase in observer quality over time. Analysis of population trends from 1966 through 1991 indicates that failure to include observers as covariables in the analysis results in an overly optimistic view of population trends.

Observer differences in transect counts, cover estimates and plant size measurements on range monitoring sites in an arid shrubland.

A field trial was held to test between-observer differences in recording counts and measurements made at range monitoring sites in grazed shrublands near Carnarvon, Western Australia. Ten pairs of observers worked independently to count perennials by species, t o measure major and minor axes of marked shrubs and to estimate foliar cover on fixed transects within three shrub communities. Following simple conventions, observers identified and counted shrubs by species with coefficients of variation (C.V.) &lt; 10%, except where species identification or inconspicuousness of young plants led to particular difficulty. Errors incurred by a rapid field technique for shrub measurement averaged 5-9% C.V. for the major axes dimensions on a range of shrub forms, while the recording of second axis diameters was subject to two to three times as much error. Estimations of total projected foliar cover varied by 25% C.V. using a wheel point technique and by 30% using intercepts on line transects. Estimations of cover by species were more variable. The significance of these findings in planning appropriate techniques for monitoring arid shrublands is discussed.