Mean Percentage Agreement Research Articles

Quantification of intramural calcification in coronary intravascular ultrasound images with automated image analysis

Background Recent studies have documented the utility of intravascular ultrasonography in quantifying coronary morphologic characteristics and determining an appropriate intervention. Unfortunately, its potential for quantifying lesion calcification is limited by subjective evaluation and manual tracing. The aim of this study was to develop an objective automated method for quantifying calcification in intracoronary images with digital image analysis. Methods Images of human coronary arteries acquired with a 30 MHz intracoronary ultrasound catheter were evaluated with digital image analysis and compared with manual tracings. Calcifications were automatically identified as highly echogenic regions detected by global thresholding within sectors of acoustic shadowing defined as regions devoid of texture. Results The mean percentage agreement, sensitivity, and specificity of detecting calcification in 1-degree sectors of calcified vessels were 82%, 73%, and 87%, respectively. Similar results were obtained in noncalcified images. Conclusion The accuracy of this automated technique was comparable to interoperator and intraoperator variability in manually tracing calcification. (Am Heart J 1998;136:78-86.)

Pediatric nurses' use of behaviors to make medication administration decisions in infants recovering from surgery.

Although behavioral observation is recommended as the primary pain assessment for the nonverbal postsurgical child, little is known about clinicians' use of observation in their medication administration decisions. Eight infants were videotaped after surgery and segments of the videotapes were categorized as medication inactive or medication active (assumed to relieve pain) based on the usual duration of infants' analgesics. Nurses (N=50) viewed these segments and mean percent agreement with the pharmacologic categorization was 54%. Agreement was high for medication active segments and low for medication inactive ones. Nurses reported using the pain behaviors described in the literature as well as other infant characteristics in their decision making. Infant behaviors observed in the medication inactive snippets were not suggestive enough of the presence of pain to result in the nurses choosing to medicate.

Assessing Metered-Dose Inhaler Technique: Comparison of Observation vs. Patient Self-Report

The purpose of this study was to develop and determine the validity of a patient-completed questionnaire that assesses metered-dose inhaler (MDI) technique. Self-reported MDI technique was compared to observed technique. The questionnaire included nine steps for MDI use, with two to three response choices for each step. A total of 159 patients were studied. Direct observation revealed that greater than 82% of patients exhibited inadequate technique (more than two out of nine steps incorrect). The mean percentage agreement between the questionnaire and observation was 77.4%. Questionnaire results where in higher agreement with observation when patients performed correct MDI technique.

Teachers' judgments of students' academic functioning: A comparison of actual and predicted performances.

The relationship between teachers' judgments of students' academic achievement and students' performances on an achievement test was investigated. The major questions examined were: (a) How accurate are teachers' ratings of students' academic achievement? (b) What is the relationship between teachers' judgments, as operationalized by item predictions on a standardized achievement test, and students' actual performances on an achievement test? (c) Does a student's educational status, high-achieving versus low-achieving, affect the accuracy of teachers' judgments? The study included 12 teachers who volunteered to participate, and 47 students randomly selected from Wisconsin public schools. The students consisted of 30 females and 17 males, ranging from first- to fourthgrade levels. Teachers filled out the Academic Competence Scale from the Social Skills Rating System-Teacher version (Gresham & Elliott, 1990) and one questionnaire for each student. The questionnaire, developed by the authors of the current study, required teachers to predict how students would do on each item of the Kaufman Test of Educational Achievement, Brief form (K-TEA; Kaufman & Kaufman, 1985). Students were then administered the K-TEA by a qualified examiner. Teachers' judgments of students' academic achievement on the Academic Competence Scale were correlated moderately high (r = .70) with students' actual K-TEA scores. Furthermore, mean percent agreement between teachers' item predictions and students' actual performances on the K-TEA was moderately high (79%) (Kappa-coefficient of 67%). Lastly, there was partial support for the prediction that teachers were better predictors of higher-achieving than lower-achieving students. Analyses demonstrated a slightly higher percent agreement between teachers and students with above-average K-TEA scores (81 %) than between teachers and students with below-average K-TEA scores (77%). This article concludes with a discussion of the practical implications of these findings for psychologists conducting academic assessments.

Methods for the measurement of consensual beliefs within groups.

Lucian Gideon Conway Ill and Mark Schaller University of British Columbia The study of consensus in groups is fundamental to the understanding of group processes and the psychological experiences of individuals within groups. Measuring consensus in groups is tricky. This article reviews strengths and weaknesses of various methods for measuring the magnitude of consensus between persons on a single target belief. Considered are methods based on mean extremity, percentage agreement, dispersion, correlation, and spatial clustering. Specific advantages, limitations, and interpretational pitfalls are considered for each measure. Despi~ a diversity of personalities, roles, and histories, people often share very similar beliefs. This fact of consensus has fundamental conse- quences on group processes and the psychologi- cal experience of individuals within groups. Indeed, the study of consensual common ground has played an important role in many areas of small group research (Hoyle & Crawford, 1994; see Baron, Kerr, & Miller, 1992, for a review). For instance, examination of consensus is central to inquiry into how groups come to have the particular members that they do (e.g., Gailbreath, Wagner, Moffett, & Hein, 1997). In addition, because small group norms are, by definition, consensually shared beliefs, tests of group norm conformity models (e.g., Prapaves- sis & Carron, 1997) often involve the measure- ment of consensus. The study of consensus in small groups is not merely an academic issue; it has implications for the health and happiness of individuals in real-life groups. It appears that, within peer group networks, there emerge consensual norms governing health-related be- haviors such as binge eating (see Crandall, 1988); consequently, the impact of therapeutic interventions relevant to these behaviors may be enhanced by an understanding of the social

Violence and Weapon Carrying in Music Videos

The positive portrayal of violence and weapon carrying in televised music videos is thought to have a considerable influence on the normative expectations of adolescents about these behaviors. To perform a content analysis of the depictions of violence and weapon carrying in music videos, including 5 genres of music (rock, rap, adult contemporary, rhythm and blues, and country), from 4 television networks and to analyze the degree of sexuality or eroticism portrayed in each video and its association with violence and weapon carrying, as an indicator of the desirability of violent behaviors. Five hundred eighteen videos were recorded during randomly selected days and times of the day from the Music Television, Video Hits One, Black Entertainment Television, and Country Music Television networks. Four female and 4 male observers aged 17 to 24 years were trained to use a standardized content analysis instrument. Interobserver reliability testing resulted in a mean (+/- SD) percentage agreement of 89.25% +/- 7.10% and a mean (+/- SD) kappa of 0.73 +/- 0.20. All videos were observed by rotating 2-person, male-female teams that were required to reach agreement on each behavior that was scored. Music genre and network differences in behaviors were analyzed with chi 2 tests. A higher percentage (22.4%) of Music Television videos portrayed overt violence than Video Hits One (11.8%), Country Music Television (11.8%), and Black Entertainment Television (11.5%) videos (P = .02). Rap (20.4%) had the highest portrayal of violence, followed by rock (19.8%), country (10.8%), adult contemporary (9.7%), and rhythm and blues (5.9%) (P = .006). Weapon carrying was higher on Music Television (25.0%) than on Black Entertainment Television (11.5%), Video Hits One (8.4%), and Country Music Television (6.9%) (P < .001). Weapon carrying was also higher in rock (19.8%) and rap (19.5%) videos than in adult contemporary (16.1%), rhythm and blues (6.9%), and country (6.3%) videos (P = .002). The videos with the highest level of sexuality or eroticism were found to be less likely to contain violence (P < or = .04). Because most music videos are between 3 and 4 minutes long, these data indicate that even modest levels of viewing may result in substantial exposure to violence and weapon carrying, which is glamorized by music artists, actors, and actresses.

46. The Accuracy of Medical Records and Police Reports in Determining Motor Vehicle Crash Characteristics

Objective: To determine the accuracy of ambulance (AR), emergency department (EDR) and police (PR) records in describing motor vehicle crash (MVC) characteristics when compared to a standard - an in-depth motor vehicle crash investigation record (CIR).Methods: Fifty-six MVC patients transported to a suburban, university hospital emergency department via ambulance and included in a crash investigation were identified. The time period was January 1993 through December 1995. Data sources were the AR, EDR, PR and CIR. The CIR was abstracted to a standard form. The other data sources were abstracted to a standard form using a retrospective, blinded review. Variables included occupant position, restraint use, air bag deployment, type of impact, ejection, and external cause of injury code. Accuracy was measured by determining percent agreement and Kappa for each data source compared to the CIR.Results: Forty-six cases (82%) had one or more episodes of discordance. The mean percent agreement for the sources studied was AR 0.813, EDR 0.893 and PR 0.932. Mean values for Kappa were AR 0.712, EDR 0.831 and PR 0.885. Among variables, restraint use was determined with the least accuracy with means for percent agreement and Kappa of 0.792 and 0.674, respectively. For the AR, discordance was due to missing information almost 50% of the time.

Occupational Therapists' Agreement in Screening Patients in General Psychiatric Care for Occupational Therapy

The screening of psychiatric patients for occupational therapy service has traditionally relied on clinical experience, and the rational of the screening is seldom documented. The present study examines agreement among occupational therapists in judging need for occupational therapy for these patients on the basis of videotaped interviews. Occupational therapists (n = 38) working in general psychiatric wards saw videotaped interviews of 10 patients. Two different interview methods were used: a traditional occupational therapy interview and a semi-structured interview. After viewing each interview the occupational therapists had to decide whether to offer the patient occupational therapy treatment or not, or if he/she wanted more assessment information before making a decision. The agreement was high; the mean percentage agreement level of whether the occupational therapists should include or exlude patients from service was 81%. However, most of the occupational therapists wanted to offer occupational the...

What teachers believe

Summary A total of 145 statements about teaching were collected from a variety of sources. They were assembled into seven questionnaires, which were administered to the academic staff of Chichester Institute of Higher Education: 126 out of the 145 statements (87 per cent) were agreed with by more than half of those responding to them; and 39 statements were agreed with by 100 per cent of those responding to them. These 39 statements with which none of the respondents had disagreed were assembled into a single questionnaire. This questionnaire was sent not only to the same list of 129 dons at Chichester, but also to 46 teachers at Carshalton High School for Boys. The total number of respondents to the second survey was 109 of 175 recipients (63 per cent). The lowest overall percentage agreement with any single statement was 73 per cent, the highest 98 per cent. The mean percentage agreement was 91 per cent among respondents at both Chichester and Carshalton High School. The authors conclude that the result...

Development of a physical performance and mobility examination.

To develop and validate the Physical Performance and Mobility Examination (PPME), an observer-administered, performance-based instrument assessing 6 domains of physical functioning and mobility for hospitalized elderly. Development of a pass-fail and 3-level scoring system and training manuals for the PPME instrument for use in both clinical and research settings. Two patient samples were used to assess construct validity and interrater reliability of the PPME. A third sample was selected to assess the test-retest reliability of the instrument. (1) 146 subjects > or = 65 years of age with impaired mobility admitted to Medical Units of Stanford University Hospital. (2) 352 subjects > or = 65 admitted to acute Medical and Surgical Services of the Palo Alto VA Medical Center. Patient samples were obtained during hospitalization and followed until 3 months post-discharge. To study test-retest reliability, 50 additional patients, whose clinical condition was stable, were selected from both settings. An expert panel selected 6 mobility tasks integral to daily life: bed mobility, transfer skills, multiple stands from chair, standing balance, step-up, and ambulation. Tasks were piloted with frail hospitalized subjects for appropriateness and safety. Test-retest and interrater reliability and construct validity were evaluated. Construct validity was tested using the Folstein Mini-Mental State Examination, Activities of Daily Living (ADL), Instrumental Activities of Daily Living (IADL), Geriatric Depression Scale, and modified Medical Outcomes Study Measure of Physical Functioning (MOS-PFR). Two scoring schema were developed for each task: (1) dichotomous pass-fail and (2) 3-level high pass, low pass, and fail. A summary scale was developed for each method of scoring. High interrater reliability and intrarater reliability were demonstrated for individual tasks. The mean percent agreement (interrater) for each pass/fail task ranged from 96 to 100% and from 90 to 100% for the 3 pairs of raters for each task using the 3-level scoring. Kappas for individual pairs of raters ranged from .80 to 1.0 for pass-fail scoring and from .75 to 1.0 for 3-level scoring (all P < 0.01). Intraclass correlation coefficients for 3-level scoring by pairs of raters ranged from .66 to 1.0. For summary scales, the mean intraclass correlation was .99 for both scoring schema. Test-retest reliability for summary scales using kappa coefficients was .99 for both pass-fail and 3-level scoring, and .99 and .98, respectively, using Pearson Product Moment Correlation. Correlations of PPME with other instruments (construct validity) suggest that the PPME adds a unique dimension of mobility beyond that measured by self-reported ADLS and physical functioning, and it is not greatly influenced by mood or mental status (r = 0.70 (ADL), r = 0.43 (IADL), r = 0.36 (MMSE), r = 0.71 (MOS-PFR), r = 0.23 (GDS)). The 3-level summary scale was sensitive to the variability in the patient population and exhibited neither ceiling nor floor effects. The PPME is a reliable and valid performance-based instrument measuring physical functioning and mobility in hospitalized and frail elderly.

Use of uniform terminology by occupational therapists.

Uniform Terminology for Occupational Therapy-Second Edition (American Occupational Therapy Association, 1989b) was published to foster consistency in terminology among occupational therapists. In this study, agreement between occupational therapists and Uniform Terminology about the definition and categorization of 15 terms was examined. Agreement was measured on 113 responses from a randomly selected sample of 180 occupational therapists. Low levels of agreement were found on both definition of terms (71.9 mean percentage of agreement, SD = 12.4) and categorization of terms into larger conceptual categories (34.9 mean percentage of agreement, SD = 11.7). Therapists selected an alternative definition more often than the one given by Uniform Terminology on 2 of 15 terms and selected an alternative categorization more often than the one given in Uniform Terminology on 8 of 15 terms. These results indicate that therapists do not agree with many Uniform Terminology definitions and categorizations.

THE SELF‐INJURY TRAUMA (SIT) SCALE: A METHOD FOR QUANTIFYING SURFACE TISSUE DAMAGE CAUSED BY SELF‐INJURIOUS BEHAVIOR

A method is described for classifying and quantifying surface tissue damage caused by self-injurious behavior. The Self-Injury Trauma Scale permits differentiation of self-injurious behavior according to topography, location of the injury on the body, type of injury, number of injuries, and estimate of severity. Fifty pairs of independently scored records were subjected to interrater reliability analyses, and the following mean (median) percentage agreement scores were obtained: overall agreement, 97% (98%); location of injury, 99% (100%); type of injury, 96% (100%); number of injuries, 89% (100%); and severity of injury, 94% (100%). Percentage agreement also was calculated for three summary scores: Number Index, 90%; Severity Index, 92%; and Estimate of Current Risk, 100%. Potential applications and limitations of the scale are discussed.

Assessment of observer variation in measuring the radiographic vertebral index in patients with multiple myeloma

The Radiographic Vertebral Index (RVI) was assessed as a possible outcome measure for bone disease in myeloma by evaluating within and between reader reproducibility. Four readers (2 radiologists and 2 clinical hematologists) independently scored, on two separate occasions, the RVI on 40 radiographs from patients with myeloma. Each vertebra from third thoracic (T3) to fifth lumbar (L5) received a score of “1” if normal, “2” if biconcave and “4” if crushed or fractured. RVI global scores, therefore, could vary from a minimum of 15, for no damage, to a potential maximum of 60 in which all vertebrae are collapsed. Agreement was determined for each vertebra using crude percentage agreement and the kappa statistic (which corrects for chance-expected agreement) for categorical data, and for global score using analysis of variance and calculating intra-class correlation. With increasing mean abnormality score on individual vertebrae there was a corresponding increase in kappa and reduction in crude percentage agreement. Within readers, the mean percentage agreement across all vertebrae varied from 85.6 to 90.3% and the observed differences just reach statistical significance ( p = 0.048). Mean kappa values ranged from 0.48 to 0.63 and were similar across readers. Differences in intra-reader agreement were not related to subspecialty. When between reader percentage agreement and kappa scores were assessed for radiologists and non-radiologist clinicians, no difference could be detected. Agreement with respect to intra-reader mean global RVI scores was excellent as illustrated by the intra-class correlation coefficient which varied from 0.89 to 0.94. The mean intra-class correlation for radiologists was 0.92, compared with 0.91 for non-radiologists. Inter-reader agreement for global RVI score was assessed by a repeated measures analysis of variance of mean global scores and a highly significant difference ( p < 0.001) was detected among the four readers. This indicates an important extra component of variance between readers over and above the variation within readers. The data show reasonably good reproducibility of the RVI within readers and suggests that sequential RVI assessments on individual patients should be performed by a single observer. The RVI may be a useful outcome measure for evaluating bone disease in myeloma, but needs to be validated prospectively in the setting of a clinical trial.

Validity of the Medical Record for Evaluation of Telephone Management

The purpose of this study was to determine whether the medical record provides accurate documentation of the telephone encounter. The study was a "blinded" comparison of audiotaped telephone encounters and corresponding medical records; it was carried out in the continuity clinic of a primary care pediatric resident-training program. During their first month of rotation through the outpatient department, 17 PL-I residents received one or two calls made by a "simulated mother" using standardized scripts. Transcripts of these calls and the corresponding written medical records were rated by an examiner unaware of the resident's identity. A standardized instrument was used to measure three aspects of performance: General History Taking, Specific History Taking, and General Management. A percentage of agreement was calculated for each scale, and rating scores of audiotapes and written records were compared. Twenty-seven pairs of audiotape and written documentation were analyzed. The mean percentage of agreement between audiotapes and written records was high: 78% for General History Taking, 78% for General Management, 77% for Specific History Taking. Rating scores of the audiotape and the medical record were significantly correlated for two of the scales: General Management (r = .55, P less than .01) and Specific History Taking (r = .50, P less than .01). Most aspects of the telephone encounter were well documented in the medical record. However, several discrepancies were noted when audiotapes and medical records were compared for the presence of specific items. With the exception of the Specific History Taking scale, there was no correlation between the length of the written record and the percentage of agreement.(ABSTRACT TRUNCATED AT 250 WORDS)

Patient-Spouse Agreement on the Drinking Behaviors of Alcoholics

Because investigators have expressed concern about the validity of alcoholism self-report measures, attention has been focused on the use of spouse or other collateral ratings of patient drinking behaviors. Previous studies have shown that spouse ratings of alcoholics on the Self-Administered Alcoholism Screening Test (SAAST) were reliable and a valid means of screening for alcoholism. In the current study, our objectives were to attempt to replicate the earlier findings in a new and larger sample and to determine whether the various content dimensions of the SAAST show similar patient-spouse agreement. The SAAST results of 240 patient-collateral pairs showed that the mean percentage agreement over the 35 items of the SAAST was 76.5%. Patient-spouse agreement ranged from 84.7% on the component "family alcohol problems" to 68.5% on "loss of control." When patients were classified as alcoholic or nonalcoholic on the basis of spouse ratings on the SAAST, correct classification occurred 98% of the time. Correct conjoint classification by both patient and spouse occurred 95.4% of the time. Thus, we found a high level of agreement between patient and spouse on the SAAST ratings and confirmed that spouse SAAST ratings of drinking behavior of patients provide a reliable diagnosis of alcoholism.

Improving patient-education assessment skills of hospital staff: A case study in diabetes

While the importance of collecting patient education data of high quality is well established, few studies have examined the quality of patient educational assessment procedures. Utilizing a standardized instrument and assessment protocol, and observational trial of nine insulin-controlled adult diabetics was conducted using pairs of trained patient assessors. Although the overall mean percent agreement for the first trial was very good (90%), lack of agreement was noted for several important subcategories. A training program was conducted to improve accuracy, addressing the assessment problems found in the first trial. A second rater trial found an overall level of rater agreement of 99%. This study confirmed that routine assessment of patients should be periodically monitored using the quality control methods described to ensure program data of high quality.

Listener Agreement on Articulation Errors of Four- and Five-Year-Old Children

This study reported differences in agreement among four experienced listeners who analyzed the articulation skills of 97 four- and five-year-old children. Mean agreement percentages based on the identification of only misarticulated sounds were contrasted with mean percentages of agreement for decisions made on both correct and incorrect articulations of each sound for the entire population. In addition, the weighted average of the modal category of agreement for the specification of error type for misarticulated responses was determined for each sound tested over all subjects. Differences in agreement for evaluations of all responses, for error responses only, and for specification of error types were calculated and reported by place and manner of articulation, voicing characteristics, and syllabic function. Place and manner of articulation revealed differences of agreement, whereas voicing and syllabic function contributed little to agreement or disagreement.

A Critique of Methods for Operationalizing the Concept of Fakeability

Past practice of operationalizing the concept of fakeability is reviewed. The strengths and weaknesses of these indices (mean difference, percent overlap, percent agreement, correlation, and variance of difference scores) are discussed in the light of a proposed new definition of fakeability based upon Naylor's (1967) model of measurement accuracy.