Intergroup Agreement Research Articles

Human performance in predicting enhancement quality of gliomas using gadolinium-free MRI sequences.

To develop and test a decision tree for predicting contrast enhancement quality and shape using precontrast magnetic resonance imaging (MRI) sequences in a large adult-type diffuse glioma cohort. Preoperative MRI scans (development/optimization/test sets: n=31/38/303, male=17/22/189, mean age=52/59/56.7years, high-grade glioma=22/33/249) were retrospectively evaluated, including pre- and postcontrast T1-weighted, T2-weighted, fluid-attenuated inversion recovery, and diffusion-weighted imaging sequences. Enhancement prediction decision tree (EPDT) was developed using development and optimization sets, incorporating four imaging features: necrosis, diffusion restriction, T2 inhomogeneity, and nonenhancing tumor margins. EPDT accuracy was assessed on a test set by three raters of variable experience. True enhancement features (gold standard) were evaluated using pre- and postcontrast T1-weighted images. Statistical analysis used confusion matrices, Cohen's/Fleiss' kappa, and Kendall's W. Significance threshold was p<.05. Raters 1, 2, and 3 achieved overall accuracies of.86 (95% confidence interval [CI]:.81-.90),.89 (95% CI:.85-.92), and.92 (95% CI:.89-.95), respectively, in predicting enhancement quality (marked, mild, or no enhancement). Regarding shape, defined as the thickness of enhancing margin (solid, rim, or no enhancement), accuracies were.84 (95% CI:.79-.88),.88 (95% CI:.84-.92), and.89 (95% CI:.85-.92). Intrarater intergroup agreement comparing predicted and true enhancement features consistently reached substantial levels (≥.68 [95% CI:.61-.75]). Interrater comparison showed at least moderate agreement (group: ≥.42 [95% CI:.36-.48], pairwise: ≥.61 [95% CI:.50-.72]). Among the imaging features in the EPDT, necrosis assessment displayed the highest intra- and interrater consistency (≥.80 [95% CI:.73-.88]). The proposed EPDT has high accuracy in predicting enhancement patterns of gliomas irrespective of rater experience.

Agreement of Pain Assessment Using the Short Form of the Canine Glasgow Composite Measure Pain Scale between Veterinary Students, Veterinary Nurses, Veterinary Surgeons, and ECVAA-Diplomates.

Several pain scoring systems have been validated to measure pain in dogs. However, pain may not be adequately assessed since these tools are associated with high-level inter-observer variation. The aim of this study is to evaluate the agreement of pain assessment using the CMPS-SF between veterinary students, veterinary nurses, veterinary surgeons, and European College of Veterinary Anaesthesia and Analgesia (ECVAA) diplomates. Forty-five client-owned dogs presented to a teaching hospital were enrolled in this prospective, observational study. All dogs were pain-scored in vivo, while a video of the assessment was recorded and subsequently evaluated by twenty assessors, with five per group. Mean scores between groups were compared, and agreement within groups and agreement of the average scores between groups were assessed by calculating the intraclass correlation coefficient (ICC). The intervention point at which dogs were deemed to require additional analgesia was also evaluated. Overall agreement of pain assessment was poor (ICC = 0.494). Nurses had the best inter-observer agreement (ICC = 0.656), followed by ECVAA diplomates (ICC = 0.540), veterinary surgeons (ICC = 0.478), and veterinary students (ICC = 0.432). The best inter-group agreement was between veterinary surgeons and nurses (ICC = 0.951) and between ECVAA diplomates and nurses (ICC = 0.951). Students were more likely to determine that additional analgesia was required compared to other groups. Pain assessment is key for animal welfare, and training in this area should be reinforced to improve consistency.

Validity and reliability of a computer-assisted system method to measure axial vertebral rotation.

Axial vertebral rotation and Cobb's angle are essential parameters for analysing adolescent idiopathic scoliosis. This study's scope evaluates the validity and absolute reliability of application software based on a new mathematical equation to determine the axial vertebral rotation in digital X-rays according to Raimondi's method in evaluators with different degrees of experience. Twelve independent evaluators with different experience levels measured 33 scoliotic curves in 21 X-rays with the software on three separate occasions, separated one month. Using the same methodology, the observers re-measured the same radiographic studies three months later but on X-ray films and in a conventional way. Both methods show good validity and reliability, and the intraclass correlation coefficients are almost perfect. According to our results, the software increases 1.7 times the validity and 1.9 times the absolute reliability of axial vertebral rotation on digital X-rays according to Raimondi's method, compared to the conventional manual measurement. The intra-group and inter-group agreement of the measurements with the software shows equal or minor variations than with the manual method, among the different measurement sessions and in the three experience groups. There is almost perfect agreement between the two measurement methods, so the equation and the software may be helpful to increase the accuracy in the axial vertebral rotation assessment.

What are the Relevant Outcomes of the Periodic Health Examination? A Comparison of Citizens' and Experts' Ratings.

PurposeDespite evidence from clinical guideline development that physicians and patients show discordance in what they consider important in outcome selection and prioritization, it is unclear to what extent outcome preferences are concordant between experts and citizens when it comes to the context of primary prevention. Therefore, the objective of this study was to assess whether expert judgments about the importance of beneficial and harmful outcomes differ from citizen preferences when considering intervention options for a periodic health examination (PHE) program.Participants and MethodsWe conducted an online survey using a modified Delphi approach. The target population for the survey consisted of citizens who had attended the PHE (n=18) and experts who made evidence-based recommendations (n=11). Citizens and experts assigned a score on a 9-point Likert scale for each outcome of 14 interventions. We analyzed the intragroup agreement based on Krippendorff’s alpha and the intergroup agreement using the cube root product measure (CRPm). We further tested for significant differences between the groups using the Mann U-test.ResultsAgreements within the groups of citizens and experts varied across the interventions and tended to be poor (α ≤0 to 0.20) or fair (α = 0.21 to 0.40), with three exceptions showing moderate agreement (α = 0.44 to 0.55). The agreements between the citizens and experts across the interventions was fair (CRPm = 0.28) during the first Delphi rating round. The mean differences between the citizens and experts on the Likert scale ranged from 0.0 to 3.8 during the first rating round and from 0.0 to 3.3 during the second. Across interventions, the citizens rated the outcomes as more important than the experts did (p<0.01). Individual participants’ ratings varied substantially.ConclusionBecause experts generally underestimated the outcomes’ importance to citizens, the involvement of citizens in guideline panels for preventive services is important.

Unexpected return visits to emergency department: A healthcare quality management challenge

ABSTRACT Objective: The aim of the study was to assess the unexpected return visits (URV) rate attributed to emergency physicians (EP) diagnostic or therapeutic errors. Methods: Retrospective review was conducted based on electronic medical records (EMR) database of an emergency department with an annual attendance of 460,000 patient visits. The EMR tracked any revisit within 48 h of discharge from ED. Two teams of three EP reviewed EMR for each identified URV independently. The cases were categorized as Category-1: URV clearly unrelated to EP quality of care; Category-2: URV attributable to EP quality of care; or Category-3: uncategorizable into either of the above. The results were presented as proportions with 95% CI for categorical outcomes and kappa (κ) testing was utilized as an indicator of the inter-group agreement. Results: Total URV was 1753 with a URV rate of 5.1% (95% CI, 4.8–5.3) of the monthly census (34,649). The n of category-2 URVs were 615 (1.7%, 95% CI 1.6–1.9%) during the study month. Out of total URVs, 575 were registered as left without being seen (LWBS) on the initial visit. None of the URV patients died within a month from index visit, and none of the LWBS were admitted to the ICU. The κ calculation for the two groups agreement was .94 (95% CI 0.93–0.96). Conclusion: We found URVs were frequently due to factors that were exceedingly unlikely due to physician care quality on the initial ED visit. Study also found that there was nearly perfect agreement between independent groups of physicians tasked to categorized URVs, supporting the use of single physician review to do the task in resource limited setting.

Non-contrast-enhanced MR angiography in the diagnosis of Budd-Chiari syndrome (BCS) compared with digital subtraction angiography (DSA): Preliminary results

Non-CE MRA techniques (true steady-state free-precession, SSFP) have been used effectively for the selective visualization of the portal venous system and inferior vena cava. Budd-Chiari Syndrome (BCS) encompasses a number of conditions that cause the obstruction of the hepatic outflow tract from the small hepatic veins to the junction of the inferior vena cava (IVC) and right atrium. The purpose of this study was to diagnose BCS with IVC obstruction using respiratory triggered three-dimensional (3D) true SSFP with T-SLIP and compare to digital subtraction angiography (DSA). The image acquisition of 3D true SSFP scans was successfully performed in 108 patients (≧2 score). The mean and SDs of the relative SNR and CNR were 55.96±2.32 and 30.72±1.56, respectively. Intergroup agreement for the detection of the 4 types (membranous obstruction, segmental occlusion, and membranous obstruction with a hole and segmental stenosis) of BCS with IVC obstruction was excellent between the Time-SLIP and the DSA. In conclusion, Time-SLIP for the detection of IVC obstruction BCS does not require the use of contrast. This procedure can achieve a high success rate, high accuracy rate and fine image quality for the diagnosis of IVC obstruction BCS.

The effects of China's grassland contract policy on Mongolian herders' attitudes towards grassland management in northeastern Inner Mongolia

China's Inner Mongolia Autonomous Region is experiencing high levels of grassland degradation partially as a result of government policies to sedentarize nomadic pastoralists and privatize collective grasslands. Previous research suggests that these policies have reduced Mongolian pastoralists' ability to effectively manage grasslands and cope with negative climatic events. Herders in New Barag Right Banner (n = 50) representing both sedentary and mobile livestock management strategies were asked to respond to a scaled survey regarding their attitudes towards the effectiveness of their current grassland management strategies and their perceptions regarding the future of pastoralism in Inner Mongolia. Inter-rater reliability and MannWhitney U Tests were utilized to compare the attitudes towards grassland management and the future viability of livestock production and to test whether or not sedentary and mobile herders share the same attitudes towards these facets of their pastoral way of life. There is both high intra and inter-group agreement on the survey variables across settlement categories, indicating that sedentary and mobile herders share the same attitudinal orientations regardless of their settlement patterns. The implications of these results for future grassland policy and sustainable livestock production are also discussed.Keywords: pastoralism, China, Inner Mongolia, grassland policy, privatization, marketization

PP12.14 – 2319: Reliability of phenotypic early onset ataxia recognition

Objective In absence of a “golden detection standard”, identification of early onset ataxia (EOA) relies on phenotypic ataxia recognition. In children, this process is complex for several reasons: 1. physiologically immature motor behavior may “overlap” with signs of initiating ataxia; 2. EOA often involves “mixed” phenotypes (i.e. concurrent with other movement disorders) and 3. movement disorders are phenotyped by specialists of diverse backgrounds. Insight in phenotypic EOA recognition may help to improve the diagnostic yield of innovative genetic techniques and may contribute to the inclusion of high quality patient data in international databases. This study aimed to investigate the inter-observer agreement on phenotypic EOA recognition and to explore whether SARA (Scale for Assessment and Rating of Ataxia) can provide a discriminative marker for EOA recognition. Methods Seven movement disorder specialists independently phenotyped motor behavior of 40 patients (mean age 15 years, range 5–34 years) in whom ataxic features were described (medical records; University Medical Center Groningen; 1998–2012). We determined Fleiss Kappa (FK) and Cohen's Kappa's according to observer-subgroups (pediatric-neurology, adult-neurology, genetics, and trainees). We compared %SARA-subscores (subscore/total score x 100%) between “indisputable” (primary ataxia recognition by at least six observers) and “mixed” (ataxia recognition, unfulfilling “indisputable” criteria) EOA phenotypes. Results Phenotypic EOA recognition was statistically significant, but of moderate strength (FK=0.414). Pediatric neurologists revealed the highest intergroup agreement (with all observer-subgroups; p Conclusion Among movement disorder professionals from different disciplines, inter-observer agreement on phenotypic EOA recognition is statistically significant, but of moderate strength. SARA gait-subscores can provide a supportive discriminative marker between EOA phenotypes. Phenotypic insight may hopefully contribute to the inclusion of uniform, high quality data in international EOA databases.

The Effects of China’s Grassland Contract Policy on Pastoralists’ Attitudes towards Cooperation in an Inner Mongolian Banner

China’s Inner Mongolia Autonomous Region is experiencing a marked decline in grassland quality partially as a result of government policies to sedentarize nomadic pastoralists and privatized collective grasslands. Previous research suggests that traditional forms of cooperation among Inner Mongolian pastoralists have deteriorated as a result of privatization and sedentarization. Herders in New Barag Right Banner (n = 50) representing both sedentary and mobile livestock management strategies were asked to respond to a scaled survey regarding their attitudes towards cooperation with other pastoralists. Inter-rater reliability and Mann–Whitney U Tests were utilized to compare the attitudes towards cooperation across sedentary and mobile settlement categories and to assess whether or not sedentary and mobile herders share the same cultural model regarding cooperation. The authors show that there is both high intra- and inter-group agreement on the survey variables across settlement categories, indicating that sedentary and mobile herders share the same cultural model regardless of their settlement pattern.

Impact of implementing smart infusion pumps in a pediatric intensive care unit

The impact of smart infusion pumps on the interception of errors in the programming of i.v. drug administrations on a pediatric intensive care unit (PICU) is investigated. A prospective observational intervention study was conducted in the PICU of a hospital in Madrid, Spain, to estimate the patient safety benefits resulting from the implementation of smart pump technology (Alaris System, CareFusion, San Diego, CA). A systematic analysis of data stored by the devices during the designated study period (January 2010-June 2011) was conducted using the system software (Guardrails CQI Event Reporter, CareFusion). The severity of intercepted errors was independently classified by a group of four clinical pharmacists and a group of four intensive care pediatricians; analyses of intragroup and intergroup agreement in perceptions of severity were performed. During the 17-month study period, the overall rate of user compliance with the safety software was 78%. The use of smart pump technology resulted in the interception of 92 programming errors, 84% of which involved analgesics, antiinfectives, inotropes, and sedatives. About 97% of the errors resulted from user programming of doses or infusion rates above the hard limits defined in the smart pump drug library. The potential consequences of the intercepted errors were considered to be of moderate, serious, or catastrophic severity in 49% of cases. The use of smart pumps in a PICU improved patient safety by enabling the interception of infusion programming errors that posed the potential for severe injury to pediatric patients.

Evaluation of the agreement by examiners according to classifications of third molars

Objectives. This study recorded and evaluated the intra- and inter-group agreement degree by different examiners for the classification of lower third molars according to both the Winter’s and Pell & Gregory’s systems. Study Design. An observational and cross-sectional study was realized with forty lower third molars analyzed from twenty digital panoramic radiographs. Four examiner groups (undergraduates, maxillofacial surgeons, oral radiologists and clinical dentists) from Aracaju, Sergipe, Brazil, classified them in relation to angulation, class and position. The variance test (ANOVA) was applied in the examiner findings with significance level of p<0.05 and confidence intervals of 95%. Results. Intra- and inter-group agreement was observed in Winter’s classification system among all examiners. Pell & Gregory’s classification system showed an average intra-group agreement and a statistical significant difference to position variable in inter-group analysis with greater disagreement to the clinical dentists group (p<0.05). Conclusions. High reproducibility was associated to Winter’s classification, whereas the system proposed by Pell & Gregory did not demonstrate appropriate levels of reliability. Key words: Agreement, reproducibility, classifications, third molars.

Risk criticality and allocation in privatised water supply projects in Indonesia

Abstract Public–private-partnership (PPP) infrastructure projects typically involve a plethora of risks. Given that many risks which are traditionally borne by the government get transferred to the private sector, careful risk allocation is considered to be the key to successful PPP projects. This paper focuses on the current risk allocation practice within Indonesia’s water supply projects. The paper captures, presents and discusses the perceptions of both regulators and operators in the context of project risk criticality and allocation. Both parties consider non-availability of raw water is of paramount concern. However, survey findings highlight low-to-moderate levels of intra-group and inter-group agreement regarding the allocation of specific project risk factors. Thus, asserting that the issue of optimal risk allocation remains elusive.

Reliability of a modified medication appropriateness index in primary care

To evaluate the inter-group and intra-group reliability of a modified Medication Appropriateness Index (MAI) for use in primary care. Elderly (>65 years) polypharmacy (> or =5 drugs) patients in Copenhagen County participated in the study. Information concerning their medical history and information regarding each drug taken by them was provided by their own general practitioners. A MAI was scored by two groups of evaluators for every drug. To evaluate inter-group agreement, 211 drugs taken by 30 patients were rated according to the ten criteria making up the MAI. Both evaluator groups provided summaries of comments on the medication of each patient. Intra-group agreement was determined from MAI ratings performed twice at two different times on 86 drugs taken by ten patients. Agreement and chance-adjusted agreement were determined, the latter through kappa statistics. The proportion of positive (ppos) and negative (pneg) agreement was also determined. The overall chance-adjusted inter-group agreement (kappa) was moderate. The agreement was good on the criteria practical directions and drug-disease interaction, moderate on the criteria dosage and duration, fair on the criteria indication, effectiveness, duplication and expense, and poor on the criterion drug-drug interaction. The overall chance-adjusted intra-group agreement was good for all criteria and very good for the criteria indication and practical directions. The MAI is used to quantify appropriate and inappropriate prescribing and changes in prescribing quality in intervention studies. However, caution should be used when comparing results across different settings and evaluators. Our study suggests that the index should only be used in intervention studies if the same group rates the appropriateness pre- and post-intervention.

Providing Evidence-Based Answers to Complex Clinical Questions: Evaluating the Consistency of Article Selection

Health care providers must maintain familiarity with current biomedical evidence, but clinicians struggle to maintain their awareness of current research because of the demands of daily practice and the exponential growth of medical knowledge. Clinical information specialists (informationists), trained experts in reviewing and filtering the medical literature in response to complex clinical queries, may be able to assist practicing clinicians. This study compared informationists and two categories of physicians in their article selection in response to two complex clinical questions. The study was performed at Vanderbilt University Medical Center. A total of 15 faculty and staff from three groups were recruited (five general physicians, five physicians trained in research methodology, and five informationists). The participants reviewed two previously selected clinical questions, worked in focus groups to define the pertinent facet questions of the questions, and then ranked the articles by pertinence to the clinical questions. In general, both informationists and physicians trained in research methodology had a high degree of intergroup agreement for ranking article pertinence, while the generalists were less likely to agree on pertinent articles. These findings suggest that informationists consistently select articles relevant to answering complex clinical queries and may assist practicing clinicians by providing information relevant to patient cases.

Peer and expert opinion and the reliability of implicit case review

ObjectiveTo compare the interrater and intergroup agreement in judging physician maloccurrence and compliance with standards of care using the implicit case review process. DesignMail survey with questionnaire. ParticipantsCase reviews and questionnaires were mailed to 140 board-certified ophthalmologists and 140 board-certified ophthalmologists with fellowship training. Main outcome measureAgreement judging maloccurrence and compliance with standard of care within each group and between general ophthalmologists and specialists. ResultsNinety-seven (35%) questionnaires were returned. Overall, 35% of respondents believed that ophthalmologists in the case reviews committed an error of either commission or omission. Forty-five percent of reviewers believed that physicians did not meet the standard of care. There was good within-group agreement for finding clinical error in management and not meeting the standard of care for all groups (kappa coefficient range: 0.55–0.83; P = < 0.004) except retina specialists (kappa coefficient = 0.12; P = 0.2). ConclusionsUnstructured implicit case review is not a reliable method for determining physician error or for measuring compliance with standards of care. The process is susceptible to bias, and results may vary with reviewer background or training. Unstructured implicit case review needs to be regarded as a rough screening tool and used accordingly.

Locational Preferences of 14-year-old School Children in Londonderry

This research attempted to replicate Stringer and Cornish’s (1990) study of locational perceptions in Northern Ireland while avoiding some of that study’s limitations. Ratings of residential preference and perceived unemployment, violence and denomination for 25 areas of Londonderry were gathered from 52 Catholic and 52 Protestant 14-year-olds. Intergroup agreement on perceptions was high, both groups viewing Catholic locations as having higher unemployment and violence. For Protestants, this reinforced a preference for in-group locations, but for Catholic subjects in-group preference was counterbalanced by negative perceptions of social conditions. This pattern was similar to that reported by Stringer and Comish (1990), and implied the behaviour observed here can be generalised beyond the original subjects and geographical locations. Additional factors for developing a model of locational preference formation were suggested.

Social Psychological Correlates of Locational Preference in Northern Ireland

The residential preferences of 60 Northern Irish university students (30 Protestant and 30 Catholic) for sixty locations within the Province were examined to assess the relationship of this variable with their perceptions of the level of unemployment, violence and denominational composition of those locations. There was a high level of intergroup agreement between Protestant and Catholic subjects on the application of all scales and subjects’ perceptions correlated well with ‘objective’ measures. The greatest difference between the two groups was in the correlation between perceived denomination and preference ratings: for Protestant subjects, the relative dominance of own-group members was highly correlated with preference, while for Catholic subjects no such relationship was found. This was explained by the differential use of judgements of ‘objective’ characteristics in the formation of residential preference.

Survey of hospital doctors' training goals in diagnosis and management of pulmonary diseases.

Although subspecialty training goals for junior hospital doctors have not been evaluated, they are potentially useful for assessing clinical competence. A questionnaire was sent to medical residents, full-time pulmonary teaching staff, and community-based physicians who were asked to rate the importance of diagnosing and managing selected pulmonary diseases for the future practices of young hospital doctors. The latter also rated their perceived preparedness for the same training goals. Generally good intra- and intergroup agreement about the relevance of most of the training goals was observed, although the hospital doctors rated a greater number of items to be more important than did the two other groups. This finding may be attributed to institutional influences and to many clinical abilities expected of all physicians. Immunological or fibrotic and paediatric respiratory disorders were rated least important by most respondents. The hospital doctors disagreed on the basis of the distribution of their ratings about their preparedness for the same goals, which probably reflects varying training experiences and background. The process of developing general professional training goals in a subspecialty requires discussion, identification, and consensus to identify and potentially correct areas of weakness, with allowance for institutional training patterns. A survey such as described in this study can provide data that can help measure clinical competence and support or define curricular changes.

Imagery and Meaningfulness Ratings of Sentences: Reliability and Relationship to Learning

Exp. 1 assessed the test-retest reliability with which individuals and groups can rate simple sentences for image-evoking power and meaningfulness. For 10 different Ss rating 100 sentences on each dimension, the mean reliability coefficients were .69 and .67 for imagery and meaningfulness respectively. Within-group test-retest reliability of mean ratings was excellent (.90, imagery; .92, meaningfulness). Rating of both dimensions was also performed by a third group of 19 Ss, and inter-group agreement (estimated through correlation of group mean ratings) was substantial (.76, imagery; .87, meaningfulness). However, agreement between individual Ss was low for both dimensions (.18, imagery; .27, meaningfulness), though significant. Exp. 2 examined Paivio's (1971) dual-coding hypothesis applied to sentence memory through correlation of both individual ratings and group mean ratings with sentence recognition scores. Neither method yielded a significant correlation between learning and imagery, or meaningfulness. Apparently the dual-coding hypothesis cannot be straightforwardly applied to memory for sentences, despite its predictive strength for single-word learning tasks.

The selection of a psychiatric curriculum for medical students: results of a survey.

A survey of 131 psychologists, psychiatrists, and nonpsychiatrist physicians taken to determine what doctors of medicine should know about psychiatry revealed that of the 21 topics assessed, there was a significant intergroup agreement on the 10 most important and the 5 least important topics. Interviewing, suicide evaluation, the chronically ill or dying patient, and psychiatric referral received high ratings by each group, and psychoanalytic theory, psychodynamics, and mental retardation received uniformly low ratings. The authors believe that these findings provide useful information for planning an undergraduate curriculum in psychiatry.