Global Judgments Research Articles

An intervention to improve the reliability of manuscript reviews for the Journal of the American Academy of Child and Adolescent Psychiatry.

The effects of methods used to improve the interrater reliability of reviewers' ratings of manuscripts submitted to the Journal of the American Academy of Child and Adolescent Psychiatry were studied. Reviewers' ratings of consecutive manuscripts submitted over approximately 1 year were first analyzed; 296 pairs of ratings were studied. Intraclass correlations and confidence intervals for the correlations were computed for the two main ratings by which reviewers quantified the quality of the article: a 1-10 overall quality rating and a recommendation for acceptance or rejection with four possibilities along that continuum. Modifications were then introduced, including a multi-item rating scale and two training manuals to accompany it. Over the next year, 272 more articles were rated, and reliabilities were computed for the new scale and for the scales previously used. The intraclass correlation of the most reliable rating before the intervention was 0.27; the reliability of the new rating procedure was 0.43. The difference between these two was significant. The reliability for the new rating scale was in the fair to good range, and it became even better when the ratings of the two reviewers were averaged and the reliability stepped up by the Spearman-Brown formula. The new rating scale had excellent internal consistency and correlated highly with other quality ratings. The data confirm that the reliability of ratings of scientific articles may be improved by increasing the number of rating scale points, eliciting ratings of separate, concrete items rather than a global judgment, using training manuals, and averaging the scores of multiple reviewers.

A COMPLEXITY ANALYSIS OF ITEMS FROM A SURVEY OF ACADEMIC ACHIEVEMENT IN THE LIFE SCIENCES.

ABSTRACTThe difficulty of 44 items from the life sciences subscale of the NAEP 1985–86 science assessment was analyzed in terms of item attributes and science educators' judgments of difficulty. The attributes included ratings of various characteristics of the items' text and option set, the items' cognitive demand, and the level of knowledge required by items. Science educators' mean judgment of item difficulty, which accounted for 52% of the variance, was the best single predictor of item difficulty. Combining item attribute information with educators' judgments of item difficulty improved the prediction of item difficulty on the order of 7% to 15% of the variance. When item difficulty was modeled in terms of discrete item attributes (global judgments of item difficulty not included in the model), the level of knowledge required was an important determinant of difficulty, while cognitive demand was not. The implications of these results for construct validation and for test design are discussed.

Analyzing clinical ratings of performance on pediatric neuropsychological tests.

This project examined the agreement among pediatric neuropsychologists when rating the functioning of normal and low-birth-weight children in seven neuropsychological domains. In Phase 1, two neuropsychologists rated 154 children; in Phase 2, three neuropsychologists rated 41 children. Intraclass correlations of agreement in Phase 1 were: attention .39, intelligence .85, auditory/linguistic .82, haptic .70, visual perceptual/visuomotor .78, mnestic .72, and global .61. Intraclass correlations observed in Phase 2 were similar to those found in Phase 1 except that agreement in rating attention increased to .53. Linear modeling of global judgments revealed that two raters emphasized the auditory/linguistic, haptic, and visual perceptual/visuomotor domains in deriving global ratings, whereas the third emphasized the intellectual and attentional domains. In general, agreement among raters was sufficient to justify using clinical ratings to combine individual test scores into more general behavioral measures.

Integration of depth modules: Stereopsis and texture

Global shape judgements were employed to examine the combination of stereopsis and shape-from-texture in the determination of three-dimensional shape. Adding textural variations to stereograms increased perceived depth. Thus, texture was not simply vetoed by the strong stereo cue. In experiments where the depth specified by texture was incongruent with that specified by stereo, the data were well described by a weighted linear combination rule. Although only a small weight was assigned to texture, this weight was somewhat greater at a farther viewing distance. This could be a consequence of the decreased reliability of stereopsis at far viewing distances.

Bankruptcy as a deliberate strategy: Theoretical considerations and empirical evidence

AbstractBankruptcy and bankruptcy reorganizations have been identified as remedies for financial distress, but there is little agreement on their value to firms, managers, and the general economy. This paper provides a brief review of proposed bankruptcy strategies and some alternative views about their costs and benefits followed by an empirical study of the outcomes of 73 bankruptcies and subsequent reorganization efforts. The evidence suggests that there are few successful reorganizations, bankruptcy is a costly response to financial distress, and managerial choice in bankrupt firms is highly constrained by forces external to the firm. The diversity of stakeholder interests limits the value of global judgements about success or failure of bankruptcy strategies. Firm size dominates all other factors in predicting success in completing the reorganization process. Delayed filings primarily reflect failed efforts to avoid bankruptcy, not deliberate strategies. Reasons for the use of bankruptcy in spite of its high costs are discussed.

高齢者におけるＢＰＰＶの平衡訓練効果と薬物療法の比較

We compared the effectiveness of 2 different therapies in elderly patients with benign paroxysmal positional vertigo.Eighteen patients were treated with an anti-vertigo drug and 18 with vestibular training (VT) and the same drug. They were classified into Group B and VT + B, respectively.The background factors, such as age, severity in symptoms, degree of disturbance in examinations, et, were similar in the 2 groups.Vertigo, dizziness, unsteadiness in Mann's test and positional nystagmus improved significantly more in Group VT+B than in Group B. The global judgment in each group revealed that both marked and moderate improvement was significantly higher in Group VT+B than in Group B (P<0.01).It was concluded that vestibular training might be useful to improve symptoms and signs in benign paroxysmal positional vertigo even in aged patients.

Preliminary Report of Guidelines for Reporting of Treatment Results in Meniere's Disease Conducted by the Committe of the Japan Society for Equilibrium Research

In order to draft the guidelines for reporting treatment results in Meniere's disease in Japan, the Committee Members were elected by the Japan Society for Equilibrium Research in 1991, and their papers were discussed in the first, second and third workshops from June 1991 to November 1992.In Septemper 1992, the Committee analyzed the preliminary criteria from the responses to a questionnaire collected from the 17 Committee Members.The preliminary draft of guidelines obtained from the analysis of the questionnaire are summarized as follows.1) Evaluation of Symptoms: Only definitive attack of vertigo with spontaneous nystagmus should be taken into account for the evaluation of the treatment of symptoms (AAOO, 1972) (Agreement: 82 %).The degree of vertiginous spells should be noted pre-and posttreatment, and the disability should be evaluated four categories (AAOHNS 1985) (Agreement: 82 %). The parameter for the assessment of the hearing level is much better with four frequencies pure-tone average (PAT) at 250, 500Hz, IKHz and 2KHz than that of the AAOHNS's four frequencies (500Hz, 1KHz, 2KHz, and 3KHz) (Agreement: 47 %).2) Evaluation Period: Valid results for the evaluation of vertiginous spells should be based on observation for over six months pre-treatment and over 12 months after the initiation of therapy (Agreement: 75%).If the duration of pretreatment is less than six months, the divisor is the average number of definitive spells per month for the period of observation.3) Evaluation Criteria: In the same way as AAOHNS (1985), the formula expressing the effect of treatment on vertiginous spells should be adopted for our criteria (Agreement: 100%).4) Global Judgment: The degree of disability should generally be evaluated in reporting the effect of therapy (Agreement: 41%).These answers to the questionnaire show that the same evaluation for medical and surgical treatment could not be adopted by all the Committee Members (Agreement: 50 %).Therefore, further discussion of these points is needed.

Inter- and intra-observer agreement in the assessment of the quality of spontaneous movements in the newborn

The inter- and intra-observer agreement for the assessment of several aspects of the quality of spontaneous movements in newborn infants was analysed by 4 observers. The agreement rates were corrected for chance by the use of kappa statistics, which revealed kappa values ranging from 0.36 to 0.84. The degree of agreement on various items differed. A moderate-to-good agreement was found for the items "onset of movement," "speed," "speed of arms compared to legs," "amplitude of arms compared to legs," "fluency," "variability in arm patterns" and "global judgement of the movement" (kappa values ranging from 0.51 to 0.84). Less agreement occurred in the items "amplitude," "variability in movement pattern" and "variability in leg patterns" (kappa values ranging from 0.36 to 0.40). Even though it is believed that the qualitative assessment of spontaneous motor behaviour is very subjective, our results suggest that several qualitative aspects of infant motor behaviour can be reliably judged. We concluded that the global judgement of the quality of general movements in full-term infants is a reproducible, and therefore reliable, clinical tool for evaluating the function of the central nervous system in the neonatal period.

Appropriateness of Care

The RAND-UCLA Health Services Utilization Study previously analyzed the appropriateness of use of carotid endarterectomy based on a literature review and global expert judgments. In this study, for 45 of the same clinical indications used in the RAND-UCLA Study, the authors compare the appropriateness judgments based on the global judgment method to appropriateness ratings based on probability estimates of specific outcomes that were provided by the same panel of experts. The authors asked these experts to estimate, for each clinical indication, the likelihood of important medical outcomes (i.e. stroke within a year) in the presence and absence of endarterectomy. Using decision analysis, the appropriateness of endarterectomy for these 45 indications was then calculated. For only two of the eight physicians were the Spearman rank-order correlations between these two methods of judging appropriateness significant and positive. (Correlations for the eight physicians ranged from 0.45 to -0.38). This result was produced by: 1) the tendency of the experts to estimate relatively poor outcomes for seriously ill patients whether or not endarterectomy was performed; and 2) a far less consistent effect of clinical factors on outcome estimates than on global judgments. Better methods for incorporating probability estimates into a global rating process must be developed. The authors found excellent agreement between the panelists' relative outcome estimates for common endarterectomy indications and the observed stroke rate for these same indications, suggesting that one promising method is to use specific outcome data to "anchor" expert judgments.

The Role of Context and Autobiography in Cognitive Assessment

The Role of Context and Autobiography in Cognitive Assessment

A therapy impact questionnaire for quality-of-life assessment in advanced cancer research

The first part of the validation procedure used for a Therapy Impact Questionnaire (TIQ) on quality-of-life assessment in advanced cancer patients is described. The TIQ is composed of 36 items which assess both disease and therapy impact according to four dimensions that operationally define quality of life: physical symptoms (24 items), functional status (3 items), concomitant emotional and cognitive factors (6 items) and social interaction (2 items). A global judgement expressed as "have you been feeling ill" further completes the TIQ. Patients answered each question using a 4-point verbal Likert scale: not at all, slight, a lot and very much. The TIQ was given to 1,000 consecutive patients who were no longer responsive to cancer treatment and presented symptoms due to disease progression. The compliance rate was quite high (87%). Results of confirmatory factor analysis were consistent with the operational dimensions identified during questionnaire construction. In particular, the dichotomized answers to 3 functional status items could be used as a Guttman scale. In a sub-sample of 50 patients, the reproducibility of functional status items was assessed using a 7-item parallel form. The intraclass correlation coefficient obtained indicated a reasonably high reproducibility. On the basis of the analyses conducted, the TIQ appears to be a reliable and concise instrument for studies aimed to assess the effectiveness of therapies in advanced cancer patients.

Explaining the evidence: Tests of the Story Model for juror decision making.

This research investigates the Story Model, Pennington and Hastie's (1986, 1988) explanationbased theory of decision making for juror decisions. In Experiment 1, varying the ease with which stories could be constructed affected verdict judgments and the impact of credibility evidence. Memory for evidence in all conditions was equivalent, implying that the story structure was a mediator of decisions and of the impact of credibility evidence. In Experiments 2 and 3, Ss evaluated the evidence in 3 ways. When Ss made a global judgment at the end of the case, their judgment processes followed the prescriptions of the Story Model, not of Bayesian or linear updating models. When Ss made item-by-item judgments after each evidence block, linear anchor and adjust models described their judgments. In conditions in which story construction strategies were more likely to be used, story completeness had greater effects on decisions.

Trends in the use of R &amp; D output indicators in EC programme evaluation

This paper rests upon a review of 15 evaluation reports of R & D programmes worked out during the 80's by the European Commission. The analysis aims at answering the main questions: Why did emerge the needs for output indicators in the middle of the 80's? What kind of output indicators were built up (or tentative)? With which methodology? What were their actual use in the evaluation reports? The linkage between EC R & D policies and evaluation is examined in order to discuss the relationships between the goals of R & D programmes and the criteria for evaluation. It is shown that the followed evaluation methodology and the evaluation goals at hand are paramount for the choice of output indicators; such goals encompass a.o. the description of the programmes, the assessment of the contractors opinion, the appraisal of the “techno-economic” effects of the programmes. As a result “expected output indicators” were developed (BRITE programme). On the other hand, one has called “meta-evaluation”, the indirect measurement of Scientific results by bibliometry (BEP_BAP programmes). Similarly, “intermediate indicators” were built up for evaluating the programmes management performance (ESPRIT programme). At last “derived output indicators” were used for techno-economic evaluation, (EURAM programmes) leading to the quantified global judgement of a “before-after” methodology, (SCIENCE-STIMULATION programmes).

Analysis of observed behaviors displayed by depressed patients during a clinical interview: relationships between behavioral factors and clinical concepts of activation

In 61 drug-free depressed patients, relationships were studied between observed behaviors and measures of common clinical concepts of activation. The behaviors were observed during a clinical interview and analyzed with ethological methods. Activation was assessed by means of self-ratings (Thayer, AD-ACL) and global judgement (Hamilton, retardation, agitation). Various aspects of patients' speaking and listening behaviors were recorded and analyzed, such as looking, head movements, yes-nodding and no-shaking, leg movements, gesturing, and body and object touching. A factor analysis was applied, enabling grouping of behaviors without using a priori concepts. Five factors reflected different aspects of a conversation: restlessness (leg and hand movements), speech, active listening (head movements and intensive body touching, during listening), speaking effort (looking, gesticulating, head movements, during speaking), and eagerness (yes-nodding and no-shaking). Significant positive relationships were found between the speech factor, the speaking effort factor and the restlessness factor on the one hand, and activation on the other. The eagerness factor was related negatively with activation. The results give insight into the organization of behavior during an interview, and show how this is related to clinical concepts of activation.

The long-term tolerability of bencyclane (‘Fludilat’) in patients with peripheral occlusive disease: A 48-week prospective double-blind controlled study versus placebo

In a controlled, multi-centre, double-blind trial, 75 patients with Stage II peripheral occlusive disease (Fontaine IIa) were treated with either 200 mg bencyclane twice daily or placebo over a period of 12 months. Undesired drug effects and concomitant phenomena were documented, and efficacy was evaluated. Bencyclane caused a slight, clinically negligible decrease in blood pressure. The pulse rate remained mostly unchanged, ECG and laboratory parameters showed no changes which would indicate a specific effect of the test substance. In the context of the generally low incidence of concomitant effects, patients in the bencyclane group mentioned symptoms such as insomnia, depressive mood, sweating and reduced motoricity more often than those in the placebo group. These symptoms are regarded as signs of the central nervous actions of the drug. The parameters used to assess the efficacy, i.e. the pain-free walking distance estimated by the patients and the physician's global judgment based on Ratschow's test, the palpability of the pedal pulse, the walking range and the patients' subjective statements about the incidence of chill, formication, and pain in the legs, showed a constant and statistically significant superiority of bencyclane over placebo.

Original article: Quality-of-life assessment during a palliative care programme

By means of a cross-sectional study, 115 terminal cancer patients (53 males, 62 females) who were no longer responsive to anticancer treatment, were investigated. The sample included all patients who had been undergoing palliative care (PC) during a single week (at the out-patient clinic, in hospital or at home). From the start of PC, the quality of the patients' lives was assessed by a weekly self-descriptive record comprising 32 items at four levels of intensity. The responses given on the questionnaire during the sample week were compared to those given by the same patients at the beginning of PC (T0). From T0 to time during treatment, figures show a significant increase in the percentage of patients who reported drowsiness, and a significant decrease in pain, weakness, functional impairment and psychological distress. The global judgment of not feeling well was reduced from 49% of the patients at TO to 31% during the period of treatment (p less than 0.01). This result shows that, although the disease progressively develops, PC can enhance the quality of the lives of patients during the terminal stages of illness. The subjective judgment of not feeling well was much more closely correlated with physical, functional and psychological symptoms. Of the physical symptoms, pain has the closest correlation with feeling bad. However, pain has a low number of statistically significant correlations with respect to the other items, in marked contrast to the high number of correlations regarding psychological and functional items.(ABSTRACT TRUNCATED AT 250 WORDS)

Clinical Decision Making: Theory vs Practice

To the Editor.— Dr Eddy 1,2 must be praised for seeking to define, or perhaps divine, a rational methodology to control and fine-tune practice policies, for patterns of practice are at the root of the inexorable and seemingly uncontrollable rise in this nation's health costs. In a private practice, fee-for-service system, it is impossible to ignore the fact that physicians will perform or request the services that maximize their income within the broad limits of legal, ethical, scientific, and clinical parameters. All practicing physicians are aware of and engage in useless and usually harmless activities, without which the patient's health would be unaffected. The litany of such practices is endless and cuts across all medical and surgical disciplines. The vast majority of such practices involves gray areas. If, using the global subjective judgment approach (Dr Eddy's term) to monitor a procedure, a panel unanimously agrees that it is unnecessary for

Clinical decision making: theory vs practice

<h3>To the Editor.—</h3> Dr Eddy^1,2must be praised for seeking to define, or perhaps divine, a rational methodology to control and fine-tune practice policies, for patterns of practice are at the root of the inexorable and seemingly uncontrollable rise in this nation's health costs. In a private practice, fee-for-service system, it is impossible to ignore the fact that physicians will perform or request the services that maximize their income within the broad limits of legal, ethical, scientific, and clinical parameters. All practicing physicians are aware of and engage in useless and usually harmless activities, without which the patient's health would be unaffected. The litany of such practices is endless and cuts across all medical and surgical disciplines. The vast majority of such practices involves gray areas. If, using the global subjective judgment approach (Dr Eddy's term) to monitor a procedure, a panel unanimously agrees that it is unnecessary for

Clinical Decision Making: Theory vs Practice

To the Editor.— I read with great interest Dr Eddy's 1,2 dissertation in which he described five methodologies for developing practice policies. These included the old "traditional method" and four new methods called "global subjective judgment," "evidence based," "outcome based," and "preference based." He states that the "traditional approach" is based on the assumption that "outcomes of medical practice can be sensed intuitively without explicit analysis or description." He used as an example of the traditional approach the current recommendations for treatment of ocular hypertension. In reality, the treatment for ocular hypertension has evolved over a number of years since the collaborative glaucoma study in the 1960s established the concept of ocular hypertension. Subsequent observation of individual patients by individual physicians showed that some of these patients would go on to develop true glaucoma, thus indicating a need for treatment in some cases. Two recent studies have confirmed that treatment

Social Support in Marriage: The Role of Social Cognition

Close relationships and self-report play a central role in the social support literature, yet little is known about how behaviors in intimate relationships lead people to feel supported. Drawing from research on cognition in marriage we argue that explanations or attributions for spouse behavior, particularly in terms of whether the behavior is viewed as voluntary and selflessly motivated, is an important determinant of whether the behavior is perceived as supportive. The impact that various cognitive processes may have on attributions, global judgments of support, and reports of behavior used to measure support, is also illustrated. Finally, the need to investigate perceived support in relation to the transactions that occur between spouses is emphasized, and the implications of a cognitive perspective on social support are outlined.