External Judges Research Articles

The Influence of Evaluation on Changing Management Systems in Educational Institutions

This article compares the influence of evaluation in two different public education contexts. One evaluation was knowledge-focused, with the evaluators acting as external judges in a context of top-down changes at post-implementation stage in infant and primary schools. The other was development-focused from a constructivist perspective in a context of bottom-up changes during the building of a shared model in a university administration department. The effects of the former were limited to the impact of disseminating scientific information. The latter evaluation had several effects, many of them indirect and diffuse. The main source of influence was participative discussion about evaluation procedures and their results. The advisability of participative evaluation to support system changes and model construction is discussed. The authors also suggest that the concept of influence be considered in the broad sense so different types of influence (multidirectional, indirect, unintended, non-instrumental) can be included as an evaluation impact.

PUBLIC SAFETY AND THE MORAL DILEMMA IN THE DEFENSE AGAINST TERROR

The economic theory of defense has traditionally described public safety as achieved through investments that deter adversaries. Deterrence is, however, ineffective and pre‐emptive defense is required when a population of intended victims confronts supreme‐value suicide terror. A moral dilemma then arises, since pre‐emption may impose collective punishment, while in the absence of pre‐emption the population of intended victims is exposed to acts of terror. We consider how a population of intended terror victims confronts the moral dilemma, and compare the threatened population's response with the public‐safety recommendations of external judges who are not personally affected by the threat of terror.

Body and gaze formations and the communicational alliance in couple-therapist triads

In order to establish and maintain what we have named a communicational alliance in marital therapy, couples and their therapist generally (a) signal their affective involvement with each other, and (b) coordinate their behaviors to constantly maintain this triadic frame of involvement. In this paper, we empirically examine the manner in which this communicational alliance was elaborated and maintained using two approaches: (a) a behavioral and micro-analytic approach based on the coding of body and gaze formations; and (b) a global assessment approach based on external judges' assessments. Results indicated the two approaches to be compatible. In the case where the judges considered the communicational alliance to be high, overall observations of body formations and gaze formations indicated that (a) everyone was included, (b) each participant adhered to his or her role, and (c) these signals were congruent. Once a communicational alliance had been established, the trend was that it was maintained throughout the therapy. The clinical and scientific importance of studying interactive indicators of the therapeutic alliance are discussed.

Psychological Adjustment to Cancer: Control Beliefs and Coping in Adult Cancer Patients

Appraisals of control, the use of problem- andemotion-focused coping strategies, and symptoms ofanxiety/depression were assessed in a sample of 83 adultcancer patients (mean age of 41.6 years) on average 10 weeks after their diagnosis.Anxiety/depression symptoms were related to patients'ratings of their use of problem-focused coping, but notto perceived control or emotion-focused coping in simplecorrelations. In hierarchical multiple-regression analyses,problem-focused coping was associated with lowersymptoms of anxiety/depression and emotion-focusedcoping with higher symptoms. The interaction ofproblem-focused coping and perceived control was a significantpredictor of lower symptoms of anxiety/depression. Thispattern was found for patients' but not external judges'ratings of patients' coping intentions. The interaction of coping and control did notpredict anxiety/depression symptoms 4 months later,after controlling for initial anxiety/depressionsymptoms. Implications for adaptive coping with cancerare highlighted.

What is binge eating? A comparison of binge eater, peer, and professional judgments of eating episodes

Binge eating is a central diagnostic feature of bulimia nervosa (BN) and binge-eating disorder (BED), yet the phenomenon of bingeing has not been adequately defined, and there is considerable variability in how individuals label eating episodes as binges. We examined the agreement among binge-eating individuals, non-eating-disordered peers, and professional dietitians over whether particular eating episodes were binges. Twenty-nine females with BED. fifteen nonclinical binge eaters (NCB), three peer judges, and three dietitians rated a sample of eating episodes of the binge-eating individuals as either binges or nonbinges based on the types and amounts of food eaten as well as the duration of each eating episode. BED participants labeled a significantly higher proportion of their intakes as binges relative to NCB participants. Peer judges were more likely than were dietitians to label participants‘ eating episodes as binges. Agreement within dietitian and peer groups was poor to fair, whereas agreement between these groups was fair. Finally, agreement between participants and the external judges (i.e., peers, dietitians) was poor. Possible explanations for these findings as well as implications for the diagnosis of BN and BED are discussed.

When to recommend compulsory versus optional CME programs? A study to establish criteria.

When should remedial continuing medical education (CME) be compulsory for family physicians? When should it be optional? Should it be structured or not? In 1993-1994, the authors addressed this need for criteria by conducting a study that used reports on 14 physicians who had undergone a structured oral interview (SOI) at the College of Physicians of Quebec. (The SOI is a day-long encounter during which two specially trained physician-interviewers present a physician with standardized clinical cases that focus on ten specific aspects of a family physician's competence.) The 14 SOI reports were reviewed by 12 external physician-judges in an attempt to see how consistently they could link the ten aspects of competence, as shown in the reports, to five particular types of recommended remedial CME programs (the strictest being "compulsory program with suspended license" and the most lenient being "simple suggestions for improvement"). There was substantial agreement among the judges when choosing between compulsory and optional programs (kappa = 0.63, p < .05). The main criteria used when recommending an optional program were overall strengths and the quality of clinical reasoning. The same two criteria were also used for recommending a compulsory program, but the judges also considered three additional factors: the physician's ability to recognize his or her limits and how he or she handled referrals and prescribed medications. Many of the criteria used by the judges were based on unique information that came out of observations and interactions during the SOIs, such as quality of argumentation, sustaining a train of thought, sense for the case as a whole, or awareness of one's limits. Finally, the external judges corroborated the decisions previously made by the College of Physicians of Quebec concerning the appropriate CME programs for the 14 physicians.

Evaluation of Supervisees

A fairly substantial body of literature has investigated sources of variability in counselor performance ratings. No comparisons of supervisors' and external judges' evaluations, however, have been reported. In this study, individual superrvisors (n = 27) evaluated their counseling students (n = 43), who were at three training levels (first practicum, second practicum, and internship). External judges rate audiotapes of students' actual counseling sessions. The correlation between the two ratings was not significant. Follow up analyses of variance revealed that, while external judges' ratings increased across training levels, supervisors' evaluations of second practicum students were significantly lower than those of their less or more experienced counterparts. Results seemed to support Holloway's (1984) assertion that supervisors' evaluations of counseling performance are influenced by their interactions with students during supervision sessions.

The Relationship of Employment Status, Social Support, and Life Events with Depressive Symptomatology among Married Turkish Women

The present study investigated the predictive power of employment status, husband's and friends' social support and number of life‐events on the levels of depressive symptomatology as measured by the Beck Depression Inventory among married Turkish women. Husband's support and number of life‐events appeared as significant predictors of depression, whereas employment status did not. The results also showed that subjects reporting negative life‐events rated by external judges to be relatively independent of individual perception had higher depression scores than subjects not reporting any of these events. The results were discussed with reference to the status of women in Turkey and the possible socio‐cultural variables that might have contributed to the differences between the results obtained from Turkish and Western samples.

A preliminary study of excess mortality using a psychiatric case register.

Systematic screening of death certificates referable to residents of two health districts covered by a psychiatric case register allows the study of excess mortality in unselected psychiatric patients from a defined area of known population. Deaths among the case register patients exceeded those expected by 80%; the relative risk was maximal during the first year after registration and was significantly raised in both sexes and in patients of all age-groups. Patients may be selected by service use and by diagnostic group: the excess mortality of "inpatients" both in relative and absolute terms exceeded that of "outpatients". For dementia patients both the relative risk and more notably the absolute risk was high, and the excess mortality was significantly raised in both sexes and all age-groups. It was again greatest during the first year after registration. Deaths of patients with schizophrenia and paranoid psychosis exceeded those expected by 40 per cent and the excess was limited to the first year after registration and occurred predominantly among men.

Use of External Examinations to Involve Faculty in Curriculum Content Evaluation

A methodfor curriculum evaluation that utilizes content and student performance analyses of both an external certifying exam (National Board, Part 1, Behavioral Sciences) and curricular components (courses in behavioral sciences) is reported. Illustrative findings demonstrate that a multidisciplinary grouping of courses can satisfactorily cover content deemed important by external judges and by internal course directors and instructors. In addition, areas of redundancy as well as omissions can be identifiedfor discussion by curriculum and course planners. Advantages of using this external certifying examfor review and evaluation of curriculum components include readily accessible review copies, student performance comparison data, a national panel of content experts, localfaculty interest in the certifying examination, and provision of a common framework for curriculum review for any department.

Evaluation of methods for exploring ill-defined problems

Methods for exploring ill-defined problems are now being developed. Claims for these methods are frequently based on analogies with optimization processes. More persuasive backing for these claims can be obtained by conducting experiments that consider both the analyst or decisionmaker and the method. The most fruitful experimental design is for each subject to apply one method to one problem. Solution quality is then assessed by external judges, using the solutions from all the subjects as a source of knowledge about the ill-defined problem.

Improvement Ratings by External Judges and Psychotherapists

Short (10 co 30 min.) pre- and posc-therapy typescripts of 24 University Mental Hygiene Clinic clients were used. These students were seen by eight therapiscs, each carrying three cases. A trained interviewer conducted both the pre- and post-interviews, the interviews being conducted approximately three and one-half months after the beginning of psychotherapy. Typescripts were submitted to two highly experienced clinicians who quartiled the cases according to improvement in the student's ability to cope with his own life, to resolve those problems which are most irnportanc to him. A third judge decided disagreements. Judges worked blindly with only pre and post typescripcs. Each psychotherapist independently quartiled his own three cases in terms of his experience with cther clients. The psychotherapists varied greatly on such dimensions as age, professional experience, activity, interpreciveness, and incer- nal-external orientation to life. The two primary external judges differed theo- retically in that one is a Jungian, the other a Personal Construct theorist. Structure of intake intmvtew.-The interviewer described the functions of the Mencal Hygiene Clinic and the general nature of the research project. Then he covered the following topics: The problem (s) as the client sees thm: How well does the client describe them? What relationships does he see between different aspects of his life? Does he perceive any background causes of rhe present problem(s), even though he may be uncertain about them? Can he report a specific episode which recently occurred which is relevant to his prob- lem(~) ? Everyday functioning: How is he performing academically? Does he see any correlates to his academic difficulties? How does he relate to his friends and peer group? What is the nature of his dorm and dating relationships? Is he aware of the feelings of his friends? Is he sensitive to them? How does he per- - ceive them? How does he relate to his superiors and to his family? How com- fortable does he feel in relationship to his superiors and family? Can he com- municate with them and in what manner? What is the nature of his dependency