Interjudge Agreement Research Articles

Validity and reliability of the Spanish version of the oral assessment guide (OAG) in cancer patients

The Oral Assessment Guide (OAG) is a widely used tool designed for evaluating problems of oral mucous in cancer patients, but it has not been validated in Spanish. The aim of this work is to translate and validate into Spanish the scale of the OAG designed by Eilers. The translation process was carried out using the method of back-translation by bilingual translators. The study was carried out with cancer patients, both outpatient and inpatients, of the Hematology/Oncology Department and with oncology nurses. The following psychometric properties of the OAG were evaluated: internal consistency, concurrent validity with WHO's mucositis scale, interjudge agreement between two different nurses. The perception of patients and nurses on the use of the OAG was also assessed. An adequate Spanish version of the OAG was obtained. All the participants (n=40) completed the study. Internal consistency measured by Cronbach's alpha was 0.71 and interjudge agreement obtained a moderate to good Kappa index in the majority of items (k=0.4-0.81), except in "tongue and gums" (k=0.33-0.37). Concurrent validity with WHO mucositis scale was acceptable (r=0.458). All the nurses (n=6) considered that the scale was easy to understand and useful in clinical practice. The patients said that oral evaluation with the scale did not cause them discomfort. The Spanish version of the OAG is a valid and reliable instrument in cancer patients. It is a scale that is easy to use in clinical practice and is well accepted by patients.

A Tutorial on Reliability Testing in AAC Language Sample Transcription and Analysis

Establishing reliability is an essential step in language sample transcription and analysis. This tutorial provides an illustration of replicable procedures for reliability testing during transcription and analysis of language samples generated by people who use augmentative and alternative communication (AAC) systems. Statistical measures used for testing agreement between raters coding categorical data are summarized. Detailed procedures for reliability testing in AAC language sample transcription and analysis are provided, beginning with the collection of raw language sample data. Procedures include guidelines for (a) establishing inter-judge agreement during the transcription process, and (b) using Cohen's kappa to establish inter-rater reliability during deeper analysis of transcribed utterances. All procedures are demonstrated in a case example using language samples from children who use AAC.

Uncertainty and agreement regarding the role of flow diversion in the management of difficult aneurysms.

The role of flow diversion in the management of aneurysms remains unknown. We sought to evaluate the community agreement regarding indications for flow diversion. A portfolio of 35 difficult aneurysm cases was sent to 40 clinicians with varying backgrounds and experience. Responders were asked whether they considered flow diversion a treatment option, whether other options were possible, whether recruitment in a randomized trial would be considered, and to select their final choice. Agreement was studied by using κ statistics. Decisions for flow diversion were more frequent (n = 300, 39%) than decisions to coil (n = 163, 21.2%), to observe (n = 121, 15.7%), to occlude the parent vessel (n = 102, 13.2%), or to clip (n = 66, 8.6%). Sidewall aneurysm morphology was associated with flow diversion as the final choice (P = .001). Interjudge agreement was fair at best (κ <0.3) for all cases and all judges, despite high certainty levels (range, 7.2-8.9 ± 2.0 on a 0-10 scale). Agreement was no better within specialties or with more experience. All patients were judged to have other treatment options. Judges were willing to offer trial participation in 417 of 741 (56.3%) scenarios, more frequently when the aneurysm was sidewall (P = .001) or in the anterior circulation (P = .028). Individuals did not agree regarding the indications for flow diversion. There is sufficient uncertainty to justify trials designed to protect patients from the potential risks of premature adoption of an innovation.

Computer-based personality judgments are more accurate than those made by humans

Judging others' personalities is an essential skill in successful social living, as personality is a key driver behind people's interactions, behaviors, and emotions. Although accurate personality judgments stem from social-cognitive skills, developments in machine learning show that computer models can also make valid judgments. This study compares the accuracy of human and computer-based personality judgments, using a sample of 86,220 volunteers who completed a 100-item personality questionnaire. We show that (i) computer predictions based on a generic digital footprint (Facebook Likes) are more accurate (r = 0.56) than those made by the participants' Facebook friends using a personality questionnaire (r = 0.49); (ii) computer models show higher interjudge agreement; and (iii) computer personality judgments have higher external validity when predicting life outcomes such as substance use, political attitudes, and physical health; for some outcomes, they even outperform the self-rated personality scores. Computers outpacing humans in personality judgment presents significant opportunities and challenges in the areas of psychological assessment, marketing, and privacy.

What to Expect in the Intake Interview? Impression Management Tactics of Psychotherapy Patients

The therapeutic alliance consists of a mutual dependency between patient and therapist. Whereas earlier studies have focused on the therapists' behavioral influence, the present study examined patients' impression management tactics. The motivation to manage the impressions one has on others is particularly strong during first contact. Patients' behavioral influence was thus examined in the intake interview. Twelve possible impression management tactics were defined on the basis of theoretical conceptions of the therapeutic alliance and discussions with practicing psychotherapists. After a comprehensive training, judges rated 60 videotaped interviews. Interjudge agreement was fair to good. Influence attempts could be observed in roughly 30% of all patients' utterances. The most frequent tactics were Supplication, Provoking a response from the therapist, and Self-promotion. Patients could be grouped into three different clusters of tactic employers: Negative self-presenters, positive self-presenters, and response provokers. Male and female patients did not differ with respect to the total amount of tactics used and to the choice of specific tactics. However, when the therapist was female, male patients used significantly more tactics overall and significantly more often the tactic Negative reports about third persons. Being sensitive to patients' behavioral influence can help therapists to better understand their interactional goals and to better tailor the therapeutic alliance.

Bolus Residue Scale: An Easy-to-Use and Reliable Videofluoroscopic Analysis Tool to Score Bolus Residue in Patients with Dysphagia.

Background. We aimed to validate an easy-to-use videofluoroscopic analysis tool, the bolus residue scale (BRS), for detection and classification of pharyngeal retention in the valleculae, piriform sinuses, and/or the posterior pharyngeal wall. Methods. 50 randomly selected videofluoroscopic images of 10 mL swallows (recorded in 18 dysphagia patients and 8 controls) were analyzed by 4 experts and 6 nonexpert observers. A score from 1 to 6 was assigned according to the number of structures affected by residue. Inter- and intrarater reliabilities were assessed by calculation of intraclass correlation coefficients (ICCs) for expert and nonexpert observers. Sensitivity, specificity, and interrater agreement were analyzed for different BRS levels. Results. Intrarater reproducibility was almost perfect for experts (mean ICC 0.972) and ranged from substantial to almost perfect for nonexperts (mean ICC 0.835). Interjudge agreement of the experts ranged from substantial to almost perfect (mean ICC 0.780), but interrater reliability of nonexperts ranged from substantial to good (mean 0.719). BRS shows for experts a high specificity and sensitivity and for nonexperts a low sensitivity and high specificity. Conclusions. The BRS is a simple, easy-to-carry-out, and accessible rating scale to locate pharyngeal retention on videofluoroscopic images with a good specificity and reproducibility for observers of different expertise levels.

Exemplarity measurement and estimation of the level of interjudge agreement for two categories of French red wines

Exemplarity measurements of wines by at least 20 judges are used to estimate the degree of interjudge (dis)agreement and to tell wines apart into two contrasting extremes. Two sets of French red wines – Pinot noir from Burgundy and Cabernet franc from the Loire Valley – are studied separately but by the same approach. Short-listing criteria are used to collate 40 starting-price and middle-range wines for each set differing a priori in olfactory terms. Wine professionals assess their local wines first orthonasally and then, independently, by global evaluation. A pool of descriptive and inferential statistics indicates there is generally neither complete divergence nor real agreement among judges. For Burgundy Pinot noir, the weak agreement observed and measured for orthonasal evaluation strengthens slightly for global evaluation. Contrariwise, for Loire Valley Cabernet franc wines, agreement is poorer for global evaluation than for olfactory evaluation. With orthonasal evaluation, responses are more consistent for Cabernet franc than for Pinot noir, whereas for global evaluation, the levels of interjudge agreement are of the same order for both sets of wines. The subjectivity of sensory responses is interpreted for each situation. The personal judgment of exemplarity (or typicality) may therefore be defined as a demanding cognitive decision varying with the circumstances of the experiment and responding to a process of perceptual categorization based on previous knowledge and on an intuitive comparison between a sample and an abstract but conscious image of the category. Agreement among judges is sufficient to bipolarize the wines. As the power to discriminate among the wines is related to the level of interjudge agreement, bipolarization is most marked for the Cabernet franc wines evaluated by olfactory evaluation.

Agreement on the perception of moral character.

This study tested for inter-judge agreement on moral character. A sample of students and community members rated their own moral character using a measure that tapped six moral character traits. Friends, family members, and/or acquaintances rated these targets on the same traits. Self/other and inter-informant agreement was found at the trait level for both a general character factor and for residual variance explained by individual moral character traits, as well as at the individual level (judges agreed on targets' "moral character profiles"). Observed inter-judge agreement constitutes evidence for the existence of moral character, and raises questions about the nature of moral character traits.

Impact of inter-judge agreement on perceptual judgment of nasality.

To investigate the effect of perceptual inter-judge agreement of hypernasality on velopharyngeal (VP) closure prediction. Two logistic regression models were developed aiming to verify the possibility of predicting the VP closure using the following characteristics: rating of VP closure (adequate, borderline, inadequate), determined by the pressure-flow technique, degree of hypernasality (absent, mild, moderate, severe), and the presence/absence of nasal air emission and nasal rustle determined perceptually by three experienced speech language pathologists. In the first model, 100 speech samples with a moderate agreement rate of hypernasality (kappa coefficient: 0.41) were used. In the second model, 43 speech samples with a perfect agreement among judges were included. The χ²-test was used to compare the models (p≤0.05). In the first model, 65 of the 100 samples were rated in the correct VP closure category, with 42 adequate and 23 inadequate. The borderline VP closure was not predicted. The second model rated 31 of the 43 samples in the correct category, with 21 adequate VP closure, 5 in the borderline VP closure, and 5 inadequate. There was no difference (p=0.526) between the two models. However, the second model showed a higher proportion of accuracy (7%) than the first one, and it has also predicted the borderline VP closure. These results showed the importance of high index of inter-judge agreement when using subjective parameters of speech evaluation, especially when compared to an instrumental evaluation. This suggests the need for strategies for training and calibration of judges in the perceptual judgment to improve the reliability of auditory-perceptual assessment.

Ocorrência de ceceio em fricativas vozeadas e não vozeadas em crianças com fissura labiopalatina operada

Objetivo investigar se o ceceio, quando identificado, difere entre as fricativas alveolares não vozeadas e vozeadas produzidas por crianças com fissura labiopalatina operada. Métodos estudo prospectivo, em que frases constituídas pelas consoantes [s] e [z] produzidas por 32 crianças com fissura labiopalatina operada (idade média, 8 anos, 8 meses) foram selecionadas de um banco de dados e posteriormente julgadas auditivamente. Todas as crianças apresentavam relação inter-arcos alteradas, conforme avaliação ortodôntica realizada por três ortodontistas (concordância inter-juiz quase perfeita, kappa= 0.81), a partir da análise de modelos de gesso. Três fonoaudiólogas julgaram auditivamente as produções áudio gravadas. A concordância inter-juízes variou entre 56% e 78% e entre 59% e 93% para as frases constituídas de [s] e [z], respectivamente. Resultados o ceceio foi identificado em 69% das crianças e, particularmente, em 72% e 50% das produções envolvendo [s] e [z], respectivamente. Houve diferença significante entre os julgamentos para as fricativas [s] e [z], com maior ocorrência de ceceio em [s]. Conclusões deformidades dentofaciais podem favorecer a ocorrência do ceceio na população com fissura labiopalatina. A maior ocorrência do ceceio em [s] em comparação à [z], a partir da identificação auditiva, pode ser justificado por razões acústicas e/ou articulatórias. Sugere-se que o ceceio é dependente do contexto fonético-fonológico da frase devendo o mesmo ser considerado para fins clínicos e de pesquisa.

Event- and interval-based measurement of stuttering: a review.

Event- and interval-based measurements are two different ways of computing frequency of stuttering. Interval-based methodology emerged as an alternative measure to overcome problems associated with reproducibility in the event-based methodology. No review has been made to study the effect of methodological factors in interval-based absolute reliability data or to compute the agreement between the two methodologies in terms of inter-judge, intra-judge and accuracy (i.e., correspondence between raters' scores and an established criterion). To provide a review related to reproducibility of event-based and time-interval measurement, and to verify the effect of methodological factors (training, experience, interval duration, sample presentation order and judgment conditions) on agreement of time-interval measurement; in addition, to determine if it is possible to quantify the agreement between the two methodologies The first two authors searched for articles on ERIC, MEDLINE, PubMed, B-on, CENTRAL and Dissertation Abstracts during January-February 2013 and retrieved 495 articles. Forty-eight articles were selected for review. Content tables were constructed with the main findings. Articles related to event-based measurements revealed values of inter- and intra-judge greater than 0.70 and agreement percentages beyond 80%. The articles related to time-interval measures revealed that, in general, judges with more experience with stuttering presented significantly higher levels of intra- and inter-judge agreement. Inter- and intra-judge values were beyond the references for high reproducibility values for both methodologies. Accuracy (regarding the closeness of raters' judgements with an established criterion), intra- and inter-judge agreement were higher for trained groups when compared with non-trained groups. Sample presentation order and audio/video conditions did not result in differences in inter- or intra-judge results. A duration of 5 s for an interval appears to be an acceptable agreement. Explanation for high reproducibility values as well as parameter choice to report those data are discussed. Both interval- and event-based methodologies used trained or experienced judges for inter- and intra-judge determination and data were beyond the references for good reproducibility values. Inter- and intra-judge values were reported in different metric scales among event- and interval-based methods studies, making it unfeasible to quantify the agreement between the two methods.

The reliability of a severity rating scale to measure stuttering in an unfamiliar language

With increasing multiculturalism, speech-language pathologists (SLPs) are likely to work with stuttering clients from linguistic backgrounds that differ from their own. No research to date has estimated SLPs’ reliability when measuring severity of stuttering in an unfamiliar language. Therefore, this study was undertaken to estimate the reliability of SLPs’ use of a 9-point severity rating (SR) scale, to measure severity of stuttering in a language that was different from their own. Twenty-six Australian SLPs rated 20 speech samples (10 Australian English [AE] and 10 Mandarin) of adults who stutter using a 9-point SR scale on two separate occasions. Judges showed poor agreement when using the scale to measure stuttering in Mandarin samples. Results also indicated that 50% of individual judges were unable to reliably measure the severity of stuttering in AE. The results highlight the need for (a) SLPs to develop intra- and inter-judge agreement when using the 9-point SR scale to measure severity of stuttering in their native language (in this case AE) and in unfamiliar languages; and (b) research into the development and evaluation of practice and/or training packages to assist SLPs to do so.

Uncertainty and agreement in the management of unruptured intracranial aneurysms

The management of unruptured intracranial aneurysms remains controversial. The goal of this study was to evaluate the clinical community agreement in decision making regarding unruptured intracranial aneurysms. A portfolio of 41 cases of unruptured intracranial aneurysms with angiographic images, along with a short description of the patient presentation, was sent to 28 clinicians (16 radiologists and 12 surgeons) with varying years of experience in the management of unruptured intracranial aneurysms. Five senior clinicians responded twice at least 3 months apart. Nineteen cases (46%) were selected from patients recruited in the Canadian UnRuptured Endovascular versus Surgery trial, an ongoing randomized comparison of coil embolization and clip placement. For each case, the responder was to first choose between 3 treatment options (observation, surgical clip placement, or endovascular coil embolization) and then indicate their level of certainty on a quantitative 0-10 scale. Agreement in decision making was studied using κ statistics. Decisions to coil were more frequent (n = 612, 53%) than decisions to clip (n = 289, 25%) or to observe (n = 259, 22%). Interjudge agreement was only fair (κ = 0.31 ± 0.02) for all cases and all judges, despite substantial intrajudge agreement (range 0.44-0.83 ± 0.10), with high mean individual certainty levels for each case (range 6.5-9.4 ± 2.0 on a scale of 0-10). Agreement was no better within specialties (surgeons or radiologists), within capability groups (those able to perform endovascular coiling alone, surgical clipping alone, or both), or with more experience. There was no correlation between certainty levels and years of experience. Agreement was lower when the cases were taken from the randomized trial (κ = 0.19 ± 0.2) compared with nontrial cases (κ = 0.35 ± 0.2). Individuals do not agree regarding the management of unruptured intracranial aneurysms, even when they share a background in the same specialty, similar capabilities in aneurysm management, or years of practice. If community equipoise is a necessary precondition for trial participation, this study has found sufficient uncertainty and disagreement among clinicians to justify randomized trials.

Reliability of brain-computer interface language sample transcription procedures.

We tested the reliability of transcribing language samples of daily brain-computer interface (BCI) communication recorded as language activity monitoring (LAM) logfiles. This study determined interrater reliability and interjudge agreement for transcription of communication of veterans with amyotrophic lateral sclerosis using a P300-based BCI as an augmentative and alternative communication (AAC) system. KeyLAM software recorded logfiles in a universal logfile format during use of BCI-controlled email and word processing applications. These logfiles were encrypted and sent to our laboratory for decryption, transcription, and analysis. The study reports reliability results on transcription of 345 daily logfile samples. The procedure was found to be accurate across transcribers/raters. Frequency of agreement ratios of 97.6% for total number of words and 93.5% for total utterances were found as measures of interrater reliability. Interjudge agreement was 100% for both measures. The results indicated that transcribing language samples using LAM data is highly reliable and the fidelity of the process can be maintained. LAM data supported the transcription of a large number of samples that could not have been completed using audio and video recordings of AAC speakers. This demonstrated efficiency of LAM tools to measure language performance benefits to BCI research and clinical communities.

The Effect of the Raters' Marginal Distributions on Their Matched Agreement: A Rescaling Framework for Interpreting Kappa

Cohen's κ measures the improvement in classification above chance level and it is the most popular measure of interjudge agreement. Yet, there is considerable confusion about its interpretation. Specifically, researchers often ignore the fact that the observed level of matched agreement is bounded from above and below and the bounds are a function of the particular marginal distributions of the table. We propose that these bounds should be used to rescale the components of κ (observed and expected agreement). Rescaling κ in this manner results in κ′, a measure that was originally proposed by Cohen (1960) and was largely ignored in both research and practice. This measure provides a common scale for agreement measures of tables with different marginal distributions. It reaches the maximal value of 1 when the judges show the highest level of agreement possible, given their marginal disagreements. We conclude that κ′ should be used to measure the level of matched agreement contingent on a particular set of marginal distributions. The article provides a framework and a set of guidelines that facilitate comparisons between various types of agreement tables. We illustrate our points with simulations and real data from two studies—one involving judges' ratings of baseball players and one involving ratings of essays in high-stakes tests.

Reliability and Validity of Nasality Ratings between a Monolingual and Bilingual Listener for Speech Samples from English-Spanish-Speaking Children

Objective: This study sought to determine if a monolingual English listener could rate nasality in English and in Spanish with the same proficiency as a bilingual English-Spanish listener, and to compare nasalance scores with nasality ratings. Patients and Methods: Speakers for this study were 26 bilingual English-Spanish-speaking children. Speech samples and nasalance scores were obtained simultaneously as each speaker recited one English sentence and one Spanish sentence. A monolingual listener and a bilingual listener rated nasality. Results: For the English sentences, the intrajudge correlation coefficient was r = 0.89 for the monolingual listener and r = 0.89 for the bilingual listener. For the Spanish sentences, the intrajudge correlation coefficient was r = 0.91 for the monolingual listener and r = 0.92 for the bilingual listener. Interjudge agreement was r = 0.86 for rating English sentences and r = 0.78 for rating Spanish sentences. All correlation coefficients were significant (p < 0.001). The correlation coefficients between nasality ratings and nasalance scores were essentially the same for both listeners and both languages. Conclusion: A monolingual and a bilingual judge had high agreement on ratings of nasality for English and Spanish speech. The relationship between nasalance and nasality was not different across languages.

Relationship between Occlusion and Lisping in Children with Cleft Lip and Palate

The literature suggests that individuals with history of cleft lip and palate who present with midfacial growth deficiency are at higher risk of presenting lisping. The relationship between distortions during production of linguoalveolar fricative sounds and the severity of malocclusion, however, has not been established for the population with cleft. To study the association between lisping and dental arch relationship. Speech samples and dental arch casts were obtained from 106 children with operated unilateral cleft lip and palate (UCLP) during the stage of mixed dentition and before orthodontic treatment. Videotaped productions of the phrase /u saci saiw sedu/ were rated by speech-language pathologists for the identification of lisping during [s]. Dental arch casts were rated by orthodontists using the Goslon Yardstick and the Five-Year Index to establish dental arch relationship. Multiple logistic regression showed no significant association between lisping and dento-occlusal index (p = .802) and age (p = .662). Substantial interjudge agreement during auditory-perceptual ratings was found (kappa = .63). Almost perfect agreement was found between orthodontists while establishing the dental arch relationship (kappa = .81). This study failed to reveal an association between lisping and dental arch relationship in children with operated UCLP. Multiple variables may play a role in determining occurrence of lisping, warranting further investigation.

Changes in Perceptions of George W. Bush’s Personality in the Wake of the September 11 2001 World Trade Center Attacks

Using data gathered just before and just after the September 11th terrorist attacks, we examine how perceptions of Bush’s personality changed in the following two weeks. Fifty participants provided ratings of Bush using the California Q-sort at various times before (including immediately before) and after the attacks. At each time interjudge agreement was strong. There was general consistency between the pre- and post-attack assessments, but the common view of Bush shifted in several important ways. Consistent with his soaring popularity, the changes were toward more positive perceptions, even for characteristics unrelated to the attacks. Findings are discussed in terms of possible mechanisms driving the changes in perception. These findings, which are based on careful assessments conducted shortly before the emergence of any hint of what was to come, provide a unique perspective on changes in Bush’s image as they unfolded in the immediate wake of the terrorist attacks.

Acoustic cues to prominence in children's speech

Previous work indicates that phrasal prominence patterns are not yet adult-like in 6-year-old children's speech. Perception of prominence in English is influenced by temporal patterning, amplitude changes, and fundamental frequency variations across the phrase. This study examined the relative weightings of these cues to prominence in adult judgments of 6-year-old children and college-aged adults' speech. Eleven adult judges listened to two—word phrases produced by 25 children and 25 adults in a counting task designed to induce stress shift (thirtéen banana versus thirtéen barbeque). The judges decided which word was most prominent in the phrase: number, noun, or equal prominence. Although the number word was always judged to be prominent regardless of speakers' ages or context, initial results indicate that agreement between judges varied systematically with different cues in children and adults' speech. More judges agreed that the number word was prominent in children's speech when the first syllable of this word was produced with an especially high F0 and long duration, but only the relative intensity of the initial syllable predicted inter-judge agreement for adult speech. The results have implications for understanding the development of cue integration in the prominence production. [Work supported by NIH/R01HD061458.]

Perceptual and acoustic reliability estimates for the Speech Disorders Classification System (SDCS)

A companion paper describes three extensions to a classification system for paediatric speech sound disorders termed the Speech Disorders Classification System (SDCS). The SDCS uses perceptual and acoustic data reduction methods to obtain information on a speaker's speech, prosody, and voice. The present paper provides reliability estimates for the two perceptual methods (narrow phonetic transcription; prosody-voice coding) and the acoustic analysis methods the SDCS uses to describe and classify a speaker's speech competence, precision, and stability. Speech samples from 10 speakers, five with significant motor speech disorder and five with typical speech, were re-measured to estimate intra-judge and inter-judge agreement for the perceptual and acoustic methods. Each of the speakers completed five speech tasks (total = 50 datasets), ranging in articulatory difficulty for the speakers, with consequences for the difficulty level of data reduction. Point-to-point percentage of agreement findings for the two perceptual methods were as high or higher than reported in literature reviews and from previous studies conducted within the laboratory. Percentage of agreement findings for the acoustics tasks of segmenting phonemes, editing fundamental frequency tracks, and estimating formants ranged from values in the mid 70% to 100%, with most estimates in the mid 80% to mid 90% range. Findings are interpreted as support for the perceptual and acoustic methods used in the SDCS to describe and classify speakers with speech sound disorders.