Indices Of Interrater Agreement Research Articles

Reliability and Interrater Agreement: Development and Validation of a Universal Values in Sport Instrument

Universal values are one of the greatest aspects that can still be obtained through various sporting activities. Therefore, it is necessary to re-emphasize how this aspect can be assessed. This research aims to develop and test the validity of an instrument measuring universal values in sport, which includes five dimensions. With this instrument, it can later be used by physical education teachers in schools to find out the impact of learning carried out in relation to universal values. This is a descriptive review of inter-rater agreement and inter-rater reliability indices. It outlines the practical application and interpretation of these indices in research. The inter-rater agreement index assesses the extent of the responses of 7 expert raters. The data on agreement between raters and the reliability index were then processed to determine the validity and reliability results of the universal values in sports instrument being developed. The results of the research explain that the average validity value of each dimension was tested and the following results were obtained: Ethics and Morals (0.784), Knowledge and Understanding (0.805), Social (0.825), Psychological (0.855), and Leadership and Organization (0.835). The total reliability value of the five dimensions is 0.990, which shows a very high level of consistency and stability of this instrument. These results indicate that the instrument developed has strong reliability and validity in measuring various aspects of universal values in sports, so that it can be used effectively to support character building and skills development through sports activities. Keywords: Development, validation, interrater, reliability agreement, instruments, universal values.

Managers' Micro-Communities Matter: The Impact of Clinical Supervision Team on Therapist Perception of the Organization.

Positive organizational climate - employee perceptions of their work environment and the impact of this environment on well-being and functioning - is associated with desirable organizational and client-level outcomes in mental health organizations. Clinical supervisors are well-positioned to impact organizational climate, as they serve as intermediaries between higher-level administrators who drive the policies and procedures and the therapists impacted by such decisions. This cross-sectional study examined the role of clinical supervisors as drivers of therapist perceptions of organizational climate within supervisory teams. Specifically, the present study investigated: (1) shared perceptions of organizational climate among therapists on the same supervisory team; (2) predictors of therapist climate perceptions. Eighty-six therapists were supervised by 22 supervisors. Indices of interrater agreement and interrater reliability of therapists on the same supervisory team were examined to determine shared or distinct perceptions of organizational climate. Multi-level models were used to examine whether supervisor attitudes towards evidence-based practices and therapist perceptions of supervisor communication predicted perceived organizational climate. Results showed perceptions of organizational cohesion and autonomy were shared among therapists on the same supervisory team and distinct from therapists on different supervisory teams. Therapist perceptions of their supervisor's communication was positively associated with perceptions of organizational cohesion and autonomy. These findings align with emerging evidence that middle managers shape their employees' experience of their work environment through communication strategies. These findings also point to the potential for intervening at lower organizational levels to improve overall organizational climate.

Cross-cultural adaptation of the Skin Cancer Index into Brazilian Portuguese for patients with cervicofacial nonmelanoma skin cancer

PurposeThis study aimed to translate the Skin Cancer Index (SCI) into Portuguese, adapt it for Brazilian culture, and clinically validate it.MethodsA five-stage cross-cultural adaptation model was followed, with subsequent clinical validation. Inter-rater agreement was assessed using the content validity index (CVI). The hypothesis of the non-inferiority of the CVI at 80% probability level was evaluated using an exact binomial test. We used Spearman’s rank-order and Pearson’s product–moment correlation analysis, internal consistency using McDonald’s ω and Cronbach’s α metric, and construct validity using confirmatory factor analysis. The factorial model was validated using the chi-squared test, root mean square error of approximation (RMSEA), comparative fit index (CFI), and standardized root mean square residual (SRMR).ResultsThe first stage yielded two independent translations. After synthesis, back-translation, and review, the prefinal version was tested on 40 patients. Inter-rater agreement indices on content validity were significantly higher than 80% (p < 0.05). The SCI remained stable, and the Spearman’s rank-order (rs), Pearson product–moment (r), and intraclass correlation coefficients were > 0.9, indicating excellent reliability. The reliability of McDonald’s ω was considered ideal (> 0.8) in all subdimensions and scale. Cronbach’s α was considered ideal in the “Emotional” and “Social” subdimensions and scale. Construct validity was observed in all subdimensions and scale through the criteria (χ2) p value > 0.05, RMSEA < 0.08, CFI ≥ 0.9, and SRMR ≤ 0.08.ConclusionThe cross-cultural adaptation of the SCI to Portuguese for Brazilian culture showed content validity and reliability, contributing to quality of life assessment in patients with NMSC.

Improving content validity evaluation of assessment instruments through formal content validity analysis.

Content validity is defined as the degree to which elements of an assessment instrument are relevant to and representative of the target construct. The available methods for content validity evaluation typically focus on the extent to which a set of items are relevant to the target construct, but do not afford precise evaluation of items' behavior, nor their exhaustiveness with respect to the elements of the target construct. Formal content validity analysis (FCVA) is a new procedure combining methods and techniques from various areas of psychological assessment, such as (a) constructing Boolean classification matrices to formalize relationships among an assessment instrument's items and target construct elements, and (b) computing interrater agreement indices. We discuss how FCVA can be extended through the implementation of a Bayesian procedure to improve the interrater agreement indices' accuracy (Bayesian formal content validity analysis [B-FCVA]). With respect to extant methods, FCVA and B-FCVA can provide a great amount of information about content validity while not demanding much more work for authors and experts. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

BUDDHIST SNAKES LADDERS PORTABLE AS LEARNING MEDIA FOR BUDDHIST SUNDAY SCHOOL'S STUDENTS

This study aimed to develop a game of snakes and ladders as a learning media through the 4-D Thiagarajan model (Define, Design, Development, and Disseminate). This study's data was collected using an interview guide and questionnaires. The data was analyzed using Aiken's V, Gregory, and descriptive statistics. Aiken's V analysis showed that all of the questions were valid. Gregory's analysis showed that the game's manual book had an excellent readability level with an interrater agreement index of 0.857. The test results showed that the games developed were practical and can be used as learning media.

Assessment of Rib Fracture in Acute Trauma Using Automatic Rib Segmentation and a Curved, Unfolded View of the Ribs: Is There a Saving of Time?

Introduction: The fast and accurate diagnosis of rib fractures in polytrauma patients is important to reduce the mortality rate and relieve long-term pain and complications. Aim: To evaluate the diagnostic accuracy and potential time savings using automatic rib segmentation and a curved, unfolded view for the detection of rib fractures in trauma patients. Methods: The multidetector computed tomography raw data of 101 consecutive polytrauma patients (72 men; mean age 45 years, age range 17 to 84 years) admitted to a university hospital were retrospectively post-processed to generate a curved, unfolded view of the rib cage. No manual corrections were performed. Patients with reconstruction errors and movement artifacts were excluded from further analysis. All fractures were identified and classified by the study coordinator using the original data set. Two readers (reader 1 and reader 2) evaluated the original axial sections and the unfolded view, separately. The fracture locations, fracture type, and reading times were recorded. Sensitivity and specificity were calculated on a per-rib basis using a ratio estimator. Cohen’s Kappa was calculated as an index of inter-rater agreement. Results: 26 of 101 patients (25.7%) were excluded from further analysis owing to breathing artifacts (6.9%) or incorrect centerline computation in the unfolded view (18.8%). In total, 107 (5.9%) of 1800 ribs were fractured in 25 (33%) of 75 patients. The unfolded view had a sensitivity/specificity of 81%/100% (reader 1) and 71%/100% (reader 2) compared to 94%/100% (reader 1; p = 0.002/p = 0.754) and 63%/99% (reader 2; p < 0.001/p = 0.002). The sensitivity (reader 1; reader 2) was poor for buckled fractures (31%; 38%), moderate for undislocated fractures (78%; 62%), and good for dislocated fractures (94%; 90%). The assessment of the unfolded view was performed significantly faster than that of the original layers (19.5 ± 9.4 s vs. 68.6 ± 32.4 s by reader 1 (p < 0.001); 24.1 ± 9.5 s vs. 40.2 ± 12.7 s by reader 2 (p < 0.001)). Both readers demonstrated a very high interobserver agreement for the unfolded view (κ = 0.839) but only a moderate agreement for the original view (κ = 0.529). Conclusion: Apart from a relatively high number of incorrect centerline reconstructions, the unfolded view of the rib cage allows a faster diagnosis of dislocated rib fractures.

Perceived and ideal conflict cultures in academe: Creating the conflict cultures survey

AbstractFocusing on conflict at the organizational level, this study explores Conflict Culture Theory by (a) conceptualizing perceived and ideal conflict cultures, (b) creating and implementing the Conflict Cultures Survey, and (c) testing Gelfand, Leslie and Keller's (2008) proposed two‐dimensional model. Tenured and tenure‐track faculty at a large, American university (N = 346) completed the survey. Ideal conflict cultures varied little whereas perceived conflict cultures varied across departments, suggesting that ideal and perceived conflict cultures are distinct constructs. Multi‐level modeling and interrater agreement indices for the conflict culture variables provide evidence that conflict cultures exist and vary by department. Results supported the two‐dimensional model rather than one‐ or four‐dimensional models, suggesting that conflict cultures vary along two dimensions: agreeableness and activeness. Practical implications for Conflict Culture Theory and the Conflict Culture Survey include predicting job satisfaction and commitment, identifying bullying or workplace harassment norms, and establishing individual‐organizational fit.

A Scoring Procedure for Malignant Narcissism Based on Personality Inventory for DSM-5 Facets

The current study focuses on the development and validation of a scoring procedure for malignant narcissism using the Personality Inventory for DSM-5, a self-report measure of Criterion B from the Alternative Model for Personality Disorders. In Study 1, a prototype matching approach was used to aggregate ratings from 15 clinicians specializing in personality disorder treatment and/or assessment. Indices of inter-rater agreement and inter-rater reliability revealed high convergence as to the most important maladaptive facets for malignant narcissism. The scoring procedure, based on additive counts for score computation, included eleven Criterion B facets covering core features of malignant narcissism. Study 2 evaluated the criterion and incremental validity of the scoring procedure in a sample of 288 patients from a personality disorder treatment clinic, as well as in a sample of 1103 participants from the community. In both samples, results from nonparametric mean comparisons, receiver operating characteristic curves, bivariate Pearson correlations, and hierarchical multiple linear regressions showed significant associations between malignant narcissism and broader components of personality functioning, as well as with relevant emotional, relational, and/or behavioral features. This new scoring procedure is a simple and valid method for measuring malignant narcissism, and is suitable for clinical and research settings.

Knowledge creation elements for enhancing community resilience towards disaster: A Delphi study.

Knowledge capacity plays a vital role in building community resilience to disasters. However, the problem is that there is no resilience framework that integrates the knowledge creation process. This article introduces a new framework for increasing community resilience based on knowledge creation theory (KCT). This research aims to define the elements that support the Knowledge Creation for Community Resilience (KCCR) and to gain consensus from experts on these factors. This study was conducted using semi-structured interviews with five panellists and three rounds of Delphi technique to determine the assessment of 26 factors (including six additional factors) that have been identified by experts (30, 18 and 11 experts in rounds I, II and III, sequentially). The data analysis was carried out in several stages, and included Spearman’s Rank Correlation Coefficient, consensus appraisal and interrater agreement (IRA) statistical evaluation. The result of the agreement level (AL) analysis shows that the majority of the constructs (96.15%) are in the ‘moderate strong’ category. This study shows that there is a significant consensus (with IRA index [awg(1)] ranging from 0.529 to 1), and panellists confirm the significance of all the key constructs. Consensus was gained from experts on seven elements that support the KCCR. This study establishes a systematic, operational and multidimensional KCCR framework that combines the concepts of knowledge creation, community resilience and disaster preparedness. This framework can be used as a qualitative instrument or guidance to build community resilience based on knowledge creation and a quantitative tool for measuring community resilience in facing disasters.

Needs and barriers of medical personnel in Catalonian occupational health services

to identify the difficulties and needs of medical professionals in occupational health services in Catalonia with respect to carrying out their functions, as prescribed by law. Delphi study in three rounds. The selection of participants was carried out sepa - rately for in-house versus outsourced occupational health services. The first round included two open-ended questions: What difficulties do you encounter when performing your duties in your occupational health service? And, what are your needs in terms of being able to per - form your duties in your occupational health service? In the second and third rounds, partic - ipants ranked the categories identified in the first round according to their importance and relevance. We calculated an inter-rater agreement index (ICC) and an "index of concern (IP)". Among in-house occupational health services, the most highly ranked difficulties were case management of workers with limitations and the low quality of medical sur - veillance systems. Among outsourced occupational health services, the most notable dif - ficulties were the lack of communication with other organizations and the prioritization of economic and profitability criteria associated with the high work volume. For both types of services, there was a clear need to establish communication channels with other institu - tions or organizations such as primary care, mutual insurance companies, public medical evaluation units and public occupational health units. The different type of occupational health services reflects important differ - ences in the types of difficulties encountered, but not in needs, where there are more points in common.

Validation of a Novel 'Supportive Oral Care Protocol' (SOCP), a Model for Care in Head and Neck Cancer Patients at Tertiary Cancer Centre in India.

India has a huge burden of head and neck cancer and specifically oral cancer. Supportive oral care is not a standard of care in our population and is often neglected. Currently, there are no specific guidelines for such care in India which could be followed. The aim of this study is to validate a novel institutional supportive oral care protocol (SOCP) for head and neck cancer patients. This protocol is specific to our population developed for head and neck/dental oncology experts working in cancer centres to provide comprehensive care. This is a cross-sectional validity study. Fifteen dental oncology experts working in cancer centres/hospitals across India and six oncology experts from our centre were enrolled. All experts provided their inputs on 41 points of the SOCP. The data was analysed for item validity, content validity index and inter-rater agreement. The statistical analyses used were kappa measure for inter-rater agreement and content validity index for item-wise agreement. Out of 861 responses from all the reviewers, 91% agreed, 8.4% agreed with modification and 0.6% disagreed. The content validity index and agreement between reviewers ranged from 0.9 to 1 for kappa measure. The SOCP of our institution was shown to be a valid protocol. SOCP addresses oral and dental supportive care and rehabilitation as part of overall comprehensive care for head and neck cancer patients in our population.

Smartphone-Based Estimation of Item 3.8 of the MDS-UPDRS-III for Assessing Leg Agility in People With Parkinson's Disease.

Goal: In this paper we investigated the use of smartphone sensors and Artificial Intelligence techniques for the automatic quantification of the MDS-UPDRS-Part III Leg Agility (LA) task, representative of lower limb bradykinesia. Methods: We collected inertial data from 93 PD subjects. Four expert neurologists provided clinical evaluations. We employed a novel Artificial Neural Network approach in order to get a continuous output, going beyond the MDS-UPDRS score discretization. Results: We found a Pearson correlation of 0.92 between algorithm output and average clinical score, compared to an inter-rater agreement index of 0.88. Furthermore, the classification error was less than 0.5 scale point in about 80% cases. Conclusions: We proposed an objective and reliable tool for the automatic quantification of the MDS-UPDRS Leg Agility task. In perspective, this tool is part of a larger monitoring program to be carried out during activities of daily living, and managed by the patients themselves.

Nature of communication performance in non-traditional procurements in South Africa

Purpose Non-traditional procurement is thought to offer enhanced benefits over traditional contracts. However, several factors significantly impacting the optimization of the benefits have been suggested. Even though effective communication is ranked high among these factors, not enough empirical insights have been given to the nature of communication performance in non-traditional procurements in construction project delivery. The purpose of this paper is to focus on understanding the taxonomy of communication performance in non-traditional procurements in the South African Construction Industry, given the centrality of communication to team effectiveness, performance, project success and managerial efficiency. Design/methodology/approach Adopting a deductive research approach, an empirical questionnaire survey was conducted drawing on the practical communication experience of project teams engaged in projects using non-traditional procurement systems. Findings Using aggregated mean scores and inter-rater agreement index and Mann‒Whitney tests, the results revealed inaccuracies, barriers and procedural challenges as frequently occurring in the interpersonal and organizational communication in the adopted non-traditional procurement systems. Additionally, untimeliness and distorted information were noted to be occurring occasionally, whereas incomplete communication, misunderstood information and gatekeeping rarely occurred. Lastly, protocol-related communication challenges were deemed to have never occurred in their interpersonal and organizational communication in their project delivery. In the assessment of communication performance, the less frequently occurring assessment indicator is a testament to good communication performance, whereas the frequently occurring indicator represents a poor performance in the communication. Originality/value This systematic approach toward understanding the taxonomy of the interpersonal and organizational communication performance in non-traditional procurement is very significant toward communication performance improvement in the industry. The insight given by the findings may help construction project practitioners to focus their attention on the necessary aspects of communication in non-traditional procurements to optimize their benefits over traditional procurement models.

Translation and cross-cultural adaptation of the International Trauma Questionnaire for use in Brazilian Portuguese.

The most recent editions of diagnostic manuals have proposed important modifications in posttraumatic stress disorder (PTSD) criteria. The International Trauma Questionnaire (ITQ) is the gold-standard measurement for assessing PTSD and complex PTSD in accordance with the model of the 11th International Classification of Diseases (ICD-11). The aim of this study was to adapt the ITQ for the Brazilian context. The translation and cross-cultural adaptation of the ITQ for use in Brazilian Portuguese was performed in trauma research facilities in Porto Alegre, Rio de Janeiro and Belo Horizonte, Brazil. The adaptation followed five steps: (1) translation; (2) committee synthesis; (3) experts' evaluation through the content validity index (CVI) and assessment of interrater agreement though kappa statistics; (4) comprehension test with clinical and community samples (n = 35); and (5) final back-translation and authors' evaluation. Two independent translations were conducted. While working on a synthesis of these translations, the committee proposed changes in six items to adapt idiomatic expressions or to achieve a more accurate technical fit. Both the expert judges' evaluation (CVI > 0.7; k > 0.55) and the pretest in the target population (mean comprehension > 3) indicated that the adapted items were adequate and comprehensible. The final back-translation was approved by the authors of the original instrument. ITQ in its Brazilian Portuguese version achieved satisfactory content validity, thus providing a tool for Brazilian research based on PTSD models of the ICD-11.

Facing Climate Change: What Drives Internal Migration Decisions in the Karst Rocky Regions of Southwest China

Global climate change and its influence on human migration have caused heated debates. There is no consensus about the role of environmental change in shaping migration decisions. To amass more evidence and develop a deeper understanding of the relations between the environment and migration, this paper seeks to evaluate the importance of various drivers (economic, social, political, demographic, and environmental drivers) and determine the internal mechanism in the decision process. The Likert scale was used as the tool for measuring each respondent’s perception of the drivers, and the within-group interrater agreement index was used to express the survey data and to select the actual driving forces. As a result, economic, social, and political factors were strong forces that promoted migration directly, while demographic and environmental factors were moderate or weak forces that promoted migration indirectly. The migrants’ core consideration was to effectively reduce family risks and sustain their livelihoods by moving to a destination to improve their household income, keep their original social networks, and obtain housing allowances from the local government. Land degradation and meteorological disasters were rooted in the vulnerability and risks of a family, and these factors indirectly influenced the people’s decisions by affecting the socioeconomic drivers. We concluded that isolating the environmental drivers from other drivers underlying migration decisions is difficult. Additionally, the internal mechanism indicated that both environmental and non-environmental factors all have an impact on choice in different ways. Future policies should be aimed at increasing sustainable livelihoods and the social resilience of migrant families at a personal level, balancing the development levels of the original locations and destinations, and strengthening international cooperation to reduce the negative effects of climate change at the regional level.

Quantification of discrete behavioral components of the MDS-UPDRS

IntroductionThe Movement Disorder Society’s Unified Parkinson’s Disease Rating Scale (MDS-UPDRS) is the current gold standard means of assessing disease state in Parkinson’s disease (PD). Objective measures in the form of wearable sensors have the potential to improve our ability to monitor symptomology in PD, but numerous methodological challenges remain, including integration into the MDS-UPDRS. We applied a structured video coding scheme to temporally quantify clinical, scripted, motor tasks in the MDS-UPDRS for the alignment and integration of objective measures collected in parallel. Methods25 PD subjects completed two video-recorded MDS-UPDRS administrations. Visual cues of task performance reliably identifiable in video recordings were used to construct a structured video coding scheme. Postural transitions were also defined and coded. Videos were independently coded by two trained non-expert coders and a third expert coder to derive indices of inter-rater agreement. Results50 videos of MDS-UPDRS performance were fully coded. Non-expert coders achieved a high level of agreement (Cohen’s κ > 0.8) on all postural transitions and scripted motor tasks except for Postural Stability (κ = 0.617); this level of agreement was largely maintained even when more stringent thresholds for agreement were applied. Durations coded by non-expert coders and expert coders were significantly different (p < 0.05) for only Postural Stability and Rigidity, Left Upper Limb. ConclusionsNon-expert coders consistently and accurately quantified discrete behavioral components of the MDS-UPDRS using a structured video coding scheme; this represents a novel, promising approach for integrating objective and clinical measures into unified, longitudinal datasets.

Comparación entre metodologías para el diagnóstico microscópico de malaria

Introduction: As part of the pre-elimination plan for malaria in Colombia, it has been proposed to develop activities within the line of work: “Improve access and quality of malaria diagnosis”. Objective: To compare the methodology recommended by PAHO/WHO with that used in Colombia for the diagnosis of malaria. Materials and methods: Samples were collected and 88 slides were prepared for malaria diagnosis, under different scenarios according to the parameters to be evaluated. After duplicate mycroscopic reading, the respective variance calculations were performed for all possible staining comparisons with the two methods used (thick smear, combined thick smear), according to the staining (modified Romanowsky or Giemsa), with the result variable being the parasite density (500, 1,000, 5,000 and 10,000 parasites/μl of blood). Results: A Cohen kappa index of inter-rater agreement of 0.923 (95% CI: 0.768-1.078) was obtained. None of the factors (A: stain, B: methodology) or interactions (AB) had a statistically significant effect on the results with a 95% confidence level. Conclusion: Based on the results of the study, the preparation of two thick smears in the same slide stained with the modified Romanowsky stain is a suitable methodology for the diagnosis of malaria in Colombia, due to its technical characteristics, of storage, low cost, use and care.

Simulated patient training: Using inter-rater reliability to evaluate simulated patient consistency in nursing education

BackgroundSimulated patients (SPs) are frequently used for training nursing students in communication skills. An acknowledged benefit of using SPs is the opportunity to provide a standardized approach by which participants can demonstrate and develop communication skills. However, relatively little evidence is available on how to best facilitate and evaluate the reliability and accuracy of SPs' performances. AimThe aim of this study is to investigate the effectiveness of an evidenced based SP training framework to ensure standardization of SPs. The training framework was employed to improve inter-rater reliability of SPs. MethodsA quasi-experimental study was employed to assess SP post-training understanding of simulation scenario parameters using inter-rater reliability agreement indices. Two phases of data collection took place. Initially a trial phase including audio-visual (AV) recordings of two undergraduate nursing students completing a simulation scenario is rated by eight SPs using the Interpersonal Communication Assessments Scale (ICAS) and Quality of Discharge Teaching Scale (QDTS). In phase 2, eight SP raters and four nursing faculty raters independently evaluated students' (N=42) communication practices using the QDTS. ResultsIntraclass correlation coefficients (ICC) were >0.80 for both stages of the study in clinical communication skills. ConclusionThe results support the premise that if trained appropriately, SPs have a high degree of reliability and validity to both facilitate and evaluate student performance in nurse education.

An Overview of Interrater Agreement on Likert Scales for Researchers and Practitioners.

Applications of interrater agreement (IRA) statistics for Likert scales are plentiful in research and practice. IRA may be implicated in job analysis, performance appraisal, panel interviews, and any other approach to gathering systematic observations. Any rating system involving subject-matter experts can also benefit from IRA as a measure of consensus. Further, IRA is fundamental to aggregation in multilevel research, which is becoming increasingly common in order to address nesting. Although, several technical descriptions of a few specific IRA statistics exist, this paper aims to provide a tractable orientation to common IRA indices to support application. The introductory overview is written with the intent of facilitating contrasts among IRA statistics by critically reviewing equations, interpretations, strengths, and weaknesses. Statistics considered include rwg, , r′wg, rwg(p), average deviation (AD), awg, standard deviation (Swg), and the coefficient of variation (CVwg). Equations support quick calculation and contrasting of different agreement indices. The article also includes a “quick reference” table and three figures in order to help readers identify how IRA statistics differ and how interpretations of IRA will depend strongly on the statistic employed. A brief consideration of recommended practices involving statistical and practical cutoff standards is presented, and conclusions are offered in light of the current literature.

An Evaluation of Interrater Reliability Measures on Binary Tasks Using d-Prime.

Many indices of interrater agreement on binary tasks have been proposed to assess reliability, but none has escaped criticism. In a series of Monte Carlo simulations, five such indices were evaluated using d-prime, an unbiased indicator of raters' ability to distinguish between the true presence or absence of the characteristic being judged. Phi and, to a lesser extent, Kappa coefficients performed best across variations in characteristic prevalence, and raters' expertise and bias. Correlations with d-prime for Percentage Agreement, Scott's Pi, and Gwet's AC1 were markedly lower. In situations where two raters make a series of binary judgments, the findings suggest that researchers should choose Phi or Kappa to assess interrater agreement as the superiority of these indices was least influenced by variations in the decision environment and characteristics of the decision makers.