Generalizability Coefficients Research Articles

관찰·추천제에 의한 수학영재 선발 시 사용되는 교사추천서와 자기소개서 평가에 대한 다변량 일반화가능도 이론의 활용

This study provides an illustrative example of using the multivariate generalizability theory. Specifically, it investigates relative effects of each error source, and finds optimal measurement conditions for the number of items within each content domain that maximizes the reliability-like coefficients, such as a generalizability coefficient and an index of dependability. The method is based on teacher recommendation letters and self-introduction letters, using an analytic scoring method in the context of selection of mathematically gifted students by observation and nomination. This study analyzed data from the 2011 academic year in the science education institute for the gifted, which is attached to the university located in the Seoul metropolitan area. It should be noted that the optimal scoring structures of this study are not generalizable to other selection instruments. However, the methodology applied in this study can be utilized to find optimal measurement conditions for the number of raters, the number of content domains, and the number of items in other selection instruments self-developed by many institutions including: the education institutes for the gifted at provincial offices of education, gifted classes, and the science education institutes for the gifted attached to universities in general. In addition, the methodology will provide bases for making informed decisions in selection instruments of the gifted based on measurement traits.

Critical Synthesis Package: American Board of Internal Medicine (ABIM) Peer Assessment

OPEN ACCESSOctober 24, 2013Critical Synthesis Package: American Board of Internal Medicine (ABIM) Peer Assessment Matt J. Rustici, MD Matt J. Rustici, MD University of Colorado School of Medicine Google Scholar More articles by this author https://doi.org/10.15766/mep_2374-8265.9590 SectionsAbout ToolsDownload Citations ShareFacebookTwitterEmail AbstractThis Critical Synthesis Package contains: (1) a Critical Analysis of the psychometric properties and the application to health science education of the American Board of Internal Medicine (ABIM) Peer Assessment, and (2) a copy of the ABIM Peer Assessment instrument developed by Eric Holmboe, MD.The ABIM Peer Assessment is an 11-item instrument with responses recorded on a 9-point numerical rating scale with descriptive anchors. Higher scores indicate a more favorable assessment. The instrument was developed as a component of the ABIM re-certification process and is designed to assess practicing internal medicine physicians. Items assess ambulatory care skills, medical knowledge, management of multiple complex problems, problem solving, management of hospitalized patients, respect, integrity, psychosocial aspects of illness, compassion, responsibility and overall clinical skills. The instrument has demonstrated an acceptable generalizability coefficient (0.7) when 10–13 instruments are collected and seems to be minimally biased when peer evaluators are self-selected by the physician being evaluated. The instrument attempts to provide feedback to physicians from peers to help guide self-improvement plans and does not have any established minimum competency scores. In one study, scores of 100 practicing physicians were positively correlated to evaluations from residency directors but there is limited supporting data correlating scores on the instrument to other measures of physician competence. Educational Objectives To describe the purpose and basic properties of the American Board of Internal Medicine (ABIM) Peer Assessment, including number of items and scales, and psychometric properties;To describe the application of the ABIM Peer Assessment to the field of health sciences education;To evaluate the relative strengths and weaknesses of the ABIM Peer Assessment; andTo provide the ABIM Peer Assessment and supplemental materials to aid in its administration. Sign up for the latest publications from MedEdPORTAL Add your email below FILES INCLUDEDReferencesRelatedDetails FILES INCLUDED Included in this publication: ABIM Peer Assessment Instrument.pdf Critical Analysis of the American Board of Internal Medicine (ABIM) Peer Assessment.pdf To view all publication components, extract (i.e., unzip) them from the downloaded .zip file. Download editor’s noteThis publication may contain technology or a display format that is no longer in use. CitationRustici MJ. Critical Synthesis Package: American Board of Internal Medicine (ABIM) Peer Assessment. MedEdPORTAL. 2013;9:9590. https://doi.org/10.15766/mep_2374-8265.9590 Copyright & Permissions© 2013 Rustici. This is an open-access publication distributed under the terms of the Creative Commons Attribution-NonCommercial-Share Alike license.KeywordsEvaluationDREAMPhysiciansDirectory and Repository of Educational Assessment MeasuresPeer AssessmentClinical CompetenceEvaluation Studies as TopicInternal MedicineFeedbackProfessionalismHumanismAmerican Board of Internal MedicineABIMNon-Clinical SkillsClinical Skills Disclosures None to report. Funding/Support None to report. Loading ...

Comparing Diagnostic Performance and the Utility of Clinical Vignette-Based Assessment Under Testing Conditions Designed to Encourage Either Automatic or Analytic Thought

Although decades of research have yielded considerable insight into physicians' clinical reasoning processes, assessing these processes remains challenging; thus, the authors sought to compare diagnostic performance and the utility of clinical vignette-based assessment under testing conditions designed to encourage either automatic or analytic thought. This 2011-2012 multicenter randomized study of 393 clinicians (medical students, postgraduate trainees, and faculty) measured diagnostic accuracy on clinical vignettes under two conditions: one encouraged participants to give their first impression (FI), and the other led participants through a directed search (DS) for the correct diagnosis. The authors compared accuracy, feasibility, reliability, and relation to United States Medical Licensing Exam (USMLE) scores under each condition. A 2 (instructional condition) × 2 (vignette complexity) × 3 (experience level) analysis of variance revealed no difference in accuracy as a function of instructional condition (F[1,379] = 2.44, P = .12), but demonstrated the expected main effects of vignette complexity (F[1,379] = 965.2, P < .001) and experience (F[2,379] = 39.6, P < .001). Pearson correlations revealed greater associations between assessment scores and USMLE performance in the FI condition than in the DS condition (P < .001). Spearman-Brown calculations consistently indicated that alpha ≥ 0.75 could be achieved more efficiently under the FI condition relative to the DS condition. Instructions to trust one's first impres-sions result in similar performance when compared with instructions to consider clinical information in a systematic fashion, but have greater utility when used for the purposes of assessment.

The Bristol Radiology Report Assessment Tool (BRRAT): Developing a workplace-based assessment tool for radiology reporting skills

To review the development of a workplace-based assessment tool to assess the quality of written radiology reports and assess its reliability, feasibility, and validity. A comprehensive literature review and rigorous Delphi study enabled the development of the Bristol Radiology Report Assessment Tool (BRRAT), which consists of 19 questions and a global assessment score. Three assessors applied the assessment tool to 240 radiology reports provided by 24 radiology trainees. The reliability coefficient for the 19 questions was 0.79 and the equivalent coefficient for the global assessment scores was 0.67. Generalizability coefficients demonstrate that higher numbers of assessors and assessments are needed to reach acceptable levels of reliability for summative assessments due to assessor subjectivity. The study methodology gives good validity and strong foundation in best-practice. The assessment tool developed for radiology reporting is reliable and most suited to formative assessments.

Reliable assessment of general surgeons' non-technical skills based on video-recordings of patient simulated scenarios

BackgroundNontechnical skills are essential for safe and efficient surgery. The aim of this study was to evaluate the reliability of an assessment tool for surgeons' nontechnical skills, Non-Technical Skills for Surgeons dk (NOTSSdk), and the effect of rater training. MethodsA 1-day course was conducted for 15 general surgeons in which they rated surgeons' nontechnical skills in 9 video recordings of scenarios simulating real intraoperative situations. Data were gathered from 2 sessions separated by a 4-hour training session. ResultsInterrater reliability was high for both pretraining ratings (Cronbach's α = .97) and posttraining ratings (Cronbach's α = .98). There was no statistically significant development in assessment skills. The D study showed that 2 untrained raters or 1 trained rater was needed to obtain generalizability coefficients >.80. ConclusionsThe high pretraining interrater reliability indicates that videos were easy to rate and Non-Technical Skills for Surgeons dk easy to use. This implies that Non-Technical Skills for Surgeons dk (NOTSSdk) could be an important tool in surgical training, potentially improving safety and quality for surgical patients.

Fiabilidad y optimización del programa PROTODEBA v 1.0 para la observación de la Toma de Decisiones en Balonmano

The aim of this study was to development, and optimizes an observational tool call PROTODEBA v 1.0. This program helps video observation to analyze behavior in decision making in handball players assessing three dimensions (location, transfer and termination) at any time (lag between passing and passing) in different sequences of a game. The study evaluated five observers each one examined six players in eighteen plays of a game. The reliability of the study was conducted in two ways. On one hand, It was calculated the Kappa index of agreement among observers that show values higher than 0,70 for all pairs of observers. For another, it has been estimated generalizability coefficients for different types of decisions and different universes, obtaining values between 0,87 and 0,99. Results support the accuracy of PROTODEBA V1.0 as a tool of observation

Using Virtual-Reality Simulation to Assess Performance in Endobronchial Ultrasound

Background: For optimal treatment of patients with non-small cell lung carcinoma, it is essential to have physicians with competence in endobronchial ultrasound-guided transbronchial needle aspiration (EBUS-TBNA). EBUS training and certification requirements are under discussion and the establishment of basic competence should be based on an objective assessment of performance. Objectives: The aims of this study were to design an evidence-based and credible EBUS certification based on a virtual-reality (VR) EBUS simulator test. Methods: Twenty-two respiratory physicians were divided into 3 groups: experienced EBUS operators (group 1, n = 6), untrained novices (group 2, n = 8) and simulator-trained novices (group 3, n = 8). Each physician performed two standardized simulated EBUS-TBNA procedures. Simulator metrics with discriminatory ability were identified and reliability was explored. Finally, the contrasting-groups method was used to establish a pass/fail standard, and the consequences of this standard were explored. Results: Successfully sampled lymph nodes and procedure time were the only simulator metrics that showed statistically significant differences of p = 0.047 and p = 0.002, respectively. The resulting quality score (QS, i.e. sampled lymph nodes per minute) showed an acceptable reliability and a generalizability coefficient of 0.67. Reliability of 0.8 could be obtained by testing in 4 procedures. Median QS was 0.24 (range 0.21-0.26) and 0.098 (range 0.04-0.21) for groups 1 and 2, respectively (p = 0.001). The resulting pass/fail standard was 0.19. Group 3 had a median posttraining QS of 0.11 (range 0-0.17). None of them met the pass/fail standard. Conclusions: With careful design of standardized tests, a credible standard setting and appropriate transfer studies, VR simulators could be an important first line in credentialing before proceeding to supervised performance on patients.

学内開発プレイスメントテスト得点解釈と使用の妥当性の評価について Evaluating Validity for In-House Placement Test Score Interpretations and Uses

Validity theory has evolved dramatically in the past few decades. The most prominent theory in recent years is an argument-based validity framework, proposed by Kane (1992, 2004, 2006). To evaluate test score interpretations and uses based on Kane’s framework, test developers first need to provide interpretive arguments and then validity arguments by proving sound warrants for the following four inferences: (a) scoring from observation to an observed score, (b) generalization from the observed score to the universe score, (c) extrapolation from the universe score to a target score, and (d) decision from the target score to use. In the field of language testing, a number of studies have been conducted to investigate the validity of test score interpretations and uses, especially for the ones considered to be high-stakes such as the TOEFL (Chapelle, 2008; Chapelle, Enright, &amp; Jamieson, 2010). However, not many studies have been conducted to validate in-house placement test score interpretations and uses, and no study has evaluated the validity of such low-stakes tests using Kane’s validity framework. Regardless of whether the tests are high or low stakes, test developers need to be responsible for validating their test score interpretations and uses in order to attest to the validity. This study uses Kane’s (2006) argument-based validity framework to evaluate the validity of in-house placement test score interpretations and uses. The research questions are as follows: (a) to what extent do examinees get placement items correct and high-scoring examinees get more placement items correct; (b) to what extent are placement items consistently sampled from a domain sufficient in number so as to reduce measurement error; (c) to what extent do the difficulty of placement items match the objectives of a reading course; and (d) to what extent do placement decisions made to place examinees in their proper level of the course have an impact on washback in the course? An in-house placement test made up of 40-item grammar, 40-item vocabulary, and 10-item reading sections was developed and administered to 428 first-year private-university students in April 2010. The item format adopted was all multiple-choice so the answer sheets could be easily scored with a reader. Based on their test scores, about 60 high-scoring students and 50 low-scoring students were placed into one of two advanced or one of two basic reading classes. The remaining students were placed into one of several intermediate classes. A 55-item grammar achievement test was administered twice (once as a pretest and then again as a posttest) to the two basic and two intermediate classes. In addition, a 51-item class evaluation survey was administered to investigate students’ participation in the reading classes and to gauge students’ satisfaction with the classes and study support. Warrants for a validity argument of score inference were based on the results of the item analysis. A warrant for a validity argument of generalization inference was based on the composite generalizability coefficient of .92. A warrant for a validity argument of extrapolation inference was based on FACETS analysis, showing that difficulty estimates of learning levels were in an expected difficulty order. A warrant for validity arguments of decision inference was based on the basic-level students’ score gain on an achievement test and their positive reactions to a class evaluation survey. All the validity arguments presented in this study support the validity of the placement test score interpretations and uses. However, to further improve the validity of the test score interpretations and uses, it is necessary to investigate washback effects of the placement test in the reading classes and to revise the test to make grammar, vocabulary, and reading sections with 30 items each.

Using Peers to Assess Handoffs: A Pilot Study

Handoffs among post-graduate year 1 (PGY1) trainees occur with high frequency. Peer assessment of handoff competence would add a new perspective on how well the handoff information helped them to provide optimal patient care. The goals of this study were to test the feasibility of the approach of an instrument for peer assessment of handoffs by meeting criteria of being able to use technology to capture evaluations in real time, exhibiting strong psychometric properties, and having high PGY1 satisfaction scores. An iPad® application was built for a seven-item handoff instrument. Over a two-month period, post-call PGY1s completed assessments of three co-PGY1s from whom they received handoffs the prior evening. Internal Medicine PGY1s at the University of Pennsylvania. ANOVA was used to explore interperson score differences (validity). Generalizability analyses provided estimates of score precision (reproducibility). PGY1s completed satisfaction surveys about the process. Sixty-two PGY1s (100 %) participated in the study. 59 % of the targeted evaluations were completed. The major limitations were network connectivity and inability to find the post-call trainee. PGY1 scores on the single item of "overall competency" ranged from 4 to 9 with a mean of 7.31 (SD 1.09). Generalizability coefficients approached 0.60 for 10 evaluations per PGY1 for a single rotation and 12 evaluations per PGY1 across multiple rotations. The majority of PGY1s believed that they could adequately assess handoff competence and that the peer assessment process was valuable (70 and 77 %, respectively). Psychometric properties of an instrument for peer assessment of handoffs are encouraging. Obtaining 10 or 12 evaluations per PGY1 allowed for reliable assessment of handoff skills. Peer evaluations of handoffs using mobile technology were feasible, and were well received by PGY1s.

A generalizability analysis of score consistency for the Balanced Inventory of Desirable Responding.

Our goal in this investigation was to evaluate the reliability of scores from the Balanced Inventory of Desirable Responding (BIDR) more comprehensively than in prior research using a generalizability-theory framework based on both dichotomous and polytomous scoring of items. Generalizability coefficients accounting for specific-factor, transient, and random-response error ranged from .64 to .75 for the BIDR's Self-Deception Enhancement (SDE) and Impression Management (IM) subscale scores, and these values were systematically lower than corresponding alpha (.66 to .83) and 1-week test-retest (.78 to .86) coefficients. Polytomous scoring provided higher reliability than dichotomous scoring on nearly all indexes reported. Random-response (8%-17%) and specific factor error (11%-17%) exceeded transient error (3%-6%) for both subscales and scoring methods. Doubling the number of items on a single occasion provided greater improvements in generalizability (.76-.83) than aggregating scores across 2 administrations (.72-.81). Both scoring methods provided reasonably high indexes of consistency (φ coefficients≥.91) at cut scores on the IM scale for detecting faked responses when all sources of error were taken into account. Implications of these results for common uses of the BIDR are discussed.

Reliability of the Thoracolumbar Injury Classification and Severity Score and Comparison With the Denis Classification for Injury to the Thoracic and Lumbar Spine

This study is a series of thoracic and lumbar spine fracture cases to assess the reliability of thoracolumbar injury classification and severity score (TLICS) in simulated clinical scenarios. To determine the inter- and intraobserver reliability of TLICS compared with the Denis classification system, and to assess differences based on rater characteristics. Thoracolumbar injury severity score and TLICS have been subjected to reliability testing using less robust statistical analysis. Both systems have demonstrated poor to good reliability, with particularly weak agreement on the status of the posterior ligamentous complex. Fifty-four spine fracture cases were selected from a chart review. These cases were scored on 2 occasions by 11 experts using both TLICS and the Denis classification systems. Reliability was assessed using a generalizability coefficient. The primary outcome was interobserver reliability. Secondary outcomes were intraobserver reliability, difference between orthopedic and neurosurgeons, as well as trainees and consultants, and correlation with treatment recommendations. TLICS demonstrated good interobserver agreement of 0.73 to 0.74. The posterior ligamentous complex component was the least reliable. The Denis classification also demonstrated good reliability between observers, but was least reliable for flexion-distraction injuries. In addition, interobserver reliability between the Denis classification and TLICS morphology subcomponent was strong. TLICS also predicted the need for operative treatment as determined by the experts scoring the injuries. TLICS is a reliable system for assessing fractures of the thoracic and lumbar spine when used by experts. Similar to previous studies, the posterior ligamentous complex subcomponent score was the least reliable component. Reliability assessment using a generalizability coefficient is a robust method for validating fracture classifications.

Reliability of observers' subjective impressions of families: A generalizability theory approach

Parenting was observed in videotaped interactions in 30 families referred for child conduct problems. Generalizability coefficients and the impact of varying numbers of raters were estimated. Two measurement designs were compared: All raters observed all families (“crossed” design) and a different rater observed each family (“nested” design). The crossed design provided higher generalizability coefficients than a nested design, implying inflated generalizability estimates if a crossed estimation model is used for a nested data collection. Three and four raters were needed to obtain generalizability coefficients in the .70–.80 range for monitoring and discipline, respectively. One rater was sufficient for a corresponding estimate for positive involvement and for an estimate in .80–.90 range for problem-solving. Estimates for skill encouragement were non-acceptable.

Reliably Measuring Ambulatory Activity Levels of Children and Adolescents With Cerebral Palsy

ObjectiveTo identify sources of variance in step counts and to examine the minimum number of days required to obtain a stable measure of habitual ambulatory activity in the cerebral palsy (CP) population. DesignCross-sectional. SettingFree-living environments. ParticipantsChildren and adolescents with CP (N=209; mean age ± SD, 8y, 4mo ± 3y, 4mo; n=118 boys; Gross Motor Function Classification System [GMFCS] levels I–III) were recruited through 3 regional pediatric specialty care hospitals. InterventionsDaily walking activity was measured with a 2-dimensional accelerometer over 7 consecutive days. An individual information-centered approach was applied to days with <100 steps, and participants with ≥3 days of missing values were excluded from the study. Participants were categorized into 6 groups according to age and functional level. Generalizability theory was used to analyze the data. Main Outcome MeasuresMean step counts, relative magnitude of variance components in total step activity, and generalizability coefficients (G coefficients) of various combinations of days of the week. ResultsVariance in step counts attributable to participants ranged from 33.6% to 65.4%. For youth ages 2 to 5 years, a minimum of 8, 6, and 2 days were required to reach acceptable G coefficient (reliability) of ≥.80 for GMFCS levels I, II, and III, respectively. For those ages 6 to 14 years, a minimum of 6, 5, and 4 days were required to reach stable measures of step activity for GMFCS levels I, II, and III, respectively. ConclusionsThe findings of the study suggest that an activity-monitoring period should be determined based on the GMFCS levels to reliably measure ambulatory activity levels in youth with CP.

A Comparison of Two Standard-Setting Approaches in High-Stakes Clinical Performance Assessment Using Generalizability Theory

Scoring clinical assessments in a reliable and valid manner using criterion-referenced standards remains an important issue and directly affects decisions made regarding examinee proficiency. This generalizability study of students' clinical performance examination (CPX) scores examines the reliability of those scores and of their interpretation, particularly according to a newly introduced, "critical actions" criterion-referenced standard and scoring approach. The authors applied a generalizability framework to the performance scores of 477 third-year students attending three different medical schools in 2008. The norm-referenced standard included all station checklist items. The criterion-referenced standard included only those items deemed critical to patient care by a faculty panel. The authors calculated and compared variance components and generalizability coefficients for each standard across six common stations. Norm-referenced scores had moderate generalizability (ρ = 0.51), whereas criterion-referenced scores showed low dependability (φ = 0.20). The estimated 63% of measurement error associated with the person-by-station interaction suggests case specificity. Increasing the number of stations on the CPX from 6 to 24, an impractical solution both for cost and time, would still yield only moderate dependability (φ = 0.50). Though the performance assessment of complex skills, like clinical competence, seems intrinsically valid, careful consideration of the scoring standard and approach is needed to avoid misinterpretation of proficiency. Further study is needed to determine how best to improve the reliability of criterion-referenced scores, by implementing changes to the examination structure, the process of standard-setting, or both.

Reliable and valid assessment of competence in endoscopic ultrasonography and fine-needle aspiration for mediastinal staging of non-small cell lung cancer

Fine-needle aspiration (FNA) guided by endoscopic ultrasonography (EUS) is important in mediastinal staging of non-small cell lung cancer (NSCLC). Training standards and implementation strategies of this technique are currently under discussion. The aim of this study was to explore the reliability and validity of a newly developed EUS Assessment Tool (EUSAT) designed to measure competence in EUS - FNA for mediastinal staging of NSCLC. A total of 30 patients with proven or suspected NSCLC underwent EUS - FNA for mediastinal staging by three trainees and three experienced physicians. Their performances were assessed prospectively by three experts in EUS under direct observation and again 2 months later in a blinded fashion using digital video-recordings. Based on the assessments, intra-rater reliability, inter-rater reliability, and construct validity were explored. The intra-rater reliability was good (Cronbach's α = 0.80), but comparison of results based on direct observations and blinded video-recordings indicated a significant bias favoring consultants (P = 0.022). Inter-rater reliability was very good (Cronbach's α = 0.93). However, one rater assessing five procedures or two raters each assessing four procedures were necessary to secure a generalizability coefficient of 0.80. The assessment tool demonstrated construct validity by discriminating between trainees and experienced physicians (P = 0.034). Competency in mediastinal staging of NSCLC using EUS and EUS - FNA can be assessed in a reliable and valid way using the EUSAT assessment tool. Measuring and defining competency and training requirements could improve EUS quality and benefit patient care.

A framework for optimising the cost and performance of concept testing

To anticipate the likely market demand better and identify the best customers to target with potential new products early in their development, concept tests need to provide adequate data quality for the objectives of measurement. Generalisability theory provides a framework where a generalisability study can be conducted to identify each possible aspect of a measurement as a factor that may be a potential source of variability. When a limited budget is available, optimising measurement designs involves a trade-off between the accuracy of the data (i.e. generalisability coefficients) and cost considerations. Building on previous and ongoing research, this study presents a multivariate optimisation procedure to achieve the most cost-efficient measurement design under the pre-specified generalisability coefficient constraint. We used the available online concept testing data to illustrate how to optimise the measurement cost by sampling along the facets that contribute to the total error variance in different iso-generalisability designs. The findings may help to facilitate decision making for the sampling of respondents, concepts, items, and occasions in the design of concept test. However, the optimisation framework may be applied more generally to improve the effectiveness and efficiency of other customer tests in marketing.

Program for laparoscopic urological skills assessment: Setting certification standards for residents

Aim: There is growing pressure from the government and the public to define proficiency standards for surgical skills. Aim of this study was to estimate the reliability of the Program for Laparoscopic Urological Skills (PLUS) assessment and to set a certification standard for second-year urological residents. Methods: Fifty participants were assessed on performance time and performance quality to investigate the reliability of the PLUS assessment. Generalisability coefficient of 0.8, on a scale of 0 to 1.0, was considered to indicate good reliability for assessment purposes. Pass/fail standards were based on laparoscopic experience: Novices, intermediates, and experts (>100 procedures). The pass/fail standards were investigated for the PLUS performances of 33 second-year urological residents. Results: Fifteen novices, twenty-three intermediates and twelve experts were included. An inter-trial reliability of >0.80 was reached with two trials for each task. Inter-rater reliability of the quality measurements was 0.79 for two judges. Pass/fail scores were determined for the novice/intermediate boundary and the intermediate/expert boundary. Pass rates for second-year residents were 63.64% and 9.09%, respectively. Conclusion: The PLUS assessment is reliable for setting a certification standard for second-year urological residents that serves as a starting point for residents to proceed to the next level of laparoscopic competency.

A Predictive and Construct Validity Study of a High-Stakes Objective Clinical Examination for Assessing the Clinical Competence of International Medical Graduates

Background: The purpose of this study was to investigate the predictive and construct validity of a high-stakes objective structured clinical examination (OSCE) used to select candidates for a 3-month clinical rotation to assess practice-readiness status. Summary: Analyses were undertaken to establish the reliability and validity of the OSCE. The generalizability coefficient (Ep2) for the assessment scores (checklist, global, and total) were all high, ranging from 0.73 to 0.84. Two discriminant analyses (promotion to the 3-month rotation and pass/fail status on the rotation) provided evidence of predictive validity with a 100% correct classification rate in the pass/fail rotation results. Factor analysis results provided evidence of construct validity with four factors identified: Clinical Skills, Internal Medicine, General Medical Knowledge, and Counseling. The known group differences between licensing status and residency experience also provided evidence of construct validity. Conclusions: The results are encouraging for the predictive and construct validity of the OSCE as an assessment of clinical competence.

How many treatment sessions and patients are needed to create a stable score of adherence and competence in the treatment of cocaine dependence?

The study utilized a generalizability theory analysis of adherence and competence ratings to evaluate the number of sessions and patients needed to yield dependable scores at the patient and therapist levels. Independent judges' ratings of supportive expressive therapy (n = 94), cognitive therapy (n = 103), and individual drug counseling (n = 98) were obtained on tapes of sessions from the NIDA Collaborative Cocaine Treatment Study. Generalizability coefficients revealed that, for all three treatments, ratings made on approximately five to 10 sessions per patient are needed to achieve sufficient dependability at the patient level. At the therapist level, four to 14 patients need to be evaluated (depending on the modality), to yield dependable scores. Many studies today use fewer numbers.

Properties of the Achievement of Therapeutic Objectives Scale (ATOS): A Generalizability Theory study

The psychometric properties of the process instrument Achievement of Therapeutic Objectives Scale (ATOS) were examined in this study. Generalizability Theory (GT) was used to evaluate variability attached to several possible sources of error. A random sample of 24 psychotherapy sessions was selected from a larger RCT trial including Cluster C patients and rated in different rating conditions (rating all scales versus rating a subgroup of the scales). Two G-study designs were used to compute variance components and generalizability coefficients. The results provided evidence that the ATOS is sensitive to differences among patients and to differences among subscales within patients (i.e., to different constructs within the ATOS). Rating condition contributed much to variability in scores when the ATOS scales were examined separately, but this variability was negligible when all scales were included in the same analysis. There was little variability due to raters, which indicates that adequately trained raters are able to apply the ATOS without contributing to measurement error.