Generalizability Analysis Research Articles

MP015: Daily encounter cards: evaluating the quality of documented assessments

Introduction: In response to concerns in the literature over the quality of completed work-based assessments (WBAs), faculty development and rater training initiatives have been developed. The Completed Clinical Evaluation Report Rating (CCERR) was designed to evaluate these interventions by providing a measure of the quality of documented assessments on In-Training Evaluation Reports (ITERs). Daily Encounter Cards (DECs) are a common form of WBA used in the Emergency Department setting. A tool to evaluate initiatives aimed at improving the quality of completion of this widely used WBA is also needed. The purpose of this study was to provide validity evidence to support using the CCERR to assess the quality of DEC completion. Methods: This study was conducted in the Department of Emergency Medicine at the University of Ottawa. Six experts in resident assessment grouped 60 DECs into three quality categories (high, average, poor) based on their perception of how informative each DEC was for reporting judgments of the resident’s performance. Eight clinical supervisors (blinded to the expert groupings) scored the 10 most representative DECs in each group using the CCERR. Mean scores were compared using a univariate ANOVA to determine if the CCERR was able to discriminate DEC quality. Reliability for the CCERR scores was determined using a generalizability analysis. Results: Mean CCERR scores for the high (37.3, SD=1.2), average (24.2, SD=3.3), and poor (14.4, SD=1.4) quality groups differed (p<0.001). A pairwise comparison demonstrated that differences between all three quality groups were statistically significant (p<0.001), indicating that the CCERR was able to discriminate DEC quality as judged by experts. A generalizability study demonstrated the majority of score variation was due to differences in DECs. The reliability with a single rater was 0.95. Conclusion: There is strong validity evidence to support the use of the CCERR to evaluate DEC quality. It can be used to provide feedback to supervisors for improving assessment reporting, and offers a quantitative measure of change in assessor behavior when utilized as a program evaluation instrument for determining the quality of completed DECs.

An Investigation of the Generalizability of Medical School Grades

ABSTRACTConstruct/Background: Medical school grades are currently unstandardized, and their level of reliability is unknown. This means their usefulness for reporting on student achievement is also not well documented. This study investigates grade reliability within 1 medical school. Approach: Generalizability analyses are conducted on grades awarded. Grades from didactic and clerkship-based courses were treated as 2 levels of a fixed facet within a univariate mixed model. Grades from within the 2 levels (didactic and clerkship) were also entered in a multivariate generalizability study. Results: Grades from didactic courses were shown to produce a highly reliable mean score (G = .79) when averaged over as few as 5 courses. Although the universe score correlation between didactic and clerkship courses was high (r = .80), the clerkship courses required almost twice as many grades to reach a comparable level of reliability. When grades were converted to a Pass/Fail metric, almost all information contained in the grades was lost. Conclusions: Although it has been suggested that the imprecision of medical school grades precludes their use as a reliable indicator of student achievement, these results suggest otherwise. While it is true that a Pass/Fail system of grading provides very little information about a student's level of performance, a multi-tiered grading system was shown to be a highly reliable indicator of student achievement within the medical school. Although grades awarded during the first 2 didactic years appear to be more reliable than clerkship grades, both yield useful information about student performance within the medical college.

Observación y análisis de las acciones a balón parado en el fútbol profesional

Utilizando la metodología observacional como medio, hemos analizado tres acciones a balón parado muy frecuentes en el fútbol como son los saques de esquina y los tiros libres directos e indirectos, con el fin de elaborar y generalizar una herramienta de observación. El registro de los datos se realizó sobre partidos de la UD Almería de la temporada 2006–2007 en la 2ª división nacional. Para ello se ha utilizado un diseño nomotético de seguimiento y se ha realizado un registro intermitente con codificación de los datos como multieventos, haciéndose posteriormente un análisis de la calidad del dato para asegurar la fiabilidad del registro, donde los datos fueron sometidos a un análisis de componentes de varianza y a un análisis de generalizabilidad. Como conclusiones podemos destacar la confirmación de la fiabilidad y generalizabilidad de la herramienta de observación construida y la inexistencia de diferencias significativas en el comportamiento general de los equipos observados en los saques de esquina y tiros libres directos e indirectos.

Competency Evaluations in the Next Accreditation System: Contributing to Guidelines and Implications

ABSTRACTConstruct: This study examines validity evidence of end-of-rotation evaluation scores used to measure competencies and milestones as part of the Next Accreditation System (NAS) of the Accreditation Council for Graduate Medical Education (ACGME). Background: Since the implementation of the milestones, end-of-rotation evaluations have surfaced as a potentially useful assessment method. However, validity evidence on the use of rotation evaluation scores as part of the NAS has not been studied. This article examines validity evidence for end-of-rotation evaluations that can contribute to developing guidelines that support the NAS. Approach: Data from 2,701 end-of-rotation evaluations measuring 21 out of 22 Internal Medicine milestones for 142 residents were analyzed (July 2013–June 2014). Descriptive statistics were used to measure the distribution of ratings by evaluators (faculty, n = 116; fellows, n = 59; peer-residents, n = 131), by postgraduate years. Generalizability analysis and higher order confirmatory factor analysis were used to examine the internal structure of ratings. Psychometric implications for combining evaluation scores using composite score reliability were examined. Results: Milestone ratings were significantly higher for each subsequent year of training (15/21 milestones). Faculty evaluators had greater variability in ratings across milestones, compared to fellows and residents; faculty ratings were generally correlated with milestone ratings from fellows (r = .45) and residents (r = .25), but lower correlations were found for Professionalism and Interpersonal and Communication Skills. The Φ-coefficient was .71, indicating good reliability. Internal structure supported a 6-factor solution, corresponding to the hierarchical relationship between the milestones and the 6 core competencies. Evaluation scores corresponding to Patient Care, Medical Knowledge, and Practice-Based Learning and Improvement had higher correlations to milestones reported to the ACGME. Mean evaluation ratings predicted problem residents (odds ratio = 5.82, p < .001). Conclusions: Guidelines for rotation evaluations proposed in this study provide useful solutions that can help program directors make decisions on resident progress and contribute to assessment systems in graduate medical education.

Selecting and Simplifying: Rater Performance and Behavior When Considering Multiple Competencies

Theory: Assessment of clinical competence is a complex cognitive task with many mental demands often imposed on raters unintentionally. We were interested in whether this burden might contribute to well-described limitations in assessment judgments. In this study we examine the effect on indicators of rating quality of asking raters to (a) consider multiple competencies and (b) attend to multiple issues. In addition, we explored the cognitive strategies raters engage when asked to consider multiple competencies simultaneously. Hypotheses: We hypothesized that indications of rating quality (e.g., interrater reliability) would decline as the number of dimensions raters are expected to consider increases. Method: Experienced faculty examiners rated prerecorded clinical performances within a 2 (number of dimensions) × 2 (presence of distracting task) × 3 (number of videos) factorial design. Half of the participants were asked to rate 7 dimensions of performance (7D), and half were asked to rate only 2 (2D). The second factor involved the requirement (or lack thereof) to rate the performance of actors participating in the simulation. We calculated the interrater reliability of the scores assigned and counted the number of relevant behaviors participants identified as informing their ratings. Second, we analyzed data from semistructured posttask interviews to explore the rater strategies associated with rating under conditions designed to broaden raters' focus. Results: Generalizability analyses revealed that the 2D group achieved higher interrater reliability relative to the 7D group (G = .56 and .42, respectively, when the average of 10 raters is calculated). The requirement to complete an additional rating task did not have an effect. Using the 2 dimensions common to both groups, an analysis of variance revealed that participants who were asked to rate only 2 dimensions identified more behaviors of relevance to the focal dimensions than those asked to rate 7 dimensions: procedural skill = 36.2%, 95% confidence interval (CI) [32.5, 40.0] versus 23.5%, 95% CI [20.8, 26.3], respectively; history gathering = 38.6%, 95% CI [33.5, 42.9] versus 24.0%, 95% CI [21.1, 26.9], respectively; ps < .05. During posttask interviews, raters identified many sources of cognitive load and idiosyncratic cognitive strategies used to reduce cognitive load during the rating task. Conclusions: As intrinsic rating demands increase, indicators of rating quality decline. The strategies that raters engage when asked to rate many dimensions simultaneously are varied and appear to yield idiosyncratic efforts to reduce cognitive effort, which may affect the degree to which raters make judgments based on comparable information.

Measuring multiple neuromuscular activation using EMG - a generalizability analysis.

Analysis of electromyography (EMG) data has been shown to be valuable in biomedical and clinical research. However, most analysis tools do not consider the non-linearity of EMG data or the synergistic effects of multiple neuromuscular activities. The SYNERGOS algorithm was developed to assess a single index based on non-linear analysis of multiple neuromuscular activation (MNA) of different muscles. This index has shown promising results in Parkinsonian gait, but it was yet to be explored whether the SYNERGOS index is generalizable. In this study, we evaluated generalizability of the SYNERGOS index over the course of several trials and over separate days with different walking speeds. Ten healthy adults aged from 18 to 40 years walked on a treadmill on two different days, while EMG data was collected from the upper and lower right leg. SYNERGOS indices were obtained and a generalizability analysis was conducted. The algorithm detected changes in MNA in response to altering gait speed and depicted a high generalizability coefficient ( ρ^2 ${\hat \rho ^2}$ ) of 0.823 with a standard error of 5.117 with nominal inter-trial or inter-day effects. We concluded SYNERGOS may be a valuable tool in EMG analysis due to its generalizability and its sensitivity to task modifications and associated neuromotor changes.

How Many Responses Do We Need? Using Generalizability Analysis to Estimate Minimum Necessary Response Rates for Online Student Evaluations

Construct:The study compares paper and online ratings of instructional units and analyses, with the G-study using the symmetry principle, the response rates needed to ensure acceptable precision of the measure when compliance is low. Background: Students' ratings of teaching contribute to the quality of medical training programs. To date, many schools have replaced pen-and-paper questionnaires with electronic forms, despite the lower response rates consistently reported with the latter. Few available studies have examined the effects of low response rates on the reliability and precision of the evaluation measure. Moreover, the minimum number of raters to target when response rates are low remains unclear. Approach: Descriptive data were derived from 799 students' paper and online ratings of 11 preclinical instructional units (PIUs). Reliability was assessed by Cronbach's alpha coefficients. The generalizability method applying the symmetry principle approach was used to analyze the precision of the measure with a reference standard error of mean (SEM) set at 0.10; optimization models were built to estimate minimum response rates. Results: Overall, response rates were 74% and 30% (p < .001) and PIUs ratings were 3.8 ± 0.5 and 3.6 ± 0.5 (p = .02), respectively in paper and online questionnaires. Higher SEM levels and significantly larger 95% confidence intervals of PIUs rating scores were observed with online evaluations. To keep the SEM within preset limits of precision, a minimum of 48% response rate was estimated for online formats. Conclusions: The proposed generalizability analysis allowed estimating the minimum response needed to maintain acceptable precision in online evaluations. The effects of response rates on accuracy are discussed.

생물자원 전통지식 추출을 위한 델파이조사의 신뢰성 연구

본 연구는 문헌 및 현장에서 얻은 생물자원지식들의 가치를 평가하기 위해 전문가들을 대상으로 실시한 델파이 조사 (Delphi method)의 신뢰성을 평가하였다. 델파이조사는 전문가들이 다른 사람의 의견과 관계없이 독립적으로 전문성 있는 평가를 내린 다음에 다음 단계에서 다른 전문가들의 의견을 참고하며 자신의 의견을 수정하는 절차를 갖는다. 본 연구에서는 문헌에서 얻은 전통지식 100건, 현장에서 취득한 지식 100건 등, 모두 200건을 우선 선정한 후 전문가 6인을 선정하여 각 지식의 가치에 대하여 두 차례 평가하도록 하였다. 그 결과 두 차례의 평가점수는 연관성이 매우 높으면서 2차에서는 다른 전문가들의 의견을 수용해 어느 정도 자체수정이 발생하여 각 문항에 대한 평가점수의 표준편차가 줄어들었다. 본 조사의 신뢰성 (reliability)을 파악하기 위해 일반적인 신뢰도 계수인 크론바하 알파와 함께 일반화가능도 (generalizability) 계수를 구하였다. 이 두 신뢰도 분석을 통해 2차 평가 후 평가의 신뢰도가 상승하여 전문가에 의한 델파이 조사의 신뢰도가 매우 높다는 사실을 지지하였으나 일반화 가능도 분석 결과를 해석하는 과정에서 다른 결과를 유추할 수 있었다. 신뢰도계수가 증가하였음에도 불구하고 평가자간의 편차는 증가하여 신뢰도가 높아진 것은 평가가 상향되고 평균에 회귀하는 경향으로 잔차변동이 줄어서이지 평가자간의 의견수렴이 이루어진 결과로 볼 수는 없다는 사실이었다. 이러한 결과를 토대로 신뢰도 계수와 함께 평가자 간의 분산을 파악하여 델파이조사의 추가적인 단계 (round)가 필요함을 제시하였다. In the knowledge and information age, to discover and protect Intellectual Properties would be very important for their economic value as a major growth engine. This study evaluated the reliability of a Delphi survey conducted by experts to assess the value of agricultural resources knowledge obtained from literature reviews and field interviews. Delphi method is collecting the opinions of experts for several rounds repeatedly, in the next round the experts have chance to modify their opinion. Scores between two rounds are highly correlated and standard deviations are declined for second round to imply that some correction of their evaluations are made. To check reliability of Delphi survey of two rounds Cronbach's reliability coefficient and Generalizability coefficient are derived. The Cronbach alpha's supported the reliability of the method, but the Generalizability analysis revealed some unexpected results while checking the variance components of sources of measurement errors. Despite the increased reliability coefficients, the deviations between the raters are increased which means that additional rounds are required to get consensus, the goal of Delphi research.

Using Generalizability Analysis to Establish Guidelines for Designing Horizontally Integrated Anatomy Assessments

Background &amp; PurposeIn the context of planning assessments for a combined gross anatomy/histology course, this study used generalizability analysis to estimate the number of gross anatomy and histology items that would be needed on a series of examinations to reliably assess discipline‐specific knowledge.MethodsScores of first‐year medical students (n=145) who sat for 3 gross anatomy and histology written and practical examinations were analyzed. A generalizability study (G‐study) estimated the reliability of the written and practical examination scores, and a decision study (D‐study) estimated the optimum assessment parameters for reaching a minimum reliability of 0.8.ResultsEstimated G coefficients for calculating reliability were 蠅0.782 for the gross anatomy examinations and 蠅0.659 for the histology examinations. Assuming that a series of 4 separate examinations would be ideal for assessing a breadth of content in a combined gross anatomy/histology course, the D‐study estimated that a minimum of 35 and 40 items for a gross anatomy written and practical examination, respectively, would be needed to reach a reliability of 0.8 for measuring gross anatomy knowledge alone. Furthermore, each of the 4 examinations would require an additional 40 written and 45 practical items to reliability assess one's knowledge of histology.ConclusionThis study helps to establish basic parameters for designing horizontally integrated anatomy assessments. Future studies will investigate how incorporating ‘truly integrated items' (that require knowledge of histology and gross anatomy to answer a single question) may affect the reliability of such assessments.

Improving QST Reliability—More Raters, Tests, or Occasions? A Multivariate Generalizability Study

The reliability of quantitative sensory testing (QST) is affected by the error attributable to both test occasion and rater (examiner) and the interactions between them. Most reliability studies account for only 1 source of error. The present study employed a fully crossed, multivariate generalizability design to account for rater and occasion variance simultaneously. Nineteen healthy volunteers were examined with a battery of 7 QST procedures 4 times on 2 occasions by 2 raters. The QST battery was composed to include a mix of different pain stimuli and response domains, including threshold, intensity, tolerance, and modulation with mechanical, thermal, and chemical stimuli. The classical test-retest and interrater reliability (.19 < intraclass correlation coefficient <.92) was in line with the literature, and generalizability analysis indicated that the universe score was generally the dominant source of variation (relative contribution = 19%, 78%). Error attributable to the interaction between study participant and occasion was also influential. Dependability coefficients indicated that a substantial increase in reliability and feasibility could be achieved by employing a composite QST battery compared to single QST procedures. Reliability was improved more by repeated testing on separate occasions than by repeated testing by different raters. PerspectivesWhen balancing reliability and feasibility, the current findings suggest that a carefully selected battery of QST procedures repeated on a few occasions may be optimal.

Development and Validation of a Trustworthy Multisource Feedback Instrument to Support Nurse Appraisals

Nurse appraisal is well established in the Western world because of its obvious educational advantages. Appraisal works best with many sources of information on performance. Multisource feedback (MSF) is widely used in business and in other clinical disciplines to provide such information. It has also been incorporated into nursing appraisals, but, so far, none of the instruments in use for nurses has been validated. We set out to develop an instrument aligned with the UK Knowledge and Skills Framework (KSF) and to evaluate its reliability and feasibility across a wide hospital-based nursing population. The KSF framework provided a content template. Focus groups developed an instrument based on consensus. The instrument was administered to all the nursing staff in 2 large NHS hospitals forming a single trust in London, England. We used generalizability analysis to estimate reliability, response rates and unstructured interviews to evaluate feasibility, and factor structure and correlation studies to evaluate validity. On a voluntary basis the response rate was moderate (60%). A failure to engage with information technology and employment-related concerns were commonly cited as reasons for not responding. In this population, 11 responses provided a profile with sufficient reliability to inform appraisal (G = 0.7). Performance on the instrument was closely and significantly correlated with performance on a KSF questionnaire. This is the first contemporary psychometric evaluation of an MSF instrument for nurses. MSF appears to be as valid and reliable as an assessment method to inform appraisal in nurses as it is in other health professional groups.

Comparing the Effectiveness of SPSS and EduG using Different Designs for Generalizability Theory

AbstractGeneralizability theory (G theory) provides a broad conceptual framework for social sciences such as psychology and education, and a comprehensive construct for numerous measurement events by using analysis of variance, a strong statistical method. G theory, as an extension of both classical test theory and analysis of variance, is a model which can deal with multiple sources of error. In conducting the analysis of the G theory, there are several software programs that can be used such as GENOVA, SPSS, SAS, EduG, and G-String. In this study, the general perspectives of G theory are first explained broadly. Then, the SPSS and EduG software programs are used to conduct generalizability analyses on the data obtained from the answers of 30 students (p) to nine open-ended questions (i) as rated by three raters (r). There are three different designs in the study. Two of them are random effects designs, pxixr and pxi:r, and the last one is pxixr design using a fixed rater . According to the findings from the study, SPSS and EduG give the same results for variance component estimates as well as for G (Generalizability) and D (Decision) studies of all designs, as expected. Besides comparing the program outputs, their weaknesses and strengths were also discussed regarding different designs and data sets in this study.Keywords: Generalizability Theory * G Study * D Study * SPSS * EduGG theory has formed a comprehensive structure by employing variance analysis which provides a broad conceptual framework for social sciences such as psychology and education (Brennan, 2000, 2001a; Cronbach, Gleser, Nanda, & Rajaratnam, 1972; Shavelson & Webb, 1991). It is also a powerful statistical tool for situations where there are numerous measurements. The theory, as an extension of classical test theory and variance analysis, stands as a model where multiple sources of error can be handled (Brennan, 2001a; Shavelson & Webb, 1991).Generalizability (G) TheoryThe reliability of measurement results in education and psychology was previously examined using classical test theory (CTT) in general. It is assumed in CTT that the observed score is composed of the actual score with no separable score for error. The restriction of this assumption, especially in performance measurements where the probability of the existence of more than one source of error is high, reveals the importance of G theory in which more than one source of error is handled and can be predicted simultaneously (Brennan, 2000). Another advantage of G theory in using performance assessment is that while there is a restrictive parallel assumption in CTT, randomly parallel assumption is adopted in G theory (Brennan, 2011; Kretchmar, 2006). The main aim of G theory is to generalize the scores of a specific measurement tool from a specific group to the universe of generalization which consist of 1) the universe of admissible observations and generalizability studies (G studies), 2) the universe of G studies and decision studies (D studies). While G studies provide an estimate of the generalizability coefficient of variances from all facets and this coefficient includes the examinee's universe score, D studies enable one to examine the interactions among all applicable facets (tasks, raters, observations, etc.) and the subject of measurement for calculating the dependability coefficient (Brennan, 2000; Crocker & Algina, 1986; Hsu, 2012).G theory has four main advantages compared to CTT. 1) It provides simultaneous evaluation of test-retest reliability, internal consistency, inter-rater reliability, and convergent validity. 2) It enables estimates of both individual measurement facets and interaction effects. 3) When assessing an examinee's performance, it gives information about the quality of their absolute structural level of knowledge as well as ranking this information in order. 4) It allows researchers to optimize the reliability of an assessment within the cost constraints of time and money. …

Development and validation of an instrument for measuring the quality of teamwork in teaching teams in postgraduate medical training (TeamQ).

BackgroundTeamwork between clinical teachers is a challenge in postgraduate medical training. Although there are several instruments available for measuring teamwork in health care, none of them are appropriate for teaching teams. The aim of this study is to develop an instrument (TeamQ) for measuring teamwork, to investigate its psychometric properties and to explore how clinical teachers assess their teamwork.MethodTo select the items to be included in the TeamQ questionnaire, we conducted a content validation in 2011, using a Delphi procedure in which 40 experts were invited. Next, for pilot testing the preliminary tool, 1446 clinical teachers from 116 teaching teams were requested to complete the TeamQ questionnaire. For data analyses we used statistical strategies: principal component analysis, internal consistency reliability coefficient, and the number of evaluations needed to obtain reliable estimates. Lastly, the median TeamQ scores were calculated for teams to explore the levels of teamwork.ResultsIn total, 31 experts participated in the Delphi study. In total, 114 teams participated in the TeamQ pilot. The median team response was 7 evaluations per team. The principal component analysis revealed 11 factors; 8 were included. The reliability coefficients of the TeamQ scales ranged from 0.75 to 0.93. The generalizability analysis revealed that 5 to 7 evaluations were needed to obtain internal reliability coefficients of 0.70. In terms of teamwork, the clinical teachers scored residents' empowerment as the highest TeamQ scale and feedback culture as the area that would most benefit from improvement.ConclusionsThis study provides initial evidence of the validity of an instrument for measuring teamwork in teaching teams. The high response rates and the low number of evaluations needed for reliably measuring teamwork indicate that TeamQ is feasible for use by teaching teams. Future research could explore the effectiveness of feedback on teamwork in follow up measurements.

Evaluating Procedures for Reducing Measurement Error in Math Curriculum-Based Measurement Probes

At present, it is unclear whether math curriculum-based measurement (M-CBM) procedures provide a dependable measure of student progress in math computation because support for its technical properties is based largely upon a body of correlational research. Recent investigations into the dependability of M-CBM scores have found that evaluating probe design procedures is necessary to estimate the contribution of probe-related factors to variance in M-CBM scores. To extend this research, the present study compared a set of commercially available probes (i.e., AIMSweb) with an experimental set that was constructed to enhance consistency in item content across alternative forms. Two probes were randomly selected from each set and administered to 43 students in fifth grade. Generalizability analyses indicated that the experimental probes generated scores with less error variance than those of the AIMSweb probes. Follow-up analyses indicated that fewer administrations of the experimental probes were necessary to attain adequate reliability for relative decisions (i.e., &gt;.80) and high reliability for absolute decisions (i.e., &gt;.90). Results support prior studies and suggest that error variance is associated with M-CBM probe content and development procedures.

Confirmatory Factor Analysis of the System for Evaluation of Teaching Qualities (SETQ) in Graduate Medical Training.

The System for Evaluation of Teaching Qualities (SETQ) was developed as a formative system for the continuous evaluation and development of physicians' teaching performance in graduate medical training. It has been seven years since the introduction and initial exploratory psychometric analysis of the SETQ questionnaires. This study investigates the validity and reliability of the SETQ questionnaires across hospitals and medical specialties using confirmatory factor analyses (CFAs), reliability analysis, and generalizability analysis. The SETQ questionnaires were tested in a sample of 3,025 physicians and 2,848 trainees in 46 hospitals. The CFA revealed acceptable fit of the data to the previously identified five-factor model. The high internal consistency estimates suggest satisfactory reliability of the subscales. These results provide robust evidence for the validity and reliability of the SETQ questionnaires for evaluating physicians' teaching performance.

Are Teacher Perspectives Useful? Incorporating EFL Teacher Feedback in the Development of a Large-Scale International English Test

This aticle presents a case study incorporating English teachers’ perspectives into the development of a large-scale international English assessment, the recently developed TOEFL Junior® Comprehensive test. It discusses how stakeholder feedback gathered during test development supports the validity argument for score interpretation and the use of a newly developed test. When the pilot version of the test was administered, focus-group interviews were conducted to gather teacher perspectives on the test. Teachers of English as a foreign language in Korea from different types of institutions—public schools, private mainstream schools, and private after-school English enrichment programs—were invited to participate. This article discusses how the interviews were conducted and summarizes the teachers’ major comments about the test. Then, it discusses how the teachers’ recommendations and suggestions were reflected in the final test design decisions. During the decision-making process, the teachers’ comments were examined in light of various statistical results, such as interitem correlations and generalizability analyses. The article concludes with a broader discussion of how teachers’ perspectives, which have not often been incorporated into the development of large-scale international assessments, can make an important difference in designing a test and, consequently, enhance the validity of the test.

A procedural skills OSCE: assessing technical and non-technical skills of internal medicine residents.

Internists are required to perform a number of procedures that require mastery of technical and non-technical skills, however, formal assessment of these skills is often lacking. The purpose of this study was to develop, implement, and gather validity evidence for a procedural skills objective structured clinical examination (PS-OSCE) for internal medicine (IM) residents to assess their technical and non-technical skills when performing procedures. Thirty-five first to third-year IM residents participated in a 5-station PS-OSCE, which combined partial task models, standardized patients, and allied health professionals. Formal blueprinting was performed and content experts were used to develop the cases and rating instruments. Examiners underwent a frame-of-reference training session to prepare them for their rater role. Scores were compared by levels of training, experience, and to evaluation data from a non-procedural OSCE (IM-OSCE). Reliability was calculated using Generalizability analyses. Reliabilities for the technical and non-technical scores were 0.68 and 0.76, respectively. Third-year residents scored significantly higher than first-year residents on the technical (73.5 vs. 62.2%) and non-technical (83.2 vs. 75.1%) components of the PS-OSCE (p < 0.05). Residents who had performed the procedures more frequently scored higher on three of the five stations (p < 0.05). There was a moderate disattenuated correlation (r = 0.77) between the IM-OSCE and the technical component of the PS-OSCE scores. The PS-OSCE is a feasible method for assessing multiple competencies related to performing procedures and this study provides validity evidence to support its use as an in-training examination.

The versatility of generalizability theory as a tool for exploring and controlling measurement error

Measurement error arises from many sources in educational assessment. It is important to estimate the importance of this error, and, if appropriate, to seek ways to reduce it. Generalizability theory represents a powerful tool in this sense, allowing identifiable error contributions to be separately quantified, and measurement error to be estimated and even predicted in response to possible changes in the measurement procedure. The paper offers examples of generalizability analysis of numeracy attainment data deriving from the Scottish Survey of Achievement, with the aim of illustrating the versatility of the methodology for error estimation and prediction in this type of sample-based programme.

THE THINKING-ABOUT-DERIVATIVE TEST FOR UNDERGRADUATE STUDENTS: DEVELOPMENT AND VALIDATION

Two studies were conducted for the development and validation of a multidimensional test to assess undergraduate students’ mathematical thinking about derivative. The first study involved two phases: question generation and refinement of the Thinking-about-Derivative Test (TDT). The second study included four phases as follows: test administration, generalizability analysis, confirmatory factor analysis, and subgroup validity analysis. Findings suggested that the 30-item multiple-choice TDT, which comprises 6 mathematical thinking aspects, enactive, iconic, algorithmic, algebraic, formal, and axiomatic thinking, demonstrates acceptable levels of reliability and validity. Followed by additional cross-validation studies, the TDT may be a useful tool for mathematics education researchers and mathematicians. Directions for future research and implications for educational practice are discussed.

Reliability and benefits of medical student peers in rating complex clinical skills

Background: Peers have been shown to be reliable raters in an objective structured clinical exam (OSCE). While the literature supports the use of senior level students in rating basic clinical skills, little is known about the reliability of peers of the same level of training in assessing complex clinical skills.Aim: To investigate the reliability of student peers of the same level of training in rating complex clinical skills in a geriatric OSCE.Methods: Peer (n = 115) ratings were compared to faculty ratings using correlation and generalizability analysis. Paired Wilcoxon Signed-Rank test was used to establish peer learning benefits.Results: Reliability of the OSCE was moderately strong (G-coefficient = 0.70) with strong correlations between peer and faculty ratings for the overall OSCE (r = 0.78, p = 0.001) and for each case (r = 0.70–0.85, p = 0.001). Generalizability analysis indicated that raters contributed minimally to score variance. Peers reported gaining learning benefits from the rating process.Conclusion: Peer raters of the same level of training can provide accurate ratings of complex clinical tasks and can serve as an important resource in assessing student performance in an OSCE. The peer review process can also serve an important role in enhancing student learning.