Generalizability Coefficients Research Articles

태권도 품새 경기의 주관적 평가결과의 오차원 분석: 일반화가능도 이론 적용

본 연구는 G-Theory를 적용하여 태권도 품새 평가항목에 대한 채점자 간 평가점수의 신뢰도를 추정하기 위해 G-연구로 경기일 수, 채점자 수를 다중오차원으로 선정하고 이들 변인간의 상호작용으로 인한 오차변량의 상대적 크기에 의해 오차원을 분석하고, G-연구의 결과를 토대로 D-연구를 수행하여 최적의 측정조건을 결정하는 데 목적이 있다. 결과를 요약하면 다음과 같다. G-Theory를 적용하여 태권도 품새 평가항목 중 정확성에 대해 분산성분 추정치를 추정한 결과, 오차의 영향력은 채점자국면에서 가장 크게 나타났으며, 각 설계에 따른 상호효과 (피험자 내), 피험자 간 순서였으며, 표현성 평가항목에 대한 분산성분 추정치 오차의 영향력은 상호효과 (피험자 내)국면에서 가장 크게 나타났으며, 각 설계에 따른 피험자 간, 채점자 국면 순으로 나타났다. 마지막으로 D-연구를 통하여 일반화가능도계수를 추정한 결과, 채점자 수에 따른 최적수준의 측정조건은 정확성 평가항목에서 8명의 채점자일 때 안정적인 신뢰도를 얻을 수 있으며, 표현성 평가항목에서는 7명의 채점자일 때 안정적인 신뢰도를 얻었다. This study aims to apply the G-theory for estimation of reliability of evaluation scores between raters on Taekwondo Poomsae rating categories. Selecting a number of game days and raters as multiple error sources, we analyzed the error sources caused by relative magnitude of error variances of interaction between the factors and proceeded with D-study based on the results of G-study for optimal determination of measurement condition. The results showed below. The estimated outcomes of variance component for accuracy among the Taekwondo Poomsae categories with G-theory showed that impact of error was the biggest influence factor in raters conditions and in order of interaction in subjects and between subjects, also impact of variance component estimation error on expression category was the major influence factor in interaction and in order of the between subjects and raters. Finally, the result of generalizability coefficient estimation via D-study showed that measurement condition of optimal level depend on the number of raters was 8 persons of raters on accuracy category, and stable reliability on expression category was gained when the raters were 7 persons.

Cross-cultural challenges in assessing medical professionalism among emergency physicians in a Middle Eastern Country (Bahrain): feasibility and psychometric properties of multisource feedback.

BackgroundMultisource feedback (MSF) is an evaluation tool whereby surveys assessing physicians are administered among medical peers and colleagues. Such evaluations provide physicians with non-biased valuations of both their strengths and their weaknesses, offering an opportunity for improvement in their work. Studies have shown that MSF is particularly effective for emergency care physicians.MethodsThe study was undertaken in a military teaching hospital in Bahrain. A total of 30 emergency physicians (the total number of emergency physicians in our hospital), 16 males and 14 females, were evaluated using multisource feedback. Each emergency physician was assessed by three groups of raters, including 4 emergency physicians, 4 referral physicians from other departments, and 4 coworkers from within the emergency department. Feasibility of the questionnaire was analyzed via response rates, average time required to complete it, and the number of raters required to produce reliable results. We used exploratory factor analysis to examine for the construct validity. Cronbach’s coefficient was calculated to measure the internal consistency reliability of the instrument.ResultsThe total mean response rate was 74.2 %, and the self-reported average time needed to fill out each survey was 4.3 min, indicating a good feasibility of the questionnaire. Reliability analysis indicated that the full-scale instrument had high internal consistency (Cronbach’s α 0.98). Factor analysis showed that the data on the questionnaire decomposed into three factors, which accounted for 72.6 % of the total variance: professionalism, collaboration, and communication. The generalizability coefficients (Ep2) were 0.76 for the surveys. Out of the 30 candidates, 26 participated in the knowledge test. The total mean score of the knowledge exam was 34.52, with scores ranging from 17 to 54.ConclusionsBased on this study’s results, we conclude that the instruments and procedures used have high reliability, validity, and feasibility in assessing the emergency physician in the emergency department in our clinical setting in the Middle East. The item analyses, reliability, and factor analyses all indicate that these instruments are effective in assessing emergency physicians.

Reliability analysis of the objective structured clinical examination using generalizability theory

BackgroundThe objective structured clinical examination (OSCE) is a widely used method for assessing clinical competence in health sciences education. Studies using this method have shown evidence of validity and reliability. There are no published studies of OSCE reliability measurement with generalizability theory (G-theory) in Latin America. The aims of this study were to assess the reliability of an OSCE in medical students using G-theory and explore its usefulness for quality improvement.MethodsAn observational cross-sectional study was conducted at National Autonomous University of Mexico (UNAM) Faculty of Medicine in Mexico City. A total of 278 fifth-year medical students were assessed with an 18-station OSCE in a summative end-of-career final examination. There were four exam versions. G-theory with a crossover random effects design was used to identify the main sources of variance. Examiners, standardized patients, and cases were considered as a single facet of analysis.ResultsThe exam was applied to 278 medical students. The OSCE had a generalizability coefficient of 0.93. The major components of variance were stations, students, and residual error. The sites and the versions of the tests had minimum variance.ConclusionsOur study achieved a G coefficient similar to that found in other reports, which is acceptable for summative tests. G-theory allows the estimation of the magnitude of multiple sources of error and helps decision makers to determine the number of stations, test versions, and examiners needed to obtain reliable measurements.

Sources and magnitude of variability in pedometer-determined physical activity levels of youth.

This study examined sources of variability in habitual physical activity of children, and the minimum number of days required for estimating children’s habitual physical activity levels using pedometers. A total of 31 children wore two pedometers during five weekdays and four weekend days. A two random facet completely crossed design was conducted with two-way analysis of variances across weekdays, weekends, and weekdays and weekend days combined. Moderate/high generalizability coefficients were estimated across all days. Primary sources of variability were variance components of the person and person by day interaction. Minimum numbers of days required for estimating habitual physical activity levels using a pedometer were five during weekdays. However, estimating habitual physical activity levels during weekends, and weekdays and weekend days combined was impractical.

Measuring multiple neuromuscular activation using EMG - a generalizability analysis.

Analysis of electromyography (EMG) data has been shown to be valuable in biomedical and clinical research. However, most analysis tools do not consider the non-linearity of EMG data or the synergistic effects of multiple neuromuscular activities. The SYNERGOS algorithm was developed to assess a single index based on non-linear analysis of multiple neuromuscular activation (MNA) of different muscles. This index has shown promising results in Parkinsonian gait, but it was yet to be explored whether the SYNERGOS index is generalizable. In this study, we evaluated generalizability of the SYNERGOS index over the course of several trials and over separate days with different walking speeds. Ten healthy adults aged from 18 to 40 years walked on a treadmill on two different days, while EMG data was collected from the upper and lower right leg. SYNERGOS indices were obtained and a generalizability analysis was conducted. The algorithm detected changes in MNA in response to altering gait speed and depicted a high generalizability coefficient ( ρ^2 ${\hat \rho ^2}$ ) of 0.823 with a standard error of 5.117 with nominal inter-trial or inter-day effects. We concluded SYNERGOS may be a valuable tool in EMG analysis due to its generalizability and its sensitivity to task modifications and associated neuromotor changes.

Cross-cultural challenges for assessing medical professionalism among clerkship physicians in a Middle Eastern country (Bahrain): feasibility and psychometric properties of multisource feedback.

BackgroundWe aimed to design, implement, and evaluate the feasibility and reliability of a multisource feedback (MSF) system to assess interns in their clerkship year in the Middle Eastern culture, the Kingdom of Bahrain.MethodThe study was undertaken in the Bahrain Defense Force Hospital, a military teaching hospital in the Kingdom of Bahrain. A total of 21 interns (who represent the total population of the interns for the given year) were assessed in this study. All of the interns were rotating through our hospital during their year-long clerkship rotation. The study sample consisted of nine males and 12 females. Each participating intern was evaluated by three groups of raters, eight medical intern colleagues, eight senior medical colleagues, and eight coworkers from different departments.ResultsA total of 21 interns (nine males and 12 females) were assessed in this study. The total mean response rates were 62.3%. A factor analysis was conducted that found that the data on the questionnaire grouped into three factors that counted for 76.4% of the total variance. These three factors were labeled as professionalism, collaboration, and communication. Reliability analysis indicated that the full instrument scale had high internal consistency (Cronbach’s α 0.98). The generalizability coefficients for the surveys were estimated to be 0.78.ConclusionBased on our results and analysis, we conclude that the MSF tool we used on the interns rotating in their clerkship year within our Middle Eastern culture provides an effective method of evaluation because it offers a reliable, valid, and feasible process.

생물자원 전통지식 추출을 위한 델파이조사의 신뢰성 연구

본 연구는 문헌 및 현장에서 얻은 생물자원지식들의 가치를 평가하기 위해 전문가들을 대상으로 실시한 델파이 조사 (Delphi method)의 신뢰성을 평가하였다. 델파이조사는 전문가들이 다른 사람의 의견과 관계없이 독립적으로 전문성 있는 평가를 내린 다음에 다음 단계에서 다른 전문가들의 의견을 참고하며 자신의 의견을 수정하는 절차를 갖는다. 본 연구에서는 문헌에서 얻은 전통지식 100건, 현장에서 취득한 지식 100건 등, 모두 200건을 우선 선정한 후 전문가 6인을 선정하여 각 지식의 가치에 대하여 두 차례 평가하도록 하였다. 그 결과 두 차례의 평가점수는 연관성이 매우 높으면서 2차에서는 다른 전문가들의 의견을 수용해 어느 정도 자체수정이 발생하여 각 문항에 대한 평가점수의 표준편차가 줄어들었다. 본 조사의 신뢰성 (reliability)을 파악하기 위해 일반적인 신뢰도 계수인 크론바하 알파와 함께 일반화가능도 (generalizability) 계수를 구하였다. 이 두 신뢰도 분석을 통해 2차 평가 후 평가의 신뢰도가 상승하여 전문가에 의한 델파이 조사의 신뢰도가 매우 높다는 사실을 지지하였으나 일반화 가능도 분석 결과를 해석하는 과정에서 다른 결과를 유추할 수 있었다. 신뢰도계수가 증가하였음에도 불구하고 평가자간의 편차는 증가하여 신뢰도가 높아진 것은 평가가 상향되고 평균에 회귀하는 경향으로 잔차변동이 줄어서이지 평가자간의 의견수렴이 이루어진 결과로 볼 수는 없다는 사실이었다. 이러한 결과를 토대로 신뢰도 계수와 함께 평가자 간의 분산을 파악하여 델파이조사의 추가적인 단계 (round)가 필요함을 제시하였다. In the knowledge and information age, to discover and protect Intellectual Properties would be very important for their economic value as a major growth engine. This study evaluated the reliability of a Delphi survey conducted by experts to assess the value of agricultural resources knowledge obtained from literature reviews and field interviews. Delphi method is collecting the opinions of experts for several rounds repeatedly, in the next round the experts have chance to modify their opinion. Scores between two rounds are highly correlated and standard deviations are declined for second round to imply that some correction of their evaluations are made. To check reliability of Delphi survey of two rounds Cronbach's reliability coefficient and Generalizability coefficient are derived. The Cronbach alpha's supported the reliability of the method, but the Generalizability analysis revealed some unexpected results while checking the variance components of sources of measurement errors. Despite the increased reliability coefficients, the deviations between the raters are increased which means that additional rounds are required to get consensus, the goal of Delphi research.

Application of generalizability theory in the examination of introduction of nursing

Objective To analysis the reliability of the examination of Introduction of Nursing by using the methods of generalizability theory and to give advice on how to modify examination paper. Methods Estimation value of variance component from different variations and precision indexes were calculated by applying the principle of generalizability theory. Results The generalizability coefficient and index of dependability were greater than 0.4 in choice question and terminology explanation, the generalizability coefficient of short-answer question was also greater than 0.4. Short-answer question's index of dependability and generalizability coefficient and index of dependability of comprehensive problem were less than 0.4. Conclusions Generalizability theory can be used in analysis the reliability of the examination of Introduction of Nursing. Its result is helpful to improve the framework of examination papers and the reliability of examination. Key words: Generalizability theory; Introduction of nursing; Examination

Simulator training for endobronchial ultrasound: a randomised controlled trial.

Endobronchial ultrasound-guided transbronchial needle aspiration (EBUS-TBNA) is very operator dependent and has a long learning curve. Simulation-based training might shorten the learning curve, and an assessment tool with solid validity evidence could ensure basic competency before unsupervised performance.A total of 16 respiratory physicians, without EBUS experience, were randomised to either virtual-reality simulator training or traditional apprenticeship training on patients, and then each physician performed EBUS-TBNA procedures on three patients. Three blinded, independent assessor assessed the video recordings of the procedures using a newly developed EBUS assessment tool (EBUSAT).The internal consistency was high (Cronbach's α=0.95); the generalisability coefficient was good (0.86), and the tool had discriminatory ability (p<0.001). Procedures performed by simulator-trained novices were rated higher than procedures performed by apprenticeship-trained novices: mean±sd are 24.2±7.9 points and 20.2±9.4 points, respectively; p=0.006. A pass/fail standard of 28.9 points was established using the contrasting groups method, resulting in 16 (67%) and 20 (83%) procedures performed by simulator-trained novices and apprenticeship-trained novices failing the test, respectively; p<0.001.The endobronchial ultrasound assessment tool could be used to provide reliable and valid assessment of competence in EBUS-TBNA, and act as an aid in certification. Virtual-reality simulator training was shown to be more effective than traditional apprenticeship training.

Measuring clinical skills in agenda-mapping (EAGL-I)

ObjectiveTo develop and validate the Evaluation of AGenda-mapping skilL Instrument (EAGL-I). MethodsEAGL-I was constructed after a literature review and piloting. Simulated consultation recordings were collected in a workshop with third-year medical students at three time points: once pre-teaching, twice post-teaching. Three raters used EAGL-I to assess student agenda-mapping. We examined reliability, ability to detect change and predict full expression of patients’ agendas. ResultsEAGL-I scores represented reliable assessment of agenda-mapping (Ep2=0.832; φ=0.675). Generalizability coefficients across items (Ep2=0.836) and raters (φ=0.797 two raters) were acceptable. A one-way repeated measure ANOVA with post hoc analysis found a statistically significant difference between the pre-teaching occasion of measurement and each post-teaching occasion (p<0.001) and no significant difference between the two post-teaching occasions (p=0.085). Multilevel logistic regression show scores predict expression of scripted hidden agendas irrespective of occasions, or patient scenario (n=60, p=0.005). ConclusionEvidence of measure validation is shown. Reliability is optimised when two or more raters use EAGL-I and agenda-mapping has been taught. EAGL-I appears sensitive to change. Higher scores predict the likelihood that a patient will disclose their full agenda in a simulated environment. Practice implicationsA validated tool for measuring agenda-mapping in teaching and research is now available.

Past-behavioural versus situational questions in a postgraduate admissions multiple mini-interview: a reliability and acceptability comparison

BackgroundThe Multiple Mini-Interview (MMI) mostly uses ‘Situational’ Questions (SQs) as an interview format within a station, rather than ‘Past-Behavioural’ Questions (PBQs), which are most frequently adopted in traditional single-station personal interviews (SSPIs) for non-medical and medical selection. This study investigated reliability and acceptability of the postgraduate admissions MMI with PBQ and SQ interview formats within MMI stations.MethodsTwenty-six Japanese medical graduates, first completed the two-year national obligatory initial postgraduate clinical training programme and then applied to three specialty training programmes - internal medicine, general surgery, and emergency medicine - in a Japanese teaching hospital, where they underwent the Accreditation Council for Graduate Medical Education (ACGME)-competency-based MMI. This MMI contained five stations, with two examiners per station. In each station, a PBQ, and then an SQ were asked consecutively. PBQ and SQ interview formats were not separated into two different stations, or the order of questioning of PBQs and SQs in individual stations was not changed due to lack of space and experienced examiners. Reliability was analysed for the scores of these two MMI question types. Candidates and examiners were surveyed on this experience.ResultsThe PBQ and SQ formats had generalisability coefficients of 0.822 and 0.821, respectively. With one examiner per station, seven stations could produce a reliability of more than 0.80 in both PBQ and SQ formats. More than 60% of both candidates and examiners felt positive about the overall candidates’ ability. All participants liked the fairness of this MMI when compared with the previously experienced SSPI. SQs were perceived more favourable by candidates; in contrast, PBQs were perceived more relevant by examiners.ConclusionsBoth PBQs and SQs are equally reliable and acceptable as station interview formats in the postgraduate admissions MMI. However, the use of the two formats within the same station, and with a fixed order, is not the best to maximise its utility as an admission test. Future studies are required to evaluate how best the SQs and PBQs should be combined as station interview formats to enhance reliability, feasibility, acceptability and predictive validity of the MMI.

A daily measure of positive and negative alcohol expectancies and evaluations: documenting a two-factor structure and within- and between-person variability.

Despite the documented importance of alcohol outcome expectancies in predicting alcohol use and related consequences, little research has explored within-person variability in expectancies. This article details the construction and psychometric analysis of a measure of alcohol expectancies specifically designed for daily assessment. We developed a 15-item instrument to measure the likelihood of experiencing various outcomes from drinking, as well as the subjective evaluation of these outcomes. College students (N = 352; mean age = 19.7 years, SD = 1.26; 53.4% female) participated in a yearlong study wherein they completed three computerized interviews daily via mobile phones for 2 weeks in each academic quarter. Multilevel exploratory factor analysis was used to examine dimensionality at between-person and within-person levels, and generalizability coefficients were calculated to establish reliability. Intraclass correlation coefficients were generally between .30 and .40, demonstrating both between-person and within-person variability. Exploratory factor analysis resulted in a two-factor solution of positive and negative effects of alcohol, and two items with equivocal loadings were dropped from the final scale. The two subscales showed excellent reliabilities at within- and between- person levels, and the measure demonstrated good convergent and discriminant validity with a commonly used expectancy measure. Drinkers hold many expectations about the effects of alcohol, and measures are needed that are designed to capture both stable and context-dependent aspects of these beliefs. Results demonstrated significant day-to-day variation in the strength and valuation of alcohol expectancies, and the scale demonstrated good psychometric properties that establish its appropriateness for use in daily process studies of alcohol use.

Reliability and validity of an extended clinical examination

Introduction: An extended clinical examination (ECE) was administered to 85 final year medical students at the Radboud University Medical Centre in the Netherlands. The aim of the study was to determine the psychometric quality and the suitability of the ECE as a measurement tool to assess the clinical proficiency of eight separate clinical skills.Methods: Generalizability studies were conducted to determine the generalizability coefficient and the sources of variance of the ECE. An additional D-study was performed to estimate the generalizability coefficients with altering numbers of stations.Results: The largest sources of variance were found in skill difficulties (36.18%), the general error term (26.76%) and in the rank ordering of skill difficulties across the stations (21.89%). The generalizability coefficient of the entire ECE was above the 0.70 lower bound (G = 0.74). D studies showed that the separate skills could yield sufficient G coefficients in seven out of eight skills, if the ECE was lengthened from 8 to 14 stations.Discussion: The ECE proved to be a reliable clinical assessment that enables examinees to compose a clinical reasoning path through self-obtained data. The ECE can also be used as an assessment tool for separate clinical skills.

Reliable and Valid Assessment of Point-of-care Ultrasonography

To explore the reliability and validity of the Objective Structured Assessment of Ultrasound Skills (OSAUS) scale for point-of-care ultrasonography (POC US) performance. POC US is increasingly used by clinicians and is an essential part of the management of acute surgical conditions. However, the quality of performance is highly operator-dependent. Therefore, reliable and valid assessment of trainees' ultrasonography competence is needed to ensure patient safety. Twenty-four physicians, representing novices, intermediates, and experts in POC US, scanned 4 different surgical patient cases in a controlled set-up. All ultrasound examinations were video-recorded and assessed by 2 blinded radiologists using OSAUS. Reliability was examined using generalizability theory. Construct validity was examined by comparing performance scores between the groups and by correlating physicians' OSAUS scores with diagnostic accuracy. The generalizability coefficient was high (0.81) and a D-study demonstrated that 1 assessor and 5 cases would result in similar reliability. The construct validity of the OSAUS scale was supported by a significant difference in the mean scores between the novice group (17.0; SD 8.4) and the intermediate group (30.0; SD 10.1), P = 0.007, as well as between the intermediate group and the expert group (72.9; SD 4.4), P = 0.04, and by a high correlation between OSAUS scores and diagnostic accuracy (Spearman ρ correlation coefficient = 0.76; P < 0.001). This study demonstrates high reliability as well as evidence of construct validity of the OSAUS scale for assessment of POC US competence. Hence, the OSAUS scale may be suitable for both in-training as well as end-of-training assessment.

Estimation of Generalizability Coefficient: An Application with Different Programs

Estimation of Generalizability Coefficient: An Application with Different Programs

Evaluation of satisfaction in an extracurricular enrichment program for high-intellectual ability participants.

The objective of this study was to evaluate the satisfaction of an extracurricular enrichment program of the cognitive and personal management of participants with high intellectual ability. At the first time point, the sample consisted of n= 38 participants, and n= 20 parents; n= 48 participants at the second time point; and n= 60 participants at the third time point. The Satisfaction Questionnaire (CSA in Spanish), both for students (CSA-S) and for parents (CSA-P), was constructed. The CSA-S scores showed adequate psychometric properties. Exploratory factor analysis yielded a unidimensional structure. Cronbach&rsquo;s alpha ranged between 85 and .86. Test-retest reliability was 0.45 (p&lt;.05). The generalizability coefficient was .98. A high percentage of the sample was satisfied with the program, perceived improvements in cognitive and emotional management, motivation and interest in learning, and in the frequency and quality of their interpersonal relationships. The evaluation of educational programs is necessary in order to determine the efficacy and the effects of their implementation on the participants&rsquo; personal and intellectual management.

Comparing the Effectiveness of SPSS and EduG using Different Designs for Generalizability Theory

AbstractGeneralizability theory (G theory) provides a broad conceptual framework for social sciences such as psychology and education, and a comprehensive construct for numerous measurement events by using analysis of variance, a strong statistical method. G theory, as an extension of both classical test theory and analysis of variance, is a model which can deal with multiple sources of error. In conducting the analysis of the G theory, there are several software programs that can be used such as GENOVA, SPSS, SAS, EduG, and G-String. In this study, the general perspectives of G theory are first explained broadly. Then, the SPSS and EduG software programs are used to conduct generalizability analyses on the data obtained from the answers of 30 students (p) to nine open-ended questions (i) as rated by three raters (r). There are three different designs in the study. Two of them are random effects designs, pxixr and pxi:r, and the last one is pxixr design using a fixed rater . According to the findings from the study, SPSS and EduG give the same results for variance component estimates as well as for G (Generalizability) and D (Decision) studies of all designs, as expected. Besides comparing the program outputs, their weaknesses and strengths were also discussed regarding different designs and data sets in this study.Keywords: Generalizability Theory * G Study * D Study * SPSS * EduGG theory has formed a comprehensive structure by employing variance analysis which provides a broad conceptual framework for social sciences such as psychology and education (Brennan, 2000, 2001a; Cronbach, Gleser, Nanda, & Rajaratnam, 1972; Shavelson & Webb, 1991). It is also a powerful statistical tool for situations where there are numerous measurements. The theory, as an extension of classical test theory and variance analysis, stands as a model where multiple sources of error can be handled (Brennan, 2001a; Shavelson & Webb, 1991).Generalizability (G) TheoryThe reliability of measurement results in education and psychology was previously examined using classical test theory (CTT) in general. It is assumed in CTT that the observed score is composed of the actual score with no separable score for error. The restriction of this assumption, especially in performance measurements where the probability of the existence of more than one source of error is high, reveals the importance of G theory in which more than one source of error is handled and can be predicted simultaneously (Brennan, 2000). Another advantage of G theory in using performance assessment is that while there is a restrictive parallel assumption in CTT, randomly parallel assumption is adopted in G theory (Brennan, 2011; Kretchmar, 2006). The main aim of G theory is to generalize the scores of a specific measurement tool from a specific group to the universe of generalization which consist of 1) the universe of admissible observations and generalizability studies (G studies), 2) the universe of G studies and decision studies (D studies). While G studies provide an estimate of the generalizability coefficient of variances from all facets and this coefficient includes the examinee's universe score, D studies enable one to examine the interactions among all applicable facets (tasks, raters, observations, etc.) and the subject of measurement for calculating the dependability coefficient (Brennan, 2000; Crocker & Algina, 1986; Hsu, 2012).G theory has four main advantages compared to CTT. 1) It provides simultaneous evaluation of test-retest reliability, internal consistency, inter-rater reliability, and convergent validity. 2) It enables estimates of both individual measurement facets and interaction effects. 3) When assessing an examinee's performance, it gives information about the quality of their absolute structural level of knowledge as well as ranking this information in order. 4) It allows researchers to optimize the reliability of an assessment within the cost constraints of time and money. …

Language Shift and the Inclusion of Indigenous Populations in Large-Scale Assessment Programs

Indicators of academic achievement for bilingual students can be inaccurate due to linguistic heterogeneity. For indigenous populations, language shift (the gradual replacement of one language by another) is a factor that can increase this heterogeneity and poses an additional challenge for valid testing. We investigated whether and how indigenous populations can be validly included in a large-scale assessment program. We gave Mexican preschool Mayan students aged 5 to 6 years the same set of mathematics items in three versions: (1) original in Spanish, (2) Mayan translation, and (3) content equivalent, developed from scratch in Mayan. Also, we collected information on the students’ and the teachers’ use of Spanish and Mayan, and on the communities’ support of the two languages. Students performed poorly on the three versions. Generalizability theory-based analyses revealed considerable performance inconsistency across items and language versions and low generalizability and dependability coefficients. This performance instability appears to stem from a dwindling support of the Mayan language in Mayan schools and communities. Fair, valid assessment of indigenous populations in either their languages or their countries’ dominant languages appears to be difficult to accomplish with current testing models and policies.

Dental students' peer assessment: a prospective pilot study.

Peer assessment is increasingly used in health education. The aims of this study were to evaluate the reliability, accuracy, educational impact and student's perceptions of undergraduate pre-clinical and clinical dental students' structured and prospective Peer assessment and peer feedback protocol. Two Direct Observation of Procedural Skills (DOPS) forms were modified for use in pre-clinical and clinical peer assessment. Ten year two dental students working in a phantom-heads skills laboratory and 16-year five dental students attending a comprehensive care clinic piloted both peer DOPS forms. After training, pairs of students observed, assessed and provided immediate feedback to each other using their respective peer DOPS forms as frameworks. At the end of the 3-month study period, students anonymously provided their perceptions of the protocol. Year 2 and year 5 students completed 57 and 104 peer DOPS forms, respectively. The generalizability coefficient was 0.62 for year 2 (six encounters) and 0.67 for year 5 (seven encounters). Both groups were able to differentiate amongst peer-assessed domains and so detect improvement in peers' performance over time. Peer DOPS scores of both groups showed a positive correlation with their mean end-of-year examination marks (r ≥ 0.505, P ≥ 0.051) although this was not statistically significant. There was no difference (P ≥ 0.094) between the end-of-year examination marks of the participating students and the rest of their respective classes. The vast majority of both groups expressed positive perceptions of the piloted protocol. There are no data in the literature on the prospective use of peer assessment in the dental undergraduate setting. In the current study, both pre-clinical and clinical students demonstrated the ability to identify those domains where peers performed better, as well as those which needed improvement. Despite no observable educational impact, most students reported positive perceptions of the peer DOPS protocol. The results of this pilot study support the need for and the potential benefit of a larger- and longer-term follow-up study utilising the protocol.

Standardised clients as assessors in a veterinary communication OSCE: a reliability and validity study

In human medicine, standardised patients (SP) have been shown to reliably and accurately assess learners’ communication performance in high-stakes certification Objective Structured Clinical Examinations (OSCE), offering a feasible way to...