Examinee Responses Research Articles

Validation of a Script Concordance Test Combined with ECG during an Emergency Medicine Clerkship Rotation

One of the emergency physician's challenges is to think, to act, and to treat patients with a high level of uncertainty (for instance: incomplete past history of the patient, life-threatening decisions with poor clinical information). In recent years, Script Concordance Tests (SCTs) are increasingly used by educators to assess clinical reasoning, especially in situations of uncertainty. They compare the responses of examinees (students, trainees, or residents) with those of a panel of experts (PE). The examinee's answers are scored based on the level of agreement with responses provided by a PE. In recent years, teachers in Emergency Medicine (EM) also use SCTs. In emergency situations, the interpretation of an electrocardiogram (ECG), which is a commonly performed test, encounters a certain level of uncertainty. A last point could not be forgotten: according to the recent literature, the training of future emergency physicians remains challenging. Facing these three corner stones (uncertainty among emergency situations, uncertainty among ECG's interpretation, and challenge in training), the authors decided to construct and validate an SCT combined with an ECG (SCT-ECG) by submitting it to students, trainees, and residents during their emergency medicine clerkship rotation.

Multilevel Modeling of Item Position Effects

In many testing programs it is assumed that the context or position in which an item is administered does not have a differential effect on examinee responses to the item. Violations of this assumption may bias item response theory estimates of item and person parameters. This study examines the potentially biasing effects of item position. A hierarchical generalized linear model is formulated for estimating item‐position effects. The model is demonstrated using data from a pilot administration of the GRE wherein the same items appeared in different positions across the test form. Methods for detecting and assessing position effects are discussed, as are applications of the model in the contexts of test development and item analysis.

Rater Comparability Scoring and Equating: Does Choice of Target Population Weights Matter in This Context?

When a constructed-response test form is reused, raw scores from the two administrations of the form may not be comparable. The solution to this problem requires a rescoring, at the current administration, of examinee responses from the previous administration. The scores from this “rescoring” can be used as an anchor for equating. In this equating, the choice of weights for combining the samples to define the target population can be critical. In rescored data, the anchor usually correlates very strongly with the new form but only moderately with the reference form. This difference has a predictable impact: the equating results are most accurate when the target population is the reference form sample, least accurate when the target population is the new form sample, and somewhere in the middle when the new form and reference form samples are equally weighted in forming the target population.

The Impact of Item Format and Examinee Characteristics on Response Times

Current research on examination response time has focused on tests comprised of traditional multiple-choice items. Consequently, the impact of other innovative or complex item formats on examinee response time is not understood. The present study used multilevel growth modeling to investigate examinee characteristics associated with response time differences on a medical certification exam comprised of two item formats. A linear model described examinee pacing on the traditional multiple-choice section, while a quadratic or curvilinear model described pacing on a diagnostic study section comprised of complex graphic-intensive multiple-response items. Examinees’ gender, ability, and age explained variability in response times for each exam section; notably, older examinees’ initial pacing was slower on the multiple-choice section but faster on the complex graphic-intensive multiple-response section. This study has implications for test developers throughout the world who intend to incorporate complex item formats into their high-stakes exams.

Critical Synthesis Package: Script Concordance Testing (SCT)

OPEN ACCESSAugust 14, 2013Critical Synthesis Package: Script Concordance Testing (SCT) Javarro Russell, PhD Javarro Russell, PhD National Board of Medical Examiners Google Scholar More articles by this author https://doi.org/10.15766/mep_2374-8265.9492 SectionsAbout ToolsDownload Citations ShareFacebookTwitterEmail AbstractThis Critical Synthesis Package contains: (1) a Critical Analysis of the psychometric properties and application to health sciences education for the Script Concordance Test (SCT), and (2) a copy of the SCT instrument and SCT Overview developed by Bernard Charlin, MD, PhD.The SCT provides a method for assessing clinical data interpretation skills. This method uses a written simulation item format to allow respondents to address ill-defined problems typical to clinical practice. The examinee is presented with ill-defined clinical scenarios. These scenarios are then followed by a set of hypotheses. For each hypothesis the examinee is provided with an additional piece of information. The examinee is then required to make judgments as to the likelihood of the hypothesis by selecting a Likert-type response option. Points awarded for the examinee's concordance with the judgments are provided by a panel of expert judges. The SCT has been used across many clinical domains and with examinees of varying levels of clinical experience. Research on the SCT has identified guidelines for developing quality test items. An advantage of using this item format is the use of expert panels to analyze and compare examinee responses. Disadvantages to using this item format include the lack of validity evidence for the response process and the complexity of the scoring process. Further research is needed to determine how best to use the SCT to improve teaching and learning, and to evaluate the thought process that is used in responding to the SCT items. Additionally, research is needed to identify how best to use examinee scores diagnostically. Current research has not obtained validity evidence for the use of scores to prescribe educational interventions to increase examinee knowledge. Educational Objectives To describe the purpose and basic properties of Script Concordance Testing (SCT), including psychometric properties;To describe the application of SCT to the field of health sciences education;To evaluate the relative strengths and weaknesses of SCT; andTo provide SCT and supplemental materials to aid in its administration. Sign up for the latest publications from MedEdPORTAL Add your email below FILES INCLUDEDReferencesRelatedDetails FILES INCLUDED Included in this publication: Critical Analysis of Script Concordance Testing.pdf Sample Script Concordance Test.docx Script Concordance Testing Overview.docx To view all publication components, extract (i.e., unzip) them from the downloaded .zip file. Download editor’s noteThis publication may contain technology or a display format that is no longer in use. CitationRussell J. Critical Synthesis Package: Script Concordance Testing (SCT). MedEdPORTAL. 2013;9:9492. https://doi.org/10.15766/mep_2374-8265.9492 Copyright & Permissions© 2013 Russell. This is an open-access publication distributed under the terms of the Creative Commons Attribution-NonCommercial-Share Alike license.KeywordsDirectory and Repository of Educational Assessment MeasuresSCTEducational MeasurementClinical ReasoningInstrument ValidationValidation StudiesDREAMScript Concordance Testing Disclosures None to report. Funding/Support None to report. Loading ...

Using Neural Network Analysis to Define Methods of DINA Model Estimation for Small Sample Sizes

The DINA model is a commonly used model for obtaining diagnostic information. Like many other Diagnostic Classification Models (DCMs), it can require a large sample size to obtain reliable item and examinee parameter estimation. Neural Network (NN) analysis is a classification method that uses a training dataset for calibration. As a result, if this training dataset is determined theoretically, as was the case in Gierl’s attribute hierarchical method (AHM), the NN analysis does not have any sample size requirements. However, a NN approach does not provide traditional item parameters of a DCM or allow for item responses to influence test calibration. In this paper, the NN approach will be implemented for the DINA model estimation to explore its effectiveness as a classification method beyond its use in AHM. The accuracy of the NN approach across different sample sizes, item quality and Q-matrix complexity is described in the DINA model context. Then, a Markov Chain Monte Carlo (MCMC) estimation algorithm and Joint Maximum Likelihood Estimation is used to extend the NN approach so that item parameters associated with the DINA model are obtained while allowing examinee responses to influence the test calibration. The results derived by the NN, the combination of MCMC and NN (NN MCMC) and the combination of JMLE and NN are compared with that of the well-established Hierarchical MCMC procedure and JMLE with a uniform prior on the attribute profile to illustrate their strength and weakness.

Obtaining Diagnostic Classification Model Estimates Using Mplus

Diagnostic classification models (aka cognitive or skills diagnosis models) have shown great promise for evaluating mastery on a multidimensional profile of skills as assessed through examinee responses, but continued development and application of these models has been hindered by a lack of readily available software. In this article we demonstrate how diagnostic classification models may be estimated as confirmatory latent class models using Mplus, thus bridging the gap between the technical presentation of these models and their practical use for assessment in research and applied settings. Using a sample English test of three grammatical skills, we describe how diagnostic classification models can be phrased as latent class models within Mplus and how to obtain the syntax and output needed for estimation and interpretation of the model parameters. We also have written a freely available SAS program that can be used to automatically generate the Mplus syntax. We hope this work will ultimately result in greater access to diagnostic classification models throughout the testing community, from researchers to practitioners.

Setting Standards for English Foreign Language Assessment: Methodology, Validation, and a Degree of Arbitrariness

Cut‐scores were set by expert judges on assessments of reading and listening comprehension of English as a foreign language (EFL), using the bookmark standard‐setting method to differentiate proficiency levels defined by the Common European Framework of Reference (CEFR). Assessments contained stratified item samples drawn from extensive item pools, calibrated using Rasch models on the basis of examinee responses of a German nationwide assessment of secondary school language performance. The results suggest significant effects of item sampling strategies for the bookmark method on cut‐score recommendations, as well as significant cut‐score judgment revision over cut‐score placement rounds. Results are discussed within a framework of establishing validity evidence supporting cut‐score recommendations using the widely employed bookmark method.

Evaluating the Consistency of Test Items Relative to the Cognitive Model for Educational Cognitive Diagnosis

Attributes and their hierarchy may present a cognitive model.Building a cognitive model is one of key steps for cognitive diagnosis as it is directly related to the validity and usefulness of test results.It is very important to detect whether the test specification coincides with the cognitive model before administering the test.In this paper,an explicit index is given to measure the extent to which the cognitive model is represented by the test items of the diagnostic test.We call this index as the theoretical construct validity(TCV).In terms of TCV,the test reported by Tatsuoka and her colleagues(1988) is reanalyzed,and the TCV of the test is only 9/24.That is,24 knowledge states are obtained from the theoretic cognitive model but the test specification could distinguish only 9 knowledge states.Cui and her colleagues established a person fit index named hierarchy consistency index(HCI) to detect the fitness of an examinee's observed response pattern(ORP) to an expected response pattern(ERP).HCI is not defined well when an examinee mastered one attribute only and there is only one item measuring the same attribute in the test,and the examinee responses correctly to the item.The original HCI could not compute for the number of comparisons being zero,hence the denominator being zero.In addition,the HCI includes the slipping only.Combining the slipping and guessing in the response to an item,we proposed a new index called modified HCI(MHCI).MHCI could detect the consistency of the response data and the cognitive model that is obtained by experts.

Item Difficulty and Interviewer Knowledge Effects on the Accuracy and Consistency of Examinee Response Processes in Verbal Reports

The Standards for Educational and Psychological Testing indicate that multiple sources of validity evidence should be used to support the interpretation of test scores. In the past decade, examinee response processes, as a source of validity evidence, have received increased attention. However, there have been relatively few methodological studies of the accuracy and consistency of examinee response processes as measured by verbal reports in the context of educational measurement. The objective of the current study was to investigate the accuracy and consistency of examinee response processes—as measured by verbal reports—as a function of varying interviewer and item variables in a think aloud interview within an educational measurement context. Results indicate that the accuracy of responses may be undermined when students perceive the interviewer to be an expert in the domain. Further, the consistency of response processes may be undermined when items that are too easy or difficult are used to elicit reports. The implications of these results for conducting think-aloud studies are explored.

Assessing clinical reasoning using a script concordance test with electrocardiogram in an emergency medicine clerkship rotation

ObjectivesScript concordance tests (SCTs) can be used to assess clinical reasoning, especially in situations of uncertainty, by comparing the responses of examinees with those of emergency physicians. The examinee's answers...

Modeling Pacing Behavior and Test Speededness Using Latent Growth Curve Models

This research explores the usefulness of latent growth curve modeling in the study of pacing behavior and test speededness. Examinee response times from a high-stakes, computerized examination, collected before and after the examination was subjected to a timing change, were analyzed using a series of latent growth curve models to detect identifiable patterns of examinee pacing behavior. To help explain how examinees progress through the examination, the influences of two important predictor variables were tested: examinees’ native language and overall proficiency. Results illustrate how group-specific changes in the relationship between proficiency and response times and a phase-specific interaction effect would have gone unnoticed if a longitudinal perspective had not been used. The findings suggest that growth curve modeling is a useful tool for modeling change in test speed as a continuous process.

A Comparison between Discrimination Indices and Item-Response Theory Using the Rasch Model in a Clinical Course Written Examination of a Medical School.

The reliability of test is determined by each items' characteristics. Item analysis is achieved by classical test theory and item response theory. The purpose of the study was to compare the discrimination indices with item response theory using the Rasch model. Thirty-one 4th-year medical school students participated in the clinical course written examination, which included 22 A-type items and 3 R-type items. Point biserial correlation coefficient (C(pbs)) was compared to method of extreme group (D), biserial correlation coefficient (C(bs)), item-total correlation coefficient (C(it)), and corrected item-total correlation coeffcient (C(cit)). Rasch model was applied to estimate item difficulty and examinee's ability and to calculate item fit statistics using joint maximum likelihood. Explanatory power (r2) of Cpbs is decreased in the following order: C(cit) (1.00), C(it) (0.99), C(bs) (0.94), and D (0.45). The ranges of difficulty logit and standard error and ability logit and standard error were -0.82 to 0.80 and 0.37 to 0.76, -3.69 to 3.19 and 0.45 to 1.03, respectively. Item 9 and 23 have outfit > or =1.3. Student 1, 5, 7, 18, 26, 30, and 32 have fit > or =1.3. C(pbs), C(cit), and C(it) are good discrimination parameters. Rasch model can estimate item difficulty parameter and examinee's ability parameter with standard error. The fit statistics can identify bad items and unpredictable examinee's responses.

F-Type Testlets and the Effects of Feedback and Case-Specificity

A novel type of item sets, "f-type" testlets, was recently introduced on the United States Medical Licensing Examination. These testlets contain two or more questions associated with a common clinical scenario. In some cases, as the scenario unfolds, examinees are indirectly provided with feedback about their response to a testlet question. The effects of this format and of the provision of feedback to examinees about their performance are investigated. Examinee behavior is predicted using an item response model, and observed examinee responses are compared with model expectations for f-type testlets. Mean model-data discrepancies among specific examinee groups are compared to study the dependencies across within-testlet items (i.e., case-specificity) and the impact of providing feedback. Findings showed that case-specificity effects were present (on average) for all examinee subgroups except examinees who both responded unsuccessfully to the initial item within an f-type testlet and received feedback. Case-specificity effects were negative for examinees who responded unsuccessfully to the initial testlet item but did not receive feedback. For those who responded successfully to the initial testlet items, case-specificity effects were positive. Results suggest that responses to test questions within an f-type testlet are not independent-even after accounting for examinee proficiency and item characteristics. Case-specificity effects (i.e., dependencies) were observed on average for all examinees except those who both responded unsuccessfully to the initial item within an f-type testlet and received feedback. Research into modeling these effects through the use of more general item response models is recommended.

The interrelations of features of questions, mark schemes and examinee responses and their impact upon marker agreement

In this paper we develop an evidence-based framework for considering many of the factors affecting marker agreement in GCSEs and A levels. A logical analysis of the demands of the marking task suggests a core grouping comprising: (i) question features; (ii) mark scheme features; and (iii) examinee response features. The framework synthesises several strands of previous empirical research. Additionally, we present new data, collected by observing the social dynamics of examiner meetings, to illustrate some effects of these interactions. We use our framework to argue that the two most important goals of research into marker agreement are prediction and control. That is, assessment agencies need to be able not only to predict the level of agreement between a marker’s mark and the ‘definitive’ mark, but also to anticipate the effects of different possible courses of action on the level of agreement. We conclude with a list of possible courses of action, considering their effect not only on marking accuracy, but also on overall validity – the overriding concern.

Assessment of clinical reasoning: A Script Concordance test designed for pre-clinical medical students

Background: The Script Concordance test (SCT) measures clinical reasoning in the context of uncertainty by comparing the responses of examinees and expert clinicians. It uses the level of agreement with a panel of experts to assign credit for the examinee's answers.Aim: This study describes the development and validation of a SCT for pre-clinical medical students.Methods: Faculty from two US medical schools developed SCT items in the domains of anatomy, biochemistry, physiology, and histology. Scoring procedures utilized data from a panel of 30 expert physicians. Validation focused on internal reliability and the ability of the SCT to distinguish between different cohorts.Results: The SCT was administered to an aggregate of 411 second-year and 70 fourth-year students from both schools. Internal consistency for the 75 test items was satisfactory (Cronbach's alpha = 0.73). The SCT successfully differentiated second- from fourth-year students and both student groups from the expert panel in a one-way analysis of variance (F2,508 = 120.4; p < 0.0001). Mean scores for students from the two schools were not significantly different (p = 0.20).Conclusion: This SCT successfully differentiated pre-clinical medical students from fourth-year medical students and both cohorts of medical students from expert clinicians across different institutions and geographic areas. The SCT shows promise as an easy-to-administer measure of “problem-solving” performance in competency evaluation even in the beginning years of medical education.

Script concordance testing: a review of published validity evidence

Script concordance test (SCT) scores are intended to reflect respondents' competence in interpreting clinical data under conditions of uncertainty. The validity of inferences based on SCT scores has not been rigorously established. This study was conducted in order to develop a structured validity argument for the interpretation of test scores derived through use of the script concordance method. We searched the PubMed, EMBASE and PsycINFO databases for articles pertaining to script concordance testing. We then reviewed these articles to evaluate the construct validity of the script concordance method, following an established approach for analysing validity data from five categories: content; response process; internal structure; relations to other variables, and consequences. Content evidence derives from clear guidelines for the creation of authentic, ill-defined scenarios. High internal consistency reliability supports the internal structure of SCT scores. As might be expected, SCT scores correlate poorly with assessments of pure factual knowledge, in which correlations for more advanced learners are lower. The validity of SCT scores is weakly supported by evidence pertaining to examinee response processes and educational consequences. Published research generally supports the use of SCT to assess the interpretation of clinical data under conditions of uncertainty, although specifics of the validity argument vary and require verification in different contexts and for particular SCTs. Our review identifies potential areas of further validity inquiry in all five categories of evidence. In particular, future SCT research might explore the impact of the script concordance method on teaching and learning, and examine how SCTs integrate with other assessment methods within comprehensive assessment programmes.

A Monte Carlo Simulation Investigating the Validity and Reliability of Ability Estimation in Item Response Theory with Speeded Computer Adaptive Tests

Imposed time limits on computer adaptive tests (CATs) can result in examinees having difficulty completing all items, thus compromising the validity and reliability of ability estimates. In this study, the effects of speededness were explored in a simulated CAT environment by varying examinee response patterns to end-of-test items. Expectedly, ability estimates became increasingly negatively biased as the CAT became more speeded, with the bias magnitude depending on the speededness condition. Ability estimates for higher-ability examinees were also influenced more than estimates for low- and middle-ability examinees with the realistic item pool. This finding is likely linked to test operational characteristics since these results were not replicated with an ideal item pool. In any case, the CAT is relatively robust to the speededness assumption, assuming that only a few items are speeded. Results also suggested that if test developers score the test at the point of speededness, ability estimates remain unbiased. This study's findings help inform test developers and test takers about the effects of test speededness within a CAT environment. Additional implications for educators and researchers are discussed.

Item-fit evaluation in biased tests: a study under Rasch model

In this paper, the power rates and distributional properties of the Outfit, Infit, Lz, ECI2z and ECI4z statistics when they are used in tests with biased or differential item functioning (DIF) were explored. In this study, different conditions of sample size, sample size ratio focal and reference group, impact between groups, DIF effect size, and percentage of DIF items were manipulated. In addition, examinee responses were generated to simulate uniform DIF. Results suggest that item fit statistics generally detected medium percents of DIF in large samples (1000/500 or 1000/1000) only when DIF effect size was relatively high and when the mean of focal and reference group was different. Moreover, when groups had equal mean, low correct identification rates were found in the five item-fit indices. In general, the results showed adequate control of false positive rates. These findings lead to the conclusion that all indices used in this study are partially adequate fit measures for detecting biased items, mainly when impact between groups is present and sample size is large.

Using Log‐Linear Smoothing to Improve Small‐Sample DIF Estimation

This study examined the extent to which log‐linear smoothing could improve the accuracy of differential item functioning (DIF) estimates in small samples of examinees. Examinee responses from a certification test were analyzed using White examinees in the reference group and African American examinees in the focal group. Using a simulation approach, separate DIF estimates for seven small‐sample‐size conditions were obtained using unsmoothed (U) and smoothed (S) score distributions. These small sample U and S DIF estimates were compared to a criterion (i.e., DIF estimates obtained using the unsmoothed total data) to assess their degree of variability (random error) and accuracy (bias). Results indicate that for most studied items smoothing the raw score distributions reduced random error and bias of the DIF estimates, especially in the small‐sample‐size conditions. Implications of these results for operational testing programs are discussed.