Test Blueprint Research Articles

Content validity of a professionalism OSCE developed in family medicine training University of Medicine and Pharmacy at Ho Chi Minh city Vietnam

Assessments of professional behaviors such as professionalism Mini clinical examination (PMEX) and OSCE (POSCE) are playing an important role in driving the practice professionalism in medical training. Simulation-based assessment or POSCE has been used to evaluate several professional attributes. However, few evidence of content validity proving that whether POSCE is really measuring specifi professional attributes have been reported in medical education literature. The four-step process of developing FM POSCE was analyzed to highlight the validity evidence according framework of Downing (2003). Group of 5 independent FM experts from Vietnam, Boston University, US and Liege, Belgium evaluated the blueprints, scenarios and item lists on a scale (1-totally disagree to 5-totally agree) regarding to what extent test blueprint, cases and item lists were relevant to the content domains and cultural context. The results of their evaluation were considered as an evidence of content validity. The mean and standard deviation of the scores given by them were calculated using SPSS, 20.0. Important evidence of content validity were found in the process of developing POSCE. Content experts’ evaluation showed that all professional attributes represented medical professionalism. However, the adequacy of professional attributes to evaluate the broad construct of professionalism was controversial (M=3.75, SD=0.95). Cases are relevant to assess these professional attributes and culture. Only the cases of “Respecting the patient” and “Making altruistic decision” contained some inappropriate marking items (M=3.75, SD=0.95; M=3.00, SD=0.92). FM POSCE developed in Vietnamese context can assess six specifi professional attributes. This study suggested a process of developing POSCE that has several features such as using both sources of expertise and medical literature to build up the content of POSCE to improve the content validity.

IRT Scoring and Test Blueprint Fidelity

This article focuses on the topic of how item response theory (IRT) scoring models reflect the intended content allocation in a set of test specifications or test blueprint. Although either an adaptive or linear assessment can be built to reflect a set of design specifications, the method of scoring is also a critical step. Standard IRT models employ a set of optimal scoring weights, and these weights depend on item parameters in the two-parameter logistic (2PL) and three-parameter logistic (3PL) models. The current article is an investigation of whether the scoring models reflect an intended set of weights defined as the proportion of item falling into each cell of the test blueprint. The 3PL model is of special interest because the optimal scoring weights depend on ability. Thus, the concern arises that for examinees of low ability, the intended weights are implicitly altered.

Test Specifications and Blueprints: Reality and Expectations

Test Specifications and Blueprints: Reality and Expectations

Creating a test blueprint for a progress testing program: A paired-comparisons approach

Context: Creating a new testing program requires the development of a test blueprint that will determine how the items on each test form are distributed across possible content areas and practice domains. To achieve validity, categories of a blueprint are typically based on the judgments of content experts. How experts judgments are elicited and combined is important to the quality of resulting test blueprints.Methods: Content experts in dentistry participated in a day-long faculty-wide workshop to discuss, refine, and confirm the categories and their relative weights. After reaching agreement on categories and their definitions, experts judged the relative importance between category pairs, registering their judgments anonymously using iClicker, an audience response system. Judgments were combined in two ways: a simple calculation that could be performed during the workshop and a multidimensional scaling of the judgments performed later.Results: Content experts were able to produce a set of relative weights using this approach. The multidimensional scaling yielded a three-dimensional model with the potential to provide deeper insights into the basis of the experts’ judgments.Conclusion: The approach developed and demonstrated in this study can be applied across academic disciplines to elicit and combine content experts judgments for the development of test blueprints.

Using Structured-Option Test Questions to Assess Competency Attainment: Part I.

Nursing is a practice profession, and nurse educators have the responsibility to ensure learner attainment of competency at all levels of nursing education. This two-part article provides basic information about designing structured-option test questions to assess and evaluate competency attainment. Part one discusses the uses, advantages, and limits of using structured-option test questions and explains how to develop a structured-option test question. Part two provides information about developing a test blueprint, administering the test, interpreting test results, and revising the test questions. J Contin Educ Nurs. 2017;48(5):207-208.

Toward Application and Testing of Measurement Scales and an Example

Extending the discussion on replication in sociology and psychology, this article focuses on the adaption of scales in research. Drawing on a recently published scale, we highlight several validity issues in the scale’s composition and development. Using data from two empirical studies, we subsequently show how scales can be tested and adapted. In doing so, this article provides a blueprint for scale testing and helps avoid the use of inappropriate scales in research.

Development and Validation of Economics Achievement Test for Secondary Schools

Economics achievement test (EAT) for assessing senior secondary two (SS2) achievement in economics was developed and validated in the study. Five research questions guided the study. Twenty and 100 mid-senior secondary (SS2) economics students was used for the pilot testing and reliability check respectively. A sample of 250 students randomly drawn was used to subject initial 80 objective test items for the test try-out that yielded the data for item analysis. 50 items with difficulty indices ranged from 0.25 to 0.79 and discrimination indices of 0.20 to 0.58 where retained. Face and content validation of EAT was ensured by constructing items in line with the test blue print, the use of subject experts in SS2 economics and two experts in test construction. The test reliability established through Kuder-Richardson formula 20 gave a coefficient of 0.81. The test was found to be of good quality, valid and highly reliable. The EAT is therefore recommended for use in assessing SS2 students’ achievement in economics and to determine/predict students that will do well in economics in their final class (SS3) as well as those that will have good performance in economics external examinations (WAEC and NECO). Economics achievement test (EAT) for assessing senior secondary two (SS2) achievement in economics was developed and validated in the study. Five research questions guided the study. Twenty and 100 mid-senior secondary (SS2) economics students was used for the pilot testing and reliability check respectively. A sample of 250 students randomly drawn was used to subject initial 80 objective test items for the test try-out that yielded the data for item analysis. 50 items with difficulty indices ranged from 0.25 to 0.79 and discrimination indices of 0.20 to 0.58 where retained. Face and content validation of EAT was ensured by constructing items in line with the test blue print, the use of subject experts in SS2 economics and two experts in test construction. The test reliability established through Kuder-Richardson formula 20 gave a coefficient of 0.81. The test was found to be of good quality, valid and highly reliable. The EAT is therefore recommended for use in assessing SS2 students’ achievement in economics and to determine/predict students that will do well in economics in their final class (SS3) as well as those that will have good performance in economics external examinations (WAEC and NECO).

Analysis of undergraduate pharmacology annual written examination papers at Pt. B. D. Sharma University of Health Sciences Rohtak

Background: Written examinations are the most commonly employed methods for assessing the knowledge and clinical competence of undergraduate medical students. The assessment tools commonly employed in the theory paper are reasoning and short note type of questions. However, a few disadvantages such as less number of questions, unfair representation or distribution of topics, different difficulty level of questions, vague questions are often encountered in question papers of written examinations. Aims and Objectives: To evaluate the content validity/adequate coverage of different subdivisions of pharmacology in undergraduate written examination. Materials and Methods: For the analysis of written examination, all question papers of pharmacology second professional MBBS examination of Pt. B. D. Sharma University of Health Sciences from 2006 to 2015 were examined. In this retrospective study, we evaluated the pharmacology annual university theory examination question papers of II nd MBBS students. The students have to answer two question papers: Papers I and II, each being of 40 marks. Results: It was observed from the question papers that major systems such as autonomic nervous system, cardiovascular systems, central nervous system, and antimicrobials had almost equal weightage throughout the 10 years. However, some important topics such as anticancer drugs, analgesics (autacoids), immunosuppressants (miscellaneous) were not at all covered in many consecutive years (2011-2015). Conclusion: Assessment should become an instrument for promoting growth rather than simply measuring it. It should not be used for merely classification, grading, and certification. Therefore, methods like test blueprinting and table of specifications should be used during test construction process to improve the assessment system.

Integrated Approach to Software Defect Prediction

Software defect prediction provides actionable outputs to software teams while contributing to industrial success. Empirical studies have been conducted on software defect prediction for both cross-project and within-project defect prediction. However, existing studies have yet to demonstrate a method of predicting the number of defects in an upcoming product release. This paper presents such a method using predictor variables derived from the defect acceleration, namely, the defect density, defect velocity, and defect introduction time, and determines the correlation of each predictor variable with the number of defects. We report the application of an integrated machine learning approach based on regression models constructed from these predictor variables. An experiment was conducted on ten different data sets collected from the PROMISE repository, containing 22 838 instances. The regression model constructed as a function of the average defect velocity achieved an adjusted R-square of 98.6%, with a $p$ -value of < 0.001. The average defect velocity is strongly positively correlated with the number of defects, with a correlation coefficient of 0.98. Thus, it is demonstrated that this technique can provide a blueprint for program testing to enhance the effectiveness of software development activities.

VLAT: Development of a Visualization Literacy Assessment Test.

The Information Visualization community has begun to pay attention to visualization literacy; however, researchers still lack instruments for measuring the visualization literacy of users. In order to address this gap, we systematically developed a visualization literacy assessment test (VLAT), especially for non-expert users in data visualization, by following the established procedure of test development in Psychological and Educational Measurement: (1) Test Blueprint Construction, (2) Test Item Generation, (3) Content Validity Evaluation, (4) Test Tryout and Item Analysis, (5) Test Item Selection, and (6) Reliability Evaluation. The VLAT consists of 12 data visualizations and 53 multiple-choice test items that cover eight data visualization tasks. The test items in the VLAT were evaluated with respect to their essentialness by five domain experts in Information Visualization and Visual Analytics (average content validity ratio = 0.66). The VLAT was also tried out on a sample of 191 test takers and showed high reliability (reliability coefficient omega = 0.76). In addition, we demonstrated the relationship between users' visualization literacy and aptitude for learning an unfamiliar visualization and showed that they had a fairly high positive relationship (correlation coefficient = 0.64). Finally, we discuss evidence for the validity of the VLAT and potential research areas that are related to the instrument.

Development and Validation of an Achievement Test in Araling Asyano with Questions Addressing the K to 12 Araling Panlipunan Skills

The study aimed to develop and validate an achievement test in Araling Asyano for Grade 7 students that can measure the mastered and least mastered skills in the K-12 Araling Panlipunan. The researcher, together with a pool of experts, categorized the learning competencies of Araling Asyano, the result of which became the basis in preparing the test blueprint. Based on the test blueprint, a set of 150 questions with four options was formulated. The draft of the instrument was submitted to a panel of experts for content validation. Also, the test underwent two try-outs. After the final administration, a total of 65 items were included in the final form of the test. To ensure reliability, the achievement test underwent the process of test-retest method, split-half method, and KR-21 formula. Similarly, the data gathered through concurrent validity and known group technique of validation supported the validity of the instrument. The researcher concluded that the achievement test is valid and reliable for it covers the areas of Araling Asyano, and the items were responsive to the skills of K-12 Araling Panlipunan.

The Development of Quantitative Instruments to Evaluate Cognition and Competency of Forensic Nursing Science

Few instruments are available to evaluate fundamental forensic nursing science practice. A cognitive exam to assess knowledge and a practical competency rubric to assess skill proficiency were developed and examined for validity and reliability. Content validity of the cognitive exam was investigated using a test blueprint as well as a panel of experts in forensic science and test construction. The same panel inspected the competency rubric. Pilot testing of both instruments was then conducted to assess reliability. Internal consistency of the cognitive exam was low, leading to revision and retesting; interrater reliability of the competency rubric was acceptable.

Analysis of undergraduate pharmacology question papers at Goa medical college as regards to their content areas

&lt;p class="abstract"&gt;&lt;strong&gt;Background:&lt;/strong&gt; Undergraduate medical education aims at sculpting the future physicians. Knowledge gained during this period helps the physician to deal with human ailments and thus human life. It is necessary to ensure that students learn what is important and thus it is very important to ask the right questions in the examinations.&lt;/p&gt;&lt;p class="abstract"&gt;&lt;strong&gt;Methods:&lt;/strong&gt; This is a retrospective analytical study involving evaluation of university theory examination question papers of pharmacology of II&lt;sup&gt;nd&lt;/sup&gt; MBBS at our institute. Question papers from the year 2000 to 2016 were analyzed for weightage to content areas as regards to the syllabus. &lt;/p&gt;&lt;p class="abstract"&gt;&lt;strong&gt;Results:&lt;/strong&gt; Autonomic nervous system (25.28%) and cardiovascular system (22.30%) in paper I and chemotherapy (33.50%) and central nervous system (23.76%) in paper II are the maximum weightage content areas. Some aspects which are of clinical importance remained uncovered in a few papers, example: autacoids, kidney, gastrointestinal tract, non-steroidal anti-inflammatory drugs, miscellaneous.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Conclusions:&lt;/strong&gt; For consistency of validity in the assessment, we suggest that it should be based on structured guidelines. Methods like test blueprinting and table of specifications may be used.&lt;/p&gt;

Influence of Gender and Cognitive Styles on Students’ Achievement in Biology

The study investigated the influence of gender and cognitive styles on students' achievement in biology in senior secondary schools in Anambra State. One research question and one null hypothesis tested at 0.05 level of significance guided the study. A causal comparative research design and a population of 12,000 (SSII) biology students in sixty-four government - owned secondary schools in Awka Education zone. The sample of the study consisted of 265 SSII biology students (141 males and 124 girls) drawn from four government owned schools comprising of two (2) boys and two (2) girls' school in Awka education zone. The sample was drawn using disproportionate stratified random sampling technique. Two instruments were employed for data collection namely: Group Embedded Figure Test (GEFT) and Biology Achievement Test (BAT). GEFT was used to access the cognitive styles of students as either field -dependent or field independent. It is a standardized instrument with a reliability of 0.89 on a test re-test method. BA T content validity was ensured by a test blueprint while face validity was ensured by two biology educators and one expert in measurement and Evaluation from the science Education Department, University of Nigeria, Nsukka. The reliability coefficient of BAT was established using Cronbach Alpha which gave 0.86. Mean and standard deviation was used to answer the research question while one-way Analysis of variance (ANOVA) was used to test the null hypothesis. Result showed among others: that gender and cognitive styles has no significant, influence on achievement scores of students in biology. Recommendations and conclusion were highlighted.Keywords: Gender, Cognitive styles, Achievement, Biology

Curriculum assessment through a capstone course: a case study in hospitality and tourism programs

ABSTRACTThe purpose of this study was to develop a systematic process that other educational institutions and programs could follow to establish a consistent and accurate evaluation method for a capstone course. Hospitality industry professionals and hospitality management faculty were interviewed through focus-group discussions, and a post-focus-group survey was conducted to determine a weighted percentage for each of the nine determined content domains. A test blueprint for a hospitality management capstone course was developed to measure programmatic student learning outcomes based on the weighted domains determined through this study. The results provide a usable instrument for hospitality and tourism management programs to enhance their current assessment methodology.

Developing a situational judgment test blueprint for assessing the non-cognitive skills of applicants to the University of Utah School of Medicine, the United States.

Purpose:The situational judgment test (SJT) shows promise for assessing the non-cognitive skills of medical school applicants, but has only been used in Europe. Since the admissions processes and education levels of applicants to medical school are different in the United States and in Europe, it is necessary to obtain validity evidence of the SJT based on a sample of United States applicants.Methods:Ninety SJT items were developed and Kane’s validity framework was used to create a test blueprint. A total of 489 applicants selected for assessment/interview day at the University of Utah School of Medicine during the 2014-2015 admissions cycle completed one of five SJTs, which assessed professionalism, coping with pressure, communication, patient focus, and teamwork. Item difficulty, each item’s discrimination index, internal consistency, and the categorization of items by two experts were used to create the test blueprint.Results:The majority of item scores were within an acceptable range of difficulty, as measured by the difficulty index (0.50-0.85) and had fair to good discrimination. However, internal consistency was low for each domain, and 63% of items appeared to assess multiple domains. The concordance of categorization between the two educational experts ranged from 24% to 76% across the five domains.Conclusion:The results of this study will help medical school admissions departments determine how to begin constructing a SJT. Further testing with a more representative sample is needed to determine if the SJT is a useful assessment tool for measuring the non-cognitive skills of medical school applicants.

Twelve tips for assessment psychometrics

It is incumbent on medical schools to show, both to regulatory bodies and to the public at large, that their graduating students are “fit for purpose” as tomorrow’s doctors. Since students graduate by virtue of passing assessments, it is vital that schools quality assure their assessment procedures, standards, and outcomes. An important part of this quality assurance process is the appropriate use of psychometric analyses. This begins with development of an empowering, evidence-based culture in which assessment validity can be demonstrated. Preparation prior to an assessment requires the establishment of appropriate rules, test blueprinting and standard setting. When an assessment has been completed, the reporting of test results should consider reliability, assessor, demographic, and long-term analyses across multiple levels, in an integrated way to ensure the information conveyed to all stakeholders is meaningful.

Design and Outcome of a Certification Preparation Program for Outpatient Nurses.

This article presents the design, implementation, and outcomes of a certification preparation program for nurses working in an outpatient clinic. A nurse educator designed curriculum using the Certified Breast Care Nurse test blueprint. Nurse administrators provided the resources and the staff coverage to allow all staff members to attend the sessions without disruption to patient care. This outpatient center has achieved and sustained 100% certification among eligible nurses over the past 5 years.

A Review of SimuMCAT

This article reviews the software package SimuMCAT that simulates unidimensional and multidimensional computerized adaptive testing with various types of items (dichotomous/polytomous) and loading structures (simple-/complex-structured). In addition, the software allows users to choose from five different item selection procedures, two stopping rules for variable-length tests, as well as test constraints to satisfy test blueprint and limit item exposure.

Deep brain stimulation for neurodegenerative disease: a computational blueprint using dynamic causal modeling.

Advances in deep brain stimulation (DBS) therapeutics for neurological and psychiatric disorders represent a new clinical avenue that may potentially augment or adjunct traditional pharmacological approaches to disease treatment. Using modern molecular biology and genomics, pharmacological development proceeds through an albeit lengthy and expensive pipeline from candidate compound to preclinical and clinical trials. Such a pathway, however, is lacking in the field of neurostimulation, with developments arising from a selection of early sources and motivated by diverse fields including surgery and neuroscience. In this chapter, I propose that biophysical models of connected brain networks optimized using empirical neuroimaging data from patients and healthy controls can provide a principled computational pipeline for testing and developing neurostimulation interventions. Dynamic causal modeling (DCM) provides such a computational framework, serving as a method to test effective connectivity between and within regions of an active brain network. Importantly, the methodology links brain dynamics with behavior by directly assessing experimental task effects under different behavioral or cognitive sets. Therefore, healthy brain dynamics in circuits of interest can be defined mathematically with stimulation interventions in pathological counterparts simulated with the goal of restoring normal functionality. In this chapter, I outline the dynamic characterization of brain circuits using DCM and propose a blueprint for testing in silico, the effects of stimulation in neurodegenerative disorders affecting cognition. In particular, the models can be simulated to test whether neuroimaging correlates of nondiseased brain dynamics can be reinstantiated in a pathological setting using DBS. Thus, the key advantage of this framework is that distributed effects of DBS on neural circuitry and network connectivity can be predicted in silico. The chapter also includes a review of how DCM has been used to characterize the effects of DBS in Parkinson's disease.