Item Analysis Methods Research Articles

Item Analysis of Multiple Choice Questions in Medical Licensing Examination

Objectives: To evaluate multiple-choice questions of the Medical Licensing Examination. Methods: The study design was a cross sectional descriptive and item analysis method was used. This study was conducted in Health Development Center of the Ministry of Health. An item analysis was done on 1500 multiple-choice questions for the internal medicine licensing exam and of 2077 subjects from 2014 to 2016. The item analysis involved identifying reliability coefficient, difficulty index, and discrimination index for each question in relation to the doctors’ exam performance. Microsoft Excel 2010, and SPSS were used for the statistical analyses. Results: Reliability coefficient (KR20) of Mongolia’s medical licensing examination in 2014-2016 ranged from 0.79 - 0.92. The frequency of multiple-choice questions with a negative or zero discrimination index was 11.6 percent and a high difficulty index was 21.9 percent. Conclusion: Even though reliability of Mongolia’s medical licensing examinations were good, one-third of the multiple-choice questions used in the exams fail to evaluate examinee’s knowledge.

Fairness Principle in Accreditation of Health Specialists: The Differential Item Functioning Method

The main purpose of this article is to present a Differential Item Functioning method of item analysis. It is designed to minimize the discriminatory effect of individual items in the accreditation of graduates of medical universities with different training programs. The one-parameter Item Response Theory model is used to align graduates’ rights in accreditation. The measure of the difference in the location of the item characteristic curves constructed for different graduates’ samples is presented using the difficulty estimates. For the development of the methodology, a number of research questions are posed, the solution of which made it possible to analyze the items bias for the “Polyclinic” discipline and the range of item difficulties from 1.5 to 2, 5 logits. The item bank was cleaned and an interpretation of decisions on items release and correction for the cases of different arrangement of their characteristic curves was presented.

Adaptation of Cyber Bullying Scale to Turkish culture: psychometric properties

Objective: The aim of this study was to perform validity and reliability studies for adapting the Cyber Bullying Scale (SBS), which consists of 15 cyberbullying behaviors and three sub-dimensions, to Turkish culture. Methods: This research was carried out in two stages. In the first stage, the scale was translated into Turkish, exploratory factor analysis for validity, internal consistency coefficient, test-retest and item analysis methods were used for reliability. In the second stage, confirmatory factor analysis and compliance validity studies were conducted for the validity of the scale. The first and second stages, respectively, 300 and 339 participants took part in the study. Results: As a result of exploratory factor analysis conducted in the first stage, approximately 48% of the total variance of the scale was explained. Item factor loads in the scale ranged from 0.42 to 0.78. For the reliability of the scale, internal consistency coefficient was calculated as 0.94, 0.90, 0.88, and 0.93 for the total, bullying, victimization and witness-ing subscales. The test-retest reliability coefficient was 0.94, 0.90, 0.88, and 0.87 for the scale total, bullying, victimization and witness subscales. As a result of confirmatory factor analysis conducted in the second stage, it can be said that the scale has a good level of goodness of fit (χ2=2830.18, sd=807, χ2/sd=3.51, RMSEA=0.08, RMR=0.02, SRMR=0.06, GFI=0.91, AGFI=0.90, CFI=0.97, NNFI=0.97, IFI=0.97). Conclusions: As a result of all he procedures, it can be said that the form of Cyber Bullying Scale adapted to Turkish culture is valid and reliable.

Measuring Patient Reported Outcomes in Multiple Myeloma: Are Legacy Instruments Fit for Purpose

Introduction:Measuring patient-reported outcomes (PROs) in oncology clinical trials is important to capture the patient perspective on quality of survival and the outcomes of health-related quality of life (HRQoL), symptoms, and functional status. In multiple myeloma (MM) randomized clinical trials (RCTs) the common PRO instruments include the EORTC QLQ-C30 and the EQ-5D as generic measures of a patient's health status. While these legacy PRO instruments have been widely used in the MM population, it is important to continue to assess the instruments psychometric properties for the specific context of use and target population.The purpose of this research was to investigate the strengths and limitations of the PRO instruments when analyzed free from developer construct and scoring recommendations to assess fit for purpose in various MM clinical trial populations.Methods:Data for the analysis included 3 MM RCTs (n=1,773) (Dimopoulos 2016, Palumbo 2016, Mateos 2018). The PRO analysis set consisted of 33 items from the EORTC QLQ-C30 and EQ-5D-5L collected at baseline and during the treatment phase of the RCTs. Item analysis and modern psychometric methods were applied to the PRO item pool as a 4-step process: (1) descriptive analysis of the items, (2) assessment of factor structure, (3) performance of factors across RCT populations (treatment naive, relapsed/refractory MM), and (4) assessment of meaningful change.Items were characterized using polychoric correlations to examine the inter-item association and relationships before determining appropriate factor structures. An exploratory factor analysis (EFA) followed by multidimensional item response theory (MIRT) and bifactor IRT models were conducted to determine how items fit into factors and to test a total symptom and total impact scoring option. Differential item functioning (DIF) was conducted to evaluation item and domain performance across study populations (treatment naïve vs. relapsed/refractory) and longitudinally (baseline vs. week 24). Meaningful change in these new domain constructs was evaluated using anchor-based techniques based on the EORTC-QLQ-C30 and EQ-5D-5L global items. A novel provisional latent class model approach creating responder groups was also developed and applied to assess meaningful change.Results:The 33 items from the EORTC QLQ-C30 and EQ-5D-5L demonstrated strong inter-item correlations supportive of pre-specified domain structures and consistent across RCT study populations. The EFA supported 3-domain structures for both symptoms and impacts separately. These domains were broadly consistent with the PRO instrument developer specifications but unique in several regards. These domains were statistically identical across RCTs. Model fit showed that a proposed bi-factor IRT (Symptom domain and Impact domain) fit better than the MIRT suggesting that subdomains are less useful than total domain scores. An extensive DIF analysis within this IRT framework found no evidence of substantial DIF effects. All evidence indicated that the items performed similarly across sample populations and time points.Anchor-based meaningful change calculations for the newly created total symptom score and the total impact score demonstrated separation between global health anchor groups but the meaningful change estimates were small. The latent class models demonstrated that treatment arms differed significant in responder rates within 1 RCT.Conclusions:The analyses confirmed psychometrically robust EORTC QLQ-C30 and EQ-5D-5L items in the MM population under newly specified factor structures. In addition, performance of the items allows for calculation of a total symptom score and total impacts score. The newly developed total domains performed identically across MM populations (treatment naïve versus relapsed/refractory). The results only focus on the on-treatment phase of the RCT so future research is needed to determine how psychometric methods can be adjusted to better accommodate survival status and censoring for progression in oncology RCTs. The psychometric properties and consistency of the findings across the newly created symptom and impact total domain structures support the results that daratumumab maintains HRQoL in patients with MM. DisclosuresGries:Janssen Research & Development, LLC: Employment. Fastenau:Janssen Research & Development, LLC: Employment. Iaconangelo:Janssen Research &Development, LLC: Consultancy. Serrano:Janssen Research & Development, LLC: Consultancy.

Abstinence-Related Motivational Engagement Scale: Validity and Reliability in Turkish People

This research aimed to conduct a validity and reliability study of the Turkish version of the abstinence-related motivational engagement (ARME) scale. This study included 122 people and was administered in a smoking cessation clinic. The sociodemographic-smoking status characteristics questionnaire and the ARME scale were used for data collection. A psycholinguistic language adaptation was performed. In the validity, analyses, content, construct, and criterion-related validities were used. For content validity, expert evaluation was performed. For construct validity, principal component analyses (exploratory factor analyses) were performed. Orthogonal (Varimax) rotation was used to explore multiple factors. The Kaiser-Meyer-Olkin test was used to assess the adequacy of the sample size. For criterion-related validity, we compared the ARME scale points of people who were abstinent and had relapse for smoking at the end of the sixth month. In the reliability analysis, standard deviation (SD) and item analysis, internal consistency, and test-retest methods were used. The four factors explain 58% of the total variance. Items have factor loading between 0.409 and 0.805. When the factor structure of the scale was assessed, the items in each factor group have a factor load of at least "0.40." Due to one-dimensional use of the original scale, it has been decided to maintain this scale in its original form. The ARME scale points of people who quit smoking were statistically higher than the points of people who had relapse at the end of the sixth month. Cronbach's alpha coefficients were between 0.846 and 0.763. Significant and positive correlation was found between the test-retest scale scores. The Turkish adaptation of the ARME scale, which was developed for adults who quitted smoking, is an adequately valid and reliable measurement instrument. It is considered that the scale might be used reliably in different cultures as well.

Validity and reliability of functional assessment of chronic illness treatment-fatigue scale in Turkish patients with type 2 diabetes

BackgroundA long-term diabetic complication, fatigue, which is a direct result of treatment and psychological processes, is a common symptom in diabetic individuals. AimsTo determine the reliability and validity of Functional Assessment of Chronic Illness Treatment-Fatigue Scale in Turkish patients with type 2 diabetes. Study designA methodological study. MethodsThe sample consisted of 133 patients with type 2 diabetes who agreed to participate in the study and met the criteria for the study. Data were collected using the descriptive information form prepared by the researcher and the Scale obtained from FACIT System. The research data were collected by face-to-face interviews with the patients. Language and content validity of the Turkish version of the Scale was provided, Cronbach's Alpha coefficient, test–retest analysis and item analysis methods were used as reliability analysis. ResultsThe total Cronbach Alpha coefficient of the scale was found to be 0.92. According to the results of factor analysis, unlike the original scale, two sub-dimensions emerged in the Turkish version of the scale. ConclusionsAs a result of the validity and reliability analysis of the FACIT-Fatigue Scale, it can be said that it provides psychometric measures that can be used to determine the level of fatigue of type 2 diabetes patients in Turkish society.

Comprehensive Assessment of Highway Construction Based on Extension Engineering Method

The extension engineering method is introduced in detail. Comprehensive assessment indexes for highway construction are established by means of extension decision theory and main item analysis method. With a project as an example, application of extension engineering method to comprehensive evaluation of highway construction is discussed. It is showed that the present method is reasonable and has vastly applicable prospect compared with other methods.

Validity and reliability of functional assessment of chronic illness treatment-fatigue scale in Turkish patients with type 2 diabetes

BackgroundA long-term diabetic complication, fatigue, which is a direct result of treatment and psychological processes, is a common symptom in diabetic individuals. AimsTo determine the reliability and validity of Functional Assessment of Chronic Illness Treatment-Fatigue Scale in Turkish patients with type 2 diabetes. Study designA methodological study. MethodsThe sample consisted of 133 patients with type 2 diabetes who agreed to participate in the study and met the criteria for the study. Data were collected using the descriptive information form prepared by the researcher and the Scale obtained from FACIT System. The research data were collected by face-to-face interviews with the patients. Language and content validity of the Turkish version of the Scale was provided, Cronbach's Alpha coefficient, test–retest analysis and item analysis methods were used as reliability analysis. ResultsThe total Cronbach Alpha coefficient of the scale was found to be 0.92. According to the results of factor analysis, unlike the original scale, two sub-dimensions emerged in the Turkish version of the scale. ConclusionsAs a result of the validity and reliability analysis of the FACIT-Fatigue Scale, it can be said that it provides psychometric measures that can be used to determine the level of fatigue of type 2 diabetes patients in Turkish society.

Item Response Theory: An Introduction to Latent Trait Models to Test and Item Development

Testing in educational system perform a number of functions, the results from a test can be used to make a number of decisions in education. It is therefore well accepted in the education literature that, testing is an important element of education. To effectively utilize the tests in educational policies and quality assurance its validity and reliability estimates are necessary. There are two generally acceptable frameworks used in evaluating the quality of test in educational and psychological measurements, these are; Classical Test Theory (CTT) and Item Response Theory (IRT). The estimates of test items validity and reliability depend on a particular measurement model used. It is vital for a test developer to be familiar with the different test development and item analysis methods in order to facilitate the development of a new test. The CTT is a traditional approach which was widely criticise in the measurement community for its shortcomings such as sample dependency of coefficient measures and estimates of measurement error. However, the IRT is a modern approach which provides solutions to most of the CTT’s identified shortcomings. This paper therefore, provides a comprehensive overview of the IRT and its procedures as applied to test item development and analysis. The paper concludes with some suggestions for test developers and test specialists at all levels to adopt IRT for its identified crucial theoretical and empirical gains over CTT. IRT based parameter estimates should be superior and reliable than CTT based parameter estimates. With these features, IRT can help resolve the problems associated with test design based on CTT.

Development of a scale of Sense of Community in university online courses

ABSTRACTCreating a sense of community in online classes contributes to student retention and to their overall satisfaction with the course itself. This study aimed to develop a scale of sense of community of students attending online university courses. A series of ordinal exploratory factor analyses were conducted on data obtained from 839 students enrolled in Italian universities. Using an item analysis method, we were able to select the 36 most valid items from an original set of 60 items we had previously defined. These items are distributed across three related factors measuring membership, influence, and fulfillment of needs. This factorial structure replicates the McMillan and Chavis’s model of sense of community, upon the basis of which this scale was developed. The three factors presented good ordinal alpha and adequate convergent/divergent validity coefficients. The scale represents an efficient tool for the design, monitoring, and evaluation of online courses.

Validity and Reliability of the Turkish Chronic Pain Acceptance Questionnaire

Background:Pain acceptance is the process of giving up the struggle with pain and learning to live a worthwhile life despite it. In assessing patients with chronic pain in Turkey, making a diagnosis and tracking the effectiveness of treatment is done with scales that have been translated into Turkish. However, there is as yet no valid and reliable scale in Turkish to assess the acceptance of pain.Aims:To validate a Turkish version of the Chronic Pain Acceptance Questionnaire developed by McCracken and colleagues.Study Design:Methodological and cross sectional study.Methods:A simple randomized sampling method was used in selecting the study sample. The sample was composed of 201 patients, more than 10 times the number of items examined for validity and reliability in the study, which totaled 20. A patient identification form, the Chronic Pain Acceptance Questionnaire, and the Brief Pain Inventory were used to collect data. Data were collected by face-to-face interviews. In the validity testing, the content validity index was used to evaluate linguistic equivalence, content validity, construct validity, and expert views. In reliability testing of the scale, Cronbach’s α coefficient was calculated, and item analysis and split-test reliability methods were used. Principal component analysis and varimax rotation were used in factor analysis and to examine factor structure for construct concept validity.Results:The item analysis established that the scale, all items, and item-total correlations were satisfactory. The mean total score of the scale was 21.78. The internal consistency coefficient was 0.94, and the correlation between the two halves of the scale was 0.89.Conclusion:The Chronic Pain Acceptance Questionnaire, which is intended to be used in Turkey upon confirmation of its validity and reliability, is an evaluation instrument with sufficient validity and reliability, and it can be reliably used to examine patients’ acceptance of chronic pain.

Automatic Chinese Multiple Choice Question Generation Using Mixed Similarity Strategy

Automatic question generation can help teachers to save the time necessary for constructing examination papers. Several approaches were proposed to automatically generate multiple-choice questions for vocalbuary assessment or grammar exercises. However, most of these studies focused on generating questions in English with a certain similarity strategy. This paper presents a mixed similarity strategy which generates Chinese multiple choice distractors with a statistical regression model including orthographic, phonological and semantic features, i.e., features that were shown in previous psycholinguistics studies to contribute to character recognition. In a first experiment, we evaluated the predictive power of the proposed features in measuring Chinese character similarity. One of the significant experimental results showed that the combination of the four proposed categories of features (structure, semantic radical, stroke and meaning) accounts for 62.5 percent of the variance in the human judgments of character similarity. In the second experiment, a user study was conducted to evaluate the quality of system-generated questions using a test item analysis method. Two hundred ninety-six Chinese primary school students (10-11-year-old) participated in this study. We have compared the mixed strategy with another three common distractor generation strategies, orthographic strategy, semantic strategy, and phonological strategy. One of important findings suggested that the mixed strategy significantly outperformed other three strategies in terms of the distractor usefulness and has a highest discrimination power among four strategies.

Comparative study of simplified new method of item analysis with conventional method

Background: Most of the entrance examinations in the world use multiple choice questions for assessment. We tried a new method in which, the middle one third test papers were included in final item analysis while calculating difficulty and discrimination index. Methods: This cross-sectional study was conducted in the month of July 2017 among 62 third year undergraduate medical students. Total 50 MCQs were framed with a four options and only one correct answer. The assessed papers arranged in descending order and two question papers with highest and lowest marks were excluded to make equal two or three groups. The item analysis was done by two methods based on difficulty indexand discrimination index. Data was analyzed for standard error of difference between two proportions by Microsoft Excel. Results: Therefore 60 students participated in the part-completion examination. The highest score was 37 whereas 15 was lowest. Out of 50 items (MCQs), 7 items were removed before final item analysis because of negative value of proportions. Out of 43 items, 25 showed that there was statistically significant difference between the discrimination index (P<0.05) with new method and conventional method. In rest of the 18 items, there was no significant difference (P>0.05) in discrimination index. Z test for two proportions showed no significant difference in the difficulty index with new and conventional method. Conclusions: To conclude, there is no difference with respect to difficulty index by both methods but regarding discrimination index, the study failed to show conclusive results.

Developing and validating a perinatal depression screening tool in Kenya blending Western criteria with local idioms: A mixed methods study

BackgroundRoutine screening for perinatal depression is not common in most primary health care settings. The U.S. Preventive Services Task Force only recently updated their recommendation on depression screening to specifically recommend screening during the pre- and postpartum periods. While practitioners in high-income countries can respond to this new recommendation by implementing one of several existing depression screening tools developed in Western contexts, such as the Edinburgh Postnatal Depression Scale (EPDS) or the Patient Health Questionnaire-9 (PHQ-9), these tools lack strong evidence of cross-cultural equivalence, validity for case finding, and precision in measuring response to treatment in developing countries. Thus, there is a critical need to develop and validate new screening tools for perinatal depression that can be used by lay health workers, primary health care personnel, and patients. MethodsWorking in rural Kenya, we used free listing, card sorting, and item analysis methods to develop a locally-relevant screening tool that blended Western psychiatric concepts with local idioms of distress. We conducted a validation study with a random sample of 193 pregnant women and new mothers to test the diagnostic accuracy of this scale along with the EPDS and PHQ-9. ResultsThe sensitivity/specificity of the EPDS and PHQ-9 was estimated to be 0.70/0.72 and 0.70/0.73, respectively. This compared to sensitivity/specificity of 0.90/0.90 for a new 9-item locally-developed tool called the Perinatal Depression Screening (PDEPS). Across these three tools, internal consistency reliability ranged from 0.77 to 0.81 and test-retest reliability ranged from 0.57 to 0.67. The prevalence of depression ranges from 5.2% to 6.2% depending on the clinical reference standard. ConclusionThe EPDS and PHQ-9 are valid and reliable screening tools for perinatal depression in rural Western Kenya, the PDEPS may be a more useful alternative. At less than 10%, the prevalence of depression in this region appears to be lower than other published estimates for African and other low-income countries.

Turkish validity and reliability of the Diabetes Self-Efficacy Scale

PurposeThe aim of this methodological study was to determine the validity and reliability of the Diabetes Self-Efficacy Scale adapted to the Turkish community. MethodsThe study sample was completed with 319 patients who had been diagnosed at least 1 year before and hospitalized in the Malatya Turgut Ozal Health Center and Malatya State Hospital in Turkey. A questionnaire that consists of items on sociodemographic characteristics, drug use and information about the disease of patients and the Diabetes Self-Efficacy Scale were used for data collection in the study. In reliability analysis of the scale, the Cronbach's α coefficient was calculated and item analysis method was utilized. Factor analysis was used for the construct validity, and Principal Component Analysis and Varimax Rotation method were used for analyzing the factor structures. ResultsAccording to data obtained in the study, item-total correlation of the items of the scale was found to be at an adequate level (0.297–0.803). The scale's Cronbach's α reliability coefficient was found to be 0.86, and there was one factor that explains 52.38% of the total variance with an eigenvalue was greater than 1.0. As a result of the analysis, the factor loadings of the items of the scale were found to be between 0.59 and 0.81. ConclusionDiabetes Self-Efficacy Scale is a valid and reliable instrument for determining the self-efficacy of patients and providing a proper care. It can be suggested to investigate and evaluate the consistency of the scale by applying it to broader sample groups representing different socioeconomic levels.

Simplified method of item analysis as compared with traditional method

Simplified method of item analysis as compared with traditional method

Development of an itemwise efficiency scoring method: Concurrent, convergent, discriminant, and neuroimaging-based predictive validity assessed in a large community sample.

Traditional "paper-and-pencil" testing is imprecise in measuring speed and hence limited in assessing performance efficiency, but computerized testing permits precision in measuring itemwise response time. We present a method of scoring performance efficiency (combining information from accuracy and speed) at the item level. Using a community sample of 9,498 youths age 8-21, we calculated item-level efficiency scores on 4 neurocognitive tests, and compared the concurrent, convergent, discriminant, and predictive validity of these scores with simple averaging of standardized speed and accuracy-summed scores. Concurrent validity was measured by the scores' abilities to distinguish men from women and their correlations with age; convergent and discriminant validity were measured by correlations with other scores inside and outside of their neurocognitive domains; predictive validity was measured by correlations with brain volume in regions associated with the specific neurocognitive abilities. Results provide support for the ability of itemwise efficiency scoring to detect signals as strong as those detected by standard efficiency scoring methods. We find no evidence of superior validity of the itemwise scores over traditional scores, but point out several advantages of the former. The itemwise efficiency scoring method shows promise as an alternative to standard efficiency scoring methods, with overall moderate support from tests of 4 different types of validity. This method allows the use of existing item analysis methods and provides the convenient ability to adjust the overall emphasis of accuracy versus speed in the efficiency score, thus adjusting the scoring to the real-world demands the test is aiming to fulfill. (PsycINFO Database Record

Development of a Problematic Mobile Phone Use Scale for university students: validity and reliability study

Objective: This study aims to develop a Problematic Mobile Phone Use Scale (PMPUS) to determine the problematic mobile phone use by university students. Method: Study participants for exploratory factor analysis (EFA) and confirmatory factor analysis (CFA) were 725 university students, for test-retest 126 university students. The psychometric characteristics of the scale were investigated using test-retest, internal consistency (Cronbach’s alpha), item analysis, EFA, CFA, and criterion-related validity methods. Results: According to results obtained from exploratory factor analysis, it was determined that PMPUS had four subdimensions. Confirmatory factor analysis confirmed this four-subdimension structure. Reliability data for the PMPUS found Cronbach’s Alpha coefficients of 0.92 for EFA and 0.93 for CFA, respectively. Test-retest coefficient for PMPUS was 0.85. Furthermore, a positive, high correlation (r=0.75) was found between PMPUS and Mobile Phone Problem Use Scale (MPPUS). Subscales of PMPUS also showed the required values for validity and reliability. Conclusion: According to the results, PMPUS is a reliable and valid scale with 26 items in four subdimensions.

Construction and Standardization of Scientific Aptitude Test (SAT) for Secondary School Students-A Pilot Study

The main objective of the present pilot study was to construct and standardize a Scientific Aptitude Test (SAT) for the secondary school science students (Grade 9 and 10). The test items (30 questions) were prepared by the researchers based on the five important aspects in scientific thinking such as Analogy, Scientific reasoning, Numerical ability, Logical reasoning and Comprehension based questions. A test of multiple choice items were used as a data collection instrument in the six different schools in the selected districts in Tamilnadu, India. The computer based Microsoft excel spreadsheet (2007) was prepared to document the obtained score and the same used for statistical analysis. Through recent method of item analysis, Cronbach’s Alpha values were computed using IBM SPSS software. The reliability coefficients were found 0.874(Spearman-Brown) and 0.672(Guttmann Split-Half). Hence, the present pilot study revealed that the scientific aptitude test for secondary students to ignite their scientific temper and also useful in preparing future competitive examinations.Int. J. Soc. Sci. Manage. Vol-3, issue-1: 17-21

50년간 영어교육에 게재된 영어 평가 관련 논문분석

This paper overviews English language assessment-related articles published in English Teaching from 1965 to 2015. One hundred thirty-eight articles are chosen for analysis, and the discernible trends epitomizing a specific era are traced. The results show articles concerning item analysis and improvement oftesting methods constituted a major portion of the periodical in the early days. The number of the assessmentrelated articles began to soar from the late 1980s, and recently a major focus seems to have shifted to test validity. In terms of research topics, a significant body of research has been conducted on item analysis, improvement of testing practices, performance assessment, evaluation methods, and test validity. Even though the overall level of the articles has advanced dramatically over the past 50 years, little research has been intended in the areas of classroom assessment, English language testing policies, washback effects, new assessment tools, and the validity of national tests. This paper concludes with a suggestion that more efforts should be made to fill the gap between research and testing practices in the field.