Average Difficulty Index Research Articles

Development of Qualified Items for Nursing Education Assessment: The Progress Testing Experience

Aimto analyze the psychometric characteristics of items in the nursing inter-institutional progress testing for the years 2019, 2021, 2022 and 2023. BackgroundProgress testing is a validated method for evaluating professional undergraduate education, aimed at identifying knowledge gain in a continuous and progressive manner, with potential benefits for nursing education. However, for its results to be useful, the evaluation items used in the test must have good psychometric performance. Designa cross-sectional study. Methodsa sample of 377 items (multiple-choice questions) was applied to 4,678 students in four years of progress testing. The difficulty and discrimination indexes were analyzed using descriptive statistics, ANOVA and simple linear regression. ResultsThe average difficulty index of the test items ranged between 0.39 and 0.46. The areas of child and adolescent health, women's health and adult health had the most difficult items, while the areas of management, mental health and public health had the least difficult items. Discrimination index ranged from 0.35 to 0.43. There was a difference between discrimination index between the years of application (p<0.001), with a significant increase in the discrimination index (p<0.001) in the trend analysis. Students in the final years showed lower levels of difficulty and discrimination when compared with students in the initial years, demonstrating that the test is easier and there is less dispersion of performance among students in the final years. ConclusionsThe items are not difficult and have good discrimination. A gradual annual increase in the discrimination index of the items was observed. This study provides useful information for the psychometric analysis and quality assurance of knowledge assessment items, both for the implementation of similar PT experiences and in the use of multiple-choice questions for other knowledge assessment purposes.

Item difficulty index, discrimination index, and reliability of the 26 health professions licensing examinations in 2022, Korea: a psychometric study.

This study presents item analysis results of the 26 health personnel licensing examinations managed by the Korea Health Personnel Licensing Examination Institute (KHPLEI) in 2022. The item difficulty index, item discrimination index, and reliability were calculated. The item discrimination index was calculated using a discrimination index based on the upper and lower 27% rule and the item-total correlation. Out of 468,352 total examinees, 418,887 (89.4%) passed. The pass rates ranged from 27.3% for health educators level 1 to 97.1% for oriental medical doctors. Most examinations had a high average difficulty index, albeit to varying degrees, ranging from 61.3% for prosthetists and orthotists to 83.9% for care workers. The average discrimination index based on the upper and lower 27% rule ranged from 0.17 for oriental medical doctors to 0.38 for radiological technologists. The average item-total correlation ranged from 0.20 for oriental medical doctors to 0.38 for radiological technologists. The Cronbach α, as a measure of reliability, ranged from 0.872 for health educators-level 3 to 0.978 for medical technologists. The correlation coefficient between the average difficulty index and average discrimination index was -0.2452 (P=0.1557), that between the average difficulty index and the average item-total correlation was 0.3502 (P=0.0392), and that between the average discrimination index and the average item-total correlation was 0.7944 (P<0.0001). This technical report presents the item analysis results and reliability of the recent examinations by the KHPLEI, demonstrating an acceptable range of difficulty index and discrimination index values, as well as good reliability.

A Test Development Study on Spatial Visualization for Second-Grade Primary School Students

This study aims to develop a new spatial visualization test (SVT) for second grade primary school students. The study employed the survey design, and the test was developed in accordance with the test development steps. According to the findings obtained as a result of the pilot study, the items were generally high difficulty levels, and they were very good items. Exploratory and confirmatory factor analysis findings confirmed the three sub-dimensions (mental integration, mental rotation, and paper folding) measuring spatial visualization skills. SVT, which was reduced to 10 items, was administered to 396 students; the KR-20 internal consistency coefficient was 0.63, the average difficulty index was 0.35 and the average discrimination index was 0.48. As a result of the study, a valid and reliable test on spatial visualization for second grade primary school students was obtained and it was concluded that students' spatial visualization was at a low level.

Assessment of medical students' Surgery knowledge based on Progress Test.

Progress Testing (PT) is an assessment tool whose use has grown throughout Brazil in the last decade. PT makes it possible to assess the students' knowledge gain throughout the undergraduate course and, for their interpretations to be valid, their items (questions) must have adequate quality from the point of view of content validity and reliability of results. In this study, we analyzed the psychometric characteristics of the items and the performance of students in the content area of surgery from 2017 to 2023. For the analyses, we used the assumptions of Classical Test Theory, Bloom's taxonomy and Cronbach's alpha reliability coefficient. The items were easy (average difficulty index between 0.3-0.4), with fair to good discrimination (discrimination index between 0.3-0.4) and with a predominance of medium to high taxonomy. Reliability remained substantial over the years (>0.6). Students' knowledge gain in surgery was found to be progressive and more important from the 3rd year of the undergraduate course, reaching approximately 70-75% in the 6th year. This measurements framework can be replicated in other contexts for a better understanding of student learning and for qualification of evaluation processes.

Development of Higher Order Thinking Skills (HOTS) Assessment Instruments to Improve Students' Mathematical Creative Thinking Skills

The purpose of learning in the millennial era is the development and improvement of students' high-level thinking skills. High-level abilities or often referred to as higher order thingking skills (HOTS) are the level of thinking of students with stages of analysis, evaluation and creation. The indicators of HOTS can be derived from the ability to think critically and think creatively. In order for students' mathematical HOTS abilities to be known, a HOTS assessment instrument is needed. The purpose of this study is the development of a mathematical HOTS assessment instrument for MTs class VII students. The type of research is a development adapted from the development of Borg and Gall. Analysis of research data using quantitative and quality analysis to determine the quality of the mathematical HOTS assessment instrument conducted a review based on the criteria of validity, reliability, difficulty index and differentiating power of the problem. The result of this development is that the HOTS assessment instrument with the form of a description of 15 questions and is suitable for use. The HOTS assessment instrument is included in a valid category that is based on expert assessment with an average score of 36.5 and an excellent category. The difficulty level of the question is moderate with an average difficulty index of 0.5 in the medium category and the differentiating power of the question is good with an average differentiating power index of 0.33 in the good category. The developed mathematical HOTS instrument also meets the reliable criteria with a reliability coefficient of 0.733.

Characteristics of National Standardized School Examination Test Items on Biology Subject in High School

A good test instrument is required to measure students’ academic abilities. For final-year students, this ability is measured by USBN tests. This study aimed to analyses the characteristics of biology test items in the National Standardized School Examination (USBN) and analyses item tests categorized as difficult and poor. Type of study was descriptive exploratory with quantitative and qualitative approaches. Quantitative data analysis was performed using the classical test and item response theories with 1 logistic parameter model. Qualitative data analysis described why such question items are categorized as difficult and poor according to the classical test theory. The data were collected by using documentation of the answers of 193 students. The study results show that, according the classical test theory, Biology USBN item tests had the average difficulty index of 0.525 (moderate category) and the average of discrimination index obtained is equal to 0.124 (good category). Based on the item response theory, the mean difficulty index was -0.00029 (moderate category). The reliability test estimation result was 0.51, categorized as moderate. The qualitative analysis result showed that items categorized as difficult are the food web in the ecosystem, nervous system, and human excretion system.

Investigating the Applicability of the Newtonian Gravity Concept Inventory to First Year College Students

We report on a study of our students’ understanding of gravity using the Newtonian Gravity Concept Inventory (NGCI). This article is supposed to serve two purposes: as a proof of applicability of the test to first year medical students outside the United States and as a general survey of our students’ understanding of gravity. The motivation for this work came initially from students’ misconceptions of gravity and related topics noticed in classrooms. NGCI has 26 multiple-choice questions probing students’ knowledge of gravity in four different domains: Directionality, Gravity as a force, Independence from other forces and threshold. Results confirmed weak overall performance of the 684 students participated in this study (average score of only 38.48%) with misconceptions related to gravity in all four domains. We were able to prove the applicability of the NGCI to our students through calculations of Classical Test Theory statistics and Cronbach’s alpha. We got a Cronbach’s alpha of 0.68, average difficulty index of 0.62 and average discriminatory index of 0.31 with three questions reported as too difficult based on the acceptable range. The test was given to students twice; before and after any gravity-related instructions in class (pre- and post-test). Post-test results will not be discussed here and left for future articles. In this article, we present the pre-test results and compare it to the original work by the authors of the test on astronomy and physics students.

Test Item Analysis of MCQS of Medical Physiology: Summative Assessment

<i>Introduction</i>: MCQs in one of popular test item of educational institution in a short time, large part of curriculum. It is an effective tool for student assessment as well as providing guidelines to teachers. <i>Objectives</i>: Our study objectives were to analyze the quality of MCQs and identify the low scorers and student learning difficulties. Standardization of MCQs is required only after test item analysis. <i>Material and Methods</i>: The study design was cross sectional analytical study. Simple random sampling technique was applied. The total participants were 100 students of 1^st year MBBS of Ghazi Khan Medical College, Dera Ghazi Khan. Fifty best MCQs were given in send up of Medical Physiology 2021. On basis of students total score in test, they were divides into two groups, high scorers (n=25) and low scorers (n=25). The difficulty index (P) and discrimination index, distracter efficiency (DE%) were calculated. <i>Results</i>: Our study 80% of test items have acceptable range of difficulty index and very high distracter efficiency. Only 20% of test item showed very poor difficulty index, which needs to be rephrased or deleted. The 20% test item had acceptable difficulty index (P=38), high D. I (0.36) and highest distracter efficiency (DE=100%). <i>Conclusion</i>: Test items with average difficulty index (P), excellent DI and all functional distracters should be given in subsequent tests.

Validation of an instrument to determine oral health knowledge, attitudes, and practices during pregnancy

BackgroundOral health of the mother-infant dyad is important to preserve general health. However, there are few instruments in Spanish for the evaluation of knowledge, attitudes and practices that determine this construct. Therefore, this research aimed to develop and evaluate the psychometric properties of the Maternal Oral Health Knowledge, Attitudes and Practices Questionnaire (CAPSOM in Spanish).MethodsIn this instrument development study that carried out in 2018–2019, involving pregnant women between the ages of 18 and 45 in the city of Leon, Guanajuato, Mexico. The sample size was calculated based on 10 women per questionnaire item (n = 10 k). The study used Cronbach’s alpha, the modified Lawshe test of validity criteria, factor analysis, and the level of difficulty and discrimination of the items.Results207 women took part with their signed, informed consent (25 ± 6 years). The internal consistency of the instrument, both total and by dimension was α = 0.70, α = 0.66 knowledge, α = 0.74 attitudes, and α = 0.66 practices. Values of Content Validity Ratio’ ≥ 0.60 were obtained for the final 10 items and Content Validity Index’ = 0.90. The average difficulty index of items was 0.40, and there were significant differences (Kruskall–Wallis, p < 0.001) in the discrimination test. Factor analysis demonstrated three main components.ConclusionsA valid and reliable 10-item Spanish questionnaire was designed to measure pregnant women’s oral health knowledge, attitudes, and practices.

Experiences from Two Ways of Integrating Pre- and Post-course Multiple-choice Assessment Questions in Educational Events for Surgeons

ABSTRACT To examine how to optimise the integration of multiple-choice questions (MCQs) for learning in continuing professional development (CPD) events in surgery, we implemented and evaluated two methods in two subspecialities over multiple years. The same 12 MCQs were administered pre- and post-event in 66 facial trauma courses. Two different sets of 10 MCQs were administered pre- and post-event in 21 small animal fracture courses. We performed standard psychometric tests on responses from participants who completed both the pre- and post-event assessment. The average difficulty index pre-course was 57% with a discrimination index of 0.20 for small animal fractures and 53% with a discrimination index of 0.15 for facial trauma. For the majority of the individual MCQs, the scores were between 30%-70% and the discrimination index was >0.10. The difficulty index post-course increased in both groups (to 75% and 62%). The pre-course MCQs resulted in an average score in the expected range for both formats suggesting they were appropriate for the intended level of difficulty and an appropriate pre-course learning activity. Post-course completion resulted in increased scores with both formats. Both delivery methods worked well in all regions and overall quality depends on applying a solid item development and validation process.

Determining the Qualifications of the Secondary School Students in the Field of Quadrilaterals: A Scale Development Study

Testing-assessment is needed to determine whether the education process is successful or at what level. For this reason, it is of great importance to determine the achievement level of the students at the secondary level. The purpose of this study is to develop an achievement test consisting of multiple choice questions about ‘quadrilateral’ subject of secondary school students by using alternative assessment techniques. The study group of this research, which is a descriptive survey type, consists of 372 secondary school students who are in the seventh grade in Aydın and Muğla during the 2018-2019 academic year. In the study, ITEMAN, SPSS and JMETRIK programs were used based on Classical Test Theory and Material Testing Theory. The formed item pool was subjected to the appearance validity by five trainers, who are experts in that field. According to the experts’ opinions, it was decided that for the achievement test, 28 items were to be included in the test primarily. It was decided as a result of analysis of the data obtained from the students that an item with very low item-specificity index, four items that do not provide model data compliance in Rasch analysis and two items identified to have item bias may be excluded from the test were excluded from the test. After these processes, the final testing tool consisted of 21 items and the average difficulty index of the test was .64; the discrimination index was calculated as .40. The Cronbach Alpha internal consistency coefficient of the final state was determined as .81. According to these results, it can be said that the achievement test has sufficient validity and reliability.

Comparison between students’ perception and examination validity, reliability and items difficulty: a cross-sectional study

Introduction: Students' perception of an examination reflects their feelings, while its item analysis refers to a statistical analysis of students’ responses to examination items. The study was conducted to compare the student’s perception towards examination to its item analysis. Material and methods: This is a cross-sectional study conducted at the college of medicine, from January to April 2019. The study used a structured questionnaire and standardized item analysis of students’ examinations. All students who had registered for semester two (2018-2019) were included in the study. Exclusion criteria included students who refused to participate in the study or those who did not fill the questionnaire.&#x0D; Result: KR-20 of the examination was 0.906. The average difficulty index of the examination was 69.4. The response rate of the questionnaire was 88.9% (40/45). Students considered the examination to be easy (70.4%). A significant correlation was reported between student perceptions towards examination difficulty and standard examination difficulty.&#x0D; Discussion: Student perceptions support the evidence of examination validity. Students were found able to estimate examination difficulty.&#x0D; Keywords: Student perception, Item analysis, Assessment, validity, reliability.

The MIR 2018 Exam: Psychometric Study and Comparison with the Previous Nine Years.

Background and Objectives: The aim of the present research is to study the questions used in the 2018 MIR exam (a test that allows access to specialized medical training in Spain), describe their psychometric properties, and evaluate their quality. Materials and Methods: This analysis is performed with the help of classical test theory (CTT) and item response theory (IRT). The answers given to the test questions by a total of 3868 physicians are analyzed. Results: According to CTT, the average difficulty index for all of the test questions was 0.629, which falls into the acceptable category. The average difficulty index with correction for random effects was 0.515, which corresponds to a value within the optimal range. The mean discrimination index was 0.277, which is in the good category, while the mean point biserial correlation coefficient, with a value of 0.275 fits in the regular category. The values of difficulty and discrimination calculated according to the model of two parameters of the IRT seem adequate with average values of −0.389 and 0.677. The Cronbach alpha score obtained for the overall test was 0.944. This value is considered as very good. Conclusions: A decrease was observed in the average values of discrimination in the last three calls, which may be related to the greater proportion of Spanish graduates that take the exam in the same year of finalization of their studies in Medicine.

Energy Success Test For Classroom Teacher Candidates: Validity And Reliability Study

Bu araştırmada, çevre-enerji ilişkisi bağlamında sınıf öğretmeni adaylarına yönelik geçerlik ve güvenirlik çalışması yapılmış bir enerji başarı testi geliştirilmiştir. Çalışmada nicel araştırma yaklaşımlarından biri olan tarama modeli kullanılmıştır. Araştırmanın çalışma grubu kolay ulaşılabilir örnekleme yöntemiyle seçilmiştir. Cumhuriyet Üniversitesi Eğitim Fakültesi Sınıf Eğitimi Ana Bilim Dalı’nda 2017-2018 güz döneminde öğrenim gören 348 sınıf öğretmeni adayı pilot uygulamada, 88 sınıf öğretmeni adayı da asıl uygulama çalışmada grubunu oluşmaktadır. Başarı testinin geliştirilmesi sırasında öncelikle 79 maddelik bir madde havuzu oluşturulmuştur. Daha sonra bu maddeler soru köküne dönüştürülerek 50 soruluk başarı testi hazırlanmıştır. Uzman görüşü sonrasında bir madde taslak formdan çıkarılmış ve 49 madde soruların anlaşılabilir olduğunu test etmek için 10 sınıf öğretmeni adayına uygulanmıştır. Verilerin analizinde SPSS ve Excel programlarından yararlanılmıştır. Madde analizi sonrasında bazı maddelerin çıkarılmasıyla Enerji Başarı Testi 36 maddelik bir hal almış ve testin ortalama güçlük indeksi 0,54 (orta güçlükte); ortalama ayırt edicilik indeksi 0,33 (oldukça ayırt edici) bulunmuştur. Daha sonra alt-üst grup ortalama farkına dayalı madde analizi, korelasyona dayalı madde analizi yapılmış ve her bir maddenin p&amp;lt;0,05 olduğu tespit edilerek herhangi bir madde çıkarımına gerek kalmadan test son halini almıştır. Madde analizlerinden sonra ölçeğin iç tutarlığını belirlemek için Kuder-Richardson-20 (KR-20) güvenirlik katsayısı hesaplanmış ve pilot uygulama için 0,79 (oldukça güvenilir) bulunmuştur. Asıl uygulamada Enerji Başarı testi pilot uygulamaya dâhil edilmeyen 88 sınıf öğretmeni adayına yeniden uygulanmış ve testin Kuder-Richardson-20 (KR-20) güvenirlik katsayısı 0,71 (oldukça güvenilir) bulunmuştur. Enerji Başarı Testinde yer alan soruların çevre eğitimi dersi kapsamındaki konuları yansıttığı ve çevre-enerji ilişkisi bağlamında sınıf öğretmeni adaylarına enerji konusunda kapsam geçerliği sağladığı ortaya konulmuştur.

Comparative Study of The Difficulty Index of Items Displayed with The Paper-Based Test and The Computer-Based Testing Paper

The focus of this study is to analyze whether there are differences in the difficulty index of items if the same items are displayed with the Paper Based Test (PBT) model and the Computer-Based Testing (CBT) model. The PBT and CBT models have the same paradigm of measuring estimated abilities but have differences in the context and feel aspects. These differences include the number of items in the range of eyesight, use of tools, how to work on items, basic knowledge needs about computer operations, and habit factors. These differences can affect the results of estimating the ability of test participants. This study uses development methods and quantitative methods. Development methods are used to develop package items and CBT software. Two groups of respondents were used with equal ability to work on the same question package. One group uses the PBT model, and the other group uses CBT so that the response choices are obtained in each test model. The results of the respondent’s answer choices were then analyzed by the ITEMAN software to get the item difficulty index. The item difficulty index of each model was tested statistically using SPSS software, whether there was a significant average difference between the two test models. From the results of the study and analysis in classical theory, it can be concluded that statistically there are differences in the average difficulty index of the items if the same item is displayed with the PBT and CBT models. It was found that the items displayed with the PBT model had a more difficult tendency than when displayed with the CBT model.

VALIDITY AND RELIABILITY OF THE 10/66 DEMENTIA RESEARCH GROUP COGNITIVE TEST BATTERY FOR EARLY IDENTIFICATION OF COGNITIVE DECLINE IN MEXICO

Several neuropsychological tests have been used to assess cognitive decline across countries, nevertheless, few countries have had mind about external validity and reliability of those assessment outcomes. It is well known that there are cross-cultural differences that might be affect those results, regarding on measuring irrelevant variance. In that sense, we proposed to identify psychometrical and statistical properties of 10/66 Dementia Research Group cognitive test battery, in order to empirically confirm the utility of the instrument for the early identification of cognitive decline in the elderly population in Mexico. Secondary analysis of the basal evaluation of the Dementia Research Study 10/66 in Mexico (n=2003). The 10/66 Dementia Research Group cognitive test battery included the administration of the Community Screening Instrument for Dementia (CSI ‘D’), the Consortium to Establish a Registry for Alzheimer's Disease (CERAD) verbal fluency (VF), word list memory (WLM, immediate recall) and recall (WLR, delayed recall) tests. The psychometric analysis was implemented to estimate the 2-parameter model for the dichotomous and polytomous items response sets (Baker, 2001) difficulty and discrimination parameters. In addition we performed a Confirmatory Factor Analysis (Abell, Springer y Kamata, 2009) to confirm factor estability and validity. It was observed that the average value of discrimination of the instrument was 1.0057, which is considered as a high power of discrimination. On the other hand, the estimated average difficulty index was -2.1288, which suggests that the neuropsychological battery can effectively differentiate between subjects with very low and average levels. The instrument had a Cronbach alpha of 0.708 which suggests the consistency between the measurements obtained. When the construct validity was performed, it was observed that the items converge positively on the factor with values greater than 0.30 factorial load. What indicates that the instrument has validity. The results showed that the items are able to discriminate effectively between subjects with low neuropsychological levels and in average values. Likewise, the factorial structure was validated, favoring the internal and external validity of the construct. These evidences could help to identify subjects with early cognitive decline.

Incorporating age-related special populations into a pharmacy skills laboratory course sequence

Background and purposeThe purpose was this study was to evaluate the incorporation of geriatric and pediatric therapeutic concepts into a pharmacy skills laboratory (PSL) course sequence. Secondary outcomes were to describe the incorporation and evaluate the assessment effectiveness of these topics. Educational activity and settingTeaching materials and assessment data from the PSL course were evaluated for inclusion of content regarding age-related special populations. FindingsMaterials reviewed from 205 PSL sessions held over four years demonstrated 89 (41.5%) sessions incorporated age-related topics using a variety of methods. Quiz and exam item performance was acceptable with an average difficulty index of 77.2% ± 21.3 and 85.8% ± 12.8, respectively. The average objective structured clinical exam (OSCE) score was also satisfactory at 83.2% ± 3. Discussion and conclusionsAge-related topics were included in several PSL sessions. These incorporation strategies were deemed to be effective as students performed well on written assessments and OSCEs. Additional PSL sessions were identified as potential targets for incorporating age-related special populations into future PSL offerings.

Systemic factors of errors in the case identification process of the national routine health information system: A case study of Modified Field Health Services Information System in the Philippines

BackgroundThe quality of data in national health information systems has been questionable in most developing countries. However, the mechanisms of errors in the case identification process are not fully understood. This study aimed to investigate the mechanisms of errors in the case identification process in the existing routine health information system (RHIS) in the Philippines by measuring the risk of committing errors for health program indicators used in the Field Health Services Information System (FHSIS 1996), and characterizing those indicators accordingly.MethodsA structured questionnaire on the definitions of 12 selected indicators in the FHSIS was administered to 132 health workers in 14 selected municipalities in the province of Palawan. A proportion of correct answers (difficulty index) and a disparity of two proportions of correct answers between higher and lower scored groups (discrimination index) were calculated, and the patterns of wrong answers for each of the 12 items were abstracted from 113 valid responses.ResultsNone of 12 items reached a difficulty index of 1.00. The average difficulty index of 12 items was 0.266 and the discrimination index that showed a significant difference was 0.216 and above. Compared with these two cut-offs, six items showed non-discrimination against lower difficulty indices of 0.035 (4/113) to 0.195 (22/113), two items showed a positive discrimination against lower difficulty indices of 0.142 (16/113) and 0.248 (28/113), and four items showed a positive discrimination against higher difficulty indices of 0.469 (53/113) to 0.673 (76/113).ConclusionsThe results suggest three characteristics of definitions of indicators such as those that are (1) unsupported by the current conditions in the health system, i.e., (a) data are required from a facility that cannot directly generate the data and, (b) definitions of indicators are not consistent with its corresponding program; (2) incomplete or ambiguous, which allow several interpretations; and (3) complete yet easily misunderstood by health workers.Taking systemic factors into account, the case identification step needs to be reviewed and designed to generate intended data in health information systems.