Item Difficulty Level Research Articles

Development of Essential Thinking Test Instrument using the Presseisen Taxonomy on Ecosystem Material for Grade X High School Students

The test instrument is a measuring tool in the assessment process used to obtain information in the form of data on student learning outcomes, both cognitive aspects and thinking skills. The use of test instrument to measure students' thinking skills is still rarely used. This study aims to develop an essential thinking test instrument using the Presseisen taxonomy on valid and reliable ecosystem material. This research uses the Research Development (RD) method with the 3D modified development model combined with the test development stages consisting of 6 stages. Data collection in this research was carried with interviews, tests, and test instrument validation sheets. The analysis included content validity analysis, interrater reliability, and Rasch model analysis. The results showed that all items were valid based on content validity. Interrater reliability has a value of 0.783 with a good category. All items are declared fit, but item number 7 contains bias so it needs to be discarded. Cronbach's alpha has a value of 0.68 in the enough category, person reliability has a value of 0.50 in the weak category, and item reliability has a value of 0.99 in the excellent category. The item difficulty level group consists of 2 very difficult questions, 4 difficult questions, 2 easy questions, and 2 very easy questions. Based on the result of the research, it can be concluded that there are 9 items that are declared valid and reliable for use.

Assessing Undergraduate Cognitive System Thinking Instruments in Genetics Lectures: A Rasch Model Analysis

This study aims to test the reliability, validity and wright person-items of the test instrument. It used to measure students' cognitive system thinking in genetics lectures. This cognitive system dimension (New Marzano Taxonomy) consists of 4 levels: retrieval, comprehension, analysis, and knowledge utilization. The research method used was descriptive quantitative, with respondents as many as 116 undergraduate students who had attended genetics lectures. The instrument used consisted of 20 multiple-choice questions developed from 4 levels of the cognitive dimension of the system. Data analysis using the Rasch Model-Dichotom to generate more accurate estimates based on individual abilities and difficulty levels of question items. The results showed that the Alpha Cronbach and Item Reliability value on the instrument was 0.81 (Very Good) to 0.92 (Best). Furthermore, person reliability is 0.63 (Moderate). It can be concluded the interaction between the question items and the quality of the question items on the test instrument indicates that the instrument effectively assesses constructs and is relevant for measuring the level of cognitive system thinking. Revisions are made to items that are too easy or too difficult, as well as a review of items with similar difficulties. So it can be assessed that this test has the potential to be widely used in measuring cognitive ability, but some improvements are needed to further improve its reliability for other population.

Development of Algebra Test Using the Item Response Theory Approach for Junior High School Students

This research aims to develop valid and reliable measuring tool for students' algebraic abilities that can be used in schools and the general public. The research follows a structured test development design, including stages such as preparing test specifications and items, field testing, revising items, and test development. The questions are aligned with the 2013 curriculum syllabus, ensuring relevance to educational standards. The test was given to 662 junior high school students in Kendari City, Indonesia, and their responses were analyzed using the item response theory (IRT) model with two logistic parameters: item difficulty level and item discriminatory power. The BILOG MG program was employed to estimate item and ability parameters. Before conducting item analysis with IRT, essential assumption tests were conducted, including unidimensional and model fit tests. The results of the development process, based on item analysis using BILOG MG, yielded 15 items covering various aspects of algebraic abilities. These items were derived from indicators such as recognizing algebraic forms, identifying elements within these forms, performing addition, subtraction, multiplication, and division operations on algebraic forms, presenting and solving real-world problems in algebraic contexts, and addressing contextual problems involving algebraic operations. The items demonstrated good fit with the model and exhibited an appropriate level of item difficulty and discriminatory power, making them suitable for use as a reliable assessment tool. Consequently, these developed tests are deemed effective for measuring students' foundational algebraic abilities.

Development and Initial Evaluation of a Fine Motor Subscale of a Comprehensive Developmental Scale for Children Aged 18–41 Months

ABSTRACT Identifying developmental delays in children is vital for timely intervention, yet there are limited local developmental tests in Hong Kong that include a fine motor subscale and are specifically tailored for children aged 18–41 months. To address this need, we developed a 50-item fine motor subscale, which underwent two rounds of pilot testing and expert reviews. This study investigated its internal-scale validity and reliability, comprising 340 young children in Hong Kong, including 82 children with fine motor delays and 258 typical developing children. Unidimensionality and test targeting, known-group validity, and internal consistency of this subscale were examined. The Rasch analysis supported the unidimensionality of the fine motor subscale which accounted for 63.9% of the total variance. Most items, except two, showed a good fit, and the item difficulty levels (between −5.91 and 5.63 logits) effectively covered children’s ability levels (between −7.17 and 5.04 logits). Subscale scores significantly differed across age groups and between children with and without fine motor delays. The internal consistency was high at 0.96. This study provides initial psychometric evidence for the newly developed fine motor subscale, with ongoing refinement and validation aimed at establishing the final version of the Hong Kong Comprehensive Assessment Scales for Toddlers.

유아교사의 디지털 놀이 지원 역량 척도 개발

The purpose of this study is to develop a scale for assessing early childhood teachers' competencies in supporting digital play. The research questions addressed are as follow: Firstly, what constitutes the structure of the scale for assessing early childhood teachers' competencies in supporting digital play? Secondly, what are the validity and reliability of this scale? To achieve these objectives, a preliminary set of 50 items was derived from a comprehensive review of existing literature on digital play support. Content aspect validity verification and a pilot study were conducted, resulting in a refined draft of 40 items. Content aspect validity was assessed through analyses of item suitability and point-biserial correlation coefficients, confirming that all items were appropriate for the purpose. Empirical validity assessments demonstrated satisfactory levels of subject fit and a normal distribution pattern in the levels of subjects and item difficulty. Internal structure aspect validity confirmed the scale's fit with the Rasch model. Generalizability aspect validity testing indicated that the scale did not exhibit differential functioning across various groups of early childhood teachers. These results affirm that the scale is a valid and reliable instrument for measuring early childhood teachers' competencies in supporting digital play.

Development of Test Instruments Utilizing Multiple Representations for Enhancing Understanding of Motion Kinematics

Problem-solving in physics remains low due to the limited use of varied or multi-representation test instruments. This study aims to develop a multi-representation-based test instrument on kinematic motion and describe the results of the logical validity and empirical trials of the developed multi-representation-based test instrument for kinematic motion. This research follows a Research and Development (R&D) approach, using stages synthesized from various expert opinions, ranging from needs analysis to test assembly. The research instrument consists of a multi-representation-based test consisting of 30 questions. The data were analyzed using ANATES after being reviewed by three experts. The results indicate that 30 test items were successfully developed, with an average logical validity score of 0.63, demonstrating that the test instrument is valid and feasible for use. Based on statistical analysis, including item validity, difficulty level, discrimination index, and reliability, the empirical validity results show that 11 test items are suitable for use.

Evaluating Elasticity-Science Process Skills (E-SPS) in Construction Instrumen through Rasch Modelling

The assessment of material elasticity science process skills in high school students requires validated and reliable instruments. This study aims to develop and validate a set of multiple-choice questions using the Rasch model to ensure their suitability for educational assessments. This quantitative research employed a model approach based on the Rasch model. The instrument consisted of multiple-choice questions designed to evaluate material elasticity science process skills. The validation process included two stages: content validation and construct validation. Content validation involved eight practicing teachers as validators, while construct validation was conducted with a sample of 32 high school students. Data analysis was performed using MNSQ infit, MNSQ outfit, delta, discrimination index, item differential power, and item difficulty level, with the assistance of Quest data analysis software. The results of the analysis from Quest revealed that only 10 questions met the Rasch model criteria. Among these, question number six was identified as the most difficult. The validation process demonstrated that the developed questions are appropriate for assessing the targeted skills based on the Rasch model. The findings indicate that the validated instrument is reliable and suitable for evaluating material elasticity science process skills in high school students. This study underscores the importance of rigorous validation processes in developing educational assessments. The developed multiple-choice questions, validated through the Rasch model, are feasible for use in assessing material elasticity science process skills. Future research should explore the application of this instrument in various educational contexts to further confirm its reliability and effectiveness across different student demographics.

The Night Eating Questionnaire through the lens of the Rasch model

Previous research focused on the conventional approaches to test psychometric characteristics of the Night Eating Questionnaire (NEQ). The purpose of this research was to examine the psychometric properties of the Night Eating Questionnaire using the Rasch model in a sample of university students. The study was carried out from November 2018 to March 2019 on 300 students in health sciences at the University of Pristina temporarily seated in Kosovska Mitrovica, who completed the NEQ. A confirmatory factor analysis (CFA) suggested that the Serbian version mirrored the original NEQ structure: Goodness of fit index = 0.978, Comparative fit index = 0.996, Tucker-Lewis index = 0.995, Root Mean Square Error of Approximation = 0.011 and Standardized Root Mean Square Residual = 0.057. The Cronbach's alpha coefficient for the total scale was 0.627. The Rasch analysis showed that the item separation index classified the items into six groups based on their level of difficulty. The person reliability index separated well night eaters from day eaters. Few items did not fit the adequate range for the infit/outfit statistics. Overall, there were several groups of NEQ items that have a distinctive difficulty level, but the difference was not a remarkable one. This means that most students did not have night eating syndrome (NES), despite various levels of item difficulty. The NEQ performs well in the efforts to distinguish people who eat and do not eat at night. Most students reported conventional eating patterns and only a few had NES. The properties of the NEQ warrant its use in further night eating research.

Baby Observational Selective Control AppRaisal (BabyOSCAR): Construct validity and test performance.

To investigate the construct validity of the Baby Observational Selective Control AppRaisal (BabyOSCAR), an assessment of independent joint motion in infants with cerebral palsy (CP). BabyOSCAR was scored for 75 infants (45 with CP and 30 without CP). Rasch analysis was used in combination with classical test theory to assess areas of strength or improvement. Overall fit and precision, unidimensionality, local independence, reliability indices, Wright's child-item map, and differential item functioning were examined as part of Rasch analysis to investigate the item properties, internal construct validity, and reliability of BabyOSCAR. Cronbach's α was used to evaluate items' internal consistency. Analysis demonstrated good fit to the Rasch model, with only one erratic item. Unidimensionality results suggest two dimensions, split between arm and leg items. Item calibration reliability was between 0.84 and 0.86, with three distinct item difficulty levels. Infant measure reliability was between 0.82 and 0.91, separating infants into three ability levels. Together, the two subscales covered the full range of skills, with redundancy mostly between the same motion on both sides of the body. Cronbach's α was between 0.90 and 0.95. BabyOSCAR's construct validity was supported. Arm and leg subscales can be translated to a logit scale. Baby Observational Selective Control AppRaisal (BabyOSCAR) has excellent construct validity with good overall fit and precision. Individual BabyOSCAR items contribute and work well together, forming an interval-level assessment. BabyOSCAR has two separate subscales, arms and legs, that complement each other. BabyOSCAR's items represent a continuum of skills with three distinct difficulty levels. BabyOSCAR's continuum of skills reliably separates infants into three ability levels.

Rasch model assessment of algebraic word problem among year 8 Malaysian students

<p>Word problems continue to be a challenge for students today. All students must meet the prerequisites for problem solving and reasoning skills, which are important components of the critical thinking component of 21st century skills. This study is being conducted to assess students’ strategies for solving word problems with numbers, consecutives, and ages. The Rasch model is used to analyze the item difficulty level of word problems and students’ strategies for solving ten-word problems at various levels of item difficulty in a similar trait. Then, Pearson correlation analysis is used to investigate the item difficulty level in relation to linguistic, algebraic, and arithmetic factors of word problems before evaluating students’ performance on solving these word problems using various strategies. Rasch model found these algebraic word-problem questions are slightly harder for year 8 Malaysian students in relative to an international standard. Meanwhile, the item difficulty of word problems is driven by linguistic and algebra factors where students can score accurately if the word problems contained explicit information. However, the students encountered difficulties while losing their solution strategy when the questions contained implicit data that demanded critical thinking ability.</p>

Implementation of the Problem-Based Learning Model in Teaching of Mathematics about Adding Fractions with Different Denominators in Elementary Schools

Research has been carried out to find the level of student ability and the level of difficulty of essay questions on fractions, through polytomous data analysis by applying the Rasch model. The research was conducted in the In-Service Professional Teacher Education program, specifically in the Elementary School Teacher Education. The research was conducted at Belang 1 State Elementary School, Southeast Minahasa Regency, North Sulawesi, Indonesia, in grade 6 with a total of 27 students. Experimental research method with one group pretest-posttest design. The experimental treatment is a Problem-Based Learning (PBL) learning model. The test instrument is in the form of an essay with material on adding fractions with different denominators. The instrument was content validated through a video conference forum attended by experts, tutors, and subject teachers. Data was collected through formative pre-tests and post-tests. Scores are processed as polytomy data. Data analysis used the Rasch model with the Ministep 4.3.2 software application. The results of the analysis, as a class group, the pre-test average was 7.0 and the post-test was 77.0. This group's data shows that the use of the Problem-Based Learning (PBL) model in mathematics learning is very effective. Another conclusion is that the Rasch Model can accurately analyze polytomous data and find 1 student who has a high ability exceeding the highest level of test item difficulty in completing the essay test on adding fractions with different denominators. Then the Rasch Model can analyze the level of difficulty of the 5 items of the essay test instrument, it was found that 2 test items had a low level of difficulty. Using the Rasch model, it is consistently detected that adding fractions with different denominators containing the number 7 is categorized as having a high level of difficulty.

Pengembangan Instrumen Assesmen Literasi Sains Berbasis Kontekstual pada Materi Koloid

This research is a development research (Research and Development) with the objectives: (1) to assess the quality of the instrument in terms of content validity and empirical validity, (2) to determine the reliability of the developed instrument, and (3) to assess the practicality of the developed instrument. The product developed is a contextual science literacy assessment instrument on the topic of colloids. The developed contextual science literacy instrument is in the form of multiple-choice questions. The research was conducted using the 4D development model, which includes the stages of define, design, develop, and disseminate. The results of rational validation by two expert validators yielded 15 valid test items with an Aiken's index ≥ 0.75. The instrument was tested on 34 students from class XII MIPA 3 of SMAN 1 Gowa. The results of empirical validation using the point biserial formula at α=0.339 yielded 14 valid test items. The results of the reliability test using KR-20 showed that the instrument was categorized as reliable with a value of 0.831. The analysis of the item discrimination power showed that there were 2 very good items, 6 good items, 4 fair items, and 3 poor items. The distribution of the item difficulty level was 47% difficult items, 40% medium items, and 2% easy items. The practicality of the contextual science literacy assessment instrument on the topic of colloids was categorized as practical based on the positive results of the student questionnaire (85%) and the teacher questionnaire (95%).

An Analysis of Difficulty Level and Discrimination Indices of Items of Questionnaires on Reproductive Health Awareness of Adolescents

The study aims to analyze the difficulty level and discrimination index of the items on the questionnaires designed and developed to ascertain the awareness of adolescents about reproductive health. This pilot study was conducted to select the items in the research tools comprising preliminary drafts of two questionnaires having multiple-choice questions on reproductive health. The data was collected through purposive sampling, a total of 109 students of class eight from three private unaided CBSE affiliated schools from Sonipat district of Haryana participated in the study. The data was analyzed for difficulty level and discrimination index of each item on both the tools. The findings of the study led to the selection of the items in the final version of each tool. Those items which did not satisfy the criteria for difficulty level and discrimination index were discarded from the final version of the tool.

ChatGPT for generating multiple-choice questions: Evidence on the use of artificial intelligence in automatic item generation for a rational pharmacotherapy exam.

Artificial intelligence, specifically large language models such as ChatGPT, offers valuable potential benefits in question (item) writing. This study aimed to determine the feasibility of generating case-based multiple-choice questions using ChatGPT in terms of item difficulty and discrimination levels. This study involved 99 fourth-year medical students who participated in a rational pharmacotherapy clerkship carried out based-on the WHO 6-Step Model. In response to a prompt that we provided, ChatGPT generated ten case-based multiple-choice questions on hypertension. Following an expert panel, two of these multiple-choice questions were incorporated into a medical school exam without making any changes in the questions. Based on the administration of the test, we evaluated their psychometric properties, including item difficulty, item discrimination (point-biserial correlation), and functionality of the options. Both questions exhibited acceptable levels of point-biserial correlation, which is higher than the threshold of 0.30 (0.41 and 0.39). However, one question had three non-functional options (options chosen by fewer than 5% of the exam participants) while the other question had none. The findings showed that the questions can effectively differentiate between students who perform at high and low levels, which also point out the potential of ChatGPT as an artificial intelligence tool in test development. Future studies may use the prompt to generate items in order for enhancing the external validity of the results by gathering data from diverse institutions and settings.

Assessment of students' science literacy skills using diagnostic items on the concept of energy with Rasch modeling

This study aims to describe the quality of diagnostic items of the concept of Energy developed based on Rasch modeling and describe students' science literacy skills on the concept of Energy. The research method used in this research is descriptive. Research data were collected through online and offline tests and analyzed using the Rasch model and diagnostic items. The results showed that (1) The science literacy test instrument made was valid with an excellent category and a reliability value of 0.97, meaning that the quality of the items in the instrument was very good. This instrument can be used to assess students' abilities at all levels of ability, from those with low to high abilities; (2) For Person reliability or the quality of students' answers, the reliability value is below 0.30, meaning that the consistency of answers from students is low; (3) The quality of item questions based on the Rasch model from the item measure statistics results that all items meet the MNSA outfit criteria and no negative PTMEA Corr occurs. That is, all items are non-deviant, appropriate, and valid. This is also confirmed based on the bubble chat, where there are no items that are overfit or underfit; (4) The Rasch model makes very thorough and good measurements to measure the difficulty level of items and person's ability; (5) Based on the measurement results, it was found that the science literacy ability of students at SMPN 1 Paleleh was divided into low ability as much as 1.6%, moderate 53.6%, high 43.8%, very high 0.82%.

Language Assessment at a Thai University: A CEFR-Based Test of English Proficiency Development

The increasing popularity of the Common European Framework of Reference (CEFR) in non-native English-speaking countries has generated a demand for concrete examples in the creation of CEFR-based tests that assess the four main English skills. In response, this research endeavors to provide insight into the development and validation of a CEFR-based test aimed at evaluating undergraduate students’ English proficiency for placement tests and exit exams. The CEFR served as the framework for item development while Classical Test Theory informed the test evaluation process. A sample of 2,248 first-year students participated in Testing 1 and 3,655 first- and second-year students took part in Testing 2. The results of the analysis of the multiple-choice listening and reading tests indicated favorable levels of item difficulty and discrimination indices, as well as high reliability coefficients obtained from Cronbach’s alpha, Kuder-Richardson, and split-half reliability. The correlation and regression analyses revealed close relationships between the subtests and between each subtest and the total score, supporting the test’s criterion validity. The study also demonstrated significant predictive validity on TOEIC scores. The findings of this study offer implications for the development of university-level English proficiency tests that integrate CEFR levels and CTT analysis.

How can organizations measure the integration of environmental, social, and governance (ESG) criteria? Validation of an instrument using item response theory to capture workers' perception

AbstractEvaluation of corporate sustainability parameters has become part of organizational management. Item Response Theory (IRT) serves as a valuable approach for quantifying and exploring observable variables associated with these parameters. Therefore, this article aims to propose, apply, and validate an instrument utilizing IRT to measure environmental, social, and governance (ESG) impacts on corporate sustainability performance, using the perspective of workers. Workers were chosen as key stakeholders for the present study, as they were pointed by previous research as central to corporate sustainability decisions. The present paper addresses a literature gap by introducing, implementing, and validating a research instrument, along with a comparative scale, to assess the effects of integrating ESG criteria on corporate sustainability performance. Workers of two major Brazilian electricity sector companies were respondents for this instrument. Multigroup IRT method was used to assess item discrimination and difficulty levels, and to construct a scale reflecting workers' perceptions of integrating ESG criteria. The questionnaire consisted of 15 items, demonstrating Content Validity Coefficients (CVC) above .97, confirming the instrument's content effectiveness. Cronbach's Alpha and McDonald's Omega were greater than .94 for both companies. Questionnaire items demonstrated satisfactory discrimination ability and varying difficulty levels, resulting in a five‐level scale to gauge the impacts of ESG criteria. This research contributes through the introduction and validation of the research instrument, the latent trait assessment scale, and the evaluation of workers' perceptions regarding ESG criteria impacts on corporate sustainability performance. In conclusion, the proposed instrument showed robust psychometric properties, displaying favorable reliability and validity parameters. It effectively measures the impacts of ESG criteria, providing interpretable levels aligned with the unique characteristics of its constituent items

Breaking Ground in Skill Assessment: The Construction and Validation of a Problem-Solving Scale for University Students

Students today is expected to possess 21st-century skills, one of which is Problem Solving Skills (PSS). The importance of mastering this skill needs to be accompanied by the availability of a measuring tool. This research addresses this need by developing and validating the Problem-Solving Skill Scale. The development of this scale is focused on guidance and counseling students. The research and development of this instrument follow the educational design research method according to Plomp and Nieveen. Four hundred students were involved to test the construct validity. Problem-solving skills were measured based on three aspects: (1) Problem-Solving Confidence, (2) Approach-Avoidance Style, (3) Personal Control. Data were analyzed using the Rasch model by the WinSteps software. The research results describe the instrument's quality, the person-item interaction , the item difficulty level (item measures), fit and misfit items, and the distribution of respondents' abilities and the distribution of item levels on the same scale, and the scale’s ability to measure what it intends to measure. The Rasch analysis results show that the PSS Scale is good, precise, and has item fit with the model. The PSS Scale is a valid and reliable measuring tool for accurately measuring students' problem-solving skills. This research discusses implications and recommendations for further research efforts to facilitate the improvement of problem-solving skills in students using this measurement tool.

Revealing the characteristics of Indonesian language test used in the national-standardized school examinations

This study aimed to determine the characteristics of Indonesian language test used in the national standardized school examinations (Ujian Sekolah Berstandar Nasional, USBN). We used the response data of 218 students from a public senior high school in the Special Region of Yogyakarta, Indonesia, to the test consisting of two packages, A and B, in the 2018/2019 academic year to investigate the characteristics of the test and its items. Quantitative analysis using classical test theory (CTT) and one-parameter logistic item response theory (1-PL IRT) model was conducted to investigate the characteristics of the test and its items based on difficulty and discriminating power. The results of the study under the CTT showed that most test items in both package A and package B have difficulty in the easy category and the portion of items in the difficult category is no more than 10%. In addition, while the majority of test items in both test packages demonstrated good discriminating power, package B contained a high number of items with poor discriminating power (47.5%). Under 1-PL IRT, our study results indicated the dominance of items with difficulty level in the moderate category in both test packages. In addition to revealing the difficulty level of the test items, our study showed that the items in the difficult category under CTT and IRT were related to the topics of types of conjunction; spelling, grammar, and sentence structure; job application letter; and observation report text and its structure. The results of this study are expected to contribute to improving the quality of the Indonesian language test and the quality of learning on topics where students have difficulty based on test items in the difficult category.

Psychometric Properties of the Connor-Davidson Resilience Scale (CD-RISC) in Spanish Adolescents.

Few studies have examined the psychometric properties of the Connor-Davidson Resilience Scale (CD-RISC) in a large adolescent community sample, finding a significant disparity. This study explores the psychometric properties of the CD-RISC among Spanish adolescents by means of exploratory factor analysis (EFA), Rasch analysis, and measurement invariance (MI) across sex, as well as internal consistency and criterion validity. The sample was comprised of 463 adolescents (231 girls), aged 12 to 18 years, who completed the CD-RISC and other measures on emotional status and quality of life. The EFA suggested that the CD-RISC structure presented a unidimensional model. Consequently, shorter unidimensional CD-RISC models observed in the literature were explored. Thus, the Campbell-Sills and Stein CD-RISC-10 showed the soundest psychometric properties, providing an adequate item fit and supporting MI and non-differential item functioning across sex. Item difficulty levels were biased toward low levels of resilience. Some items showed malfunctioning in lower response categories. With regard to reliability, categorical omega was. 82. Strong associations with health-related quality of life, major depressive disorder symptoms, and emotional symptoms were observed. A weak association was found between resilience and the male sex. Campbell-Sills and Stein's CD-RISC-10 model emerges as the best to assess resilience among Spanish adolescents, as already reported in adults. Thus, independently of the developmental stage, the core of resilience may reside in the aspects of hardiness and persistence.