Item Difficulty Research Articles

A TEST TO MEASURE THE KNOWLEDGE LEVEL OF DAIRY FARMERS ABOUT ZOONOTIC DISEASES

One hundred and twenty-four items were initially constructed based on promoting thinking rather than memorization and differentiating the well-known target people from the poorly known ones to knowledge about the zoonotic disease. The scores from sample respondents were subjected to item analysis, item selection comprising of item difficulty index, item discrimination index and point biserial correlation. Sixty-two items with a difficulty index ranging from 20 to 80 per cent; discrimination index ranging from 0.2 to 0.8 and items with significant point biserial correlation either at 1 per cent or 5 per cent level were included in the final format of the knowledge test. The reliability of the knowledge test was found to be 0.96 using split–half method which was highly significant. The knowledge test was developed and standardized which is highly stable and dependable for measurement.

Knowledge relation rank enhanced heterogeneous learning interaction modeling for neural graph forgetting knowledge tracing.

Knowledge tracing models have gained prominence in educational data mining, with applications like the Self-Attention Knowledge Tracing model, which captures the exercise-knowledge relationship. However, conventional knowledge tracing models focus solely on static question-knowledge and knowledge-knowledge relationships, treating them with equal significance. This simplistic approach often succumbs to subjective labeling bias and lacks the depth to capture nuanced exercise-knowledge connections. In this study, we propose a novel knowledge tracing model called Knowledge Relation Rank Enhanced Heterogeneous Learning Interaction Modeling for Neural Graph Forgetting Knowledge Tracing. Our model mitigates the impact of subjective labeling by fine-tuning the skill relation matrix and Q-matrix. Additionally, we employ Graph Convolutional Networks (GCNs) to capture intricate interactions between students, exercises, and skills. Specifically, the Knowledge Relation Importance Rank Calibration method is employed to generate the skill relation matrix and Q-matrix. These calibrated matrices, alongside heterogeneous interactions, serve as input for the GCN to compute exercise and skill embeddings. Subsequently, exercise embeddings, skill embeddings, item difficulty, and contingency tables collectively contribute to an exercise relation matrix, which is then fed into an attention mechanism for predictions. Experimental evaluations on two publicly available educational datasets demonstrate the superiority of our proposed model over baseline models, evidenced by enhanced performance across three evaluation metrics.

Fourteen-item perceived stress scale assessment using item response theory among pregnant women.

The current study aimed to assess the psychometric properties of the Spanish language version of the 14-item Perceived Stress Scale (PSS-S) in a population of pregnant women who speak Spanish in Peru using item response theory (IRT). Our study consisted of 5,435 pregnant women who participated in the Pregnancy Outcomes Maternal and Infant Study (PrOMIS) cohort in Peru. Exploratory and confirmatory factor analyses were conducted to determine dimensionality of the scale in this population, and item response theory was conducted to determine the applicability of the PSS. The PSS consisted of a 2-factor questionnaire measuring perceived stress and coping capacity accounting for 77% of variability. The IRT analysis showed differences in item difficulty and discrimination. Item difficulty represents the level of the latent construct where 50% of respondents endorse a particular response, and item discrimination determines the rate of change of the probability of endorsing an item for differing ability levels. For the first factor, perceived stress, item 12 was the least difficult and item 2 was the most difficult. For the second factor, coping capacity, item 9 was the least difficult and item 6 was the most difficult. The Spanish version of the 14-item PSS can be a useful assessment tool for perceived stress, but more IRT should be done to delve further into the psychometric properties of the questionnaire to inform clinicians and policy makers more appropriately.

Measuring visually guided motor performance in ultra low vision using virtual reality.

Ultra low vision (ULV) refers to profound visual impairment where an individual cannot read even the top line of letters on an ETDRS chart from a distance of 0.5 m. There are limited tools available to assess visual ability in ULV. The aim of this study was to develop and calibrate a new performance test, Wilmer VRH, to assess hand-eye coordination in individuals with ULV. A set of 55 activities was developed for presentation in a virtual reality (VR) headset. Activities were grouped into 2-step and 5-step items. Participants performed a range of tasks involving reaching and grasping, stacking, sorting, pointing, throwing, and cutting. Data were collected from 20 healthy volunteers under normal vision (NV) and simulated ULV (sULV) conditions, and from 33 participants with ULV. Data were analyzed using the method of successive dichotomizations (MSD), a polytomous Rasch model, to estimate item (difficulty) and person (ability) measures. MSD was applied separately to 2-step and 5-step performance data, then merged to a single equal interval scale. The mean SD of completion rates were 98.6 1.8%, 78.2 12.5% and 61.1 34.2% for NV, sULV and ULV, respectively. Item measures ranged from -1.09 to 5.7 logits and - 4.3 to 4.08 logits and person measures ranged from -0.03 to 4.2 logits and -3.5 to 5.2 logits in sULV and ULV groups, respectively. Ninety percent of item infits were within the desired range of [0.5,1.5], and 97% of person infits were within that range. Together with item and person reliabilities of 0.94 and 0.91 respectively, this demonstrates unidimensionality of Wilmer VRH. A Person Item map showed that the items were well-targeted to the sample of individuals with ULV in the study. We present the development of a calibrated set of activities in VR that can be used to assess hand-eye coordination in individuals with ULV. This helps bridge a gap in the field by providing a validated outcome measure that can be used in vision restoration trials that recruit people with ULV, and to assess rehabilitation outcomes in people with ULV.

Topic Familiarity Effects on Performance in Speaking Assessment Tasks

Speaking assessment is believed to be difficult in its expansion and execution. Thus, it is a challenge for teachers in preparing students for speaking tests. This study’s purpose was to identify whether topic familiarity affects speaking performance among students who were preparing to sit for the high-stakes Malaysian University English Test speaking test. The study aimed to investigate the validity and reliability of the speaking assessment measures and, subsequently, to examine the speaking tests’ item difficulty measures differences according to topic familiarity level. Data were collected from 40 non-native speakers of English among Malaysian Form Six pre-university students who were preparing for their MUET test. The researcher conducted two practice speaking tests, which used retired papers of the MUET CEFR-aligned speaking tests, to the 40 participants who were grouped into 10 speaking test groups. The practice speaking tests were video and audio-recorded. Topic familiarity was measured using self-report questionnaires. In the second phase, each of the seven appointed raters scored all 40 students’ responses in two speaking tests consisting of two speaking tasks assessed across four criteria: task fulfillment, language, communicative ability, and group discussion. Many-facet Rasch measurement results revealed significant differences translating to significant influence of topic familiarity on speaking performance. The present study’s results not only confirmed the significance of topic familiarity in the preparation for speaking assessments, but also highlighted the need for formal teaching on potential topic themes that are commonly encountered in such assessments, particularly those that are at the post-secondary level. These findings imply importance in designing speaking tests taking into account test-taker topic familiarity.

Estimation of factors contributing to level differences in Japanese university decathlon athletes

This study aimed to estimate the factors that cause differences in competition level based on the competition performance structure among university student decathletes in Japan. The results of factor analysis using the maximum likelihood method (Oblimin rotation), assuming a five-factor structure estimated from parallel analysis and the information criterion, revealed the following competitive performance structures: "running speed and body projection,” "running endurance,” "rotational throwing power,” "translational throwing power,” and "vertical leaping power.” Some of these were similar to the competitive performance structures of the world’s top athletes, but they were found to have a unique structure: two throwing powers. The analysis results using latent rank theory allowed us to evaluate them on a seven-point scale. The calculated item reference profile, difficulty index, and discrimination index for each rank indicated that shot put and 100 and 400 m performance formed the basis of decathlon performance. In addition, high jump and pole vault, which fall under the category of "vertical leaping power," retained high difficulty and discrimination and were revealed to affect the stage of achievement of overall performance in the decathlon.

The metacognitive awareness of reading strategy among pre-service primary teachers and the possibility of rating improvement using Rasch analysis

This study investigated pre-service primary teachers' reading strategies using the Metacognitive Awareness of Reading Strategy Inventory (MARSI) in the Indonesian version. The participants were 4377 pre-service primary teachers in Indonesia. The data were collected via an online form, and the invitations were sent out randomly. The participants were divided into two studies: 2912 participants were in study 1, which used a 5-point Likert scale, and 1465 participants were in study 2, using a 4-point Likert scale. The Rasch measurement approach was used in investigating the validity and reliability, rating improvement analysis, checking item difficulty functioning based on gender, and classifying pre-service primary teacher reading strategies. The adapted questionnaire satisfied all psychometric requirements based on Rasch modeling. The questionnaire with a 4-point scale showed better function than the 5-point one based on the observed measure average and curve probability category. The teachers' abilities ranged from − 6 logit to 6.45 logit.

Identifying Misunderstandings and/or Misconceptions Among Test-Takers: An Analysis of Selected Items from a High-Stakes Mathematics Multiple-Choice Test

This study conducted an analysis of selected items from the Caribbean Secondary Education Certificate (CSEC) mathematics multiple-choice test to detect misunderstandings or misconceptions among Jamaican test-takers. The sample of the data included 7,000 CSEC test-takers’ test scores for the examination period 2010–2016. The study applied the Classical Test Theory (CTT) and the Item Response Theory (IRT) in the data analysis process. Results from the study showed that the CSEC mathematics multiple-choice test recorded a relatively high distractor efficiency (DE) for each exam year over the period 2010–2016; and the items were less prone to guessing, based on Jamaican test-takers’ responses. The CTT item difficulty analysis showed that Jamaican test-takers generally found topics such as measurement; and function, relations, and graphs more difficult in comparison to other topics. They displayed more of their misunderstandings or misconceptions in specific concepts tested in these areas on the test. The findings from this study have implications for the teaching and learning of mathematics in Jamaica before, at, and after the CSEC level.

ANALISIS MODEL RASCH UNTUK MENGUKUR KOMPETENSI PENGETAHUAN SISWA SMKN 1 KALIANGET PADA MATA PELAJARAN PERAWATAN SISTEM KELISTRIKAN SEPEDA MOTOR

Vocational students are prepared to work in certain positions through a curriculum in which its learning materials consist of 30% theory and 70% skill. It makes their mindset more concerned with skill-based subjects, not motivated to study the material theoretically, and reluctant to take knowledge-based tests. Knowledge is an important aspect for understanding, reasoning, and leading to rational actions when practicing. There is a need for the appropriate measuring tool for diagnosing the knowledge of vocational students in mapping knowledge so that it can be used to provide improved learning services in vocational high schools. The Rasch model is a modern evaluation tool using probability responses to items in a distribution map, so that the characteristics of vocational students in answering questions can be evaluated, starting from the level of students' abilities, distribution maps of ability and difficulty items, as well as a map of the distribution of answers that reflects their honesty. The research was conducted at the Motorcycle Engineering and Business Skills Competency Program (TBSM) of SMKN 1 Kalianget on the subject of Electrical System Maintenance, on the theme of Ignition System Maintenance, with 49 respondents. The instrument was 30 items of a multiple-choice test which was developed according to the Minister of Education and Culture of the Republic of Indonesia Number 23 of 2017 and the second Level of the Qualification Certification Scheme on TBSM Competence from BNSP 2017. The test result shows that instruments have good quality, by fulfilling the elements of reliability, validity (MNSQ outfit and ZSTD outfit, DIF item, unidimensional test), separation, and item difficulty level. Students’ knowledge ability is in the high category (44.9%), medium (44.94%), low as well as very low (only 8.16%). The results of Wright Map depict respondents’ opportunity to answer correctly by 57.14%. Based on the Guttman matrix scalogram, the percentage of careless students is just 2.04%, but that of students guessed the answers correctly is 12.24%

Higher Order Thinking Skill (Hots) Ability Instrument for Theme 2 (Unity in Diversity) in Class VI Elementary School

The lack of focus on HOTS assessment in the education system can lead to a lack of encouragement for students to develop critical, analytical and creative thinking skills. This research aims to develop a HOTS instrument for HOTS abilities for sixth grade elementary school students with theme 2 (Unity in Difference). The development of instruments in this research was guided by the Borg and Gall model. The HOTS instrument developed is a test in essay form. Meanwhile, the object in this research is the quality of the question items. The data collected in this research is data regarding the HOTS ability instrument. The method used to collect data in this research is the test method. Data were analyzed using content validity analysis tests, item validity tests, reliability tests, discrimination tests, and difficulty level tests. The content validity test results tested by five HOTS Instrument experts were declared relevant, the average content validity test results were declared valid, the average instrument validity test results were declared valid, the average instrument reliability test results were at very high criteria, the average results of the power test results which are in the good different power criteria, the average results of the difficulty level scores of the questions which are in the medium level category of difficulty. It was concluded that the HOTS ability instrument for sixth grade elementary school students with theme 2 (Unity in Difference) is suitable for measuring students' HOTS abilities in solving problems and training students to think creatively.

Analisis Butir Soal Ulangan Harian Semester Gasal Menggunakan Google Formulir di SMA Ipiems Surabaya

This research aims to (1) determine the quality of the daily test items for the even semester of class The sample for this research was students of class X Merdeka 5 at SMA Ipiems Surabaya with 33 students as respondents. This research instrument includes multiple choice questions and answer keys resulting from students' work, then tabulated analysis, validity, reliability, difficulty of questions, and differentiability of question items using the help of Microsoft Excel software. The results of this research show that of the 20 multiple choice questions, 21 students completed the multiple choice questions with a reliability value of r11 0.611, rtable level 1% 0.424, 1% reliable category, and the item difficulty index obtained was a value of 11 items in the medium category, 9 questions in the easy category, and 0 questions in the difficult category.

Analysis of Students’ Ability and Multiple-Choice Numeracy Test based on Item Difficulty Using Rasch

Analysis of Students’ Ability and Multiple-Choice Numeracy Test based on Item Difficulty Using Rasch

Education Research: Quality and Validity Evidence for a National In-Training Examination for Epilepsy Fellows.

Epilepsy education has been transformed over the past 2 decades, leading to a need for structured formative assessment tools. The American Epilepsy Society developed the Epilepsy Fellowship In-Training Examination (EpiFITE) to provide high-quality formative assessment for fellows, to stimulate program improvement, and to guide future learning and teaching. The aim of this study was to explore validity evidence for the EpiFITE in meeting these goals. Validity evidence was sought from multiple sources. The content of the examination was linked to the American Board of Psychiatry and Neurology blueprint for initial certification in epilepsy, and items were developed by trained experts. Internal structure was studied using internal consistency and item analysis. Surveys of fellows and fellowship directors focused on the examination experience (response process) and how results influenced fellow assessment, future learning, and program improvement (relationship to other variables and consequences). The EpiFITE was first administered in 2020, with 172 examinees from 67 programs. By 2022 (year 3), the EpiFITE was completed by 195 epilepsy fellows from 77 programs. The overall mean score of the examination was stable from year to year, and the committee predicted the difficulty of individual items with a high degree of accuracy. The examination had high internal consistency (Cronbach α 0.76-0.81). The median item discrimination index ranged from 0.17 in 2020 to 0.21 in 2022. Discrimination indices were lower (mean ≤0.10) for items that were either very easy or very difficult and significantly higher (mean >0.20) for other items. Program directors and epilepsy fellows agreed the examination questions were appropriate and agreed that the EpiFITE helped them identify areas for self-directed learning. Program directors also found the examination helpful in identifying areas of strength and areas for improvement within their programs. There are several sources of evidence of the quality and validity of the EpiFITE. By exploring this validity evidence, we have identified several best practices in the development and evaluation of a subspecialty examination, and this experience could be helpful for developers of in-training examinations in other subspecialties.

Comparative Analysis of NCLEX-RN Questions: A Duel Between ChatGPT and Human Expertise.

Artificial intelligence (AI) has the potential to revolutionize nursing education. This study compared NCLEX-RN questions generated by AI and those created by nurse educators. Faculty of accredited baccalaureate programs were invited to participate. Likert-scale items for grammar and clarity of the item stem and distractors were compared using Mann-Whitney U, and yes/no questions about clinical relevance and complex terminology were analyzed using chi-square. A one-sample binomial test with confidence intervals evaluated participants' question preference (AI-generated or educator-written). Qualitative responses identified themes across faculty. Item clarity, grammar, and difficulty were similar for AI and educator-created questions. Clinical relevance and use of complex terminology was similar for all question pairs. Of the four sets with preference for one item, three were generated by AI. AI can assist faculty with item generation to prepare nursing students for the NCLEX-RN examination. Faculty expertise is necessary to refine questions written using both methods. [J Nurs Educ. 2023;62(12):679-687.].

Measurement properties of Brief-BESTest scores from children, adolescents, and youth with visual impairments

Purpose: This study aimed to vet the measurement properties of Brief-BESTest scores in children, adolescents, and youth with visual impairment and blindness (YVI). Methods: A cross-sectional sample of YVI (N = 101) completed the Brief-BESTest, a modified version of the Y-Balance Test, the 360-degree turn test, bipedal quiet stance, and Activity-Specific Balance Confidence Scales. Thirty-seven YVI completed test-retest of the Brief-BESTest using a one-year interval. Using classical test theory, various forms of total and item-level Brief-BESTest score reliability and validity were investigated in YVI. Results: All inter-rater reliability coefficients were ≥ .80. When considering the eight items of the Brief-BESTest, 27 of the 28 possible correlations were statistically significant (p<.05). Various internal consistency and item difficulty results were strong. When taking total Brief-BESTest scores and their association with the complementary balance tasks/metrics into account, 11 of 13 associations were statistically significant (p<.05) providing strong convergent validity evidence. Being multimorbid and degree of vision significantly predicted total Brief-BESTest scores (p<.001) suggesting construct (i.e. known groups) validity. Numerous test-retest results (e.g. coefficients, limits of agreement) following the one-year interval were indicative of score stability. Conclusion: Practitioners and researchers should have confidence in, and consider adopting, the Brief-BESTest to examine multidimensional balance in YVI.

The Development of the Inadvertent Perioperative Hypothermia Knowledge Test (IPH-KT)

PurposeThis study aimed to develop the inadvertent perioperative hypothermia knowledge test (IPH-KT) for health care professionals and examine its validity and reliability. DesignThe methodological and cross-sectional study design was used. MethodsThe research was carried out with 326 nursing students in Turkey. Data for the study were collected using a demographic information form and the IPH-KT. FindingsThe item difficulty index and item distinctiveness index of the questions included in the draft form of the 25-item test ranged from 0.03 to 0.81 and from 0.25 to 0.99, respectively. After the analyses, 8 items were excluded from the draft test, and the final version of the test consisted of 17 questions. The item difficulty index of the final test ranged from 0.30 to 0.96, and the item distinctiveness index ranged from 0.33 to 0.81. The reliability of the test was determined using the Kuder-Richardson formula 20 and found to be 0.72, indicating high internal consistency. Therefore, the IPH-KT was accepted as a reliable test. ConclusionsThe study revealed that the questions in the IPH-KT had varying difficulty levels and a high ability to discriminate between individuals with knowledge of IPH and those without. The results demonstrated that the test had good content and face validity and showed high reliability for measuring the IPH knowledge of nursing students.

Health-Related Quality of Life in Patients With Ventricular Assist Device: Psychometric Evaluation of the German Version of the Quality of Life With a Ventricular Assist Device Questionnaire.

Ventricular assist device (VAD) implantation has become an alternative treatment for patients with end-stage heart failure. In Germany, valid and reliable instruments to assess health-related quality of life in patients with VAD are lacking. The aim of this study was to present the psychometric validation of the German version of the Quality of Life with a Ventricular Assist Device questionnaire. In a multicenter, cross-sectional study, 393 participants (mean age, 58.3 years; 85.8% male, 60.3% bridge to transplant, and 72.8% living with VAD for ≤2 years) completed the German Quality of Life with a Ventricular Assist Device questionnaire of physical, emotional, social, cognitive, and meaning/spiritual domains. Item and confirmatory factor analyses were conducted to test item difficulty and discrimination and the underlying structure, respectively. To examine internal consistency, Cronbach α was assessed. Convergent construct validity was tested using the Kansas City Cardiomyopathy Questionnaire and the Patient Health Questionnaire-9. Readability was examined using Flesch Reading Ease index and Vienna Factual Text Formula. The Quality of Life with a Ventricular Assist Device showed reasonable item difficulty (Ptotal = .67) and mostly moderate to high discriminatory power (rit > 0.30). In confirmatory factor analysis, root-mean-square error of approximation (0.07) was acceptable for model fit, but no other indices. Acceptable internal consistency was found (α ≥ 0.79), with the exception of the cognitive domain (α = 0.58). The overall questionnaire and single domains demonstrated convergent validity (r ≥ 0.45, P < .001). The questionnaire showed adequate readability (Flesch Reading Ease, 64.11; Vienna Factual Text Formula, 6.91). Findings indicate a promising standardized clinical instrument to assess health-related quality of life in patients with VAD.

Item difficulty index, discrimination index, and reliability of the 26 health professions licensing examinations in 2022, Korea: a psychometric study.

This study presents item analysis results of the 26 health personnel licensing examinations managed by the Korea Health Personnel Licensing Examination Institute (KHPLEI) in 2022. The item difficulty index, item discrimination index, and reliability were calculated. The item discrimination index was calculated using a discrimination index based on the upper and lower 27% rule and the item-total correlation. Out of 468,352 total examinees, 418,887 (89.4%) passed. The pass rates ranged from 27.3% for health educators level 1 to 97.1% for oriental medical doctors. Most examinations had a high average difficulty index, albeit to varying degrees, ranging from 61.3% for prosthetists and orthotists to 83.9% for care workers. The average discrimination index based on the upper and lower 27% rule ranged from 0.17 for oriental medical doctors to 0.38 for radiological technologists. The average item-total correlation ranged from 0.20 for oriental medical doctors to 0.38 for radiological technologists. The Cronbach α, as a measure of reliability, ranged from 0.872 for health educators-level 3 to 0.978 for medical technologists. The correlation coefficient between the average difficulty index and average discrimination index was -0.2452 (P=0.1557), that between the average difficulty index and the average item-total correlation was 0.3502 (P=0.0392), and that between the average discrimination index and the average item-total correlation was 0.7944 (P<0.0001). This technical report presents the item analysis results and reliability of the recent examinations by the KHPLEI, demonstrating an acceptable range of difficulty index and discrimination index values, as well as good reliability.

Cross-Cultural Validation of the TeamSTEPPS Teamwork Perceptions Questionnaire-Spanish in Colombia.

Teamwork influences health care quality and patient safety. Yet, validated instruments for assessing teamwork in Colombia are lacking. The purpose of this study was to validate the Spanish version of the TeamSTEPPS-Teamwork Perceptions Questionnaire (T-TPQ-S) for the Colombian health care context. The T-TPQ-S underwent translation, cultural adaptation, and comprehensive psychometric testing, including reliability and confirmatory factor analyses and item difficulty and discrimination analyses. The T-TPQ-S demonstrated high internal consistency and excellent fit to the theoretical model. Item discrimination was within expected ranges, with response thresholds displaying an ascending order. The tool better differentiated subjects with low and high teamwork perceptions. The T-TPQ-S is an effective, reliable, and valid instrument for assessing teamwork perception among Colombian health care workers.

Cancer advance care planning: development of a screening tool

ObjectivesAdvance care discussions are a useful communication tools for medical preferences and beneficial for shared decision-making processes in hospital settings. The present study developed the first screening tool for need...