Difficulty Level Of Questions Research Articles

Analysis the quality of critical thinking and creativity questions in high school biology subjects with the Rasch model

A good instrument is an instrument that can measure students' abilities accurately. The research aims to analyze the quality of critical thinking and creativity items on water pollution material in terms of validity, reliability, and level of difficulty using the Rasch model. The research used a quantitative descriptive method involving biology learning experts and 173 class X high school students. The research was carried out in December 2023-January 2024. The data collection method used expert assessment sheets and critical thinking and creativity essay questions. Data were analyzed using the Rasch model assisted by Quest. The results showed that of the 10 critical thinking essay questions, 3 questions were invalid, while of the 8 creativity essay questions, 1 question was invalid based on the suitability of the INFIT MNSQ and OUTFIT T scores. The results of the reliability test with Cronbach’s alpha showed that the reliability value of the critical thinking instrument was 0.74 and creativity 0.79. Average difficulty level of questions with threshold values in the medium and difficult categories. Overall, the instrument was feasible and question items that did not fit the Rasch model were removed.

Optimizing Educational Assessment: The Practicality of Computer Adaptive Testing (CAT) with an Item Response Theory (IRT) Approach

This research aims to develop a Computer Adaptive Test (CAT) system using the Items Response Theory (IRT) approach. This study is part of developing a web-based system using the Research and Development (R&D) method, employing the Four-D (4-D) model. At its core, this system is similar to a Computer-Based Test (CBT). Still, the critical difference lies in its ability to randomize and provide questions that align with the test-taker's skill levels using the Items Response Theory (IRT) algorithm. The system employs the 3-PL model from the Items Response Theory, considering the difficulty level of questions, the discriminative power of questions, and the likelihood of guessing or interference in the questions. The examination system randomly assigns questions to students based on their responses to previous questions, ensuring that each test-taker receives a unique question sequence. The exam concludes when a test-taker accurately estimates their ability, i.e., SE <= 0.01, or when all questions have been answered. The outcome of this research is a Computer Adaptive Test (CAT) system based on the Items Response Theory (IRT), which can be used to assess students' learning outcomes. This research was implemented in the Multimedia Department of SMK Negeri 1 Gunung Talang, with 90 students as the research sample. The evaluation of the practicality of this system received very high scores, indicating that the Computer Adaptive Test (CAT) system based on the Items Response Theory (IRT) is considered highly practical and effective in achieving the established measurement goals.

Educational Data Mining: Multiple Choice Question Classification in Vocational School

Data mining on student learning outcomes in the education sector can overcome this problem. This research aimed to provide a solution for selecting quality multiple choice questions (MCQ) using the results of students’ mid-semester exams in vocational high schools using a Data Mining approach. The research method used was the Cross-Industry Standard Process for Machine Learning (CRISP-ML) model. Steps to assess the accuracy of analyzing the difficulty level of questions based on student profile data and midterm test results. The data used in this research were the findings of basic computer tests on mid-term exams in mathematics disciplines at vocational high schools. This research used several classification algorithms, including SVM, Naive Bayes, Random Forest, Decision Three, Linear Regression, and KNN. The results of evaluating the classification

Racking Analysis Instrument Mastery Test Concepts in Learning Science Using the RADEC Model in Elementary School Students

The study's objective was to assess the content's validity, reliability, and degree of difficulty for third-grade students in one of the East Jakarta elementary schools utilizing the RADEC learning model by analyzing the pre-test and post-test items using the Rasch Racking model of analysis. The results of the Radec learning model showed that students with low and high abilities experienced a change in the concept of student effectiveness for the better. This research is a quantitative study, namely the evaluation of learning outcomes using descriptive quantitative methods with data collection techniques using pre-test and post-test questions. In early 2023, The learning activity was conducted using data from 32 student respondents. The result indicated the validity of the suitability level of the item, with ten questions found to be valid. The difficulty level of the questions, in general, is in the medium category, meaning that they are excellent because they are relatively straightforward. The conclusion is that the analysis of the items using the Rasch model in the learning evaluation activities through the pre-test and post-test is valid with excellent categories of 50% and good 50%. The question's difficulty level is excellent

Multiple Choice Question Difficulty Level Classification with Multi Class Confusion Matrix in the Online Question Bank of Education Gallery

Multiple Choice Question Difficulty Level Classification with Multi Class Confusion Matrix in the Online Question Bank of Education Gallery

The Analysis of Chemistry Test Item Difficulty on XI MIPA Year-End Assessment at SMAN 8 Pontianak

This study aims to categorize the level of difficulty of the questions and describe the cognitive level of the KKO indicators of the 2021/2022 year-end assessment (PAT) of chemistry class XI MIPA at SMAN 8 Pontianak. This research is quantitative descriptive. Data collection techniques were interviews and documentation. Documentation was obtained from chemistry teachers through google forms containing answers and PAT questions. The research subjects amounted to 150 students of class XI MIPA who had taken the End of Year Assessment. The results of the research obtained are difficult categories totaling 4 questions (13.3%), moderate 10 questions (33.3%) and easy 16 questions (53.4%). The results of the analysis obtained in order to improve the distribution of the difficulty level of questions in the future.

The Effect of Computational-Based Learning Media Using the STAD Type Cooperative Model on Student Learning Outcomes and Motivation on the Subject of Intermolecular Forces

This study aims to determine the effect of computational chemistry-based learning media using the STAD type cooperative learning model on student learning outcomes and motivation on the subject of forces between molecules. The population in this study were grade X students in high schools in Medan city which amounted to 7 classes. The samples of this study were students of class X IPA 3 and X IPA 5, each of which amounted to 36 students. The research instruments used were test instruments in the form of 20 multiple choice questions and non-test in the form of 30 statements. The quality of the research instrument was tested using validity, reliability, difficulty level of questions and differentiating power to determine the questions that would be used as instruments in the study. The data analysis technique shows that in the learning outcomes hypothesis test, the t-count> t-table value is 5.03>1.666 and the results of the learning motivation hypothesis test, the t-count> t-table value is 3.12>1.666. This data shows that there is an effect of computational chemistry-based learning media using the STAD type cooperative learning model on student learning outcomes and motivation on the subject of forces between molecules.

Analisis Taksonomi Bloom Dalam Mengidentifikasi Tingkat Kesulitan Pertanyaan Soal Dalam Mata Pelajaran Matematika Di Sekolah Dasar

This study aims to analyze the use of Bloom's Taxonomy in identifying the difficulty level of questions in mathematics in elementary schools. Bloom's taxonomy is a classification framework used to classify learning objectives into six different levels, namely knowledge, understanding, application, analysis, synthesis, and evaluation. The research subjects were 15 students in class V at SDN Karet 2, Tangerang Regency. The research method used was content analysis, in which questions in mathematics textbooks for elementary schools were analyzed based on Bloom's Taxonomy categories. The data collected included the difficulty level of the questions based on Bloom's taxonomy and the distribution of the difficulty levels of the questions in mathematics textbooks. The results showed that the level of difficulty of the questions in mathematics in elementary schools varied and could be classified based on Bloom's taxonomy. Most of the items focus on levels of knowledge and understanding, with a few items testing students' abilities to apply, analyze, synthesize, and evaluate mathematical concepts. This shows that there is a tendency to emphasize understanding concepts rather than applying concepts in learning mathematics in elementary schools.

Difficulty Level Identification of Indonesian and Mathematics Multiple Choice Questions using Machine Learning Approach

Examination question design is an important factor that could improve education, which could help teachers to analyze student understandings. Designing question should consider difficulty level, which commonly classified into three types: easy, medium, difficult. Predicting the difficulty level of questions is very important to help teachers form questions and know the level of student ability. In this study, we tackle question difficulty level identification as a classification problem. We use a dataset of Indonesian and mathematic question from elementary and junior or school exercise questions set and employ several machine learning methods on classification. We use Random Forest, Logistic Regression, SVM, Gaussian, and Dense NN on the experiment, with embeddings, lexical, and syntactic feature. The evaluation result shows that the best method on identifying question difficult level on Indonesian subject is Random Forest with 83% accuracy, while on mathematic subject the best method is Random Forest with 83% accuracy. Result analysis shows that embedding feature affect the model accuracy.

ANALYSIS OF MATHEMATICS QUESTIONS ON THEME 9 AT GRADE V OF RAUDHATUL RAHMAH ISLAMIC ELEMENTARY SCHOOL PEKANBARU

This paper discusses learning evaluation through test techniques, which are conducted on fifth-grade elementary school students on Mathematics learning subjects. The research is a type of quantitative descriptive research using the ex post facto method. The instrument used was 20 items of math subject matter questions, which were collected through a written test technique on students. The data obtained were analyzed using the Rasch Model assisted by the Winstep program through an empirical validity process, which includes validity, reliability, and difficulty level tests. The results indicate that in the validity test, items were invalid. In the reliability test, there were 2 reliabilities, Person Reliability was categorized as very good and Item Reliability was categorized as weak. Whereas on the results of the question difficulty level was categorized as very difficult. The implication of the research is that the research can be a reference for other researchers in conducting research on identifying and analyzing question items. When designing the questions, it is expected that a review will be carried out regarding the material that will be discussed on the test.

Objective evaluation of tongue diagnosis ability using a tongue diagnosis e-learning/e-assessment system based on a standardized tongue image database.

In Kampo medicine, tongue examination is used to diagnose the pathological condition "Sho," but an objective evaluation method for its diagnostic ability has not been established. We constructed a tongue diagnosis electronic learning and evaluation system based on a standardized tongue image database. This study aims to verify the practicality of this assessment system by evaluating the tongue diagnosis ability of Kampo specialists (KSs), medical professionals, and students. In the first study, we analyzed the answer data of 15 KSs in an 80-question tongue diagnosis test that assesses eight aspects of tongue findings and evaluated the (i) test score, (ii) test difficulty and discrimination index, (iii) diagnostic consistency, and (iv) diagnostic match rate between KSs. In the second study, we administered a 20-question common Kampo test and analyzed the answer data of 107 medical professionals and 56 students that assessed the tongue color discrimination ability and evaluated the (v) correct answer rate, (vi) test difficulty, and (vii) factors related to the correct answer rate. In the first study, the average test score was 62.2 ± 10.7 points. Twenty-eight questions were difficult (correct answer rate, <50%), 34 were moderate (50%-85%), and 18 were easy (≥85%). Regarding intrarater reliability, the average diagnostic match rate of five KSs involved in database construction was 0.66 ± 0.08, and as for interrater reliability, the diagnostic match rate between the 15 KSs was 0.52 (95% confidence interval, 0.38-0.65) for Gwet's agreement coefficient 1, and the degree of the match rate was moderate. In the second study, the difficulty level of questions was moderate, with a correct rate of 81.3% for medical professionals and 82.1% for students. The discrimination index was good for medical professionals (0.35) and poor for students (0.06). Among medical professionals, the correct answer group of this question had a significantly higher total score on the Kampo common test than the incorrect answer group (85.3 ± 8.4 points vs. 75.8 ± 11.8 points, p < 0.01). This system can objectively evaluate tongue diagnosis ability and has high practicality. Utilizing this system can be expected to contribute to improving learners' tongue diagnosis ability and standardization of tongue diagnosis.

Predictability of Discrimination Coefficient and Difficulty Index of Psychiatry Multiple-Choice Questions

Background: Multiple-choice questions are among the most common written tests. This study aimed to evaluate the faculty members’ability to determine and predict the level of difficulty and discrimination coefficient of multiple-choice tests at Psychiatry Department. Methods: All faculty members at Psychiatry Department of Iran University of Medical Sciences participated in this study. The difficulty and discrimination coefficient of all questions (150 questions) of the mid-term exam of psychiatric residents were measured with both software program and formulas by hand. Then, from each group of questions with high, medium, and low difficulty coefficient, 10 questions (30 questions in total) were selected and provided to faculty members for ranking each question in terms of difficulty and discrimination coefficient. Finally, the correlation between faculty members’ evaluation and standard results was measured by the Spearman’s correlation. To calculate the discrimination coefficient, the number of people who answered a question correctly in the low-score group was subtracted from the high-score group and then the result was divided by the number of people in a group. Results: Twenty-five faculty members participated in this study. There was a significant negative correlation between difficulty level and discrimination coefficient in the whole group (r=-0.196, p=0.045), but this was not the case in the upper and lower groups (r=-0.063, p=0.733). In addition, the correlation between the discrimination coefficient obtained from the formula and the average discrimination coefficient of faculty members was not significant (r=-0.047, p=0.803). Conclusion: It seems that the ability of faculty members to predict the discrimination coefficient and difficulty level of questions is not sufficient.

Determinants of grader agreement: an analysis of multiple short answer corpora

The ’short answer’ question format is a widely used tool in educational assessment, in which students write one to three sentences in response to an open question. The answers are subsequently rated by expert graders. The agreement between these graders is crucial for reliable analysis, both in terms of educational strategies and in terms of developing automatic models for short answer grading (SAG), an active research topic in NLP. This makes it important to understand the properties that influence grader agreement (such as question difficulty, answer length, and answer correctness). However, the twin challenges towards such an understanding are the wide range of SAG corpora in use (which differ along a number of dimensions) and the hierarchical structure of potentially relevant properties (which can be located at the corpus, answer, or question levels). This article uses generalized mixed effects models to analyze the effect of various such properties on grader agreement in six major SAG corpora for two main assessment tasks (language and content assessment). Overall, we find broad agreement among corpora, with a number of properties behaving similarly across corpora (e.g., shorter answers and correct answers are easier to grade). Some properties show more corpus-specific behavior (e.g., the question difficulty level), and some corpora are more in line with general tendencies than others. In sum, we obtain a nuanced picture of how the major short answer grading corpora are similar and dissimilar from which we derive suggestions for corpus development and analysis.

The performance of some machine learning approaches and a rich context model in student answer prediction

Web-based learning systems with adaptive capabilities to personalize content are becoming nowadays a trend in order to offer interactive learning materials to cope with a wide diversity of students attending online education. Learners’ interaction and study practice (quizzing, reading, exams) can be analyzed in order to get some insights into the student’s learning style, study schedule, knowledge, and performance. Quizzing might be used to help to create individualized/personalized spaced repetition algorithm in order to improve long-term retention of knowledge and provide efficient learning in online learning platforms. Current spaced repetition algorithms have pre-defined repetition rules and parameters that might not be a good fit for students’ different learning styles in online platforms. This study uses different machine learning models and a rich context model to analyze quizzing and reading records from e-learning platform called Hypocampus in order to get some insights into the relevant features to predict learning outcome (quiz answers). By knowing the answer correctness, a learning system might be able to recommend personalized repetitive schedule for questions with maximizing long-term memory retention. Study results show that question difficulty level and incorrectly answered previous questions are useful features to predict the correctness of student’s answer. The gradient-boosted tree and XGBoost models are best in predicting the correctness of the student’s answer before answering a quiz. Additionally, some non-linear relationship was found between the reading learning material behavior in the platform and quiz performance that brings added value to the accuracy for all used models.

Evaluation of a 3D-MC examination format in anatomy

IntroductionThe quality of education in medical anatomy is a fundamental pillar of good clinical practice. Current reforms of the medical curriculum have resulted in major methodological changes in the teaching and testing of anatomy. A number of recent studies have however described a decrease in positive metrics of anatomical knowledge among students so taught. It has been suggested that the reduced anatomical knowledge measured in these studies may endanger patient safety. As proxy measures of exam quality, evaluation of the levels of students ‘achievement in the examinations, assessment of the subjectively perceived level of question difficulty and analysis of exam satisfaction are each suitable parameters of investigation of medical education. Material and methodsTo address these issues with regard to medical education at the Charité-Universitätsmedizin Berlin, we have analyzed students’ levels of achievement in the anatomical Three Dimensional Multiple Choice (hereafter, 3D-MC)-examination of 2,015 students matriculated in medical studies from Summer Semester of 2014 through Summer Semester of 2017. We either compared students’ achievement levels of identical 3D-MC questions using models or prepared anatomical specimen. Furthermore, we have analyzed the type and frequency of cognitive levels used in the anatomical questions in relation to the students’ level of achievement. Finally, we conducted an anonymous survey to measure students’ (n = 207) and instructors’ (n = 16) satisfaction with the 3D-MC-examination in comparison to other employed anatomical testing strategies. ResultsStudents’ achievement is significantly enhanced with anatomical questions using models relative to those utilizing anatomical specimen. Over 80% of the anatomical questions in the 3D-MC-examinations assessed the lowest cognitive levels and higher cognitive question levels were accompanied by a significant decrease of the levels of students’ performance. Our survey further revealed that both, students and instructors preferred the practical examinations in anatomy and that the difficulty levels of the 3D-MC-examination was perceived as being the lowest in comparison to the other anatomical testing strategies. DiscussionTesting levels of anatomical understanding using anatomical models is not comparable to human specimen, and thus using specimen before models should be preferred to learn and test close to an authentic medical situation. The application of anatomical models and low cognitive question levels in the examination reduces the subjectively perceived level of difficulty, encourages superficial learning, and therefore decreases the retention of anatomical knowledge. ConclusionAlthough students and instructors prefer practical examinations in anatomy, the current development does not reflect these results. Therefore, it would be recommendable to rethink the development of anatomical testing strategies based on the existing evidence.

Analysis the Difficulty of Indonesian Language Class V Elementary School / Analisis Tingkat Kesulitan Bahasa Indonesia Kelas V Madrasah Ibtidaiyah

Indonesian Language subject is often considered easy by studensts, students are able to answer questions even though they don’t study it, and get the highest score compared to other subjects. This study aims to explain the level of difficulty and cognitive level of question of Bahasa Indonesia subject in Class V MIN 2 Banda Aceh. The question were given in the examination of odd semester 2019. The type of this research is quantitative descriptive by examining the relationship between one variable with another variable. Data collection techniques to get more detailed information wasgained by interviewing Teacher of Bahasa Indonesia Subject of Class V MIN 2 Banda Aceh. Object of this study was question sheet of Bahasa Indonesia in semester 2019, and student answers. The results showed that the difficulty level of the questions was 25% in the easy category, 65% the medium category, and 10% in the difficult categories. On the other hand, the cognitive level of questions is87.5% in the categories C1, 5% in the categories C2 questions and 7.75% in the categories C6 questions while C3, C4, and C5 categories are not found.

Automatic Grading and Adaptive Question Selection for English Language Testing

Computer based evaluation of students’ performance is playing a vital role worldwide for ELP examinations. This method is more efficient than manual evaluating process. In this study, a new method is proposed to evaluate the examinees descriptive answers using Text Mining and Text processing algorithms with the help of Natural Language Processing tools. This system includes three sections of ELP, namely: reading, writing and speaking. Dataset for the examination process is extracted using the UiPath platform from reference books which are stored into the database. On taking a test, the system automatically calculates the result and polarity of the subjective answers using NLP and fuzzy logic. In this study, we used Rasch algorithm for adapting to the ability of the examinee for question difficulty level selection and also evaluate the same answer using Natural language processing (NLP) algorithm to check grammar mistakes, polarity and stores the marks in database and finally represent them graphically for better visualization of results.

Should we be banking on it? Exploring potential issues in the use of ‘item’ banking with structured examination questions

ABSTRACT Item banking involves tests being constructed by selecting from a bank of pre-written questions. There are various examples of multiple-choice tests where item banking is used, but few examples involving other question types. This research explored the use of banking with structured questions. Three question writers were asked to construct examination papers using a bank of structured physics questions and were interviewed. Factors considered when constructing examinations included the topic and skills to be assessed, the intended difficulty level of questions, and task type (e.g., definition, explanation, calculation, graph use). Participants’ strategies varied with one spending time viewing a range of questions, the second using information about the questions to narrow down which to view, and the third relying mostly on information about the questions. Several issues arose such as challenges around achieving the necessary total mark and potential wastage of questions if the topics in a syllabus change.

Using Wearable Physiological Sensors for Affect-Aware Intelligent Tutoring Systems

Intelligent Tutoring Systems (ITS) have shown great potential in enhancing the learning process by being able to adapt to the learner's knowledge level, abilities, and difficulties. An aspect that can affect the learning process but is not taken into consideration by traditional ITS is the affective state of the learner. In this work, we propose the use of physiological signals and machine learning for the task of detecting a learner's affective state during test taking. To this end, wearable physiological sensors were used to record electroencephalography (EEG), electrocardiography (ECG), and electromyography (EMG) signals from 27 individuals while participating in a computerised English language test. Features extracted from the acquired signals were used in order to train machine learning models for the prediction of the self-reported difficulty level of the test's questions, as well as for the prediction of whether the questions would be answered correctly. Supervised classification experiments showed that there is a relation between the acquired signals and the examined tasks, reaching a classification F1-score of 74.21% for the prediction of the self-reported question difficulty level, and a classification F1-score of 59.14% for predicting whether a question was answered correctly. The acquired results demonstrate the potential of the examined approach for enhancing ITS with information relating to the affective state of the learners.

OPINI PESERTA SELEKSI CALON APARATUR SIPIL NEGARA (ASN) TERHADAP SISTEM COMPUTER ASSISTED TEST (CAT)

The purposes of using Computerized Assisted Test (CAT) system in civil servant selection were to create transparency, objectivity, accountability and efficiency. Some of the problems formulated in this study are how the public opinion of facilities and services; transparency and purity; and the level of difficulty of the CAT system exam questions. This study aims to determine the opinion of civil servant candidates for Bandung District about facilities, transparency and difficulty level of exam questions with the CAT system. The study was conducted in the Gedung Serbaguna Telkom University Bandung Regency. The population is all civil servant candidates who take the exam by CAT system in Bandung Regency in 2018. The number of samples is 100 people. The variables studied were facilities and services; transparency and purity; exam and graduation questions. Descriptive statistical analysis was performed with the help of SPSS 24 software. The results of the study obtained an average opinion of the participants on the facilities and service variables as well as transparency and were above 3 (three). The lowest average participant opinion is an indicator of the difficulty level of the questions (average = 1.91).