Characteristics Of Test Items Research Articles

Argument-based validation of Academic Collocation Tests

Despite extensive research on assessing collocational knowledge, valid measures of academic collocations remain elusive. With the present study, we begin an argument-based approach to validate two Academic Collocation Tests (ACTs) that assess the ability to recognize and produce academic collocations (i.e., two-word units such as key element and well established) in written contexts. A total of 343 tertiary students completed a background questionnaire (including demographic information, IELTS scores, and learning experience), the ACTs, and the Vocabulary Size Test. Forty-four participants also took part in post-test interviews to share reflections on the tests and retook the ACTs verbally. The findings showed that the scoring inference based on analyses of test item characteristics, testing conditions, and scoring procedures was partially supported. The generalization inference, based on the consistency of item measures and testing occasions, was justified. The extrapolation inference, drawn from correlations with other measures and factors such as collocation frequency and learning experience, received partial support. Suggestions for increasing the degree of support for the inferences are discussed. The present study reinforces the value of validation research and generates the momentum for test developers to continue this practice with other vocabulary tests.

PERBANDINGAN KEBERFUNGSIAN DISTRAKTOR DALAM TES: PENDEKATAN TEORI TES KLASIK DAN MODERN DALAM KONTEKS PENDIDIKAN SEKOLAH DASAR

This study aims to examine the characteristics of formative test items in elementary schools by comparing the functionality of distractors based on classical and modern test theories. This research is a quantitative descriptive study. The data collection technique used in this study was documentation. The results of the analysis indicate that out of 25 test items, 16% were easy, 72% were moderate, and 12% were difficult, with 3 items not fitting. The measurement results using modern test theory showed that 60% of the distractors were not functioning effectively, and 24% of the items needed to have their answer keys checked, while the measurement using classical test theory showed that 56% of the distractors were not functioning effectively, and 12% of the items needed to have their answer keys checked. Less effective distractors indicate that they are selected more frequently by high-ability groups than low-ability groups. In this context, compared to classical test theory, modern Rasch model is capable of providing more complex results and detecting a greater number of ineffectively functioning distractors. However, in its implementation, the modern Rasch model is more challenging than classical test theory.

Testing Replicability and Generalizability of the Time on Task Effect.

The time on task (ToT) effect describes the relationship of the time spent on a cognitive task and the probability of successful task completion. The effect has been shown to vary in size and direction across tests and even within tests, depending on the test taker and item characteristics. Specifically, investing more time has a positive effect on response accuracy for difficult items and low ability test-takers, but a negative effect for easy items and high ability test-takers. The present study sought to test the replicability of this result pattern of the ToT effect across samples independently drawn from the same populations of persons and items. Furthermore, its generalizability was tested in terms of differential correlations across ability tests. To this end, ToT effects were estimated for three different reasoning tests and one test measuring natural sciences knowledge in 10 comparable subsamples with a total N = 2640. Results for the subsamples were highly similar, demonstrating that ToT effects are estimated with sufficient reliability. Generally, faster answers tended to be more accurate, suggesting a relatively effortless processing style. However, with increasing item difficulty and decreasing person ability, the effect flipped to the opposite direction, i.e., higher accuracy with longer processing times. The within-task moderation of the ToT effect can be reconciled with an account on effortful processing or cognitive load. By contrast, the generalizability of the ToT effect across different tests was only moderate. Cross-test relations were stronger in relative terms if performance in the respective tasks was more strongly related. This suggests that individual differences in the ToT effect depend on test characteristics such as their reliabilities but also similarities and differences of their processing requirements.

The Mathematical Culture in Test Items of National College Entrance Examination in China from 1978 to 2021

As one of the most important examinations in China, the National College Entrance Examination (commonly known as the Gaokao, hereafter denoted Gaokao) has a long history and has attracted attention from the Chinese educational community, among others. This study focused on mathematics test items of the Gaokao from the perspective of mathematical culture and examined the national papers (drafted by the Ministry of Education of the People’s Republic of China) of tests from 1978 to 2021 in order to investigate the content and time variation characteristics of mathematics test items reflecting mathematical culture since its restoration. A mathematical culture categorization conceptual framework was established based on previous studies and was applied to test item analysis. Mathematical culture in test items was classified using four categories: Historical Topics, Interdisciplinary Connections, Social Roles, and Aesthetics & Recreation. These were used for data coding and analysis. The results showed that mathematical culture in the mathematics test items of the Gaokao over a 44-year period, in terms of content categories, was diverse but uneven in distribution, with greater focus on demonstrating the social roles of mathematics and less on the historical development of mathematical knowledge. Moreover, the average number of items with mathematical culture during this period was small and fluctuated over time. The content of topics related to social context and technology also changed over time with the distinct characteristics of the times and changes in society. This study provides empirical evidence on how mathematical culture is integrated into test items and how they have changed over time.

Characteristics of National Standardized School Examination Test Items on Biology Subject in High School

A good test instrument is required to measure students’ academic abilities. For final-year students, this ability is measured by USBN tests. This study aimed to analyses the characteristics of biology test items in the National Standardized School Examination (USBN) and analyses item tests categorized as difficult and poor. Type of study was descriptive exploratory with quantitative and qualitative approaches. Quantitative data analysis was performed using the classical test and item response theories with 1 logistic parameter model. Qualitative data analysis described why such question items are categorized as difficult and poor according to the classical test theory. The data were collected by using documentation of the answers of 193 students. The study results show that, according the classical test theory, Biology USBN item tests had the average difficulty index of 0.525 (moderate category) and the average of discrimination index obtained is equal to 0.124 (good category). Based on the item response theory, the mean difficulty index was -0.00029 (moderate category). The reliability test estimation result was 0.51, categorized as moderate. The qualitative analysis result showed that items categorized as difficult are the food web in the ecosystem, nervous system, and human excretion system.

Exploring the final examination test item characteristics of Pancasila and civic education

The study aims at exploring the final examination test items to identify the quality of the test items as an evaluation instrument. The study is descriptive quantitative research that pursues the document analysis to view the Final Semester Examination test item characteristics of Pancasila and Civic Education in Grade VIII of State Madrasah Tsanawiyah (Madrasah Tsanawiyah Negeri 3 or MTs Negeri 3) in the Regency of Gunungkidul, the Province of Yogyakarta Special Region, within the 2020/2021 Academic Year. In analyzing the data that have been collected, the researchers have adopted the approach of Item Response Theory (*IRT) using the R Program assistance. The results of the study show that the Final Semester Examination test items of Pancasila and Civic Education are more appropriate to be analyzed by using the SPL Model. The degree of difficulty for these items falls into the “Good” quality within the range -4.0 until +4.0. On the contrary, the item discrimination capacity ranges between 0.079 and 4.891 with the “Moderate” quality.

The characteristics of chemistry test items on nationally-standardized school examination in Yogyakarta City

AbstractThis study aimed to describe the characteristics of chemistry test items in Nationally Standardized School Examination or Ujian Sekolah Berstandar Nasional (USBN), consisted of discrimination index, difficulty index, and question reliability. The data was collected by using documentation of the answers of 194 students. The type of research was descriptive exploratory with quantitative and qualitative approaches. Quantitative data analysis was performed by using classical test theory approach and item response theory with 1 logistical parameter. Meanwhile, qualitative data analysis was conducted in order to describe the items that were categorized as difficult and bad categories. The results showed that according to classical test theory, USBN Chemistry item test had an average level of difficulty of 0.57 which was categorized as moderate. Regarding to the discrimination index, the average of different test power obtained were equal to 0.146, and belonged to the good category. Based on the item response theory, the average of difficulty index of -0,00086 was categorized as moderate. The results of the estimated reliability of the questions amounted to 0.48 included in the moderate category. The results of the qualitative analysis showed that the items which belonged to difficult category were the items on salt hydrolysis material, acid-base titration, the concept of periodicity, manufacture and use of chemical compounds according to classical test theory. Meanwhile, according to the theory of response, items which belonged to difficult category were found in acid-base titration material, the solution colligative phenomenon, and the concept of periodicity of the main metal elements.

Item response analysis of computational thinking practices: Test characteristics and students’ learning abilities in visual programming contexts

Computational thinking (CT) practices have gained increasing importance in primary school education. However, there are few tools to measure primary school students' CT practices that are based on rigorous psychometric approaches. This study developed an operational definition for CT practices by exploring the key dimensions of their composition and, more importantly, developed and validated a generic test tool that can effectively assess students’ CT practices. Items to measure CT practices were developed using the evidence-centered design (ECD) approach. These items prompted students to exhibit behaviors that demonstrated their skills in the four dimensions of CT practice, namely testing and debugging, algorithmic thinking, reusing and remixing, and abstracting and modularizing. A test consisting of these items was administered to 13,956 Grade 4–6 primary school students from 56 public schools in Hong Kong. The data were analyzed based on the item response theory (IRT) approach, together with other psychometric tests. A bi-factor three-parameter logistic model was ultimately adopted for test validation, as it fitted the data better than the corresponding two-parameter and one-parameter logistic models. The results validated the four-dimensional structure of the CT practices test. The reliability, test item characteristics, and CT practice abilities of students are also discussed.

The Application of Item Response Theory in Analysis of Characteristics of Mathematical Literacy Test Items

This study aims to analyze the mathematical literacy abilities of students using IRT with the GPCM approach. The results of the model compatibility test show that the GPCM is suitable for analyzing mathematical literacy test instruments with 4 levels of mathematical literacy. The results of the item parameter estimation show that there is 1 level of mathematical literacy which has a high category for item discrimination, while the other 3 levels have a medium category. Based on item difficulty levels, 1 level of mathematical literacy has a difficult category, while 3 levels have easy categories. Most students are at the level of ability 0 < θ < 1, and there are 52% of students above the average mathematical literacy. Based on the test information function and the standard error shows that the mathematical literacy instrument is suitable for students with the ability -2 < θ < -0.5.

Characteristics of Math national-standardized school exam test items in junior high school: What must be considered?

National Standardized School Exam (USBN) is used to determine Student’s graduation. This research aims to determine the characteristics of items for Math USBN in SMP on grade 9. This kind of research is a descriptive-explorative quantitative research. The data collected is a USBN test instrument at SMP Negeri 3 Pati and participants' answers, which are collected by documentation. The USBN instrument was validated by experts and the characteristic items of USBN instrument were analyzed using the classical test theory approach. The question items of math USBN test at SMP Negeri 3 Pati is generally moderately good. Based on the classical theory approach, the result of the validity is 0.924. 56.7% of items are very valid. The reliability is 0.78 categorized as high reliability. Generally, Math USBN items are in the easy category with a percentage of 83.3%. The results of discrimination index indicate that in general, USBN items are in a moderate category with a percentage of 60%. The distraction effectiveness shows that USBN items are in the functional category with a percentage of 50%.

Characteristics of non-compulsory mathematics test items on nationally standardized school examination in Kaimana Regency, West Papua Indonesia

Nationally Standardized School Examination result is used as a students’ graduation requirement. Therefore, it is absolutely necessary that its test items must be well constructed to assess. The aim of this study was to describe the characteristics of non-compulsory mathematics test items on Nationally Standardized School Examination of senior high school in Kaimana Regency, West Papua based on the items analysis. This study was a descriptive exploratory type of quantitative research. Data obtained from a set of Non-compulsory Mathematics test items including documentation of students’ answers of the following test items on Nationally Standardized School Examination of senior high school in Kaimana Regency, West Papua. A set of this test item was validated by 5 experts and analyzed by applying Classical Test Theory. Results showed that the content validity was valid for 0.9 as the average of index Aiken. The reliability of test items was in less reliable level. The items difficulty indicated that 29 items (85.3%) were hard and 5 items (14.7%) were moderate. The value of items discrimination index for 12 items (53.3%) were classified as good and 22 items (64.7%) were poor. For multiple-choice items, the distractor was made by the developer, worked effectively.

Item parameters of Yureka Education Center (YEC) English Proficiency Online Test (EPOT) instrument

Yureka Education Center (YEC) is one of the institutions which has developed an online-based English proficiency test. The test is called English Proficiency Online Test (EPOT) which follows the TOEFL ITP (Institutional Testing Program) framework. So, this study aimed to analyze the characteristics of EPOT instruments consisting of Listening, Structure, and Reading subtests, which later the quality of each EPOT test item is identified. This study used a descriptive quantitative approach by describing the characteristics of EPOT test items in terms of item difficulty index, item discrimination index, test information’s function and test measurement’s errors. The data were collected through EPOT trials conducted by 2652 online test takers as partisipant from 20 provinces in Indonesia. The collected data were then analyzed using the Item Response Theory (IRT) approach using BILOG program on all logistic parameter models which began with the item compatibility test against the model. Based on the results of the analysis, it can be concluded that all subtests matched the 3-PL model. Most of EPOT’s test items had a good range of difficulty index and discrimination index. The EPOT information’s function showed that accurate items were used on the 3-PL model for certain capability range. This study was expected to point out that EPOT test could be used as an alternative English proficiency test that is easy to use and useful.

The Characteristics of High-Risk Tryout Test Items for Indonesian Elementary Schools Students

One of the measurement forms applied in the educational field to know students' ability is a high-risk tryout test. This research examined the quality of Mathematics tryout test questions used by Indonesian teachers to prepare for the national examination as a high-risk examination. This study used a quantitative approach and analyzed based on item response theory to determine the fitness of the model and item eligibility. The data used were the answers of elementary school students who took the 2016 national examination tryout of mathematics subject with the total number of 924 students from 20 elementary schools in Indonesia. The mathematics subject consisting of 40 questions, which are adjusted to the blueprint of elementary school national examination, was chosen. The results obtained indicate that the mathematics tryout test items meet the assumptions of uni-dimension, local independence, and parameter invariance items. The logistic parameter model used was the 2-PL model, with 31 out of 40 eligible items of questions. The novelty of this research is the use of item response theory to analyze the quality of an instrument. The aim of the use of item response theory is to avoid examinee sample dependent and item sample dependent. The educational stakeholders can use this research as a reflection in preparing high-risk examination.

Self-care behaviors related to air pollution protection questionnaire: a psychometric analysis

BackgroundAir pollution is an environmental health issue that has received much attention due to its harmful effects on public health. The aim of this study was to determine the psychometric properties of the self-care behaviors related to air pollution protection. To this mean, questionnaire responses provided by Iranian college students were analyzed and a Health Belief Model (HBM) determined.MethodsThe self-care behavior related to air pollution protection was determined from responses from 301 university students using test item characteristics, face validity, reliability (internal consistency) and construct validity. Moreover, we carried out an exploratory factor analysis (EFA) to determine the factorial structure and explained variance.ResultsBased on Eigenvalues of ≥1.00 and factor loadings of ≥0.40, seven factors were extracted. The calculated Kaiser–Meyer–Olkin value was 0.804. Overall, the seven factors explain 66.39% of the variance in the hypothesized model. Cronbach’s alpha for the measured factors: the perceived susceptibility, severity, barriers, benefits, self-efficacy, cues to action and behavior were 0.85, 0.79, 0.86, 0.66, 0.74, 0.83, and 0.75, respectively.ConclusionThe analysis of the questionnaire’s reliability and validity properties resulted in good values. The questionnaire is a promising instrument to assess self-care behavior related to air pollution protection from the perspective of college students.

Dissemination of symbolic representation ability in high school physics subjects

The purpose of this study is to look at the validity and reliability of the symbolic representation ability test instrument, as well as the spread of the symbolic representation abilities of high school students. The research method used is the development of test instruments. Nine items were developed with 5 alternative answers. The physics material being tested is Newton’s Law of Motion. The results of the validation analysis showed that the instrument developed was valid, with Aiken’s V value greater than 0.88. Also, the instrument developed was compatible with the Rasch model. The characteristics of the difficulty index test items are in the good category. The reliability of the instrument is 0.70, and is feasible to be used to measure the symbolic representation ability of students with a minimum ability of -1.25. The trial was conducted on 62 students of SMA Negeri 1 Banguntapan. The average measurement result of the symbolic representation ability of students in high school physics subjects is -0,001 ± 0,144. It can be concluded that the symbolic representation ability of students of SMA Negeri 1 Banguntapan in physics is in the medium category.

A diagnosis of students' errors in answering the mathematics test in senior high school

This study aims to reveal: (1) the characteristics of diagnostic test items used in mathematics subject of the first grade of senior high school (SHS); (2) the degree of errors from several types of students’ errors in answering mathematics test; and (3) the dominant factor causing the students to make mistakes in answering mathematics test. This study used a quantitative approach involving the first graders of eight SHSs in Mataram as the population. The sample was collected by a proportionate random sampling technique, consisting of 350 students for preliminary field testing and 450 students for main field testing. The research instrument was a mathematics diagnostic test, questionnaire, and interview guidelines. The results show that: (1) the diagnostic test instrument meets the qualitative and quantitative content validity, proves empirically fit with Partial Credit Model (PCM), has reliability index of 0.92 (high category), and all items in the diagnostic test instrument are categorized in moderate difficulty; (2) misrepresentation becomes dominant errors if compared with misconceptions, the counting errors, and procedural errors. The errors dominant in the topic domain are Inequalities Linear System Two Variables if compared with Rational and Irrational Inequality One Variable, Equations and Inequalities of Absolute Value, and Linear Equations System Three Variables; and (3) most of the students do not understand how to solve the problem of inequality, determining factor quadratic equations, determining the members of the set completion, problems concerning graphs, and problems that require the capability of language interpretation into the mathematics model.

The characteristics of final mathematics test items based on classical test theory

Final Examination is the part of an evaluation that exists in school to assess student’s knowledge. This research aimed to show the characteristic of the final mathematics test in Junior High school, grade 8. This research was an explorative descriptive using a quantitative approach. Data collection was gathered using the documentation of mathematics’ test in Binjai, North Sumatera including student’s answers. The characteristics of test items were analyzed by expert judgment using the classical test theory approach. The average of content analysis was 0.81, there were three items without key answers so it can not be analyzed. The reliability average was 0.84 which has very good quality and difficulty index indicated the difficulty level of questions was on three categories (easy, moderate, and difficult). The difficulty is in the cartesian position, the relation of two sets, determining gradient, determining the solution of linear equation systems of two variables.

Using Alignment Index and Polytomous Item Response Theory on Statistics Essay Test

Purpose: Essay test in mathematics, both in the form of restricted-response and extended-response, generally consist of polytomous scored items. However, the essay test used by teachers in Indonesia has not been fully supported by sufficient quality evidence. There have been many studies focusing on the development of the essay test, but not many of them have applied the use of relevant measurement theory for the polytomous data. The evidence of content validity also has not been supported by its alignment with the curriculum. This study used alignment ind ex to prove the content validity and IRT polytomous GPCM to determin e the characteristics of test items in order to produce a n essay test that could accurately measure the achievement of students on statistical materials. Method : Procedures of this study: (1) preparation of preliminary test, (2) trials, (3) interpretation. Trial was conducted involving 688 Junior High School students in Yogyakarta, Indonesia. Result s: The content validity of the test wa s good, supported by V Aiken index of 0.88–1.00 and Porter alignment index of 0.93. The test items ha d good construct validity . Test reliability wa s categorized as good with the Construct Reliability coefficient of 0.88 and the Alpha coefficient of 0.78. Judging from its characteristics, all test items were categorized as good . Implications for Research and Practice : The use of the alignment index contribution to the verifica tion of content validity of essay test and the use of the IRT polytomous GPCM may provide reference for the use of appropriate measurement theory to determine the item characteristics of essay test.

Developing higher-order thinking skill (HOTS) test instrument using Lombok local cultures as contexts for junior secondary school mathematics

The study was aimed at producing a valid and reliable higher-order thinking skill (HOTS) test instrument using Lombok local cultures as contexts in the junior secondary school mathematics subject matter. The study is developmental research involving a field try-out of 75 students of Grade VIII. Data were analyzed using classical test theories of difficulty levels, discriminating powers, and functioning distractors. The test validity is assessed using the Aiken formula and reliability is estimated by Cronbach Alpha. Findings show that, of the 20 initial multiple-choice items, 15 were valid and reliable and had the characteristics of good test items with a medium-rated difficulty level average of 0.28, a good-rated discriminating power of 0.31), a good-rated reliability coefficient of 0.79, and all distractors well-functioning.

Determinants of Difficulty and Discriminating Power of Image-based Test Items in Postgraduate Radiological Examinations

The psychometric characteristics of image-based test items in radiological written examinations are not well known. In this study, we explored difficulty and discriminating power of these test items in postgraduate radiological digital examinations. We reviewed test items of seven Dutch Radiology Progress Tests (DRPTs) that were taken from October 2013 to April 2017. The DRPT is a semiannual formative examination, required for all Dutch radiology residents. We assessed several stimulus and response characteristics of test items. The response format of test items included true or false, single right multiple choice with 2, 3, 4, or ≥5 answer options, pick-N multiple-choice, drag-and-drop, and long-list-menu formats. We calculated item P values and item-rest-correlation (Rir) values to assess difficulty and discriminating power. We performed linear regression analysis in image-based test items to investigate whether P and Rir values were significantly related to stimulus and response characteristics. Also, we compared psychometric indices between image-based test items and text-alone items. P and Rir values of image-based items (n = 369) were significantly related to the type of response format (P < .001), and not to which of the seven DRPTs the item was obtained from, radiological subspecialty domain, nonvolumetric or volumetric character of images, or context-rich or context-free character of the stimulus. When accounted for type of response format, difficulty and discriminating power of image-based items did not differ significantly from text-alone items (n = 881). Test items with a relatively large number of answer options were generally more difficult, and discriminated better among high- and low-performing candidates. In postgraduate radiological written examinations, difficulty and discriminating power of image-based test items are related to the type of response format and are comparable to those of text-alone items. We recommend a response format with a relatively large number of answer options to optimize psychometric indices of radiological image-based test items.