Reading Comprehension Items Research Articles

Investigating the capability of ChatGPT for generating multiple-choice reading comprehension items

The development of multiple-choice reading comprehension items based on specific reading subskills is crucial in teaching, learning, and testing reading. But it remains a challenging task because this process is time-consuming and costly. This study aims to investigate the capability of ChatGPT (Chat Generative Pre-trained Transformer) for generating multiple-choice reading comprehension items. Psychometric models and human review were adopted to evaluate the item quality based on the benchmark of human-authored items. The results showed that ChatGPT-authored multiple-choice items were acceptable and comparable to human-authored items in terms of psychometric properties, and human review by questionnaire, expert judgment, and interview found that ChatGPT had the potential capability to serve as a test developer and assistant for teaching and learning reading. However, some shortcomings and potential pitfalls were also identified and room for improvement was discussed when ChatGPT is applied to generate items for educational purposes.

Using OpenAI GPT to Generate Reading Comprehension Items

AbstractThe purpose of this study is to introduce and evaluate a method for generating reading comprehension items using template‐based automatic item generation. To begin, we describe a new model for generating reading comprehension items called the text analysis cognitive model assessing inferential skills across different reading passages. Next, the text analysis cognitive model is used to generate reading comprehension items where examinees are required to read a passage and identify the irrelevant sentence. The sentences for the generated passages were created using OpenAI GPT‐3.5. Finally, the quality of the generated items was evaluated. The generated items were reviewed by three subject‐matter experts. The generated items were also administered to a sample of 1,607 Grade‐8 students. The correct options for the generated items produced a similar level of difficulty and yielded strong discrimination power while the incorrect options served as effective distractors. Implications of augmented intelligence for item development are discussed.

Iraqi Graduate Students’ Analysis of the Graduate Record Examination (GRE) Verbal Reasoning Content: A Think-Aloud Study

This study describes difficulties Iraqi graduate students encounter while answering Verbal Reasoning Measure (VRM) questions of the Graduate Record Examination (GRE). Participants thought aloud while answering questions from three VRM subsections: Reading Comprehension (RC), Text Completion (TC), and Sentence Equivalence (SE). Data were collected via audio recordings, from seven Iraqi graduate students, and transcribed orthographically, coded, and analyzed. The analysis yielded several themes: Difficulties related to the structure of the question and/or the meaning of its answers, the content is pertinent to USA culture and history, SE is the most difficult part followed by TC, and RC is time consuming. These findings are consistent with previous research results that the structure of the GRE VRM questions and/or the answers are difficult to understand, SE items were much harder than TC and RC items due to their unfamiliar content and vague meanings, and RC is time consuming for international graduate students. Pedagogical implications were identified and recommendations for ETS and students were suggested.

Lessons Learned about Evaluating Fairness from a Data Challenge to Automatically Score NAEP Reading Items

Natural language processing (NLP) is widely used to predict human scores for open-ended student assessment responses in various content areas (Johnson et al., 2022). Ensuring algorithmic fairness based on student demographic background factors is crucial (Madnani et al., 2017). This study presents a fairness analysis of six top-performing entries from a data challenge involving 20 NAEP reading comprehension items that were initially analyzed for fairness based on race/ethnicity and gender. This study describes additional fairness evaluation including English Language Learner Status (ELLs), Individual Education Plans, and Free/Reduced-Price Lunch. Several items showed lower accuracy for predicted scores, particularly for ELLs. This study recommends considering additional demographic factors in fairness scoring evaluations and that fairness analysis should consider multiple factors and contexts.

Construction of a reading literacy test item bank for fourth graders based on item response theory.

Reading literacy is not only central to students' academic success during their school years but also crucial to their personal development in their later life. The development of assessment instruments for reading literacy has been of interest to researchers, educators and educational administrators. The purpose of the present study was to construct and validate a comparable item bank for assessing fourth-grade students' reading literacy. One hundred fifteen reading comprehension items were developed and administered to 2,174 Grade 4 students to construct an item bank. Using the test equating technique and balanced incomplete block design, we divided participants into 10 subgroups, and the 115 items were further assigned into 10 test forms. Item response theory software was used to estimate discrimination, items' threshold parameters, and students' ability parameters. The criterion-related validity was also examined in 135 Grade 4 students who completed the reading literacy test and verbal self-description questionnaire. The final item bank included 99 reading performance indicators to express high achievement. The correlation between the students' reading literacy and the verbal self-description questionnaire was significant and demonstrated the item bank's good criterion-related validity. The item bank developed in this study shows good psychometric characteristics and can be used to assess the reading literacy of fourth graders.

Diagnosing Primary Students’ Reading Progression: Is Cognitive Diagnostic Computerized Adaptive Testing the Way Forward?

Cognitive diagnostic computerized adaptive testing (CD-CAT) is a cutting-edge technology in educational measurement that targets at providing feedback on examinees’ strengths and weaknesses while increasing test accuracy and efficiency. To date, most CD-CAT studies have made methodological progress under simulated conditions, but little has applied CD-CAT to real educational assessment. The present study developed a Chinese reading comprehension item bank tapping into six validated reading attributes, with 195 items calibrated using data of 28,485 second to sixth graders and the item-level cognitive diagnostic models (CDMs). The measurement precision and efficiency of the reading CD-CAT system were compared and optimized in terms of crucial CD-CAT settings, including the CDMs for calibration, item selection methods, and termination rules. The study identified seven dominant reading attribute mastery profiles that stably exist across grades. These major clusters of readers and their variety with grade indicated some sort of reading developmental mechanisms that advance and deepen step by step at the primary school level. Results also suggested that compared to traditional linear tests, CD-CAT significantly improved the classification accuracy without imposing much testing burden. These findings may elucidate the multifaceted nature and possible learning paths of reading and raise the question of whether CD-CAT is applicable to other educational domains where there is a need to provide formative and fine-grained feedback but where there is a limited amount of test time.

Generating reading comprehension items using automated processes

Over the last five years, tremendous strides have been made in advancing the AIG methodology required to produce items in diverse content areas. However, the one content area where enormous problems remain unsolved is language arts, generally, and reading comprehension, more specifically. While reading comprehension test items can be created using many different item formats, fill-in-the-blank remains one of the most common when the goal is to measure inferential knowledge. Currently, the item development process used to create fill-in-the-blank reading comprehension items is time-consuming and expensive. Hence, the purpose of the study is to introduce a new systematic method for generating fill-in-the-blank reading comprehension items using an item modeling approach. We describe the use of different unsupervised learning methods that can be paired with natural language processing techniques to identify the salient item models within existing texts. To demonstrate the capacity of our method, 1,013 test items were generated from 100 input texts taken from fill-in-the-blank reading comprehension items used on a high-stakes college entrance exam in South Korea. Our validation results indicated that the generated items produced higher semantic similarities between the item options while depicting little to no syntactic differences with the traditionally written test items.

Response Demands of Reading Comprehension Test Items: A Review of Item Difficulty Modeling Studies

ABSTRACT Item difficulty modeling studies involve (a) hypothesizing item features, or item response demands, that are likely to predict item difficulty with some degree of accuracy; and (b) entering the features as independent variables into a regression equation or other statistical model to predict difficulty. In this review, we report findings from 13 empirical item difficulty modeling studies of reading comprehension tests. We define reading comprehension item response demands as reading passage variables (e.g., length, complexity), passage-by-item variables (e.g., degree of correspondence between item and text, type of information requested), and item stem and response option variables. We report on response demand variables that are related to item difficulty and illustrate how they can be used to manage item difficulty in construct-relevant ways so that empirical item difficulties are within a targeted range (e.g., located within the Proficient or other proficiency level range on a test’s IRT scale, where intended).

Validating a Reading Assessment Within the Cognitive Diagnostic Assessment Framework: Q-Matrix Construction and Model Comparisons for Different Primary Grades.

Cognitive diagnostic assessment (CDA) has been developed rapidly to provide fine-grained diagnostic feedback on students’ subskills and to provide insights on remedial instructions in specific domains. To date, most cognitive diagnostic studies on reading tests have focused on retrofitting a single booklet from a large-scale assessment (e.g., PISA and PIRLS). Critical issues in CDA involve the scarcity of research to develop diagnostic tests and the lack of reliability and validity evidence. This study explored the development and validation of the Diagnostic Chinese Reading Comprehension Assessment (DCRCA) for primary students under the CDA framework. Reading attributes were synthesized based on a literature review, the national curriculum criteria, the results of expert panel judgments, and student think-aloud protocols. Then, the tentative attributes were used to construct three booklets of reading comprehension items for 2–6 graders at three key stages. The assessment was administered to a large population of students (N = 21,466) in grades 2–6 from 20 schools in a district of Changchun City, China. Q-matrices were compared and refined using the model-data fit and an empirical validation procedure, and five representative cognitive diagnostic models (CDMs) were compared for optimal performance. The fit indices suggested that a six-attribute structure and the G-DINA model were best fitted for the reading comprehension assessment. In addition, diagnostic reliability, construct, internal and external validity results were provided, supporting CDM classifications as reliable, accurate, and useful. Such diagnostic information could be utilized by students, teachers, and administrators of reading programs and instructions.

The Role of Glossing and Working Memory Capacity in Second Language Reading Comprehension

Glossing, providing information for unfamiliar lexical items to promote reading comprehension, has long been investigated as a textual modification technique to promote second language (L2) reading comprehension. Thus far, however, inconclusive findings have been produced as to whether L2 readers’ working memory capacity plays as a moderator of the effects of glossing on L2 reading comprehension. To fill the gap, the present study explored the moderating impact of working memory capacity on the efficacy of glossing in L2 reading comprehension. Eighty-eight Korean university students read two English passages in either a glossed or an unglossed version, while answering multiple-choice reading comprehension items. Participants’ phonological short-term memory was assessed with a digit span task and a Korean nonword repetition task, while their complex working memory was measured with a backward digit span task and an automated operation span task. The results of mixed-effects modeling revealed that forward digit span scores moderated the effects of glossing on L2 reading comprehension scores.

The impact of glossing and reading activity manipulation on learning of L2 lexico-grammatical and lexical items

The present study investigates the impact of glossing and reading activity manipulation on learning of target lexico-grammatical and lexical items. Eighty-eight Korean undergraduate students read either glossed or unglossed English texts. Each paragraph of the texts was split into two (less careful condition) or three to four subparts (more careful condition), and participants were asked to rearrange them into a coherent order and answer multiple-choice reading comprehension items. Glossing was conducted by providing Korean translations of target items, i.e. 15 English unaccusative verbs and 10 pseudowords. Learning was assessed with an untimed grammaticality judgment test and word form and meaning recognition tests. The results revealed that glossing affected word meaning recognition scores positively, but form recognition scores negatively. In addition, more careful reading facilitated learning of the target unaccusative verbs while having a negative influence on word meaning recognition scores. Neither glossing nor reading activity manipulation affected reading comprehension scores.

A comparison of higher-order reading comprehension performance for different language of instruction models in South African primary schools

In this article, preProgress in Reading Literacy Study (prePIRLS) 2011 data is used to compare the performance of different language of instruction groupings (English, Afrikaans and African languages) in primary schools on the more complex, higher-order reading comprehension items tested in a large-scale international test. PrePIRLS 2011 (N=15 744) was conducted in South Africa’s eleven official languages. Schools were sampled according to the language of learning and teaching (LoLT) in Grades 1–3 and the reading comprehension test was administered in that same language. To examine bilingual effects, a sub-sample was drawn from the national dataset that consisted of low socio-economic status (SES) learners whose first language was not English (but who had received instruction in English from grades 1–3) as well as low SES learners who received their Foundation Phase instruction in one of the African languages as a mother tongue. A linear regression (n = 6 342) showed that low socio-economic status (SES) learners whose language of instruction is English, despite it not being their mother tongue, benefitted by 20.35 score points with a t-value of 3.19. This is significant at the 0,01 level (equivalent to half a year) from being in the English L2 group, in comparison to the African languages L1 group, as a measure of achievement on the higher-order subscale. It is argued in this paper that learners whose LoLT is English, but who do not speak English as a home language and tend to be part of the most disadvantaged sector of the population, perform better on the higher-level reading comprehension processes when compared with African language mother tongue instruction across the same grades and socio-economic status. The findings highlight the importance of improved English second language instruction for all LoLT groupings.

The analysis of Higher-Order Thinking skills in English reading comprehension tests in Malaysia

Abstract The current English language education in Malaysia has integrated the Higher-Order Thinking Skills (HOTS) as part of its testing and evaluation system, which aims to enhance students’ cognitive skills and abilities in learning. Having a higher-order thinking ability at the earliest stage of formal education (primary and elementary levels) is essential in order to meet the growing demand for highly intelligent and professional workers in the future. The present study aims to identify the use of HOTS items in selected English reading comprehension examinations for Standard 6 students in Malaysia. 80 reading comprehension items were selected from various final examination papers distributed in eight different states in the country. Data were quantitatively analysed using content analysis based on the levels of cognitive listed in Bloom’s Taxonomy. Apart from that, the data were also analysed based on a series of professional evaluations made by four content experts in the country. The findings indicate that most reading comprehension questions in the English examination papers need further revisions in order to achieve the standard of HOTS that have become part of the new curriculum and national education policy. Based on the findings, some practical suggestions have also been developed to further enhance the use of HOTS in English teaching and learning practices in Malaysia. Keywords: Bloom's Taxonomy, content analysis, English language assessment, Higher-Order Thinking skills, national education policy, reading comprehension test

The Impact of Using Chunking Technique on Developing Reading Skill and Perception of Intermediate School Students

Teaching reading skill to the students is very difficult thing. The difficulties might come from the teachers who are not creative to simulate in teaching reading, lack of using affective strategies to attract students’ attention to the lesson and keep using traditional ways of teaching. Other difficulties came up from the students who did not interest to read a long passage, lack of vocabulary, and understanding the meaning of what they have read. Students who learn English as foreign language face difficulties in reading long stories, texts and paragraphs. The presented study aims at investigating effects of using chunking technique on developing English language reading skill of Iraqi EFL students. The study uses quantitative research method. The samples of the study were randomly selected from the second intermediate school students. The study adopted a pretest treatment- post-test design. Through the results of the study, the first hypothesis was accepted and was concluded that chunking technique was better than the recommended teaching in both the overall achievement and reading comprehension items

The Impact of Reading Strategy Knowledge and Science Knowledge on Developing Reading Skills of School Students

The samples of the study were randomly selected from the second intermediate school students. The study adopted a pretest treatment-post-test design. The first hypothesis was accepted and was concluded that picture strip story activity was better than the recommended teaching in both the overall achievement and reading comprehension items. Both English teachers and students need to improve and to master the four important language skills, i.e. listening, speaking, reading, and writing. Reading is one of the four language skills required when the students learn the target. The presented study aims at investigating the effects of picture strip story on reading comprehension of Iraqi EFL students. The study uses quantitative research method. By reading texts which is categorized as a receptive skill, the students are able to get a lot of information.

Interplay of Working Memory, Strategy Use, and Task Difficulty in L2 Reading Comprehension

In pursuit of the precise mechanism by which working memory (WM) functions in second language (L2) reading comprehension (RC), the current study examined whether strategy use or task difficulty influences the role of WM in L2 RC. Eighty Korean adult EFL learners participated, and their WM capacity, RC performance, and strategy use were measured. The variable of task difficulty was operationalized by the relative difficulty of texts on which RC items were based. The results showed that difference in strategy use brought about significant difference in RC performance when the readers with insufficient WM capacity had to comprehend difficult texts, whereas no such difference was found with easier texts or among L2 readers with more WM capacity. These findings not only illustrate intricate interrelationships among WM and other variables in L2 RC, but also provide a noteworthy implication for L2 reading instruction that L2 learners with deficient WM capacity could benefit from active strategy use when faced with a difficult L2 reading task. The compensatory role of strategy and the necessity of remedial strategy training for the learners with limited WM is further implicated for L2 reading classrooms.

Identifying the Relationships Between Response Behaviors for Reading Comprehension Items and Student Characteristics

The study aimed to identify the students’ characteristics (perception of self-efficacy, resistance, persistence, taking responsibility and competence for their cognitive processes) that might affect the response behaviors of fifth grade students for open ended questions written in different cognitive levels. This study was designed as relational survey model research. Sample consisted of 392 5th grade students selected by purposeful sampling from Yenimahalle, Mamak and Cankaya districts in Ankara city. Data was collected with reading comprehension achievement test, response behaviors form and student information survey. Analysis was conducted with 366 students’ data after missing values were removed. MPlus package program was utilized to analyze data and structural equation modelling was used. The findings of the research revealed that response behaviors for open ended questions written in different cognitive levels in Turkish learning area were found to be predicted low and at a significant level by students’ perception of self-efficacy, persistence, resistance, taking responsibility and cognitive skills. It might be concluded that improving children’s perception for cognitive and metacognitive skills can enhance their thinking process, lead them to perform complex response behaviors and affect positively their higher order thinking process. In other words, students with high cognitive and metacognitive skills show higher level response behaviors such as putting themselves into author’s shoes, relating their own life to the event in the text or using imagination. Therefore, they can comprehend what they read easily and express it back.

Relationship Between Vocabulary Size, Syntactic Awareness and Reading Comprehension in Low- and High-Level Korean High School EFL Students

This present study sought to investigate the relative contribution of vocabulary size and grammar awareness for L2 reading comprehension (RC) for Korean high school EFL learners. 74 eleventh grade students attending a Korean high school were asked to take a vocabulary test, a grammar awareness test, and a reading comprehension test. Vocabulary knowledge was measured using the Korean version of the Vocabulary Levels Test and syntactic awareness was estimated with a subset of the Grammaticality Judgement Test (20 items). RC items were adopted from the Korean College Scholastic Ability Test (GSAT) Mock Test. The results showed a significant correlation between vocabulary size and RC. However, grammar awareness and RC showed no significant correlation. This study also found that for high-level students, there was a significant correlation between vocabulary size and RC especially for words taken from the 3,000-word family level. In contrast, for low-level students, there was a significant correlation between vocabulary size and RC but for words taken from the 5,000-word family level. Furthermore, there was not a significant correlation between syntactic awareness and RC for either low or high-level students. Therefore, vocabulary knowledge predicts both low and high-level students RC, whereas syntactic awareness does not predict the students'' RC.

중국 대학생 영어독해 평가모형 구인타당도 연구

중국 대학생 영어독해 평가모형 구인타당도 연구

The Impact of Incidental and Intentional L2 Vocabulary Learning on Depths and Breadth Dimensions of Vocabulary Knowledge

Through the introduction of different dimensions of vocabulary knowledge, various studies attempted to examine numerous effective factors on these dimensions. The present study aimed to show the effects of different vocabulary learning styles through extensive and intensive reading programs on depth and breadth aspects of vocabulary knowledge. To achieve this goal, 45 sophomore undergraduate students of English language teaching and translation in Payam-e-Noor University participated in this study. Initially, in order to homogenize the learners based on their level of language proficiency, MEPT was administered. Then, by measuring the mean and standard deviation of participants’ scores, the number of participants reduced to 35. The ultimate subjects’ scores on the reading comprehension items of MEPT show that they are all at the intermediate level of reading ability. Participants divided into three experimental groups randomly: two groups were in the extensive reading program with different form-focused and meaning-focused tasks as incidental vocabulary learning style. And the third group was in the intensive reading program as intentional vocabulary learning style. Participants in these experimental groups read long stories or passages per week with ten goal-oriented words. After 8 weeks, Word Associates Test (WAT) and Vocabulary Knowledge Scale (VKS) were administered to measure the acquired knowledge of new words and also, determining the effects of various learning styles on different dimensions of vocabulary knowledge. The results of Paired-samples and Independent T-tests revealed that both incidental and intentional groups developed in the period between the pre- and post-test, but, there was a significant difference between the effects of incidental vocabulary learning in the form of ER program and intentional vocabulary learning in the form of IR program. Moreover, there was a significant difference between the effects of the form-focused and meaning-focused task.