Reading Comprehension Assessment Research Articles

Scanpath Regularity as an Index of Reading Comprehension

ABSTRACT Purpose Recent research on the potential of using eye-tracking to measure reading comprehension ability suggests that the relationship between standard eye-tracking measures and reading comprehension is influenced by differences in task demands between comprehension assessments. We compared standard eye-tracking measures and scanpath regularity as predictors of reading comprehension scores. Method We used a dataset in which 79 participants (mean age: 22 years, 82% females, 76% monolingual English speakers) were administered three widely-used reading comprehension assessments with varying task demands while their eye movements were monitored: the York Assessment of Reading for Comprehension; (YARC), the Gray Oral Reading Test; (GORT-5), and the sentence comprehension subtest of the Wide Range Achievement Test; (WRAT-4). Results Results showed that scanpath regularity measures, similarly to standard eye-tracking measures, were influenced by differences in task demands between the three tests. Nevertheless, both types of eye-tracking measures made unique contributions as predictors of comprehension and the best set of predictors included both standard eye-tracking measures and at least one scanpath measure across tests. Conclusion The results provide evidence that scanpaths capture differences in eye-movement patterns missed by standard eye-tracking measures. Overall, the results highlight the effect of task demands on eye-movement behavior and suggest that reading goals and task demands need to be considered when interpreting eye-tracking data.

A deep-learning-based grading system (ASAG) for reading comprehension assessment by using aphorisms as open-answer-questions

Today reading comprehension is considered an essential skill in modern life, therefore, higher education students require more specific skills to understand, interpret and evaluate texts effectively. Short answer questions (SAQs) are one of the relevant and proper tools for assessing reading comprehension skills. Unlike multiple-choice questions, SAQs allow for the assessment of cognitive abilities such as attention, language, perception, and problem solving. However, the task of SAQs scoring is time-consuming and susceptible to ambiguity. Automatic Short Answer Grading (ASAG) is a new paradigm that could help solve these problems. This experimental analysis aims to implement ASAG using several approaches to sentence embedding based on deep learning with a multilayer perceptron regression layer on the top, trained with a reading comprehension dataset based on aphorisms. For experimental testing, the available dataset is composed of answers given by 199 undergraduate students in Spanish. BERT and Skip-Thought models are tested with different hyperparameters to find the best performance in terms of Pearson correlation coefficient and RMSE against human experts grades. The result of the current study showed that BERT model performed better than other approaches.

Reading comprehension processes

Reading comprehension is a complex, multifaceted process in which a number of components are appropriately and often simultaneously activated. The importance of different background subprocesses, its complex nature, as well as the interconnection of various components has been demonstrated by various ‘models of reading and reading comprehension’. Based on these models, assessment methods and materials for reading comprehension are developed. However, the models and assessment materials developed so far are mostly derived from research on languages with non-transparent orthography (e.g., English). Therefore, the question arises regarding the extent to which they can or should be applied in languages with shallow, transparent orthography (e.g., Croatian) that have clear and consistent relationships between letters and sounds, as opposed to languages with deep orthography. The main aim of this study was to present a brief review of prominent reading comprehension models and their interconnections through specific levels of language processing (single word, sentence, discourse), as well as to discuss the methodological aspects of assessing reading comprehension processes arising from the presented theoretical models. Motivated by the fact that there is a lack of studies on models and assessment materials in languages with transparent orthography, the application of existing models and assessment methods will be discussed in the context of transparent orthography languages. This study provides comprehensive insights, based on theory, on the key elements to consider when developing an assessment method/tool for reading comprehension, both for research or diagnostic purposes

Assessments of English Reading and Language Comprehension in Bilingual Children: A Systematic Review 2010 to 2021

Learning to read marks an important milestone in children. Extensive research with monolingual and bilingual children has demonstrated that language comprehension (LC) forms fundamental building blocks for reading comprehension (RC). However, mixed findings are reported among studies that compare readings skills in children with and without diverse language experiences. Depending on how researchers operationalize the construct of LC and RC, studies use different standardized tests or assessments to assess reading skills in children, which may lead to different findings across studies. The current review systematically examined tests of LC and RC that empirical studies have used to assess bilingual children who speak English as their second language. Out of an initial sample of 374 studies, 25 were eligible for inclusion. We extracted LC and RC assessments from the studies and documented task- and administration-related factors. Moreover, participant characteristics, definition of LC as described by authors, and findings related to the relationship between LC and RC were examined for each study. Our results demonstrated variability in the measures and definitions used to assess and describe LC and RC, potentially explaining the mixed findings in the literature. We underscore the importance of considering the multidimensional nature of LC and the need to further explore how different administrative and task characteristics of LC tests relate to RC. Furthermore, this review provides researchers and practitioners with an original and extensive survey of the literature on how LC and RC were assessed among bilingual children. Lastly, we highlight limitations in the current literature and discuss practical implications in the field of school psychology in supporting children with diverse language experiences.

A General Mixture Model for Cognitive Diagnosis

Although the generalized deterministic inputs, noisy “and” gate model (G-DINA; de la Torre, 2011) is a general cognitive diagnosis model (CDM), it does not account for the heterogeneity that is rooted from the existing latent groups in the population of examinees. To address this, this study proposes the mixture G-DINA model, a CDM that incorporates the G-DINA model within the finite mixture modeling framework. An expectation–maximization algorithm is developed to estimate the mixture G-DINA model. To determine the viability of the proposed model, an extensive simulation study is conducted to examine the parameter recovery performance, model fit, and correct classification rates. Responses to a reading comprehension assessment were analyzed to further demonstrate the capability of the proposed model.

Reading comprehension performance of elementary and senior high school students

IntroductionIn Brazil, reading has been widely discussed, mainly due to the published results of national and international performance exams of Brazilian schoolchildren. Learning to read is therefore a continuous process, and the ability to make inferences while reading a text develops with age. The textual complexity involving the syntactic structuring, vocabulary and types of text progressively increases from the initial years of elementary school to high school students, also increasing the cognitive demand of the students; this occurs in parallel to their development and school advancement, which allows improvements in their teaching/learning processes. Based on the above, the following questions were raised: (1) How is the semantic process of reading established among elementary school students in elementary school (cycle II) and high school? Aim: to characterize the performance of elementary and senior high school students on semantic process tests from the Brazilian adaptation of the evaluation of reading processes (PROLEC-SE-R).MethodsA total of 436 students of both sexes, aged between 11 and 18 years, participated. They were evaluated with Assessment of Reading Processes-PROLEC-SE-R.Results and discussionThe results indicated that the semantic process was equally established among high school students, with a higher average performance compared with that of elementary school students. Among elementary school students, there was progression in the average correct answers with advancement in schooling. In the two levels of education, narrative texts allowed a greater number of correct answers, followed by multiple-choice and literal questions. The PROLEC-SE-R semantic process tests proved to be effective for assessing reading comprehension in elementary and high school students and reflected the Brazilian reality with regard to the gaps and weaknesses in the educational system.

Revisiting complex text instruction: A study of 11th‐grade history students

AbstractThe reading of complex text has become part of the reading curriculum since the introduction of the Common Core State Standards. However, little research has been conducted to determine the effect of complex text instruction on comprehension and fluency in secondary readers. This study randomly assigned 53 11th‐grade US history students to one of two classroom conditions. In one classroom, students were taught using a business‐as‐usual approach while the treatment classroom taught students to use text annotation and writing to improve their understanding of complex texts over a 24‐week period. Students were pre‐ and post‐tested using a standardized reading comprehension assessment and curriculum‐based measurement fluency measures. Results revealed statistically significant improvement favoring the treatment group for both reading comprehension and fluency. Results and implications for instruction are discussed.

Comprehensive Assessment of Reading in Aphasia (CARA) reading questionnaire-German version.

Reading comprehension is frequently impaired in persons with aphasia (PWA). For goal-setting and outcome measurement, speech and language therapists (SLTs) need to determine an individual's perspective of their reading difficulties and everyday reading activities. The Comprehensive Assessment of Reading in Aphasia (CARA) reading questionnaire provides a person-centred tool to find out the individual perception of reading functions, reading-related emotions and reading activities in PWA. It was developed and evaluated in English. So far, there is no equivalent instrument in German. To translate and adapt the CARA reading questionnaire into German language and culture, to evaluate its practicability and acceptance, and to provide the first psychometric properties of the German version. Based on translation and adaptation guidelines, we conducted two forward translations that were merged and then adapted. A back translation was prepared and compared with the original version. It was found to be semantically equivalent by one of the authors of the original version. We performed pilot testing with 12 PWA, and the pilot version was adapted according to the comments of these participants. We then collected data on self-reported perception of reading and on psychometric properties of the translated and adapted German version. A total of 22 German-speaking PWA completed the questionnaire at least five times during an intervention study. We analysed retest reliability with Spearman correlation, internal consistency with Cronbach's alpha, internal responsiveness with the standardized response mean, as well as the relationship between outcomes of the questionnaire and text comprehension measures using repeated measures correlations. Our data suggest good practicability and acceptance of the German version of the CARA reading questionnaire as well as appropriate validity, reliability and sensitivity to measure therapy-induced change. We found moderate correlations between outcomes of the questionnaire and text-level reading speed. The German version of the CARA reading questionnaire could be helpful in intervention planning and goal-setting with German-speaking PWA. By using the questionnaire, SLTs can find out about a person's individual perception of reading difficulties as well as individually relevant reading activities. The questionnaire provides a tool to measure change and is therefore valuable to demonstrate self-reported individual progress. As reading speed seems to be an indicator of personal perception of reading difficulty, it is important to consider reading speed in reading interventions and in reading comprehension assessments. What is already known on the subject Reading comprehension is frequently impaired in PWA. Reading preferences, the perception of difficulties and the impact on everyday life reading activities are specific to the individual and thus need to be known for goal-setting, intervention planning and monitoring of change. As part of a comprehensive assessment of reading, Morris etal. developed a person-centred English language questionnaire for this purpose. So far, there is no equivalent tool in German. What this paper adds to the existing knowledge In this study, we translated and adapted the questionnaire to German language and culture, and analysed its validity and reliability with German-speaking PWA. We demonstrated that the German version is accessible for German-speaking PWA, and that it has appropriate validity, reliability and sensitivity to measure self-reported change. Outcomes of the questionnaire correlate with text level reading speed. What are the potential or actual clinical implications of this work? The German version of the questionnaire could be a valuable self-reported outcome measure to assess individual perceptions of reading and to measure progress (as perceived by an individual) as a consequence of recovery or intervention in either clinical or research settings. As reading speed might be an indicator of everyday life reading as perceived by an individual, it should be considered in reading assessments and interventions.

A Diagnostic Tree Model for Adaptive Assessment of Complex Cognitive Processes Using Multidimensional Response Options

A tree model for diagnostic educational testing is described along with Monte Carlo simulations designed to evaluate measurement accuracy based on the model. The model is implemented in an assessment of inferential reading comprehension, the Multiple-Choice Online Causal Comprehension Assessment (MOCCA), through a sequential, multidimensional, computerized adaptive testing (CAT) strategy. Assessment of the first dimension, reading comprehension (RC), is based on the three-parameter logistic model. For diagnostic and intervention purposes, the second dimension, called process propensity (PP), is used to classify struggling students based on their pattern of incorrect responses. In the simulation studies, CAT item selection rules and stopping rules were varied to evaluate their effect on measurement accuracy along dimension RC and classification accuracy along dimension PP. For dimension RC, methods that improved accuracy tended to increase test length. For dimension PP, however, item selection and stopping rules increased classification accuracy without materially increasing test length. A small live-testing pilot study confirmed some of the findings of the simulation studies. Development of the assessment has been guided by psychometric theory, Monte Carlo simulation results, and a theory of instruction and diagnosis.

Using Eye‐Tracking Measures to Predict Reading Comprehension

ABSTRACTThis study examined the potential of eye‐tracking as a tool for assessing reading comprehension. We administered three widely used reading comprehension tests with varying task demands to 79 typical adult readers while monitoring their eye movements. In the York Assessment of Reading for Comprehension (YARC), participants were given passages of text to read silently, followed by comprehension questions. In the Gray Oral Reading Test (GORT‐5), participants were given passages of text to read aloud, followed by comprehension questions. In the sentence comprehension subtest of the Wide Range Achievement Test (WRAT‐4), participants were asked to provide a missing word in sentences that they read silently (i.e., a cloze task). Linear models predicting comprehension scores from eye‐tracking measures yielded different results for the three tests. Eye‐tracking measures explained significantly more variance than reading‐speed data for the YARC (four times better), GORT (three times better), and the WRAT (1.3 time better). Importantly, there was no common strong predictor for all three tests. These results support growing recognition that reading comprehension tests do not measure the same cognitive processes, and that participants adapt their reading strategies to the tests' varying task demands. This study also suggests that eye‐tracking may provide a useful alternative for measuring reading comprehension.

An Application of Eye Movement Parameters Collected from Mass Market Devices for the Estimation of a Text Comprehension.

The growing interest in evaluating the reader's comprehension leads to the search for new methods that allow such estimation in real-time (or pseudo-real-time). This can be used for more effective educational processes and to adopt textual content for various purposes. The present study used the Oken Reader eye-tracking application (60 Hz) for mass-market devices to assess reading comprehension processes. Twenty-three (23) respondents aged between 19 and 31 (mean = 24. 5, SE = 1. 4, 65% female) participated in the study. The mean, mu, and sigma parameters differed significantly depending on the level of text comprehension. This result indicates the possibility of using mass-market devices with eye-tracking technology to assess comprehension in reading. Furthermore, the study's results confirm the possibility of searching the correlations between physiological indicators such as eye movements and comprehension.

Pre-school Skills and School-Age Reading Comprehension in Children on the Autism Spectrum: A Preliminary Investigation

We explored reading comprehension development in children on the spectrum from pre-school to the first (YOS1) and third year of schooling (YOS3). Children were first assessed on meaning-related skills in pre-school. Forty-one children completed follow-up assessments of reading comprehension, reading accuracy, and listening comprehension in YOS1. Nineteen returned for assessments of reading accuracy, reading comprehension, and listening comprehension in YOS3. Children showed poorer reading comprehension than reading accuracy at both timepoints. Reading comprehension, reading accuracy, and listening comprehension were significantly concurrently correlated. Pre-school receptive vocabulary was a significant predictor of YOS3 reading comprehension. Results from this preliminary investigation highlight the potential for early identification of children on the spectrum at risk for reading comprehension difficulties.

The role of inferencing in struggling adult readers' comprehension of different texts: A mediation analysis

Inferencing skills uniquely contribute to the reading comprehension skills of older grade-school and college students. Evidence also suggests that children's reading component skills, such as decoding and language comprehension, differentially contribute to various reading comprehension assessments. However, additional research is needed to investigate the complex relations of foundational reading skills and inferencing skills to sentence-level and passage-level reading comprehension assessments with struggling adult readers. This study examined the relations between struggling adult readers' (N = 125) text-based (decoding, fluency), language-based (morphological awareness, vocabulary, language comprehension), and inferencing skills. The indirect effects of language-based reading component skills to sentence-level and passage-level reading comprehension measures through inferencing were also examined. Vocabulary and morphological awareness indirectly predicted passage-level reading comprehension through inferencing. Word reading fluency and vocabulary knowledge uniquely predicted sentence-level comprehension whereas inferencing skills predicted passage-level comprehension. These results suggest that inferencing is an important contributor to the reading comprehension skills of struggling adult readers. These findings also emphasize a need for multiple measures of comprehension to understand the complexity and underlying component processes involved in struggling adults' reading comprehension skills. Educational implications for adult literacy programs are discussed.

Refocusing reading comprehension: Aligning theory with assessment and intervention

In this commentary, we propose that the field of reading comprehension requires refocusing on theories that help describe and explain the dynamic interactions among reader characteristics, texts, and contexts in which comprehension occurs across development. Further, we assert that reading comprehension assessments and interventions have not been aligned sufficiently with each other or with theory, thus limiting researchers' capacity to detect potentially important intervention effects. We integrate findings from the four empirical articles in this issue, highlighting the need for greater alignment of theory, assessment, and intervention, which can, in turn, promote transfer of reading comprehension skills across texts and contexts.

The impact of test-taking strategies on eye movements of elementary students during reading comprehension assessment.

Teachers often encourage students to use test-taking strategies during reading comprehension assessments, but these strategies are not always evidence-based. One common strategy involves teaching students to read the questions before reading an associated passage. Research findings comparing the passage-first (PF) and questions-first (QF) strategies are mixed. The present study employed eye-tracking technology to record 84 third- and fourth-grade participants' eye movements (EMs) as they read a passage and responded to multiple-choice (MC) questions using PF and QF strategies in a within-subject design. Although there were no significant differences between groups in accuracy on MC questions, EM measures revealed that the PF condition was superior to the QF condition for elementary readers in terms of efficiency in reading and responding to questions. These findings suggest that the PF strategy supports a more comprehensive understanding of the text. Ultimately, within the PF condition, students required less time to obtain the same accuracy outcomes they attained when reading in the QF condition. School psychologists can improve reading comprehension instruction by encouraging the importance of teaching children to gain meaning from the text rather than search the passage for answers to MC questions. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Technological tool for reading comprehension assessment of an argumentative text in Spanish as a native language

Resumen El procedimiento cloze, que se emplea habitualmente como herramienta tecnológica para evaluar la comprensión lectora de segundas lenguas, no suele usarse con lenguas maternas. Los motivos que descartan este instrumento con hablantes nativos se basan en que solo detecta el reconocimiento de información y habilidades lingüísticas microestructurales, y no mide la comprensión global ni inferencial del texto. En esta investigación se cuestiona esta idea y, con el fin de dilucidar el valor procedimental, se creó una herramienta de prueba basada en el modelo de exámenes DELE (nivel B2) y se aplicó a estudiantes universitarios españoles (N = 125) con dos objetivos: 1) medir la comprensión lectora de un texto argumentativo; 2) valorar la comprensión de la coherencia y cohesión discursivas a partir del uso de conectores. La herramienta fue diseñada en Moodle y ejecutada presencialmente. Se midió estadísticamente el índice de acierto y error de cada frase y conector y se aplicaron el test Chi-cuadrado de homogeneidad y el test de Cochran para valorar la significatividad de la confusión entre conectores y nociones argumentales. Los resultados muestran argumentos y conectores más transparentes y patrones de confusión argumental. Se demuestra que esta herramienta basada en el cloze mide la comprensión de textos argumentativos en lengua materna y puede ser un instrumento tecnológico replicable.

Implementing Virtual Literature Circles to Support English Language Development

The research from this mixed-methods, quasi-experimental study explores the effectiveness of literature circle (LC) strategies to support English language development, reading comprehension, and student engagement in an online synchronous post-secondary course. A total of 40 linguistically and culturally diverse participants were split evenly between the control group and the experimental group. Using constructivist grounded theory methodology, the study draws upon data from surveys, reading comprehension assessments, observations, and document analysis. To compare the mean scores between the less proficient and more proficient writers, independent samples t-test was used. Results indicated that there are significant differences between the two groups in content knowledge, vocabulary, elaborative discussion of text, reading experience and confidence, and in all the measures of literary competence. The results show that LC strategies also improve English language development while engaging students in virtual learning processes.

Reading Comprehension Assessment Rubric for Foreign Language Level A1 Based on Ger (Gemeinsame Europäische Referenzrahmen)

This research is to produce an assessment rubric for reading comprehension in German with the standard GER (Gemeinsame europäische Referenzrahmen). This arises because the same standard will make it easier for students and educators to provide qualifications for learning German. The research method taken is qualitative. The technique used in this research is documentation and interviews. The series of assessment rubrics will help make it easier for educators to find out students' weaknesses in language and for students to deepen the knowledge they feel is lacking. Expert validation shows a figure of 97% for this reading comprehension rubric—subjects taken by students of a course institution.
 Keywords: Foreign Language Level, GER (Gemeinsame europäische Referenzrahmen), Reading comprehension assessment

Supplemental Material for The Impact of Test-Taking Strategies on Eye Movements of Elementary Students During Reading Comprehension Assessment

Supplemental Material for The Impact of Test-Taking Strategies on Eye Movements of Elementary Students During Reading Comprehension Assessment

Emotional factors of causal coherence in text comprehension

The current study tested the effects of positive and negative emotion at the beginning and end of texts, as well as the consistency in valence throughout a text on readers’ response choice to items on a multiple-choice reading comprehension assessment. Multinomial logistic regressions were performed on 1,161 college participants’ assessment responses to test whether emotion in narrative and informational text items significantly predicted which distractor response options readers chose in comparison to the correct response option. Overall, consistent emotion throughout text and positive and negative emotion at the beginning and end of text were significant predictors of readers’ response choices. The results are discussed in terms of emotion being an indicator of readers’ causal processing. Specifically, findings provide a better understanding of how emotional features in narrative and informational texts may influence how readers develop causal coherence and comprehension during reading. This understanding could also help inform the development of instructional tools that encourage readers to focus on aspects of text (i.e., emotion) which could, in turn, help improve comprehension for readers who struggle.