Instructional Sensitivity Research Articles

Measuring students' learning progressions in energy using cognitive diagnostic models.

This study applied cognitive diagnostic models to assess students’ learning progressions in energy. A Q-matrix (i.e., an item attribute alignment table) was proposed based on existing literature about learning progressions of energy in the physical science domain and the Trends in International Mathematics and Science Study (TIMSS) assessment framework. The Q-matrix was validated by expert review and real data analysis. Then, the deterministic inputs, noisy ‘and’ gate (DINA) model with hierarchical relations was applied to data from three jurisdictions that had stable, defined science curricula (i.e., Australia, Hong Kong, and Ontario). The results suggested that the hypothesized learning progression was consistent with the observed progression in understanding the energy concept. We also found similarities in students’ attribute mastery across the three jurisdictions. In addition, we examined the instructional sensitivity of the selected item. We discuss several curriculum-related issues and student misconceptions that may affect students’ learning progressions and mastery patterns in different regions of the world.

Mapping from proximity traces to socio‐spatial behaviours and student progression at the school

AbstractIdentifying students facing difficulties and providing them with timely support is one of the educator's key responsibilities. Yet, this task is becoming increasingly challenging as the complexity of physical learning spaces grows, along with the emergence of novel educational technologies and classroom designs. There has been substantial research and development work focused on identifying student social behaviours in digital platforms (eg, the learning management system) as predictors of academic progression. However, little work has investigated such relationships in physical learning spaces. This study explores the potential of using wearable trackers for the early detection of low‐progress students based on their social and spatial (socio‐spatial) behaviours at the school. Positioning data from 98 primary school students and six teachers were automatically captured over a period of eight weeks. Fourteen socio‐spatial behavioural features were extracted and processed using a set of machine learning classifiers to model students’ learning progression. Results illustrate the potential of prospectively identifying low‐progress students from these features and the importance of adapting classroom learning analytics to differences in pedagogical designs. Practitioner notesWhat is already known about this topic Learning analytics research on predicting students’ academic progression is emerging in both digital and physical learning spaces. Students’ social behaviours in learning activities is a key factor in predicting their academic progression. Emerging sensing technologies can provide opportunities to study students’ real‐time social behaviours in physical learning spaces. What this paper adds Fourteen progression‐related socio‐spatial behavioural features are extracted from students’ physical (x‐y) positioning traces. Predictive learning analytics that achieved 81% accuracy in prospectively identifying low‐progress students from their real‐time socio‐spatial behaviours. Empirical evidence to support the need for classroom learning analytics to have instructional sensitivity (ie, be calibrated according to the learning design). Implications for practice and/or policy Sensing technologies and machine learning algorithms can be used to capture and generate valuable insights about higher‐order learning constructs (eg, performance and collaboration) from students' physical positioning traces in classrooms. Researchers and practitioners should be cautious with generalised classification algorithms and predictive learning analytics that do not account for the pedagogical differences between different subjects or learning designs. Researchers and practitioners should consider the potentially unforeseen ethical issues that can emerge in using sensing technologies and predictive learning analytics in authentic, physical classroom settings.

When academic achievement (also) reflects personality: Using the personality-achievement saturation hypothesis (PASH) to explain differential associations between achievement measures and personality traits.

Students’ academic achievement is a key predictor of various life outcomes. The two most prominent measures of academic achievement are grades and standardized test scores. Both measures are commonly used during selection processes as well as for educational monitoring and accountability. Research has suggested that grades and test scores are strongly related to students’ characteristics (e.g., cognitive abilities) but might differentially reflect personality. In order to better explain differential personality-achievement associations, it is important to move beyond the dichotomy of grades versus test scores. To this end, we propose the personality-achievement saturation hypothesis (PASH), which suggests that associations between personality and achievement vary, depending on four main features of the achievement measure used: level of standardization, relevance for the student, curricular validity, and instructional sensitivity. The PASH suggests that conscientiousness should typically be more strongly associated with grade point average, followed by course grades and final examination grades, whereas openness should be more strongly associated with test scores. We used data from three large-scale studies (total N = 14,953) and aggregated our findings. In line with the PASH, conscientiousness was most strongly related to grades, which have lower standardization, moderate to high relevance, high curricular-validity, and high instructional sensitivity, whereas it had substantially weaker associations with more highly standardized, less curriculum-valid, and less instructionally sensitive measures. In addition, openness was most strongly related to highly standardized, less relevant, less curriculum-valid, and less instructionally sensitive measures in English. Implications for the ways in which achievement measures can be used are discussed.

The Instructional Sensitivity of the Test of Gross Motor Development-3 to Detect Changes in Performance for Young Children With and Without Down Syndrome.

The Test of Gross Motor Development is among the most commonly used measures of gross motor competency in children. An important attribute of any developmental assessment is its sensitivity to detect change. The purpose of this study was to examine the instructional sensitivity of the Test of Gross Motor Development-third edition (TGMD-3) performance criteria to changes in performance for 48 children (age 4-7years) with and without Down syndrome following 10 weeks of physical education. Paired t tests identified significant improvements for all children on locomotor (p < .01) and ball skills (p < .01). These significant differences were associated with moderate to large effect sizes. SEM was low relative to the maximum raw score for each subtest, indicating high confidence in the scores. These findings provide evidence that the TGMD-3 is sensitive to change in performance for children with and without Down syndrome.

Assessing subjective and objective information literacy at upper secondary schools - an empirical study in four German-speaking countries

Information literacy (IL) is a key 21st-century skill. This paper contributes to the body of knowledge by specifying the construct of IL. It defines IL and presents a seven-dimensional IL model. On this basis, two types of test instruments are evaluated: a performance test for objective IL (OIL) and a self-assessment instrument for subjective IL (SIL). Both instruments are validated by means of item response theory. We utilise a sample with N = 432 upper secondary students from German-speaking countries. Rasch scalability of the instruments is demonstrated. Reliability equals in the case of OIL for EAP/PV .84 and for WLE .88; in the case of SIL, .75 and .77, respectively. For OIL, we present a proficiency level model to ensure a criterion-based interpretation of test scores. In terms of instructional sensitivity, we provide evidence for a positive influence of students' grade and IL-related content in class on OIL. The correlation between OIL and SIL equals .52.

Sustainability of the Belt and Road Initiative: An Integrated, Conceptual Framework for Instructional Communication in China’s Universities

China is fast becoming a coveted destination and a hub for higher education among international students, particularly since the announcement of its Belt and Road Initiative (BRI) in September 2013. Consequently, China’s higher-education institutions are seeking ways to make international students’ educational experience more consistent with their expectations. Nonetheless, instructional communication—that is, communication for the purpose of engaging students academically while reducing problematic misunderstandings in the classroom—is a bane of the educational experience of international students in China. Therefore, this article extends instructional communication and intercultural sensitivity models to pedagogical, learner-centered contexts in an attempt to develop an integrated conceptual framework on sustaining international student–Chinese faculty interactions in the classroom. That framework has three key constructs: (a) the faculty’s classroom behaviors and international students’ characteristics, (b) international students’ instructional beliefs, and (c) learning outcomes. They will serve as the basis for positioning instructional practices in responding more appropriately to enhancing the experience of international students as global learners and toward deepening and sustaining the internationalization of China’s higher-education institutions, specifically within the context of BRI.

Sensitivity of test items to teaching quality

Instructional sensitivity is the psychometric capacity of tests or single items of capturing effects of classroom instruction. Yet, current item sensitivity measures’ relationship to (a) actual instruction and (b) overall test sensitivity is rather unclear. The present study aims at closing these gaps by investigating test and item sensitivity to teaching quality, reanalyzing data from a quasi-experimental intervention study in primary school science education (1026 students, 53 classes, Mage = 8.79 years, SDage = 0.49, 50% female). We examine (a) the correlation of item sensitivity measures and the potential for cognitive activation in class and (b) consequences for test score interpretation when assembling tests from items varying in their degree of sensitivity to cognitive activation. Our study (a) provides validity evidence that item sensitivity measures may be related to actual classroom instruction and (b) points out that inferences on teaching drawn from test scores may vary due to test composition.

Assessing Instructional Sensitivity Using the Pre-Post Difference Index: A Nontechnical Tool for Extension Educators

This article provides an illustrative description of the pre-post difference index (PPDI), a simple, nontechnical yet robust tool for examining the instructional sensitivity of assessment items. Extension educators often design pretest-posttest instruments to assess the impact of their curricula on participants' knowledge and understanding of the concepts taught. Although the use of pretests and posttests is common in Extension evaluation, the validity and reliability of these tests are rarely reported or discussed, mostly due to many Extension educators' limited knowledge of various statistical methods. The PPDI method described in this article should be a useful addition to Extension educators' evaluation toolboxes.

A new video-based tool to enhance teaching economics

We present a new video-based tool developed in Germany to enhance the teaching of economics. The tool consists of authentic teacher-student interactions in economics classes to which instructors need to react. With 18 videos, the tool provides an easily accessible opportunity to assess and enhance teaching performance in economics. The tool was rated by 41 highly experienced secondary school economics teachers according to different criteria relevant for curriculum design and instructional practice, including authenticity of the simulated classroom situations, and typical difficulties and misconceptions students have in learning economics. The teachers confirm the curricular and instructional sensitivity, validity, quality and innovative approach of the tool for formative assessment and enhancement of performance in teaching economics. Based on German practical experience, the use of the tool in secondary school teacher training programs in the United States is currently being investigated.

Instructional sensitivity in vocational education

Apprentices' performance after vocational educational training (VET) is commonly attributed to the effectiveness of the training. This implies the assumption that learners’ development of vocational knowledge and ability is significantly affected by vocational instruction. However, the few analyses that have been made of instructional sensitivity within the general school-based educational system, have in most cases shown little or no effect of instruction (time in school) on performance in assessments. The question as to whether, and to what extent, VET in adult education is effective (in the sense that it fosters the development of vocational knowledge and ability), as well as the related question—whether we are able to track the resulting learning progress with adequate measures (i.e., assessments)—has hardly been investigated. In the present study, we propose modeling of instructional sensitivity via differential item functioning (DIF), and apply this method to a sample of n = 534 apprentices. We find that during vocational instruction, apprentices significantly improved their performance in an assessment of vocational knowledge and ability, and that we were able to track these changes in the quality of their abilities over the span of a three year initial VET program: that is, the first program of vocational study in which apprentices become qualified to work in a given trade. Moreover, with this proposed method, it is possible to identify items that are particularly sensitive to instruction and that appear therefore to be amenable to the future development of vocational assessments.

What About the "Instruction" in Instructional Sensitivity? Raising a Validity Issue in Research on Instructional Sensitivity.

In instructional sensitivity research, it is important to evaluate the validity argument about the extent to which student performance on the assessment can be used to infer differences in instructional experiences. This study examines whether three different measures of mathematics instruction consistently identify mathematics assessments as being sensitive to instruction. Mixed findings across fourth-grade (n = 8,298) and fifth-grade (n = 9,336) students and their teachers across three school districts raise questions as to whether different ways of measuring instruction provide similar inferences about the instructional sensitivity of assessments. This raises validity concerns about the quality of inferences based on different measures of instruction.

Absolute and Relative Measures of Instructional Sensitivity

Valid inferences on teaching drawn from students’ test scores require that tests are sensitive to the instruction students received in class. Accordingly, measures of the test items’ instructional sensitivity provide empirical support for validity claims about inferences on instruction. In the present study, we first introduce the concepts of absolute and relative measures of instructional sensitivity. Absolute measures summarize a single item’s total capacity of capturing effects of instruction, which is independent of the test’s sensitivity. In contrast, relative measures summarize a single item’s capacity of capturing effects of instruction relative to test sensitivity. Then, we propose a longitudinal multilevel item response theory model that allows estimating both types of measures depending on the identification constraints.

An Examination of the Instructional Sensitivity of the TIMSS Math Items: A Hierarchical Differential Item Functioning Approach

ABSTRACTIn recent years, students’ test scores have been used to evaluate teachers’ performance. The assumption underlying this practice is that students’ test performance reflects teachers’ instruction. However, this assumption is generally not empirically tested. In this study, we examine the effect of teachers’ instruction on test performance at the item level using a hierarchical differential item functioning approach. The items are from the U.S. TIMSS 2011 4th-grade math test. Specifically, we tested whether students who had received instruction on a given item performed significantly better on that item compared with students who had not received such instruction when their overall math ability was controlled for, whether with or without controlling for student-level and class-level covariates. This study provides preliminary findings regarding why some items show instructional sensitivity and sheds light on how to develop instructionally sensitive items. Implications and directions for further research are also discussed.

A Psychometric Framework for the Evaluation of Instructional Sensitivity

Although there is a common understanding of instructional sensitivity, it lacks a common operationalization. Various approaches have been proposed, some focusing on item responses, others on test scores. As approaches often do not produce consistent results, previous research has created the impression that approaches to instructional sensitivity are noticeably fragmented. To counter this impression, we present an item response theory–based framework that can help us to understand similarities and differences between existing approaches. Using empirical data for illustration, this article identifies three perspectives on instructional sensitivity: One perspective views instructional sensitivity as the capacity to detect differences in students' stages of learning across points of time. A second perspective treats instructional sensitivity as the capacity to detect differences between groups that have received different instruction. For a third perspective, the previous two are combined to consider differences between both time points and groups. We discuss linking sensitivity indices to measures of instruction.

Evaluating the Instructional Sensitivity of Four States' Student Achievement Tests

As state tests of student achievement are used for an increasingly wide array of high- and low-stakes purposes, evaluating their instructional sensitivity is essential. This article uses data from the Bill and Melinda Gates Foundation's Measures of Effective Project to examine the instructional sensitivity of 4 states' mathematics and English language arts assessments to 5 measures of pedagogical quality. The results indicate modest overall levels of sensitivity, with some state assessments insensitive to most or all measures of pedagogical quality. Furthermore, there are sometimes substantial differences across states in the sensitivity of their assessments. The article calls for more research on what makes state tests instructionally sensitive and cautions against the use of insensitive assessments for consequential decisions.

Initial Considerations When Applying an Instructional Sensitivity Framework: Partitioning the Variation Between and Within Classrooms for Two Mathematics Assessments

ABSTRACTDrawing inferences about the extent to which student performance reflects instructional opportunities relies on the premise that the measure of student performance is reflective of instructional opportunities. An instructional sensitivity framework suggests that some assessments are more sensitive to detecting differences in instructional opportunities compared to other assessments. This study applies an instructional sensitivity framework to compare student performance on two different mathematics achievement measures across five states and three grade levels. Results suggest a range of variation in student performance among teachers on the same mathematics achievement measure, variation between the two different mathematics achievement measures, and variation between grade levels within the same state. Findings highlight initial considerations for educators interested in selecting and evaluating measures of student performance that are reflective of instructional opportunities.

The Need for Instructional Sensitivity and Construct Clarity in PACT

This commentary on the article titled “Examining the Internal Structure Evidence for the Performance Assessment for California Teachers: A Validation Study of the Elementary Literacy Teaching Event for Tier 1 Teacher Licensure” provides an overview of Performance Assessment for California Teachers (PACT), its relationship to edTPA and the construct (domains) measured, a summary of the major findings of the article commented on, a discussion of PACT validity issues and rubrics, and the need for instructionally sensitive and formative assessments. It concludes with recommendations to the State of California that include attention to construct validity, improved rubrics, formative use of the data, and attention to utility as well as validity and reliability.

Modeling Instructional Sensitivity Using a Longitudinal Multilevel Differential Item Functioning Approach

Students’ performance in assessments is commonly attributed to more or less effective teaching. This implies that students’ responses are significantly affected by instruction. However, the assumption that outcome measures indeed are instructionally sensitive is scarcely investigated empirically. In the present study, we propose a longitudinal multilevel‐differential item functioning (DIF) model to combine two existing yet independent approaches to evaluate items’ instructional sensitivity. The model permits for a more informative judgment of instructional sensitivity, allowing the distinction of global and differential sensitivity. Exemplarily, the model is applied to two empirical data sets, with classical indices (Pretest–Posttest Difference Index and posttest multilevel‐DIF) computed for comparison. Results suggest that the approach works well in the application to empirical data, and may provide important information to test developers.

An Examination of Students' Adaptation, Aggression, and Apprehension Traits with their Instructional Feedback Orientations

Feedback orientations refer to students' perceptions of instructional feedback utility, retention, sensitivity, and confidentiality. In this paper, we report three studies that investigated the relationships among feedback orientations and communication traits. Specifically, we examined the associations among communication adaptation traits (Study 1), aggression traits (Study 2), and apprehension traits (Study 3). The results of Study 1 (N =149) indicated that students high in cognitive flexibility and responsiveness reported retaining and using instructors' feedback and were less sensitive to feedback than other participants. Findings from Study 2 (N =182) showed that students who were high in verbal aggressiveness, Machiavellianism, and tolerance for disagreement found their instructors' feedback less useful and retained less feedback than other participants. The results of Study 3 (N =172) revealed that students who were high in communication apprehension and low in intellectual flexibility reported being sensitive to receiving feedback, preferred to receive feedback privately, and did not find feedback to be overly useful. Results may be used by instructors to better provide students with useful, memorable, nonthreatening, and private feedback in the classroom.

A Mixture Rasch Model With a Covariate

Mixtures of item response theory (IRT) models have been proposed as a technique to explore response patterns in test data related to cognitive strategies, instructional sensitivity, and differential item functioning (DIF). Estimation proves challenging due to difficulties in identification and questions of effect size needed to recover underlying structure. In particular, the impact of covariates for examinees in estimation has not been systematically explored. The goal of this study is to carry out a systematically designed simulation study to investigate the performance of mixture Rasch model (MRM) under Bayesian estimation. Insights and suggestions on model application and model estimation are discussed. The foci of this study are to use a flexible logistic regression structure to include examinees’ covariate in MRM, to study Markov chain Monte Carlo (MCMC) estimation behavior in light of effect size, and to provide an effective and applicable method for dealing with chain switching.