Educational Assessment Research Articles

Improving Teaching and Learning in Higher Education through Machine Learning: Proof of Concept’ of AI’s Ability to Assess the Use of Key Microskills

Advances in artificial intelligence (AI), including intelligent machines, are opening new possibilities to support teaching and learning in higher education. This research has found a ‘proof of concept’ in the application of machine learning in the assessment of educators’ use of four key microskills, drawn from an internationally established framework. The analysis of teaching videos where these microskills were demonstrated multiple times in front of a green screen or in a space formed the data set. Multiple videos of this nature were recorded to allow for increased analysis and deconstruction of the video components to enable the application of machine learning. The results showed how AI can be used to support the collaborative and reflective practice of educators in a time when online teaching has become the norm. Having achieved a ‘proof of concept’, this research has laid the groundwork to allow for the whole framework of ten microskills to be applied in this way thus adding a new dimension to its use. Providing such critical information that is not currently available in such a systematic and personalised way to educators in the higher education sector can also support the validity of formative assessment practices.

EVALUATION IN EARLY CHILDHOOD EDUCATION CURRENT CHALLENGE AND PERSPECTIVES

The purpose of this work was to consider the influence of assessment in Kindergarten classes, since educational assessment is understood as a continuous teaching-learning process. Specifically, we sought to correspond to the forms of evaluation found in the Early Childhood Education modality, the need for the conception of the pedagogical practice developed by teaching professionals from the public and private network in the sector. Currently there are several concepts about the practice of evaluating, as researchers and educators we are improving and innovating techniques to better monitor the cognitive development of students. The influence of learning assessment in Early Childhood Education has the mediating function of making situations favorable that value and respect the previous experiences of each student. That is why it is essential for teachers to reexamine their assessment tactics, if necessary, seek other mechanisms that help the evolutionary process of children and that contemplate a global, consecutive and unified constitution. This study portrays multiple aspects of the functionality of the educational process.

Reading Comprehension and Linguistic Abilities of Children with and without Specific Learning Difficulties: Theoretical and Educational Implications

The aim of the present study has been to investigate reading comprehension (RC) of children with specific learning difficulties (SLD), considering linguistic factors, such as receptive vocabulary, morphosyntax, and pragmatics (i.e., figurative language). Participants included 90 students (9–12 years old; Μyears = 10.8, SD = 0.95), 45 with SLD and 45 typically developing (TD) controls, matched on age, gender, and non-verbal cognitive ability. Results indicated that students with SLD had significantly lower performance on RC and across all linguistic measures compared to TD peers. Scores of the figurative language comprehension task predicted RC for TD children, whereas morphosyntactic ability emerged as a unique predictor of RC for SLD children. The two groups utilize distinct linguistic resources in their effort to extract meaning from written texts. The differentiated language profile of children with SLD suggests the implementation of differentiated educational assessment and intervention practices, which are discussed.

Commentary: Modernizing Educational Assessment Training for Changing Job Markets

Commentary: Modernizing Educational Assessment Training for Changing Job Markets

Health literacy pedagogies and assessment embedded in a school-based health education program

AbstractStudies have focused on the development of teachers’ Health Literacy (HL) pedagogies but have yet to focus on building teachers’ assessment capacity. The aims of this research study were three-fold: 1) professionally develop teachers to enhance the HL planned in a Health and Physical Education (HPE) program and assessment task; 2) validate a previously developed curricular model of HL assessment (rubric) using student work samples; 3) measure students HL learning using an adaptation of the validated rubric. Researchers used a descriptive single school case study, using a mixed methods approach in an Independent, coeducational early learning, primary and secondary day school in Sydney, Australia to work closely with participants to review the pedagogical knowledge, learning activities and assessment within a HPE program using Nutbeam’s three level HL hierarchy. Using Messick’s six aspects of construct validity testing of the rubric was completed using 20 sample assessment tasks, as well as receiving feedback on the adaptability and feasibility of the rubric. The validated rubric was then adapted to the assessment task to measure students’ HL. Results demonstrated that professional development had a positive impact on the development of planned HL pedagogies as shown by a 17% increase in planned critical HL learning activities. Findings suggest that the validated rubric is a feasible and adaptable tool that teachers can use to measure students’ HL. This is the first study to validate a curricular model of measuring HL. Future research needs to develop teachers’ HL capabilities and assessment literacy.

Iterative Attribute Hierarchy Exploration Methods for Cognitive Diagnosis Models

Attribute hierarchies are commonly assumed to exist in many fields of psychological and educational assessment. Several theory-driven and data-driven approaches have been used to validate or explore attribute hierarchies, such as validating attribute hierarchies in the cognitive diagnostic model (CDM) through statistical hypothesis testing or even learning attribute hierarchies directly from data. A class of structural parameter standard error estimation methods for CDMs is useful for exploring attribute hierarchies, with the limitation that the information matrices of some model parameters may be unstable or singular, leading to biased hypothesis testing. An iterative method of attribute hierarchy testing was proposed to modify the original z-statistic method. The simulation study systematically compares the performance of the z-statistic and the iterative z-statistic in exploring the attribute hierarchy. The results show that the iterative z-statistic provides a better Type I error control rate and statistical power, and it partially solves the problem that the z-statistic is too conservative. In addition, the iterative z-statistic method also achieves satisfactory results on real data.

‘Now you’ve said it, it’s like a big light bulb!’: enacting post observation feedback suggestions

Teaching observations have become a ubiquitous feature of teacher education programmes, development schemes and assessment regimes. Whilst the processes and procedures of classroom observation are well documented, the feedback which follows teaching observations has been given less attention. Most research into teaching observations focuses on eliciting teachers’ perspectives on their experiences of being observed. In contrast, we examine two aspects vital to teacher development and enhanced teaching practice: (1) post observation feedback talk; (2) teachers’ enactment of feedback following the feedback session. This article argues that examining feedback talk and how talk may influence enactment can help both observers and teachers maximise the effectiveness of teaching observations. We focus on suggestions, a common way of helping teachers to develop and improve. We analyse empirical examples of authentic post observation feedback talk to explore how suggestions are made and responded to, identifying features of suggestions which prompt teacher understanding and enactment. Analysis enables us to provide observers with concrete advice on how to make suggestions, thus showing the practical affordances and methodological warrant of analysing feedback talk.

The Evolving Nature of Threat and Violence Risk Assessment in College Counseling Centers

ABSTRACT This paper examines the historical and current contexts of threat assessment and violence risk in higher education, highlighting the evolving roles of college counseling centers in the threat assessment process for students displaying signs of concerning behaviors. Given research showing many college mental health professionals feel under-prepared for this work, we build foundational knowledge about the differing purposes of psychological evaluations and threat assessments, and provide recommendations for ways to continue to build and demonstrate understanding, competency and proficiency with threat assessments… Our review shares recent updates in the field regarding high-risk groups and the use of AI in assessment, and highlights the need for more attention to be given to training community members as well as counseling center staff and other members of campus teams involved in decision-making about students who threaten violence. We recommend the deliberate practice model of skill building to better prepare college mental health professionals when needing to respond under stress and pressure.

Where is the Middle School Movement Today? Interrogating Consequential Organizations

In an effort to advance the middle school movement, this paper examines the research trends from three organizations that support middle grades educational innovation, assessment, and research: The Carnegie Corporation of New York, the Educational Testing Service, and the Institute of Education Sciences. Using an instrumental, multiple case study design, trends within and across these organizations regarding research on middle grades education were identified. We found trends across these organizations that reveal the following: They privilege quantitative methodologies; the research topics examined are limited in focus; and the end-result for much of their work results in exclusive products for sale. Further, each of these organizations has deep pockets that provide significant funds for their research projects and products.

Informing Evidence-based Practice in Nutritional Genomics: An Educational Needs Assessment of Nutrition Care Providers in Canada.

Purpose: To investigate why Canadian nutrition care providers choose, or not, to integrate nutritional genomics into practice, and to evaluate the nutritional genomics training/education experiences and needs of nutrition providers in Canada, while comparing those of dietitians to non-dietitians.Methods: A cross-sectional online survey was distributed across Canada from June 2021 to April 2022.Results: In total, 457 healthcare providers (HCPs) [n = 371 dietitians (81.2%)] met the inclusion criteria. The majority (n = 372; 82.1%) reported having no experience offering nutritional genomics to clients (n = 4 did not respond). Of the 81 respondents with experience (17.9%), the most common reason to integrate nutrigenetic testing into practice was the perception that clients would be more motivated to change their eating habits (70.4%), while the most common reason for not integrating such tests was the perception that the nutrigenetic testing process is too complicated (n = 313; 84.1%). Dietitians were more likely than non-dietitians to view existing scientific evidence as an important educational topic (p = 0.002). The most selected useful educational resource by all HCPs was clinical practice guidelines (n = 364; 85.4%).Conclusions: Both dietitians and non-dietitians express a desire for greater nutritional genomics training/education; specific educational needs differ by type of HCP. Low implementation of nutrigenetic testing may be partly attributed to other identified barriers.

Introduction to the Contemporary Assessment and Recommendations to Enhance Surgical Education and Training: Reports of the Subcommittees of the Blue Ribbon Committee II.

The BRC II was a panel of 67 experts selected on the basis of experience and leadership in surgical education and training. It was organized into subcommittees, each of which was asked to prepare a manuscript on their findings and recommendations. The BRC II Subcommittees were: Blue Ribbon Committee 1 Review and Assessment; Surgical Workforce; Medical Student Education; Work Life Integration; Resident Education; Goals, Structure and Financing of Training; Education Support and Faculty Development; Research Training; Educational Technology and Assessment. BRC II used the Delphi approach with consensus defined as equal to or greater than 80% and identified and recommended 31 priorities for surgical education in 2024. The initial findings were presented to a general surgery and related specialty resident and fellow focus group for comments and written feedback, and they were asked to prepare a manuscript as well. The reports of the Subcommittees of the BRC II provide an assessment and key recommendations concerning surgical education and training in 2024.

ChatGPT, Copilot, Gemini, SciSpace and Wolfram versus higher education assessments: an updated multi-institutional study of the academic integrity impacts of Generative Artificial Intelligence (GenAI) on assessment, teaching and learning in engineering

ABSTRACT More than a year has passed since reports of ChatGPT-3.5’s capability to pass exams sent shockwaves through education circles. These initial concerns led to a multi-institutional and multi-disciplinary study to assess the performance of Generative Artificial Intelligence (GenAI) against assessment tasks used across 10 engineering subjects, showcasing the capability of GenAI. Assessment types included online quiz, numerical, oral, visual, programming and writing (experimentation, project, reflection and critical thinking, and research). Twelve months later, the study was repeated using new and updated tools ChatGPT-4, Copilot, Gemini, SciSpace and Wolfram. The updated study investigated the performance and capability differences, identifying the best tool for each assessment type. The findings show that increased performance and features can only heighten academic integrity concerns. While cheating concerns are central, opportunities to integrate GenAI to enhance teaching and learning are possible. While each GenAI tool had specific strengths and weaknesses, ChatGPT-4 was well-rounded. A GenAI Assessment Security and Opportunity Matrix is presented to provide the community practical guidance on managing assessment integrity risks and integration opportunities to enhance learning.

Application and Effect Evaluation of Machine Learning in Mathematics Education Assessment and Feedback

Application and Effect Evaluation of Machine Learning in Mathematics Education Assessment and Feedback

Assessing Population Invariance of 2021 Agricultural Science Examination of National Examination Council (NECO) in North Central Nigeria

Purpose: This study examines the challenges and implications of equating scores from different test forms, focusing on the 2021 NECO Agricultural Science examinations in North Central Nigeria. It considers factors such as school location, school type (public vs. private), and test administration (internal vs. external) to evaluate the extent to which these equated scores maintain invariance across different populations. Methodology: The study employs a non-equivalent anchor test (NEAT) group equating design and analyzes data from a population of 97,413 Senior Secondary students using the IRT statistical tool. Findings: The key findings reveal significant disparities in the equated scores between internal and external examinations, as well as between urban and rural locations. These discrepancies challenge the fairness and comparability of the examinations, raising concerns about potential biases and inequities in educational assessment. Significance: The research highlights the importance of achieving population invariance in test equating methodologies. It recommends adopting efficient equating methods, such as mean equating, and utilizing the Item Response Theory (IRT) approach for standardized test development to ensure fairness and equity in educational evaluations.

Investigating response behavior through TF-IDF and Word2vec text analysis: A case study of PISA 2012 problem-solving process data

The process data in computer-based problem-solving evaluation is rich in valuable implicit information. However, its diverse and irregular structure poses challenges for effective feature extraction, leading to varying degrees of information loss in existing methods. Process-response behavior exhibits similarities to textual data in terms of the key units and contextual relationships. Despite the scarcity of relevant research, exploring text analysis methods for feature recognition in process data is significant. This study investigated the efficacy of Term Frequency-Inverse Document Frequency (TF-IDF) and Word to Vector (Word2vec) in extracting response behavior features and compared the predictive, analytical, and clustering effects of classical machine learning methods (supervised and unsupervised) on response behavior. An analysis of the PISA 2012 computer-based problem-solving dataset revealed that TF-IDF effectively extracted key response behaviors, whereas Word2vec captured effective features from sequenced response behaviors. In addition, in supervised machine learning using both methods, the random forest model based on TF-IDF performed the best, followed by the SVM model based on Word2vec. Word2vec-based models outperformed TF-IDF-based ones in the F1-score, accuracy, and recall (except for precision) across the logistic regression, k-nearest neighbor, and support vector machine algorithms. In unsupervised machine learning, the k-means algorithm effectively clustered different response behavior patterns extracted by these methods. The findings underscore the theoretical and methodological transferability of these text analysis methods in educational and psychological assessment contexts. This study offers valuable insights for research and practice in similar domains by yielding rich feature representations, supplementing fine-grained assessment evidence, fostering personalized learning, and introducing novel insights for educational assessment.

Effect of short-term intensive design-based STEM learning on executive function: an fNIRS study of the left-behind children.

Design-based STEM learning is believed to be an effective cross-disciplinary strategy for promoting children's cognitive development. Yet, its impact on executive functions, particularly for disadvantaged children, still need to be explored. This study investigated the effects of short-term intensive design-based STEM learning on executive function among left-behind children. Sixty-one Grade 4 students from a school dedicated to the left-behind children in China were sampled and randomly assigned to an experimental group (10.70 ± 0.47years old, n = 30) or a control group (10.77 ± 0.43years old, n = 31). The experimental group underwent a two-week design-based STEM training program, while the control group participated in a 2-week STEM-related reading program. Both groups were assessed with the brain activation from 4 brain regions of interest using functional near-infrared spectroscopy (fNIRS) and behavioral measures during a Stroop task before and after the training. Analysis disclosed: (i) a significant within-group time effect in the experimental group, with posttest brain activation in Brodmann Area 10 and 46 being notably lower during neutral and word conditions; (ii) a significant between-group difference at posttest, with the experimental group showing considerably lower brain activation in Brodmann Area 10 and Brodmann Area 46 than the control group; and (iii) a significant task effect in brain activity among the three conditions of the Stroop task. These findings indicated that this STEM learning effectively enhanced executive function in left-behind children. The discrepancy between the non-significant differences in behavioral performance and the significant ones in brain activation implies a compensatory mechanism in brain activation. This study enriches current theories about the impact of Science, Technology, Engineering, and Mathematics (STEM) learning on children's executive function development, providing biological evidence and valuable insights for educational curriculum design and assessment.

Nurse perspectives on the assessment of emergency medicine residents: A qualitative study.

Multisource and 360-degree feedback are emerging methods in the medical education assessment literature. Nursing feedback in particular has been identified as a point of emphasis regarding what medical education leadership would most value for incorporating into their evaluations. There is no currently accepted tool for multisource feedback in the emergency department (ED). To answer this call, we conducted a qualitative study to elucidate nurse perspectives on evaluation of emergency medicine residents. We conducted individual semistructured interviews of ED nurses with experience working alongside residents via volunteer recruitment. Transcripts were coded independently by each author and then reviewed collaboratively to resolve differences. Codes were created by inductive thematic analysis and subsequently underwent constant comparison and classical content analysis. Recruitment continued until analysis of transcripts showed thematic saturation. We organized the interview transcript into chunks of information, arranging 407 quotes from the 10 interview transcripts into 17 codes and six themes regarding observable resident behaviors. Representative quotes and exemplar cases added detailed description to the meaning of these codes. The identified themes for observable resident behaviors included "general communication style," "medical knowledge," "efficiency," "patient communication," "nurse communication," and "professionalism." An additional 13 codes addressed the themes of barriers and catalysts for feedback. The identified themes on observable resident behaviors aligned well with prior literature and with the Accreditation Council for Graduate Medical Education (ACGME) Milestones. The interviews also highlighted nurse perspectives on barriers and catalysts for feedback that would be useful to incorporate into the design of an assessment tool.

Evaluating Human and Machine Assessment: Introducing a Hybrid Approach for Enhanced Educational Evaluation

This essay examines the evolving landscape of educational assessment, focusing on the analysis and comparison of human expertise and machine capabilities. Traditional educational assessment has predominantly relied on human evaluators who utilize their judgment and experience to assess student performance. However, advancements in technology have introduced machine assessments that leverage artificial intelligence to analyze data and provide feedback efficiently. This paper explores various assessment methods, including formative, summative, and peer assessments, and the role technology plays in enhancing these processes. A critical analysis of the strengths and weaknesses of both human and machine assessments is presented, highlighting scenarios where machines excel in efficiency and objectivity, particularly in handling large datasets and standardizing evaluations. Conversely, the essay emphasizes the irreplaceable depth of human insight necessary for assessing complex cognitive skills like creativity and critical thinking. The research advocates for a hybrid approach that combines the rapid analytical capabilities of machines with the nuanced judgment of human assessors. This integrative strategy aims to enhance reliability, personalize learning feedback, and optimize strategic educational planning. The proposed hybrid model not only addresses the shortcomings of both systems but also underscores the potential for improved assessment practices in various professional fields beyond education. This study calls for further exploration into the implementation of such hybrid assessment methods across diverse educational settings to maximize the benefits of both human and machine contributions.

Restorative Quality Assessment of Academic - An Islamic Motivation for Circular Nation-Building

A circular nation-building initiative emphasising zero-dysfunctionality is an urgent step to restore national education. The nose-diving performance in the 2022 Programme for International Student Assessment, PISA 2022, is a wake-up call to power up the people manning the education sector. The study analysed and identified the alarming situations signalling the urgent need for rectification based on the empirical data acquired over 30 years. The analysis points to evidence of fake claims of performance in academia as the main hub of human resource development and the primary factor that shadows the loopholes of education quality. Springboarding from there, the study recommends steps for the betterment of the sector. Qualitative assessment of academic staff via restorative assessment (RES-Q) measure assimilating the functional from the dysfunctional academics is the identified first step. Functionality, in this regard, is defined by the functions aligned to the fulfilment of national needs. Local universities' declining popularity prompted RES-Q map construction for root cause-to-solution to uplift the education sector. Detailing educator assessment needs leads to RES-Q inquisitive analysis to detect and fix issues without derailing from the national strategic alignment (NSA). Deploying RES-Q today should empower educators and in turn, revive learners’ interests in tertiary education. Through the corrected cycle of critical mass, the reshuffling of academics based on their dominant strengths must be pursued. The talent appropriation strategy is commensurate with the gratefulness in Islamic morality that simultaneously forbids the waste of resources.

Analysis of differential item functioning in agricultural science examination across southwestern nigeria's senior schools

Differential Item Functioning poses a threat to test fairness and validity in educational assessments. To this end, this study probes the incidence of DIF in the Senior School Certificate Agricultural Science Examination (SSCASE) across school locations in Southwestern Nigeria. It determined the magnitude of DIF using item parameters in SSCASE. The study applies one, two and three-parameter logistic models to analyze the dichotomous data responses of 818 randomly selected students from urban and rural locations. The ex-post facto design adopted the 2015 National Examination Council SSCE in Agricultural Science as its instrument (α =0.887) in this study. The study population comprised SSCE candidates from Osun, Ondo, and Oyo. The results showed the magnitude of DIF in SSCASE was large and moderate (1PLM=0.52 significant DIF, 2PLM=0.45 moderate DIF, 3PLM=0.47 moderate DIF) across school locations. The study concluded that 1, 2 and 3 PLMs can produce fair items in SSCASE.