Educational Assessment Research Articles

Developing and validating the construct maps to assess mathematical proficiencies

The study was aimed to develop and validate a measurement test to assess mathematical proficiencies in the numbers and algebra strand. A total of 125 seventh-grade learners with diverse learning abilities from secondary educational institutions under the management of the Khon Kaen Educational Service Area 25, Thailand was chosen as respondents. The researchers employed design-based research consisting of four building blocks to construct a prototype utilizing a construct modeling approach. Firstly, the researchers developed construct maps to identify the learners’ mathematical proficiency (MP) level. This was followed by the creation of the items of measurement test according to the proficiency levels. Next, the researchers allocated scoring measures and formed the conversion of learners’ MP stages. Lastly, the researchers validated the superiority of the measurement test through Wright map consuming the multi-dimensional random coefficient multinomial logit model. The construct maps of the MP level consisted of two dimensions, namely mathematical procedures and structure of learning outcome. Findings discovered that there were 20 items in the assessment tool and its quality passed the determined education and psychological assessment criteria. It can be determined that every item is capable to measure the learners’ multi-dimensional mathematical proficiencies.

Endoscopic sleeve gastroplasty: the identification of the key procedural steps through a modified Delphi method

BackgroundEndoscopic sleeve gastroplasty (ESG) is an innovative, minimally invasive bariatric procedure with an excellent safety and efficacy profile in adults with obesity. The purpose of the procedure is to shorten and tubularize the stomach along its greater curvature. Nevertheless, there are some heterogeneities in the approach to ESG, which will be important to address as the procedure sees increasingly widespread clinical adoption. Here, an expert consensus on standardized ESG techniques is presented. MethodsThe modified Delphi method was used to establish the key procedural steps of an ESG. A panel of 8 experts was selected, of which 6 participated. The panel was selected based on their experience with performing the procedure and consisted of 1 bariatric surgeon and 5 interventional gastroenterologists. A neutral facilitator was designated and produced a skeletonized initial version of the key steps that was sent to each expert. Each survey began with the experts rating the given steps on a Likert scale of 1 to 5, with 1 being the most inaccurate and 5 being the most accurate. Furthermore, the final product was rated. The survey continued with open-ended questions designed to revise and polish the key steps. Areas of discrepancy were addressed using binary questions and a majority vote. The respondents were given 10 days to complete each survey. At the end of each round, the survey was redistributed with updated key steps and questions. This process was continued for a predesignated 3 rounds. ResultsOf the 8 experts who were queried, 6/8, 5/8, and 5/8 replied to each round. The given ratings for the accuracy of the steps in each round were 4.2, 4.6, and 4.4. The final rating was 4.8. Although expert opinion varied around smaller portions of the procedure, such as the placement of an overtube and the shape of each suture line, there was consensus on the need for full-thickness bites and appropriate swirling of the tissue with the helix device. Whether or not to include the fundus in the gastroplasty was an additional area of discrepancy. Of note, 4 of 5 experts agreed that the fundus should remain intact. The final protocol consisted of 21 steps curated from the summarized responses of the experts. ConclusionUsing the modified Delphi method, 21 key steps have been described for a safe and effective ESG. This rubric will be standardized across institutions and practitioners. Furthermore, these findings allow for the generation of educational assessment tools to facilitate training and increase the adoption of ESG by endoscopists.

Higher education assessment practice in the era of generative AI tools

Higher education assessment practice in the era of generative AI tools

The Teacher Challenges in Implementing Educational Evaluation Assessment Standards in the Affective Domain at Pondok Mutiara Quran Putri Wonogiri

Teacher challenges in implementing educational evaluation assessment standards in the affective domain is an important issue that needs attention. This study aims to determine how teachers implement educational evaluation standards in the affective realm at Pondok Mutiara Quran Putri Wonogiri. This research method uses qualitative descriptive analysis with a phenomenological approach so as to produce data conclusions to answer problems related to this title. The results of this study show that some of the difficulties faced by teachers in implementing educational evaluation assessment standards. Efforts to improve the implementation of affective education evaluation assessment standards include comprehensive approaches, such as teacher skill development, awareness raising about authentic assessment, and training on data collection techniques

Exploring Equity in Existing Assessment Practices for Visually Impaired Students at Higher Education Level

ABSTRACT The educational assessment procedure provides insight into the achievement and competency of visually impaired students to their educators. This study aimed to promote equity in assessment procedures for visually impaired students at higher education levels and the ways of executing inequity. A qualitative approach with an explanatory nature was used for this study. Participants of study were visually impaired students (N=20) from higher education levels. A self-developed semi-structured interview protocol was administered for the data collection process. The instrument’s validity was confirmed through the expert opinion (N=02). Instrument’s reliability was assured through an extensive literature review. Five themes emerged from the interviews of the participants i.e., perception of visually impaired students, institutes’ support, barriers for visually impaired students, effects on career development, and suitable ways for educational assessment. This study in its findings revealed that the existing assessment procedure is quite challenging for them. This study gives its recommendations to Higher education institutes should provide knowledge and orientation about assessment policy and procedure to the faculty and non-teaching staff. The assessment reflects the ultimate learning for the students at higher education level including visually impaired students. A need is always to conduct assessments for both students with and without visual impairment as per their access level. For this reason, modification in the assessment procedure for visually impaired students is mandatory for promoting equity in the procedures of assessment for visually impaired students. Dawadi et al., (2024) suggested pedagogical implications for the future learning environment through the promotion of equity in assessment. Design/Methodology/Approach: This study was qualitative and exploratory. A self-developed semi-structured interview was used as an instrument for this study. Findings: The findings revealed that visually impaired students perceived existing assessment procedures as quite difficult for them. The results revealed that visually impaired students are only allowed to get permission from their institutes to avail writer. The findings revealed that the writers’ mistakes and non-seriousness of visually impaired students affect the career development of the visually impaired students.

The Use of Linear Regression Equation in Physical Education Teaching in Higher Education Institutions

This study investigates the application of linear regression equations in physical education teaching within higher education institutions, utilizing the Integrated Linear Logistic Regression AdaBoost Classifier (ILLR-ABC). The research aims to enhance teaching methodologies and performance assessment in physical education through the integration of statistical modelling techniques. Through simulated experiments and empirical validations, the effectiveness of the ILLR-ABC model is evaluated in predicting student performance and guiding instructional interventions. Results demonstrate significant improvements in accuracy and precision compared to traditional teaching methods. For instance, the ILLR-ABC model achieved an average accuracy rate of 90% in predicting student achievement levels, enabling educators to tailor teaching strategies to individual student needs effectively. Additionally, the model provides valuable insights into the factors influencing student performance, facilitating targeted interventions and curriculum adjustments. These findings highlight the potential of integrating statistical modelling techniques like ILLR-ABC to optimize physical education teaching practices and enhance student learning outcomes.

Optimizing Educational Assessment: The Practicality of Computer Adaptive Testing (CAT) with an Item Response Theory (IRT) Approach

This research aims to develop a Computer Adaptive Test (CAT) system using the Items Response Theory (IRT) approach. This study is part of developing a web-based system using the Research and Development (R&D) method, employing the Four-D (4-D) model. At its core, this system is similar to a Computer-Based Test (CBT). Still, the critical difference lies in its ability to randomize and provide questions that align with the test-taker's skill levels using the Items Response Theory (IRT) algorithm. The system employs the 3-PL model from the Items Response Theory, considering the difficulty level of questions, the discriminative power of questions, and the likelihood of guessing or interference in the questions. The examination system randomly assigns questions to students based on their responses to previous questions, ensuring that each test-taker receives a unique question sequence. The exam concludes when a test-taker accurately estimates their ability, i.e., SE <= 0.01, or when all questions have been answered. The outcome of this research is a Computer Adaptive Test (CAT) system based on the Items Response Theory (IRT), which can be used to assess students' learning outcomes. This research was implemented in the Multimedia Department of SMK Negeri 1 Gunung Talang, with 90 students as the research sample. The evaluation of the practicality of this system received very high scores, indicating that the Computer Adaptive Test (CAT) system based on the Items Response Theory (IRT) is considered highly practical and effective in achieving the established measurement goals.

Exploring teachers' (future) digital assessment practices in higher education: Instrument and model development

AbstractDigital technologies are increasingly used in assessment. On the one hand, this use offers opportunities for teachers to practice assessment more effectively, and on the other hand, it brings challenges to the design of pedagogically sound and responsible digital assessment. There is a lack of validated instruments and models that explain, assess and support teachers' critical pedagogical practice of digital assessment. This explorative work first develops and validates a survey instrument to examine teachers' digital assessment practices. Secondly, we build a model to investigate to what extent teachers' pedagogical digital assessment knowledge is a foundation for the future of digital assessment (ie, authentic, accessible, automated, continuous and responsible). A total of 219 university teachers at a large European university participated in the survey study. Factor exploratory analysis and structural equation modelling were used to validate the reliability and validity of items and internal causal relations of factors. The results show the survey is a valid and reliable instrument for assessing teachers' digital assessment practice in higher education. Teachers' pedagogical knowledge and pedagogical content knowledge of digital assessment is critical, while teachers' technological pedagogical knowledge seems to have a more limited impact on the future of digital assessment.Practitioner notesWhat is already known about this topic Digital technologies are increasingly used in assessment in higher education. Teachers are the key stakeholders in the assessment for learning. There is a limited transformative nature of digital assessment practices. What this paper adds A validated instrument to assess teachers' pedagogical practice of digital assessment in higher education. Teachers' pedagogical and pedagogical content knowledge of digital assessment is critical to the future of digital assessment. Teachers' technological pedagogical knowledge of digital assessment has a more limited impact on the future of digital assessment. Implications for practice and policy Teachers need to be supported in developing pedagogical digital assessment practices in higher education. Based on this study's outcomes, educators, institutions and policymakers can inform the design and implementation of effective and responsible digital assessment practices. The design and implementation of responsible digital assessments will enhance the quality of higher education in the digital age.

Thread of Influence: Understanding the Dynamics Shaping Current Academic Performance in Faisalabad’s Matriculation Students through Sibship, Prior Academic Performance, and Parental Involvement

The study assessed the impact of sibship size, prior academic performance, and parental involvement on the current academic performance of matriculation students in urban schools in Faisalabad. A multistage sampling technique was used to select respondents. The study used a quantitative and cross-sectional research design. The questionnaire had items related to the study variables. The questionnaire was pretested on 30 students to assess its validity and reliability. The findings of the study revealed that prior academic performance and parental involvement significantly positively impact current academic performance. However, the study underscores the robust negative impact of sibship size on current academic performance. These findings offer significant new information for the field of educational assessment and evaluation. The study helps to focus student interventions and promote a comprehensive comprehension of the various factors that influence the current academic performance of high school students.

Development of e-worksheet based on search, solve, create, and share (SSCS) Islamic context to improve science process skills on excretory system material

Electronic student worksheets (e-worksheet) are needed as alternative teaching materials in learning. This study aims to produce teaching materials in the form of e-worksheet based on Search Solve Create and Share (SSCS) Islamic context to improve science process skills on class XI excretory system material that is worthy of the results of validity, responses of students and biology teachers. This research was conducted at MAN 1 Bogor. The research model used is a 4D model (Define, Design, Develop, Disseminate), but in research only to the development stage (Develop). The e-worksheet teaching materials developed are tested for validity. The results of validity tests by design and media experts obtained 89% in the very decent category, material expert assessments obtained 84% in the very feasible categories, and expert assessments of Islamic religious education obtained 100% in the very feasible categories. The results of student responses obtained percentages of 89% in limited trials and 89.7% in expanded trials. Based on the findings of the research results, it can be concluded that E e-worksheet based on Search Solve, Create and Share (SSCS), an Islamic context to improve science process skills on the excretory system material developed, is very feasible to be used and applied to biology learning.

Online assessment in higher education during the COVID-19

The COVID-19 pandemic has accelerated the digital transformation of education, prompting universities worldwide to swiftly transition to online and blended learning environments. The adaptation process to online education varies depending on several factors, including the level of digitalization and technological penetration in each country, the availability of technologies at universities, and the digital skills of students and staff. Additionally, the specific field of knowledge influences the adaptation process, with differences observed between disciplines such as health sciences and humanities. In this study, we conduct an exploratory qualitative analysis on how universities have responded, specifically focusing on the coordinators of Master’s programs in Spain in the specific area of assessment. This unprecedented situation has forced educators and institutions to rethink their assessment strategies and adapt to a new educational landscape. The insights gained from this study could provide valuable guidance for future crises and contribute to the ongoing discourse on the role of digital technologies in education in general, and in assessment in particular.

A Data-Driven Autonomous Assessment Framework for Education Quality Based on Multiscale Deep Learning

Nowadays, computational intelligence-assisted autonomous assessment of education quality has become a more and more general concern in the area of smart education management. As education quality assessment is a complicated process with multiple heterogeneous factors, it remains challenging to make effective assessment using simple information modality and criteria. To deal with this issue, this paper introduces multiscale deep learning, and proposes a novel data-driven autonomous assessment framework for education quality. In particular, four aspects of heterogeneous indicators are selected as the basic indexes and a dilated convolutional neural network structure is formulated to perform multiscale feature extraction. Then, the structural equation is adopted to make multivariate characterization and output the final assessment results. At last, some simulations are carried out on realistic education operation data to evaluate the efficiency of the proposed autonomous assessment framework. Two aspects of findings can be deduced from the results. For one thing, multisource fusion of multiple indicators well makes sense to autonomous education assessment. For another, multiscale deep learning can provide some beneficial promotion to assessment efficiency.

Assessment of education in a community hospital on healthcare providers' knowledge of and attitudes toward medical marijuana

IntroductionAlthough medical marijuana usage continues to become more socially acceptable and is increasingly regarded as a legitimate treatment alternative for certain medical conditions such as cancer and epilepsy, stigma remains. MethodsThis study examined whether offering an educational session addressing several key aspects of medical marijuana (e.g., pharmacology, legality) to healthcare providers in a community hospital setting improved knowledge about and/or altered attitudes toward marijuana use. This information was collected through administration of pre- and post-education session surveys. Multiple choice questions were utilized to assess knowledge of marijuana pharmacotherapy, pharmacodynamic interactions, and regulations, whereas descriptive analyses via Likert-scale questions determined attitudes toward medical marijuana. A secondary outcome entailed analysis of drug interactions with marijuana users via retrospective chart review. ResultsA total of 43 healthcare providers participated in the study. There were statistically significant increases in healthcare provider knowledge post-education session across multiple occupations (physician, P < .01; nurse, P < .001; pharmacist, P < .01; and nurse technician, P < .05). A total of 72 patients (83%) who self-reported marijuana use for medical or recreational purposes had a potential drug-drug interaction with marijuana. ConclusionsAs both medical and recreational marijuana continue to become legalized across the United States, formal education surrounding marijuana use and laws becomes increasingly important for healthcare providers and can be highly effective in preventing misinformation.

Artificial intelligence for education and its emphasis on assessment and adversity quotient: a review

PurposeThe purpose of this study was to analyze importance of artificial intelligence (AI) in education and its emphasis on assessment and adversity quotient (AQ).Design/methodology/approachThe study utilizes a systematic literature review of over 141 journal papers and psychometric tests to evaluate AQ. Thematic analysis of quantitative and qualitative studies explores domains of AI in education.FindingsResults suggest that assessing the AQ of students with the help of AI techniques is necessary. Education is a vital tool to develop and improve natural intelligence, and this survey presents the discourse use of AI techniques and behavioral strategies in the education sector of the recent era. The study proposes a conceptual framework of AQ with the help of assessment style for higher education undergraduates.Originality/valueResearch on AQ evaluation in the Indian context is still emerging, presenting a potential avenue for future research. Investigating the relationship between AQ and academic performance among Indian students is a crucial area of research. This can provide insights into the role of AQ in academic motivation, persistence and success in different academic disciplines and levels of education. AQ evaluation offers valuable insights into how individuals deal with and overcome challenges. The findings of this study have implications for higher education institutions to prepare for future challenges and better equip students with necessary skills for success. The papers reviewed related to AI for education opens research opportunities in the field of psychometrics, educational assessment and the evaluation of AQ.

Comparison of OBGYN postgraduate curricula and assessment methods between Canada and the Netherlands: an auto-ethnographic study.

Although the Dutch and the Canadian postgraduate Obstetrics and Gynecology (OBGYN) medical education systems are similar in their foundations [programmatic assessment, competency based, involving CanMED roles and EPAs (entrustable professional activities)] and comparable in healthcare outcome, their program structures and assessment methods considerably differ. We compared both countries' postgraduate educational blueprints and used an auto-ethnographic method to gain insight in the effects of training program structure and assessment methods on how trainees work. The research questions for this study are as follows: what are the differences in program structure and assessment program in Obstetrics and Gynecology postgraduate medical education in the Netherlands and Canada? And how does this impact the advancement to higher competency for the postgraduate trainee? We found four main differences. The first two differences are the duration of training and the number of EPAs defined in the curricula. However, the most significant difference is the way EPAs are entrusted. In Canada, supervision is given regardless of EPA competence, whereas in the Netherlands, being competent means being entrusted, resulting in meaningful and practical independence in the workplace. Another difference is that Canadian OBGYN trainees have to pass a summative written and oral exit examination. This difference in the assessment program is largely explained by cultural and legal aspects of postgraduate training, leading to differences in licensing practice. Despite the fact that programmatic assessment is the foundation for assessment in medical education in both Canada and the Netherlands, the significance of entrustment differs. Trainees struggle to differentiate between formative and summative assessments. The trainees experience both formative and summative forms of assessment as a judgement of their competence and progress. Based on this auto-ethnographic study, the potential for further harmonization of the OBGYN PGME in Canada and the Netherlands remains limited.

Creativity Self Assessments in Design Education: A Systematic Review

In the context of design education, the integration of Creativity Self Assessments (CSAs) is pivotal, offering a mechanism for reflective learning that facilitates the development and evaluation of students'’ creative capabilities. Despite recognising the value of CSAs in design education, current research lacks coherence and consolidation in their application within the domain. This systematic review aims to provide a thorough analysis of the application of CSAs within the design education landscape. Through systematic search and meticulous screening, ten CSAs from nine eligible studies were identified. The review classifies these assessments into four distinct types, using the 4 P's framework: product, person, process, and press (environment), and establishes a nuanced relationship between CSAs and creative thinking skills. Furthermore, this review highlights existing research limitations and proposes potential future research directions. These directions underscore the potential necessity for the development of press-based CSAs, and advocate for the employment of CSAs in pre-post reflective learning in improving creative thinking skills, The review provides actionable insights for design educators, encouraging the adoption of CSAs to measure changes in creative thinking skills and foster a reflective learning environment.

Unlikely allies: ChatGPT and higher education assessment

In the context of higher education, the emergence and spread of generative AI, including Large Language Models (LLMs), is revolutionising our relationship with knowledge. From spell-check and formatting tasks to rapid literature reviews, all forms of written assessments and even assessment feedback can be produced at the click of a button. We could understandably marvel at this new technology as an equaliser (at least, for those who have access to it). Second-language learners can write with native fluency. Non-traditional learners can overcome casual discrimination and academic elitism. Working class and mature learners can transition back into higher education. All this without recourse to pernicious ‘essay mills’. Yet, what of assessment? If Generative AI writes submissions for students and then marks them for lecturers, is the assessment dead? If so, what will become of learning developers - are their roles also dead, joining both assessment and the lecture in the academic afterlife? In this opinion piece we acknowledge those concerns but balance them with a message of hope. Generative AI and learning development may seem unlikely allies but there is still time to forge an alliance in the name of impactful learning and we conclude with three recommendations on how this might be achieved.

The use of a project management tool in distance education to enhance students’ engagement in group work

Group work is a widely embraced learning and assessment method that holds considerable importance in both in-person and online settings in higher education. However, group assignments introduce distinctive challenges, including unequal contributions, team conflict and divergent work ethics, leading to a pervasive reluctance among students to engage in university-level group work. Despite these concerns, group work continues to be a vital assessment approach for students, equipping them with essential skills for professional environments that prioritise teamwork and collaboration. In this opinion piece, I explore students' reluctance to engage in group work and introduce strategies and tools to transform unproductive group work into a collaborative learning experience. The discussion outlines implications for academics and learning developers, contributing to the ongoing discourse on collaborative tools in higher education assessments. With a primary focus on the challenges faced by distance-learning students, the opinion piece delves into the potential of Trello, a collaborative project management tool, to address these challenges and enhance collaborative learning in distance education. Its purpose is threefold: (1) enhance academics’ understanding of students’ reluctance to engage in group work, (2) enrich the experience of distance-learning students by highlighting the multifaceted benefits of collaborative learning in their educational journey that extend beyond traditional classroom boundaries and (3) propose innovative solutions to educators for fostering a culture of collaboration for more effective group work.

Generative artificial intelligence (AI) in higher education: a comprehensive review of challenges, opportunities, and implications

This paper explores recent advancements and implications of artificial intelligence (AI) technology, with a specific focus on Large Language Models (LLMs) like ChatGPT 3.5, within the realm of higher education. Through a review of the academic literature, this paper highlights the unprecedented growth of these models and their wide-reaching impact across various sectors. The discussion sheds light on the complex issues and potential benefits presented by LLMs, providing a overview of the field's current state. In the context of higher education, the paper explores the challenges and opportunities posed by LLMs. These include issues related to educational assessment, potential threats to academic integrity, privacy concerns, the propagation of misinformation, EDI aspects, copyright concerns and inherent biases within the models. While these challenges are multifaceted and significant, the paper emphasizes the availability of strategies to address them effectively and facilitate the successful adoption of LLMs in educational settings. Furthermore, the paper recognises the potential opportunities to transform higher education. It emphasises the need to update assessment policies, develop guidelines for staff and students, scaffold AI skills development, and find ways to leverage technology in the classroom. By proactively pursuing these steps, higher education institutions (HEIs) can harness the full potential of LLMs while managing their adoption responsibly. In conclusion, the paper urges HEIs to allocate resources to handle the adoption of LLMs effectively. This includes ensuring staff AI readiness and taking steps to modify their study programmes to align with the evolving educational landscape influenced by emerging technologies.

Implementing the Concept of Integration of Science and Islamic Values at Islamic University

The integration of scientific principles within Islamic religious universities governed by the Ministry of Religion is a widespread practice. However, the application of this concept across universities offering programs under the auspices of both the Ministry of Education and the Ministry of Religion remains unexplored. This study delineates the framework for integrating scientific knowledge at an Islamic University in Madura, focusing on the execution and assessment of education that amalgamates scientific insights and Islamic values. Employing observational techniques on policy documents and teaching resources of instructors, alongside semi-structured interviews with fifteen educators from the physics, biology, and mathematics departments, this investigation reveals a unique five-pillar knowledge integration concept. These pillars encompass Islam, Indonesia, Islamic Boarding School, Madura, and Aswajaan perspectives. The formal documentation supporting this integration at Madura Islamic University is encapsulated within its Statutes and chancellor's guidelines for curriculum enhancement. Evidence of implementation is observed in instructors' Syllabi and teaching materials, albeit only 53.3% have successfully linked scientific content with Qur'anic verses. The primary challenge identified is the considerable time required to develop materials that effectively integrate Islamic values. Solutions proposed include the creation of a comprehensive guide for integrating the five knowledge pillars and extensive training for all faculty members to facilitate this integration process.