Assessment Rubrics Research Articles

Enhancing instructional design learning: a comparative study of scaffolding by a 5E instructional model-informed artificial intelligence chatbot and a human teacher

ABSTRACT To address the limitations of general-purpose artificial intelligence (AI) tools, we developed a task-oriented AI chatbot based on the 5E (i.e. “engage”, “explore”, “explain”, “elaborate” and “evaluate”) model to scaffold students’ instructional design process. We examined the impact of integrating the 5E instructional model-informed AI chatbot on students’ learning performance and perceptions. The results indicated that the AI chatbot, when combined with human teacher scaffolding, significantly improved the students’ instructional design performance relative to receiving human teacher scaffolding only. The chatbot provided valuable suggestions on instructional design frameworks, class activities and teaching topics during the “explore” phase. In the “evaluate” phase, the chatbot offered immediate feedback on the students’ design plans and proposed alternative instructional frameworks regarding areas for improvement. However, the students expressed concerns about the chatbot’s evaluation quality, noting that it needed to be better aligned with the course assessment rubric. We recommend using AI chatbots for instructional design conceptualisation, although we emphasise the critical role of human teachers in evaluating final design work and providing timely support.

Implementasi Model Project Based Learning Menggunakan Mind Map terhadap Keterampilan 4C Geografi Siswa Kelas X Fase E SMAN 15 Padang

Geography, as a discipline that studies the interaction between humans and the environment, as well as physical phenomena on Earth, plays an important role in developing 4C skills (Communication, Collaboration, Critical Thinking, and Creativity). However, many students still face challenges in applying 4C skills in geography learning. These challenges include a lack of student participation, as seen from their minimal involvement in class discussions, which hinders the development of communication and collaboration skills. This issue is often caused by low self-confidence or insufficient understanding of the material. Additionally, students frequently struggle with critical thinking, particularly in identifying and analyzing complex geographical problems. The method used in this research is descriptive qualitative, with the subjects selected using purposive sampling, resulting in a sample of Class X Phase E 8. Data collection was carried out using assessment rubrics in terms of attitudes, knowledge, and skills to evaluate student activities in creating Mind Maps. The results of this study show that the assessment of Mind Mapping, using the Project-Based Learning model, yielded an overall average score of 81.72, categorized as good, and positively impacted the development of students' 4C skills. Students demonstrated creativity (C6) by creating Mind Maps, which reflect their ability to think critically and creatively, core aspects of Higher Order Thinking Skills (HOTs). Furthermore, they showed improved communication and collaboration abilities in learning, thus achieving the 4C skills.

A Framework for the Communication Capabilities of scientists

ABSTRACT The ability to communicate science effectively is a key attribute for science graduates, and considerable work has been done to establish the techniques and skills associated with effective communication. What is less clear however, is the learning progression students traverse as they develop proficiency in science communication. In this study we evaluated aspects of university-level student science communication capabilities pre- and post- explicit teaching of core science communication capabilities. We used this data to identify the characteristics of early science communication learners and synthesise a robust learning progression Framework for Communication Capabilities (FCC). The FCC was created to support educators in identifying the characteristics of their learners to facilitate the constructive alignment of learning objectives, curriculum, and assessment rubrics for learners of science communication through early phases of their development.

PELATIHAN PENULISAN KARYA TULIS ILMIAH DALAM MENINGKATKAN KEMAMPUAN MENULIS PIMPINAN DAN KADER PK IMM FTIK UIAD SINJAI

This research examines the effectiveness of scientific writing training in improving participants' writing skills. The training method is a direct practice approach with mentoring, intensive training, discussion, and reflection sessions. The instruments used to measure writing ability before and after training are scientific writing assessment rubrics and self-assessment questionnaires. The research results showed a significant increase in participants' writing abilities after attending the training. Before the training, most participants had limited writing skills, especially in writing structure, data presentation, and scientific argumentation. However, after the training, participants significantly improved written structure, argumentation clarity, and analysis depth. Apart from that, this training also helped participants to develop a critical attitude towards literature and hone their ability to integrate knowledge from various sources to support the argument. Participants also received positive responses regarding the interactive training methods and in-depth approach to scientific writing techniques. This research concludes that scientific writing training effectively improves participants' writing skills. Therefore, it is recommended that similar training focusing on various aspects of scientific writing be carried out to support the development of participants' academic and professional competencies.

Analytic Rubrics for the Practical Assessment of Students’ Performances in Engineering Survey Course

The use of analytic rubrics as an assessment tool has been an integral component of education. In Surveying courses, it is essential to evaluate the psychomotor skills under simulated conditions of actual practise. The main objective of this study is to determine the comparative and efficacy of practical assessment conducted at the diploma and degree levels in the Civil Engineering programme. In order to achieve the objective, implementing rubrics in the continuous assessment of the Surveying courses to facilitate a profound comprehension of physics, mathematics, and engineering principles needs to be executed. Previously, a practical assessment has been conducted in a traditional method which gauges the student performance according to the time records during certain practical tasks given resulting in inadequate assessment tools that failed to capture the psychomotor component of practical assessments effectively. As for methodology, a comparative analysis was conducted to examine the consistency and usefulness of analytic rubrics in assessing the course and program outcomes of Surveying courses at both diploma and degree levels. The analysis aimed to determine the impact of these rubrics on student performance. This comparison should be conducted in order to prevent redundant actions, as it pertains to diploma level students who will be integrated into the degree program. This assessment reveals that most students met all performance criteria at an acceptable level. This indicates that the analytical rubric used for the Practical Test in Surveying courses is adequate and effective as a grading instrument. As for the results, the comprehensive analysis indicates that a significant majority, ranging from 75% to 81%, exhibited commendable performance in achieving the criteria for various Programme Outcomes (PO), particularly in PO4, PO5, PO9, and PO10.

The Role of Elements of Text Analysis in Consecutive Interpreting Performance: A Mixed-Methods Study

Despite the theoretical importance of text analysis in translation process, there is a paucity of experimental research into the role that the elements of the text can play in enhancing interpreting performance. The present study, adopting a mixed-methods design, aimed to investigate whether knowledge of text elements enhances consecutive interpreting performance. In so doing, a sample of 36 homogeneous undergraduate translation students participated in this study based on convenience sampling, and a pool of six students participated in the qualitative phase of the study based on purposive sampling, following the data saturation method. Following Nord’s guidelines for text analysis, the experimental group was exposed to the elements of text during the intervention. A validated assessment rubric was utilized to measure students’ interpreting performance, taking the inter-rater reliability of the scores also into consideration. The results of one-way ANOVA confirmed that there was a statistically significant difference between the control group and the experimental group; the experimental group outperformed the control group. Having calculated the inter-coder reliability of the responses emerged from the interviews, the number of common factors included six themes, namely unique, new information, message analysis, motivating, general lexis, and semiotics. In the end, some practical implications are offered for interpreting students and teachers.

Preliminary Validation of a Clinical Reasoning Theory-Based Assessment Rubric: An e-Delphi Study.

Questions persist regarding the evaluation of cognitive processes related to clinical reasoning when resolving situations in a learning by concordance tool. This Delphi technique study aimed to validate a clinical reasoning assessment rubric based on script theory. Seventeen experts participated in the study. Two rounds of consultation were conducted to obtain a consensus on the accuracy and clarity of the rubric descriptors (clarity index and content validity index ≥ 0.9). The results inform future research procedures and the intended use of the rubric to facilitate evaluator inferences, provide student feedback, and support the development of learners' clinical reasoning.

Reliability of ChatGPT in automated essay scoring for dental undergraduate examinations

BackgroundThis study aimed to answer the research question: How reliable is ChatGPT in automated essay scoring (AES) for oral and maxillofacial surgery (OMS) examinations for dental undergraduate students compared to human assessors?MethodsSixty-nine undergraduate dental students participated in a closed-book examination comprising two essays at the National University of Singapore. Using pre-created assessment rubrics, three assessors independently performed manual essay scoring, while one separate assessor performed AES using ChatGPT (GPT-4). Data analyses were performed using the intraclass correlation coefficient and Cronbach's α to evaluate the reliability and inter-rater agreement of the test scores among all assessors. The mean scores of manual versus automated scoring were evaluated for similarity and correlations.ResultsA strong correlation was observed for Question 1 (r = 0.752–0.848, p < 0.001) and a moderate correlation was observed between AES and all manual scorers for Question 2 (r = 0.527–0.571, p < 0.001). Intraclass correlation coefficients of 0.794–0.858 indicated excellent inter-rater agreement, and Cronbach’s α of 0.881–0.932 indicated high reliability. For Question 1, the mean AES scores were similar to those for manual scoring (p > 0.05), and there was a strong correlation between AES and manual scores (r = 0.829, p < 0.001). For Question 2, AES scores were significantly lower than manual scores (p < 0.001), and there was a moderate correlation between AES and manual scores (r = 0.599, p < 0.001).ConclusionThis study shows the potential of ChatGPT for essay marking. However, an appropriate rubric design is essential for optimal reliability. With further validation, the ChatGPT has the potential to aid students in self-assessment or large-scale marking automated processes.

Podcast and Translation Learning as a Means of Developing Students' Reading, Speaking, and Listening Skills

The purpose of this study is to evaluate the efficacy of employing podcast content and translation to improve reading, listening, and speaking abilities. An analysis using a mix of methods was developed to address such a query. To address such an issue, a treatment has to be administered. In the tenth grade of high school, there were ten students. For the lesson, the students used books with podcasts and translation-method content. To improve students' reading and listening abilities, the teacher began using podcasts. The teacher used translation to make understanding easier. The teacher instructed the students in grammar by taking the text's message to heart. In order to improve students' speaking abilities, the teacher also taught retelling, explaining pictures, and discussion. Pre and post tests were administered as part of the treatment. An assessment rubric was utilized to evaluate the pre-test and post-test. Analysis was done on the pre-test and post-test results. It is possible to conclude from the analysis of this type of result that the use of podcasts and translation to improve reading and listening comprehension and speak more fluently is successful. This outcome can be used by the teachers to teach the pertinent subjects. It can also be developed for comparable applications by other researchers.

Implementation Of Differentiated Learning In Class Iv Fiqh Subjects Material On Procedures For Prayer In Sickness At Mi Muhammadiyah Semondo Gombong Kebumen

The research objectives are: (1) to find out how differentiated learning is implemented in the Merdeka Curriculum in terms of content, process, product and learning environment in class IV fiqh subjects at MI Muhammadiyah Semondo; (2) to determine the supporting and inhibiting factors faced in implementing differentiated learning. This research is in the form of Qualitative Research. The students studied were class IV MI Muhammadiyah Semondo with a total of 50 children. The method used in this research is a qualitative method using field research. The data collected was implementation data, supporting and inhibiting factors for differentiated learning in class IV fiqh subjects at MI Muhammadiyah Semondo. Data collection techniques in this research used observation, interviews and documentation. The data analysis used is inductive/qualitative data analysis. The data analysis techniques used are data collection, data condensation, data presentation, and drawing conclusions. Based on the research results, it can be concluded that: (1) implementing differentiated learning in class IV fiqh subjects at MI Muhammadiyah Semondo, the teacher begins by carrying out an initial assessment to determine the students' learning styles. Next, the teacher prepares learning steps that contain differentiation of content, processes, products and learning environments, which are in accordance with the needs and interests of students; (2) supporting factors for differentiated learning, namely the existence of a policy of implementing an independent curriculum, support from the madrasa head, and a supportive learning environment. While the inhibiting factors are managing time to be effective and difficulties in the assessment process, the solution is to make clear learning plans and detailed assessment rubrics.

Rethinking Assessment: Community-Engaged Experiential Learning and Complex Environments in Business from a Student Perspective

The COVID-19 pandemic has disrupted the academic landscape profoundly, prompting a shift toward new learning environments and evolving pedagogy, including a notable increase in experiential community-engaged learning (CEL) projects within business curricula. Despite these innovations, traditional assessment practices often neglect the development of critical skills, focusing instead on outdated knowledge certification. This study examines the integration of student voice in assessment practices, particularly within CEL projects. It aims to explore student perceptions of the opportunities provided by CEL, offering insights into specific elements that should be assessed to measure student success, accomplishment, and growth. Using a qualitative approach, data were collected through end-of-term reflections, focus groups, and interviews with 52 undergraduate business students engaged in CEL projects. Thematic analysis revealed three key themes that can impact the design, selection, implementation, and interpretation of assessment methods: Adaptation, Balancing Teamwork with Individual Accountability, and Creativity and Innovative Problem-Solving. Students emphasized the importance of flexibility and proactive problem-solving in dynamic project environments. They also valued effective collaboration and personal accountability, preferring individual assessments within team projects. Creativity and innovative solutions were highlighted as essential competencies, suggesting the need for assessment tools that recognize these skills. The research indicates that incorporating student voice in the assessment process enhances engagement and relevance, aligning evaluations with students' actual experiences while fostering essential life skills. The study recommends the development of assessment rubrics that measure these competencies along with longitudinal studies to examine the impact of revised assessment processes on student experience and satisfaction over time.

Learning strategy regulation processes with rubrics for learning assessment

Learning assessment can be done with the help of the development of rubrics. Rubrics are used to evaluate a wide variety of aspects of the pedagogical process; they are even very efficient for standardizing assessment and creating feedback. The adoption of rubrics helps establish educationally appropriate criteria and performance levels for both metrics and performance measurements, including complex learning structures such as the pedagogical framework. Regulated learning frameworks has come to be cornerstone for examining learning forms of regulation. Despite recent advances in the area of self-regulated and co-regulation, more research is still needed on how groups of students regulate their learning in learning environments in the classroom context. In this work we apply, we re-designed Pintrich’s self-regulation framework, placing a special emphasis on the crucial role of performance measurement in learning with the aid of rubrics, providing a more holistic view of regulatory processes. The extended framework presented here has implications for practice, being especially beneficial in creating strategies for facilitating students’ self-regulation and co-regulation. This study provides valuable information for educators regarding instructional design and the selection of appropriate regulatory processes to shape and measure learning activities within a framework. It was developed a complete prototype called the Learning Assessment System by Rubrics.

What matters for competent teaching? A multinational comparison of teaching practicum assessment rubrics

Practicum assessment rubrics have a backwash effect on preservice teachers' learning through the criteria they transmit. This article presents a documentary analysis of ten rubrics used across six countries: South Africa, India, England, Singapore, Canada, and Sweden. We compare the dispositions, knowledge, outcomes, and reasoning. We use Legitimation Code Theory (LCT) to show how practicum assessments are legitimated differently. Some rubrics emphasise preservice teachers’ dispositions and whether they implement protocols correctly. Others emphasise their capacity for reasoning in context. These positions call for teacher educators and policymakers to interrogate where the emphasis is in their own assessments.

The Effect of Picture Word Card Media on Beginning Reading Ability at Grade II Students Of SDN 09 Koto Baru

Education The problem with learning Indonesian is students' lack of interest in reading. The use of media in learning is also inappropriate. Through this problem, the aim of this research is to overcome the low initial reading ability, so interesting and appropriate media is needed so that students can easily understand the material. The researcher will apply picture word card media. The type of research used is experiment, Pre Experimental Design. The population in this study was all class II students totaling 24 students. The development of the instrument used is a reading test in accordance with the initial reading assessment rubric. The data collection technique used was an initial test (pre-test) and a final test (post-test). The data analysis technique for this research uses the Normality Test and hypothesis testing. Based on the results of the Wilxocon test which obtained a significance of 0.000&lt;0.05, it can be concluded that there is an influence of picture word card media on the initial reading ability of class II students at SDN 09 Koto Baru.

Development and Evaluation of a Surgical Simulator and Assessment Rubric for Standing Castration of the Horse

In veterinary education, simulators are models or devices that can imitate a real patient or scenario and allow students to practice skills without the need for live patients. Castration is a common surgical procedure in all species, and the standing, open technique is frequently performed in horses. Although a simulator has been developed for equine closed castration, a simulator for standing castration in the horse has not yet been described. This two-part study focused on the design, creation, and evaluation of a simulator for teaching standing castration in the horse. A low-technology simulator was created using molded silicone testicles, cohesive bandage, stockings, and socks. A rubric was created for assessing performance using the simulator. Participants were recruited from three groups: university academic staff members ( n = 12, majority equine veterinarians), equine veterinarians working in private practice ( n = 9), and final-year veterinary students ( n = 28). Each group tested the simulator while being graded using the developed rubric, and participants completed an anonymous online feedback questionnaire. Feedback was positive overall, with 98% of respondents ( n = 48/49) stating that the model would be a useful addition to the veterinary curriculum. Furthermore, 100% of students reported that using the simulator increased their confidence in performing standing castration in horses. Evaluation of the model included assessment of responses from veterinarians and students regarding realism and usefulness of the simulator, comparison of rubric scores between veterinarians and students, and assessment of the reliability of the rubric. Median student rubric score was significantly lower than qualified veterinarians ( p &lt; .001), and Cronbach's alpha demonstrated that there was adequate internal reliability in rubric scoring (α = .85). It was determined that the simulator is effective for teaching the steps of the surgical procedure and for increasing student confidence.

A Snapshot of Introductory Pharmacy Practice Experience Program Characteristics

ObjectiveTo provide schools and colleges of pharmacy a snapshot of Introductory Pharmacy Practice Experience (IPPE) program characteristics to generate and share ideas for IPPE program modifications to improve student learning outcomes and comply with Standards 2025. MethodsA 41-item electronic survey was distributed to 140 Accreditation Council for Pharmacy Education fully accredited schools and colleges of pharmacy in 2023. IPPE program characteristics such as structure, learning activities, assessments, and requirements were described. ResultsEighty-seven programs participated in the survey (62.1% response rate). Notable findings included 64.4% of programs starting IPPEs during the first professional year, 78.6% did not count simulation hours toward Accreditation Council for Pharmacy Education hour requirements, 24.1% incorporated patient care electives, over 90% incorporated various patient care activities, 40.2% exceeded a 2:1 student-to-preceptor ratio, 15.7% incorporated layered learning, 50.0% incorporated interprofessional education, 57.5% used summative assessment rubrics based on Entrustable Professional Activities, and 77.0% used pass/fail grading. ConclusionOwing to evolving accreditation standards, the study provided valuable information about the current state of IPPEs. The study results included examples and components that programs can use to ensure that they comply with Standards 2025.

Implementation of Differentiated Learning in Identifying Sentence Structures for Fifth Grade Elementary School Students: Analysis of Obstacles and Implementation Solutions

Differentiated learning in fifth-grade elementary schools in the Untung Surapati cluster, East Denpasar District, has not yet been maximized. Teachers face challenges in implementing differentiated learning when teaching sentence structure. This issue needs to be examined to determine solutions that can be used in sentence structure teaching. The methods used for data collection include observation, interviews, and document analysis. The challenges faced by teachers include: 1) limited teaching staff. To address this, students are divided into several groups. 2) Difficulty understanding the concept of sentence structure. To overcome this difficulty, teachers can use the picture-and-picture method with the assistance of keywords. 3) Limited teaching time. This can be addressed by collaborating with fellow teachers in developing teaching materials. 4) With a large number of students, implementing cooperative learning can be a solution to handling large classes. 5) Teachers' skills in implementing differentiated learning are not yet optimal. Teachers are given opportunities to attend training sessions. 6) Each student has different needs, interests, and skill levels. To address this, teachers create flexible assessment rubrics. 7) Teachers use various teaching approaches to meet the diverse needs of students. Adjusting assessments to different teaching approaches becomes a challenge for teachers. Teachers should create assessment instruments based on the methods applied in teaching.

Machine learning approaches for influenza A virus risk assessment identifies predictive correlates using ferret model in vivo data

In vivo assessments of influenza A virus (IAV) pathogenicity and transmissibility in ferrets represent a crucial component of many pandemic risk assessment rubrics, but few systematic efforts to identify which data from in vivo experimentation are most useful for predicting pathogenesis and transmission outcomes have been conducted. To this aim, we aggregated viral and molecular data from 125 contemporary IAV (H1, H2, H3, H5, H7, and H9 subtypes) evaluated in ferrets under a consistent protocol. Three overarching predictive classification outcomes (lethality, morbidity, transmissibility) were constructed using machine learning (ML) techniques, employing datasets emphasizing virological and clinical parameters from inoculated ferrets, limited to viral sequence-based information, or combining both data types. Among 11 different ML algorithms tested and assessed, gradient boosting machines and random forest algorithms yielded the highest performance, with models for lethality and transmission consistently better performing than models predicting morbidity. Comparisons of feature selection among models was performed, and highest performing models were validated with results from external risk assessment studies. Our findings show that ML algorithms can be used to summarize complex in vivo experimental work into succinct summaries that inform and enhance risk assessment criteria for pandemic preparedness that take in vivo data into account.

EKRANISASI TEKS EKSPLANASI MENJADI GENRE FILM MELALUI MODEL PROJECT BASED LEARNING DAN PENGARUHNYA PADA KEMAMPUAN BERPIKIR KREATIF SISWA KELAS VIII SMP PRIBADI BANDUNG

Students experience the degradation of digital technology. Students are becoming more proficient in operating devices and the applications they possess. Students of this generation need to be given stimuli to transform an explanatory text into another genre to make it more interesting and encourage creative thinking. This issue is observed among eighth-grade students at SMP Pribadi Bandung. The aim of this research is to describe ability to screenize (adapt into visual or multimedia form) explanatory texts on creative thinking through the project-based learning model among eighth-grade students at SMP Pribadi Bandung. The research method used is a quasi-experimental research method. Data collection techniques involve mixed methods (quantitative and qualitative). The results of the study show that eighth-grade students at SMP Pribadi Bandung were able to screenize explanatory texts into film genres based on the assessment rubric. The highest score obtained was 96, and the lowest score was 87, with an average score of 92. The students' ability to screenize explanatory texts into film genres is excellent. The students' ability to meet the structural and language rules in the films produced through the project-based learning model in eighth grade at SMP Pribadi Bandung achieved the highest post-test score of 100, with the lowest score being 80, and an average score of 89.9. The creative thinking ability of the students in the films they produced through the project-based learning model in eighth grade at SMP Pribadi Bandung is excellent. Out of 39 students, 36 students were categorized as capable of creative thinking, while three students were not yet able to think creatively. There is a very strong influence of the ability to screenize explanatory texts on creative thinking through the project-based learning model among eighth-grade students at SMP Pribadi Bandung, as confirmed by the hypothesis test.

NAVIGATING THE PERCEPTION-PRACTICE DILEMMA: AN INQUIRY INTO ASSESSORS’ IMPLEMENTATION OF ORAL PRESENTATION RUBRICS

Background and Purpose: Various factors affect the assessor’s ability to assess an oral presentation and arrive at an accurate assessment of the interconnected skills such as fluency, coherence, syntax, grammar, diction, and task achievement concurrently. The oral presentation rubric aids in achieving consistency in assessing this complex skill. This research investigates assessors’ application and perceptions of the speaking assessment rubric prepared in-house and used in the Foundation Program (FP) English Unit at Dhofar University (DU) for levels 1, 2, and 3 in order to identify the rubric’s possible weakness and ambiguities to provide insights to plug the loopholes and ensure fair, objective and uniform assessment. Methodology: Data was collected in a 3-phase data collection cycle through a pre-session task (n=18), a hands-on focus groups session (n=16) to explore qualitative data regarding actual application of the rubric, and a questionnaire (n=40) to obtain quantitative data for further investigation of teachers’ perceptions to cross check with implementation. This makes the present study a mixed-methods one in an exploratory sequential design. Findings: The findings of this research clearly showed the loopholes in the current oral presentation rubric used by the FP for formative and summative assessments. This research has exposed a notable inconsistency in grading practices of the oral presentation using the in-house rubric despite the surprising finding regarding assessors’ confidence in the rubric’s reliability and their level of training. The identified gap between assessors’ perceptions and the application of rubric criteria highlights potential issues in the design of the rubric leading to inconsistencies in the assessment outcomes. Contributions: This research emphasizes the importance of the regular refinement of assessment rubrics to minimize ambiguity and subjective elements, coupled with comprehensive user training to enhance assessment practices, support professional development, promote clearer learning expectations for students, and mitigate the impact of grading disparities in evaluation processes. Keywords: Rubric, oral presentation, assessment, discrepancy, raters. Cite as: Veetil, M., Iqbal, R., &amp; Abugohar, M. (2024). Navigating the perception-practice dilemma: An inquiry into assessors’ implementation of oral presentation rubrics. Journal of Nusantara Studies, 9(2), 267-296. http://dx.doi.org/10.24200/jonus.vol9iss2pp267-296