Development Of Rubric Research Articles

Minding summaries: The development of a summary map rubric for an academic literacy intervention

Mind mapping or summary mapping has proved to be an effective and powerful tool for meaningful visual summarising in a structured way. However, to determine whether students have reached the set outcomes for summarising in this way, an objective measuring instrument is required. The assessment of summary maps is known to be a problematic activity: (i) students have difficulty to differentiate between essential and non-essential information; (ii) lecturers tend to rely too heavily on rewarding content and not the structure in which the content is presented; (iii) as a result of the interpretative nature of summary maps, there are many levels of subjectivity imbedded in the drawing, teaching and assessment process; (iv) the nature of the summarising activity calls for a need to improve inter-marker reliability; and (v) current assessment practices may result in a wasted opportunity for constructive feedback. In search of a suitable marking rubric, several examples are available in the literature to assess summary mapping. These, however, proved to still have a high level of subjectivity and are not appropriate for North-West University’s foundational academic literacy module. Consequently, an assessment tool for summary maps (the Summary Mapping Assessment Rubric Tool—SMART) was developed that addresses typical difficulties that markers experience. The newly developed tool proves to fulfil our most important and immediate needs with regard to the fair and effective assessment of summary maps. Keywords: academic literacy, summary maps, assessment, rubric, scoring consistency, feedback

Development and Initial Validation of the Early Elementary Writing Rubric to Inform Instruction for Kindergarten and First-Grade Students

This article describes the development of the Early Elementary Writing Rubric (EEWR), an analytic assessment designed to measure kindergarten and first-grade writing and inform educators’ instruction. Crocker and Algina’s (1986) approach to instrument development and validation was used as a guide to create and refine the writing measure. Study 1 describes the development of the 10-item measure (response scale ranges from 0 = Beginning of Kindergarten to 5 = End of First Grade). Educators participated in focus groups, expert panel review, cognitive interviews, and pretesting as part of the instrument development process. Study 2 evaluates measurement quality in terms of score reliability and validity. Data from writing samples produced by 634 students in kindergarten and first-grade classrooms were collected during pilot testing. An exploratory factor analysis was conducted to evaluate the psychometric properties of the EEWR. A one-factor model fit the data for all writing genres and all scoring elements were retained with loadings ranging from 0.49 to 0.92. Internal consistency reliability was high and ranged from .89 to .91. Interrater reliability between the researcher and participants varied from poor to good and means ranged from 52% to 72%. First-grade students received higher scores than kindergartners on all 10 scoring elements. The EEWR holds promise as an acceptable, useful, and psychometrically sound measure of early writing. Further iterative development is needed to fully investigate its ability to accurately identify the present level of student performance and to determine sensitivity to developmental and instruction gains.

Profil Kesiapan dan Pemahaman Guru Biologi terhadap Pelaksanaan Asesmen Psikomotorik dan Afektif Selama Pembelajaran Online di SMA Seluruh Kecamatan Comal, Pemalang

<p class="Default"><em><span lang="IN">Assessment is part of a teacher's pedagogic competence and curriculum-13 must carry out cognitive, psychomotor and affective assessments. Online learning causes the implementation of psychomotor and affective assessments to be hampered. This study aims to determine the readiness and understanding of Biology teachers on psychomotor and affective assessments during online learning. The method used is qualitative with descriptive analysis. The sampling technique is a saturated sampling technique. The subject of the study, namely the Biology teacher of the SMA in Comal District, opened 5 teachers through filling out questionnaires distributed online on google forms and RPP documentation as well as evidence of the implementation of the assessment, while the student questionnaires were used as secondary data. Data analysis using descriptive analysis consists of reduction, data display, and verification. This study resulted in the finding that the teacher's assessment readiness profile during online learning can be declared ready to carry out psychomotor and affective assessments. Meanwhile, the profile of teachers' understanding of psychomotor and affective assessments can generally be obtained at a low level. An understanding profile that has an impact on misconceptions in the realm of assessment and the development of an assessment rubric that is not yet good so that human application cannot run optimally. In addition, the application of psychomotor and affective assessments during online learning is influenced by the factors of the teacher's IT ability and the facilities owned by the students.</span></em></p>

Formative Assessment of Social-Emotional Skills Using Rubrics: A Review of Knowns and Unknowns

Educational practitioners have been increasingly interested in the use of formative assessment and rubrics to develop social-emotional skills in children and adolescents. Although social-emotional rubrics are nowadays commonly used, a thorough evaluation of their psychometric properties has not been conducted. In this scoping review, we examine the knowns and unknowns of the use of formative assessment approaches and rubrics in social-emotional learning. We first review initiatives of formative assessment and development of rubrics to assess social-emotional skills. Then, we discuss challenges associated with the development and use of rubrics to evaluate social-emotional skills in youth focusing on 1) assessment of single skills versus assessment of a comprehensive taxonomy of skills; 2) developing rubrics’ performance level descriptions that accurately describe increasing mastery of skills; 3) obtaining adequate internal consistency and discriminant validity evidence; 4) self-reports versus observer reports of skills; and finally 5) how to account for adolescents’ development in the construction of rubrics. This review outlines a research agenda for the psychometric study of rubrics to be used in social-emotional skill assessment.

Development of the Proposed 13 Entrustable Professional Activities for Physician Assistant Graduates.

Competency-based training has not consistently produced medical professionals who are ready to practice. To address this situation, entrustable professional activities (EPAs) have been introduced as a means to assess medical trainee readiness for their next phase of training or practice. Thirteen EPAs were developed by the Physician Assistant Education Association (PAEA) Presidents Commission in 2016 to indicate what activities they thought future physician assistant (PA) students should be expected to perform at the completion of their PA education. The 13 EPAs as proposed by the PAEA Presidents Commission were adopted by the Mayo Clinic PA program and further developed through multiple steps, including (1) Discovery and Alignment; (2) EPA Development; (3) Curriculum Development; (4) Assessment and Rubric Development, and (5) Implementation.

Enhancing assessment literacies through development of quality rubrics using a Triad based peer review process

Enhancing assessment literacies through development of quality rubrics using a Triad based peer review process

Development of Rubrics for Dental Hygienists' Competence: Based Dental Hygiene Process

Objectives: This study aims to secure the basic data for developing a rubric as an assessment tool to be applied to or used for the dental hygiene curriculum and develop an assessment rubric by steps of ‘dental hygiene management process’. Methods: The rubric was developed after literature review based on dental hygienists Competence and the dental hygiene process. Results: The development of the ‘dental hygiene management process’ assessment rubric was conducted to allow learners to spontaneously determine the direction of their own learning, understand their learning levels and reach their academic achievement levels, based on the performance standard and level of dental hygienists tasks. Conclusions: The ‘dental hygiene management process’ assessment rubric can motivate learners to receive fair evaluation and feedback about their learning levels and enhance their academic achievement, and would be used as a reliable assessment tool by securing the equity and validity in the evaluation of ‘dental hygiene management process’.

On the Cover: Having Fun with Bubbles

Every time we celebrated Easter together, my friend would hide eggs filled with candy and special coins for the kids to have fun egg-hunting. After all the eggs had been found—sometimes with kind help from those who had done the hiding work—we would enjoy seeing the kids blow bubbles. They never tire of having fun with bubbles. It turns out psychometricians can have fun with bubbles too, although gaining insights is really the main purpose. Digital assessments allow for the development of Technology-Enhanced Items (TEIs) that create richer interactions with content, often resulting in response patterns beyond conventional multiple-choice items. Traditionally, test developers specify scoring rubrics that map response patterns into item scores (e.g., 1,2,3). Scoring rubrics are often self-evident for multiple-choice items and simple TEIs. However, for items eliciting large numbers of response patterns (some unanticipated), developing scoring rubrics based on expert judgment alone (absent data) can be challenging. Bubble plots are particularly helpful in informing scoring rubric development and/or item revision. In bubble plots, each distinct observed response pattern is represented by a bubble; in this example, there are 306 bubbles - one for each observed response pattern. Bubble size is proportional to the number of test-takers providing the given response pattern. Bubbles are aligned along the x-axis according to the average EAP score of test-takers represented by the bubble. Bubbles are placed on the y-axis according to the item score awarded to that specific response pattern, determined by the scoring rubric. Average ability estimates across all observed response patterns for each item score are on the y-axis in parentheses. For visual ease, all responses earning the same item score have the same color (e.g., all green bubble responses receive full credit). EAP is used as the ability estimate, but other estimates such as raw test score could be used as well. Bubble plots provide an intuitive way to connect psychometric properties of items with response patterns. For example, test takers with high ability tend to score higher on an item. But bubbles spanning a wide ability range at a given item score level might suggest an item has low power to differentiate among test takers at different ability levels. Such observations can sharpen focus on how specific responses are handled in scoring rubrics. Interested readers may contact Edward Kulick (ekulick@ets.org) for questions on this graphic. For bubble plots in general, I would recommend both authors and readers pay attention to the bubble size that can be determined by either the diameter or the area. For kids who have fun with bubbles, they do not care about the difference between the diameter and the area—the larger the bubble is, the better. But for scientific presentation, it is crucial for the authors and readers to be on the same scale because doubling the diameter of a bubble (shown as a disk) is equivalent to quadrupling its area!

Development of a Formative Assessment Rubric for Peer Evaluation of Teaching (FARPET) and Pilot Use in Veterinary Online Teaching.

Peer evaluation of teaching (PET) serves an important role as a component of faculty development in the medical education field. With the emergence of COVID-19, the authors recognized the need for a flexible tool that could be used for a variety of lecture formats, including virtual instruction, and that could provide a framework for consistent and meaningful PET feedback. This teaching tip describes the creation and pilot use of a PET rubric, which includes six fixed core items (lesson structure, content organization, audiovisual facilitation, concept development, enthusiasm, and relevance) and items to be assessed separately for asynchronous lectures (cognitive engagement-asynchronous) and synchronous lectures (cognitive engagement-synchronous, discourse quality, collaborative learning, and check for understanding). The instrument packet comprises the rubric, instructions for use, definitions, and examples of each item, plus three training videos for users to compare with authors' consensus training scores; these serve as frame-of-reference training. The instrument was piloted among veterinary educators, and feedback was sought in a focus group setting. The instrument was well received, and training and use required a minimum time commitment. Inter-rater reliability within 1 Likert scale point (adjacent agreement) was assessed for each of the training videos, and consistency of scoring was demonstrated between focus group members using percent agreement (0.82, 0.85, 0.88) and between focus members and the authors' consensus training scores (all videos: 0.91). This instrument may serve as a helpful resource for institutions looking for a framework for PET. We intend to continually adjust the instrument in response to feedback from wider use.

Leveling the field: Development of reliable scoring rubrics for quantitative and qualitative medical education research abstracts.

Research abstracts are submitted for presentation at scientific conferences; however, criteria for judging abstracts are variable. We sought to develop two rigorous abstract scoring rubrics for education research submissions reporting (1) quantitative data and (2) qualitative data and then to collect validity evidence to support score interpretation. We used a modified Delphi method to achieve expert consensus for scoring rubric items to optimize content validity. Eight education research experts participated in two separate modified Delphi processes, one to generate quantitative research items and one for qualitative. Modifications were made between rounds based on item scores and expert feedback. Homogeneity of ratings in the Delphi process was calculated using Cronbach's alpha, with increasing homogeneity considered an indication of consensus. Rubrics were piloted by scoring abstracts from 22 quantitative publications from AEM Education and Training "Critical Appraisal of Emergency Medicine Education Research" (11 highlighted for excellent methodology and 11 that were not) and 10 qualitative publications (five highlighted for excellent methodology and five that were not). Intraclass correlation coefficient (ICC) estimates of reliability were calculated. Each rubric required three rounds of a modified Delphi process. The resulting quantitative rubric contained nine items: quality of objectives, appropriateness of methods, outcomes, data analysis, generalizability, importance to medical education, innovation, quality of writing, and strength of conclusions (Cronbach's α for the third round=0.922, ICC for total scores during piloting=0.893). The resulting qualitative rubric contained seven items: quality of study aims, general methods, data collection, sampling, data analysis, writing quality, and strength of conclusions (Cronbach's α for the third round=0.913, ICC for the total scores during piloting=0.788). We developed scoring rubrics to assess quality in quantitative and qualitative medical education research abstracts to aid in selection for presentation at scientific meetings. Our tools demonstrated high reliability.

Designing Performance Metrics and Rubrics to Assess Student Outcome Attainment in Engineering Project Design Course

The identification and development of efficient assessment and evaluation procedures (Performance Metrics) is crucial for analyzing Student Outcome attainment in Engineering Programs. The present paradigm shift in the education system has diverted the focus from teacher-centric approach to learner-centric approach. It involves the active involvement of students for their respective learning. Students are intended to perform better when they get opportunity to develop skills through experiential learning. In an engineering curriculum, Project-based courses are embedded to incorporate “learning by doing” approach and provide additional orientation to the theoretical courses studied. With this vision, the objective of this article is to present a process for continuous assessment and evaluation of graduates learning in their respective Project Design course. A sequenced procedure from defining Performance Metrics/Indicators (as per Bloom’s learning levels) for Project course to evaluate results for assessing subsequent student outcome attainment level is presented. This includes development of Rubrics for selected student outcome (ABET-EAC2) related to Project design course, collating the data using direct and indirect assessment tools and their result analysis. Engineering Accreditation Commission (EAC) as defined by ABET (Accreditation Board for Engineering and Technology) has listed seven student outcomes. EAC2 defines capability of student to produce problem-specific potential design solutions to address functional requirement of societal and environmental needs. The methodology is described and implemented by taking sample data of one pass out batch of one of the Engineering Program at our Institute. The results are evaluated and analyzed according to the expected attainment level. A comprehensive analysis leads to designing and incorporation of subsequent improvement measures to establish accountability for Outcome-Based Education in terms of student’s employability.

The Development of an Assessment Rubric for the Core and Contingency Team Interaction Among Rapid Response Teams.

Teamwork training is critical in the development of high-functioning rapid response teams (RRT). Rapid response teams involve interactions between a patient's core care team and a hospital contingency team, which can lead to disorganized and unsafe resuscitations, largely due to problems with communication and information dissemination. An extensive literature search found no assessment tools specific to the unique communicative challenges of an RRT, and thus, this study sought to develop an assessment rubric validated for training RRTs. This study elucidates the development, implementation, and testing of an RRT rubric based on Kane's framework for validating testing instruments. Twenty-four inpatient code teams underwent team training using a Team Strategies and Tools to Enhance Performance and Patient Safety (TeamSTEPPS) didactic, an online module on the TeamSTEPPS RRT program, and a subsequent presimulation and postsimulation experience. Two raters were randomized to give a bedside assessment for each team using the proposed RRT rubric. Simulation scores were assessed with Wilcoxon signed-rank tests. Interrater reliability was assessed using intraclass correlation coefficients. These analyses were then used to argue Kane's scoring, generalization, and extrapolation inferences. All teams significantly improved from the presimulation to postsimulation scenarios across all TeamSTEPPS domains. Content validity was obtained from 5 resuscitation experts with a scale-level content validity index of 0.9, with individual content validity index of 0.8 to 1.0. Intraclass correlation coefficient for "pre" scores were 0.856 (n = 24, P < 0.001), "post" scores of 0.738 (n = 24, P < 0.001), and a total of 0.890 (n = 48, P < 0.001). The authors argue for the validity of a new RRT rubric based off Kane's framework, with a specific focus on teamwork training to improve coordination and function of core and contingency teams. A follow-up study with longitudinal data along with external validation of this rubric is needed.

Textuality Rubric Development in PISA Criterion

Çalışmada ders kitaplarındaki metinleri değerlendirmek üzere “PISA Ölçütünde Metinsellik Rubriği”nin (PÖMR) geliştirilmesi amaçlanmıştır. Veriler 2019-2020 eğitim-öğretim yılında okutulan ortaokul (5-8. sınıf) Türkçe ders kitaplarında serbest okuma metni olarak yer alan ve PISA’nın sürekli metin özelliklerini taşıyan 20 okuma metninin 4 bağımsız puanlayıcı tarafından puanlanmasıyla elde edilmiştir. Analitik rubrik için ölçüt atanırken PISA 2018 okuma becerileri değerlendirme çerçevesindeki bilişsel süreçler ve okuma yeterlik düzeyleri dikkate alınmıştır. Her ölçüt için 3 farklı başarı derecesi tanımlanmıştır. Geliştirilen analitik rubriğin geçerlik kanıtlarının sağlanması için öncelikle 8 uzmanın görüşüne başvurulmuştur. Toplanan uzman tepkileri üzerinden her bir ölçüt için kapsam geçerlik oranı hesaplanmıştır. Uzman görüşünden gelen dönütler ile geçerlik kanıtı olması amacıyla temel bileşenler analizi uygulanmıştır. Analitik rubriğin güvenirlik kanıtları için hesaplanan sınıf içi korelasyon katsayısı .96 ve Cronbach alfa değeri .96 olarak bulunmuştur. Puanlayıcılar arası güvenirliğe kanıt sağlamak için 2 Türkçe öğretmeninin yaptığı puanlamalar üzerinden ağırlıklandırılmış Kappa hesaplanmıştır. Elde edilen ağırlıklandırılmış Kappa katsayısı testine göre analitik rubriğin ölçütleri iyi ve çok iyi uyuşma göstermiştir. Yapılan çalışmalarla “bilgiye ulaşma”, “anlama” ve “derinlemesine düşünme ve değerlendirme” olmak üzere 3 ana ölçüt ve toplam 8 alt ölçütten oluşan “PISA Ölçütünde Metinsellik Rubriği” geliştirilmiştir. Geliştirilen analitik rubriğin geçerli ve güvenilir bir ölçme aracı olduğu sonucuna ulaşılmıştır.

Theory and Model’s: Development of Soft Skill Rubric for The Co-Curricular Subject Among’s Matriculation Program Student of the Ministry of Education Malaysia

Theory and Model’s: Development of Soft Skill Rubric for The Co-Curricular Subject Among’s Matriculation Program Student of the Ministry of Education Malaysia

Alternative and Online Assessment in the Context of Outcome Based Education: A Practical Guide

Alternative assessment measures the learner’s ability and proficiency in performing complex tasks that are related to the intended learning outcomes and relies on direct measures of target skills in a specific knowledge or skill-domain. Meanwhile, online assessment is an assessment conducted in an online environment on a fully automated platform. Both types of assessments are getting a substantial attention worldwide due to their many advantages. However, several factors must be taken into consideration in implementing the alternative and online assessment in an outcome- based education (OBE) system. Thus, this paper provides a practical guide to alternative and online assessment in the context of OBE - including the constructive alignment, student learning time and development of assessment rubrics. Other considerations are also discussed, namely the validity and reliability of the alternative and online assessment, and deterring plagiarism and cheating.

A Rubric to Assess and Improve Technical Writing in Undergraduate Engineering Courses

Although there is consensus in the literature that writing skills are important in STEM (Science, Technology, Engineering, and Mathematics) studies, they are often neglected. However, some efforts have been made to correct this deficiency, one of them being the development of assessment rubrics. This study seeks to contribute to the discussion by presenting the results of the application of a rubric designed to assess the writing skills of a group of 3rd year engineering students. This rubric, which includes linguistic and rhetorical-organizational criteria alongside the mathematical and technical, was used to assess a number of written exercises and essays submitted by students in a 15-week course. The main interest of this study was to test the efficacy of the rubric as a diagnostic tool, conceived to detect the areas of improvement in the students’ written performance and, ultimately, to also help them to achieve higher levels of competence. This goal was achieved, as one of the main conclusions of the study is that, although students usually master the technical aspects of the course, they must improve the linguistic and rhetorical aspects of their written communication. It can likewise be said that all the participants involved in the study profited in one way or another from the application of the rubric and contributed to identifying the ways in which the rubric itself can be improved for future application.

Development of Rubric for the Evaluation of Concept Cartoons in Mathematics

Bu araştırmanın amacı matematik dersi için hazırlanan kavram karikatürlerini değerlendirebilmek için geçerli ve güvenilir bir dereceli puanlama anahtarı (DPA) geliştirmektir. Araştırma verileri Sivas il merkezinde görev yapan matematik öğretmenlerinin oluşturduğu 36 kavram karikatürünün değerlendirilmesiyle elde edilmiştir. Değerlendirme işlemini matematik eğitimi alanında yüksek lisans yapmış iki matematik eğitimcisi gerçekleştirmiştir. Geçerlik çalışmalarında öncelikle uzmanların DPA hakkındaki görüşleri içerik, yapı ve ölçüt bağlamında tek tek incelenmiştir. Bunların DPA’ya uygunluğu ve yeterliği konusunda uzmanların görüşleri arasındaki uyum yüzdelerinin kabul edilebilir düzeyde ve %82 ile %95 arasında değiştiği görülmüştür. Geçerlik çalışması kapsamında ayrıca kapsam geçerlik indeksi de hesaplanış ve her bir alt boyuta ilişkin kapsam geçerlik indeksleri 0.80 ile 1 arasında bulunmuştur. Hem uyum yüzdeleri hem de kapsam geçerlik indeksleri dikkate alındığında geliştirilen DPA’nın kavram karikatürlerini değerlendirmek için geçerli bir ölçme aracı olduğu görülmüştür. Güvenirlik çalışmalarında klasik test kuramı kapsamında iki puanlayıcı arasındaki uyum toplam puanlar için “Spearman Korelasyon Katsayısı” ile belirlenirken her bir alt ölçütteki uyum “Cohen’s Kappa” ile belirlenmiştir. DPA’ya iki değerlendiricinin verdiği toplam puanlar arasında pozitif, anlamlı ve yüksek (Rho=0.856, p&amp;lt;0.01) bir ilişkinin olduğu görülürken, alt ölçütlere ilişkin de Cohen’s Kappa sonuçları 0.706 ve 0.797 arasında yer aldığı görülmüştür. Güvenirlik çalışmalarında ayrıca madde tepki kuramı kapsamında Çok Yüzeyli Rasch Ölçme Modeli dikkate alınarak puanlayıcı, puanlanan (kavram karikatürü) ve madde yüzeyleri için güvenirlik hesaplamaları yapılmıştır. Bu yüzeylere ait güvenirlik indeksleri de sırasıyla 0.87, 0.80 ve 0.89 bulunmuştur. DPA’nın toplam puan için hesaplanan Spearman korelasyonu ve alt ölçüt puanları için hesaplanan Cohen’s Kappa değerleri ile puanlayıcı, puanlanan(kavram karikatürü) ve madde yüzeylerine ait güvenilir indeksleri bir arada değerlendirildiğinde oluşturulan DPA’nın güvenilir bir puanlama anahtarı olduğu görülmüştür. Araştırmanın bulguları ilgili literatür dikkate alınarak tartışılmış ve çeşitli önerilerde bulunulmuştur.

The Case for Authentic Student Assessment in Distance Digital Pharmacy Education.

This article was migrated. The article was marked as recommended. Distance online assessment is facing serious challenges. This opinion piece discusses these challenges and focuses on potential authentic assessment methods that can be used effectively in distance digital pharmacy education. The authentic assessment methods discussed are case studies, role-play, discussion panels and debates, student e-portfolios, computer-based simulation, diagrams and graphical presentations. These methods allow assessment to drive learning by performing tasks that are related to real professional situations and challenges followed by quantitative and qualitative feedback. Authentic assessment will require the development of clear rubrics to maintain transparency, consistency and fair evaluation of student achievement. The implementation of online assessment involves some challenges such as the level of institutional, faculty and student's technological capacity. Financial burden on low-to-medium income families and some institutions, meaningful training on the software and hardware, and students' compliance with the code of ethics and academic honesty are major issues. That said, well-planned authentic assessment tools can overcome many of these challenges. As these methods amass momentum, more creativity and innovation in assessment will be introduced, enriching and facilitating digital pharmacy education. The author calls for utilizing more formative feedback and reflection methods, focusing on demonstrating learning and professional competencies rather than on superficial letter grades.

Enhancing assessment literacies through development of quality rubrics using a Triad based peer review process

Peer review is viewed as a valid quality assurance mechanism in higher education. Peer review of teaching is common practice at universities in Australia. However, peer review of assessment is a relatively new innovation in tertiary education. Peer review of assessment in a Triad structure utilised data, via interviews with academics and students, to develop a peer review of assessment framework. This project was modelled on a Triad based peer review of teaching process at a major university in Brisbane Australia. A 10 question framework was used initially to facilitate conversations between assessors in a range of undergraduate courses (teacher education, business, visual arts, occupational therapy, outdoor education). The benefits for all stakeholders were widespread and significant, impacting students and assessors and provided a response to sector wide, national and international criticisms of tertiary assessment by students, who are driven by successful assessment experiences and shaped by the clarity of assessment rubrics.

Development of Rubric for enhancing Sensemaking among Team members in PBL in Entrepreneurship Education

Development of Rubric for enhancing Sensemaking among Team members in PBL in Entrepreneurship Education