Rubric Method Research Articles

Impact of reflective writing skill on emotional quotient

BackgroundThe skill of reflective writing and emotional drivers of behavior is enhanced by boosting the awareness of one's feelings and emotions. Individuals having characteristic activity of reflection have more emotional quotient (EQ). EQ has more significance than intelligence quotient in personal and professional life. This study is conducted to analyze the impact of reflective writing skills on components of EQ among first-year MBBS students and to find out the impact of reflective writing between components of EQ and different levels of reflective writing skills among first-year MBBS students. MethodsAn observational-analytical study was carried out for EQ self-assessment and reflections of first-year MBBS students. A paired t-test was used to check the difference between different segments of EQ. Analysis of reflections was done by modifying the REFLECT rubric method. ResultsThere was a significant difference in the motivation component between pre and post-test (p-value = 0.013), whereas no significant difference was observed in other components, viz. self-awareness, empathy, self-control, self-confidence, and social competency. The average total scores among EQ domains compared were more at the level of reflection than habitual and introspection levels. ConclusionAlong with intelligence quotient, EQ is critical for students' overall development. The average total scores of EQ were higher among the students who had a higher level of reflection. Developing reflective writing skills had an impact on the motivation component of the EQ of the students within a short duration.

Identification of influencing components on water scarcity management performance for sustainability in limited water resource area: a review

Water management by policy makers is based on multi-perspective information. In the case of water, scarcity causes public health, economic, and ecosystem problems because of the lack of safe water when the demand exceeds its availability. This study identified the influencing components of water scarcity management performance by systematic review methods based on two English full-text databases: Scopus and ScienceDirect from 2012 to 2022. A total of 22 studies were selected from 850 records by using the inclusion criteria and performance evaluation by the scoring rubric method. Moreover, the article quality was assessed using a critical appraisal tool by the Joanna Briggs Institute (JBI) critical appraisal checklist method. The result of this review revealed that the main influencing components of water scarcity management are related to the availability of water quantity and demand. Water availability is associated with the hydrological cycle which has uncertainty due to climate change. While the demand components depend on water use, population, industry and urbanization. Most of the studies in the literature review considered only water balance of water availability and demand. Such an approach is insufficient to achieve water security for sustainable development goals. Additional components including environmental components such as water quality degradation should be considered as these components can reduce the availability of freshwater resources. From an economic point of view, the benefit of water utilization should be prioritized in scarce water management in consideration of fairness in water rights. It is also essential to consider the contribution to socio-economic aspects from stakeholder meetings and participation.

Radiography education with VR using head mounted display: proficiency evaluation by rubric method

BackgroundThe use of head mounted display (HMD)-based immersive virtual reality (VR) coaching systems (HMD-VRC) is expected to be effective for skill acquisition in radiography. The usefulness of HMD-VRC has been reported in many previous studies. However, previous studies have evaluated the effectiveness of HMD-VRC only through questionnaires. HMD-VRC has difficulties in palpation and patient interaction compared to real-world training. It is expected that these issues will have an impact on proficiency. The purpose of this study is to determine the impact of VR constraints in HMD-VRC, especially palpation and patient interaction, on radiographic skills proficiency in a real-world setting.MethodsFirst-year students (n = 30) at a training school for radiology technologists in Japan were randomly divided into two groups, one using HMD-VRC (HMD-VRC group) and the other practicing with conventional physical equipment (RP group) and trained for approximately one hour. The teachers then evaluated the students for proficiency using a rubric method.ResultsIn this study, it was found that some skills in the HMD-VRC group were equivalent to those of the RP group and some were significantly lower than those of the RP group. There was a significant decrease in proficiency in skills related to palpation and patient interaction.ConclusionsThis study suggests that HMD-VRC can be less effective than real-world training in radiographic techniques, which require palpation and patient interaction. For effective training, it is important to objectively evaluate proficiency in the real world, even for HMD-VRC with new technologies, such as haptic presentation and VR patient interaction.Trial registrationThe study was conducted with the approval of the Ethics Committee of International University of Health and Welfare (Approval No.21-Im-035, Registration date: September 28, 2021).

Distilling vector space model scores for the assessment of constructed responses with bifactor Inbuilt Rubric method and latent variables

In this paper, we highlight the importance of distilling the computational assessments of constructed responses to validate the indicators/proxies of constructs/trins using an empirical illustration in automated summary evaluation. We present the validation of the Inbuilt Rubric (IR) method that maps rubrics into vector spaces for concepts’ assessment. Specifically, we improved and validated its scores’ performance using latent variables, a common approach in psychometrics. We also validated a new hierarchical vector space, namely a bifactor IR. 205 Spanish undergraduate students produced 615 summaries of three different texts that were evaluated by human raters and different versions of the IR method using latent semantic analysis (LSA). The computational scores were validated using multiple linear regressions and different latent variable models like CFAs or SEMs. Convergent and discriminant validity was found for the IR scores using human rater scores as validity criteria. While this study was conducted in the Spanish language, the proposed scheme is language-independent and applicable to any language. We highlight four main conclusions: (1) Accurate performance can be observed in topic-detection tasks without hundreds/thousands of pre-scored samples required in supervised models. (2) Convergent/discriminant validity can be improved using measurement models for computational scores as they adjust for measurement errors. (3) Nouns embedded in fragments of instructional text can be an affordable alternative to use the IR method. (4) Hierarchical models, like the bifactor IR, can increase the validity of computational assessments evaluating general and specific knowledge in vector space models. R code is provided to apply the classic and bifactor IR method.

Enhancing topic-detection in computerized assessments of constructed responses with distributional models of language

Usually, computerized assessments of constructed responses use a predictive-centered approach instead of a validity-centered one. Here, we compared the convergent and discriminant validity of two computerized assessment methods designed to detect semantic topics in constructed responses: Inbuilt Rubric (IR) and Partial Contents Similarity (PCS). While both methods are distributional models of language and use the same Latent Semantic Analysis (LSA) prior knowledge, they produce different semantic representations. PCS evaluates constructed responses using non-meaningful semantic dimensions (this method is the standard LSA assessment of constructed responses), but IR endows original LSA semantic space coordinates with meaning. In the present study, 255 undergraduate and high school students were allocated one of three texts and were tasked to make a summary. A topic-detection task was conducted comparing IR and PCS methods. Evidence from convergent and discriminant validity was found in favor of the IR method for topic-detection in computerized constructed response assessments. In this line, the multicollinearity of PCS method was larger than the one of IR method, which means that the former is less capable of discriminating between related concepts or meanings. Moreover, the semantic representations of both methods were qualitatively different, that is, they evaluated different concepts or meanings. The implications of these automated assessment methods are also discussed. First, the meaningful coordinates of the Inbuilt Rubric method can accommodate expert rubrics for computerized assessments of constructed responses improving computer-assisted language learning. Second, they can provide high-quality computerized feedback accurately detecting topics in other educational constructed response assessments. Thus, it is concluded that: (1) IR method can represent different concepts and contents of a text, simultaneously mapping a considerable variability of contents in constructed responses; (2) IR method semantic representations have a qualitatively different meaning than the LSA ones and present a desirable multicollinearity that promotes the discriminant validity of the scores of distributional models of language; and (3) IR method can extend the performance and the applications of current LSA semantic representations by endowing the dimensions of the semantic space with semantic meanings.

Concept mapping toward competency: Teaching and assessing undergraduate evidence‐informed practice

AbstractBackgroundUndergraduate students in healthcare professions need to develop critical thinking skills in order to be prepared to deliver patient‐centered high‐quality care upon graduation. The new Canadian Athletic Therapy Association (CATA) competency framework includes a “scholar” role, placing emphasis on knowledge and skills related to evidence‐informed practices (EIP). Educators are expected to develop their student's EIP skills; however, little is known about the optimal educational approaches to accomplish this task. The study objective was to examine concept mapping as a teaching and learning strategy to deepen the understanding of EIP and examine the validity and reliability of a concept mapping scoring rubric.MethodA concept mapping approach to teaching EIP was piloted in an upper‐level research methods and statistics course. Students (N = 15) participated in a pretest post‐test concurrent nested mixed‐method intervention study that analyzed the impact of concept mapping on students' understanding of EIP and assessed initial validity and reliability of the grading rubric.ResultsStudents demonstrated a deeper understanding of EIP and its relationship to healthcare practice following the concept mapping activity compared to a note‐taking activity (Cohen's d = 1.79). The concept mapping rubric used to assess EIP competence demonstrated strong construct validity and content validity with moderate inter‐rater reliability.ConclusionsEmploying the concept mapping technique as a teaching and learning tool proved to be effective to teach EIP principles and concepts based on student grades. An analytic rubric method was found valid and reliable for grading student concept maps underpinning EIP competency.

Developing reflective skills among first year MBBS students

Introduction: Transitional period of students towards becoming professional should be aimed for making self-motivated learners by medical education. It is known that reflective writing expands the affective domain of persons and promotes mind mapping which are integral parts of good professional. The present study aimed to develop reflective writing skills of the medical students and to increase awareness about reflective writing skills. Method: Prospective analytical study. Students were educated with a reflective writing skill workshop and requested to write four reflections after a specific interval with guidance. Reflections of students were analyzed for levels by the modified REFLECT Rubric method. Students were given individual feedback and /or by focus group discussion in between successive reflections. Feedback of whole process was taken from students. Results: At the end of first reflection, 65% students had a clear concept of reflection whereas at the end of the fourth reflection, all students had a clear concept with achievement of a higher level of reflection; 33% students were transferred to the level of introspection; 11% students were found in reflection. However no student reached to the level of critical reflection. Conclusion: Formal training and guiding for writing reflections over a period of time helped medical students to get sensitized and develop reflective writing skills. They had perception of enhanced skills of effective communication, ethical issues in medicine, empathy in medicine and patient’s need. For holistic growth of students, teaching reflective writing skills to students should be incorporated.

Real-Time Selective Sequencing with RUBRIC: Read Until with Basecall and Reference-Informed Criteria

The Oxford MinION, the first commercial nanopore sequencer, is also the first to implement molecule-by-molecule real-time selective sequencing or “Read Until”. As DNA transits a MinION nanopore, real-time pore current data can be accessed and analyzed to provide active feedback to that pore. Fragments of interest are sequenced by default, while DNA deemed non-informative is rejected by reversing the pore bias to eject the strand, providing a novel means of background depletion and/or target enrichment. In contrast to the previously published pattern-matching Read Until approach, our RUBRIC method is the first example of real-time selective sequencing where on-line basecalling enables alignment against conventional nucleic acid references to provide the basis for sequence/reject decisions. We evaluate RUBRIC performance across a range of optimizable parameters, apply it to mixed human/bacteria and CRISPR/Cas9-cut samples, and present a generalized model for estimating real-time selection performance as a function of sample composition and computing configuration.

Automated summary evaluation with inbuilt rubric method: An alternative to constructed responses and multiple-choice tests assessments

Automated summary evaluation is proposed as an alternative to rubrics and multiple-choice tests in knowledge assessment. Inbuilt rubric is a recent Latent Semantic Analysis (LSA) method that implements rubrics in an artificially-generated semantic space. It was compared with classical LSA’s cosine-based methods assessing knowledge in a within-subjects design regarding two validation sources: a comparison with the results of rubric scores and multiple-choice tests, and the sensitivity of predicting the academic level of the test-taker. Results showed a higher reliability for inbuilt rubric (from Pearson correlation coefficient .81 to .49) over the classical LSA method (from .61 to .34), and a higher sensitivity using binary logistic regressions and effect sizes to predict academic level. It is concluded that inbuilt rubric has a qualitatively higher reliability and validity than classical LSA methods in a way that is complementary to models based on semantic networks. Thus, it is concluded that new automated summary evaluation approaches such as the inbuilt rubric method can be practical in terms of reliability and efficiency, and, thus, they can offer an affordable and valuable form of knowledge assessment in different educational levels.

Analyzing Two Automatic Latent Semantic Analysis (LSA) Assessment Methods (Inbuilt Rubric vs. Golden Summary) in Summaries Extracted from Expository Texts

The purpose of this study was to compare two automatic assessment methods using Latent Semantic Analysis (LSA): a novel LSA assessment method (Inbuilt Rubric) and a traditional LSA method (Golden Summary). Two conditions were d using the Inbuilt Rubric method: the number of lexical descriptors needed to better accommodate an expert rubric (few vs. many) and a weighting to penalize off-topic contents d in the student summaries (weighted vs. non-weighted). One hundred and sixty-six students divided in two different samples (81 undergraduates and 85 High School students) took part in this study. Students summarized two expository texts that differed in complexity (complex/ easy) and length (1,300/500 words). Results showed that the Inbuilt Rubric method simulates human assessment better than Golden summaries in all cases. The similarity with human assessment was higher for Inbuilt Rubric (r = .78 and r = .79) than for Golden Summary (r = .67 and r = .47) in both texts. Moreover, to accommodate an expert rubric into the Inbuilt Rubric method was better using few descriptors and the weighted .

Do targeted written comments and the rubric method of delivery affect performance on future human physiology laboratory reports?

We investigated how students performed on weekly two-page laboratory reports based on whether the grading rubric was provided to the student electronically or in paper form and the inclusion of one- to two-sentence targeted comments. Subjects were registered for a 289-student, third-year human physiology class with laboratory and were randomized into four groups related to rubric delivery and targeted comments. All students received feedback via the same detailed grading rubric. At the end of the term, subjects provided consent and a self-assessment of their rubric viewing rate and preferences. There were no differences in laboratory report scores between groups (P = 0.86), although scores did improve over time (P < 0.01). Students receiving targeted comments self-reported viewing their rubric more often than students that received no comments (P = 0.02), but the viewing rate was independent of the rubric delivery method (P = 0.15). Subjects with high rubric viewing rates did not have higher laboratory report grades than subjects with low viewing rates (P = 0.64). When asked about their preference for the future, 43% of respondents preferred the same method again (electronic or paper rubric) and 25% had no preference. We conclude that although student laboratory report grades improved over time, the rate and degree of improvement were not related to rubric delivery method or to the inclusion of targeted comments.

A Comparison of Selected Methods of Scoring Classroom Assessments

This set of studies compared the findings from generalizability (G) studies and alternative decision (D) studies for 4 approaches to scoring classroom assessments. In the 1st experiment, the unconstrained point method was compared to the generic rubric method. Under the unconstrained point method, raters assigned total possible points to items and then scored the responses of 10 students to those items to compute an overall percentage score for each student. Under the generic rubric method, raters were given a generic rubric that they applied to student responses for all items as a set. In the 2nd experiment, the constrained point method was compared to the topic-specific rubric method. Under the constrained point method, raters scored the responses of 10 students to items whose point values had been previously established. In the topic-specific rubric condition, raters used a rubric that was specific to the topic of the assessment. In terms of less Rater × Person variability and higher G and D coefficients, the topic-specific rubric method was found superior to the constrained point method that was found superior to the generic rubric method that was found superior to the unconstrained point method.