Accuracy Of Self-assessments Research Articles

Strengthening the Culture of Learning Outcome Assessment in LIS Programs through Student Self- Assessments

Self-assessment empowers students to take ownership of their learning by assessing strengths and weaknesses in each area of competence they expect to achieve. This poster will illustrate perspectives gathered from a comprehensive examination and literature review to explore how LIS programs employ self-assessments, empowering students to understand their learning and competencies. Additionally, it aims to determine what lessons LIS programs can learn from other disciplines when creating an effective platform for facilitating student self-assessment. The review revealed that LIS programs use self-assessment tools such as self-reflective essays and journals, often combined with other direct assessment methods. Other professional disciplines, such as Medicine and Engineering, employ technology-based personalized self-assessment tools, aiding students in self-assessing at different program milestones. However, literature and best practices emphasize that self-assessment should not be individualistic but should be collaborative, involving peers and instructors. In addition to self-assessment methods, the review highlighted the importance of metacognitive awareness to ensure the accuracy of self-assessments and help students become more mindful of their learning. Many professional programs support students by facilitating rubric-guided judgments, self-rating, self-efficacy inventories, and metacognitive awareness inventories. LIS programs can adopt best practices from other disciplines to enhance the effectiveness and accuracy of student self-assessment practices.

Prototype development of a multi-user augmented reality framework for collaborative medical visualization in shared environments

Abstract This study developed and evaluated a prototype for augmented reality (AR) interaction among multiple users sharing a three-dimensional hologram within a shared physical space. A systems requirements analysis was conducted to determine necessary interactions, followed by an iterative implementation plan with testing at each step. The application for the HoloLens 2 (HL2) headset was developed using Unity for 3D design, C# for programming, and Photon for multiplayer capabilities. The prototype was evaluated in a user study utilizing custom and NASA TLX questionnaires, comparing participants' self-reported task success with recorded data to assess the accuracy of selfassessments. The prototype effectively facilitated multi-user interactions within a shared environment, achieving acceptable synchronization for two users. Most participants successfully completed tasks, with TLX scores indicating low cognitive and mechanical load during AR task execution. This study demonstrates AR's viability for displaying and manipulating complex three-dimensional structures within a collaborative medical context among multiple individuals in the same physical space.

Differences between resident self-assessments and faculty- assessments on Anesthesiology Milestones and associated factors: a cross-sectional study.

Accurate self-assessment is crucial for the professional development of physicians. There has been sparse data on the accuracy of self-assessments on Anesthesiology Milestones. The aim of this study was to investigate the differences between resident self-assessments and faculty-assessments on Anesthesiology Milestones and the associated factors. This was a cross-sectional study conducted in a general tertiary university-affiliated hospital. We included anesthesia residents who were enrolled in the standardized residency training program in postgraduate year two and three at the time of the Milestone evaluation. We requested evaluations of competencies from both the Clinical Competency Committee faculty and the anesthesia residents themselves, utilizing the Chinese version of Anesthesiology Milestones in January 2023 and January 2024. The primary outcome was the differences between self- and faculty-assessments, calculated by subtracting the faculty-rated score from the self-rated score on each subcompetency. A total of 46 and 42 residents were evaluated in year 2023 and 2024, respectively. The self-rated sum score was significantly higher than that rated by faculty [mean (standardized deviation): 120.39 (32.41) vs. 114.44 (23.71), P = 0.008 in paired t test] with an intraclass correlation coefficient of 0.55 [95% confidence interval (CI): 0.31 to 0.70]. The Bland-Altman plots revealed significant overestimation in patient care (bias 0.32, 95% CI: 0.05 to 0.60), practice-based learning and improvement (bias 0.45, 95% CI: 0.07 to 0.84), and professionalism (bias 0.37, 95% CI: 0.02 to 0.72). Ratings from residents with master's degrees (mean difference: -1.06, 95% CI: -1.80 to -0.32, P = 0.005) and doctorate degrees (mean difference: -1.14, 95% CI: -1.91 to -0.38, P = 0.003) were closer to the faculty-assessments than residents with bachelor's degrees. Compared with patient care, the differences between self- and faculty- rated scores were smaller in medical knowledge (mean difference: -0.18, 95% CI: -0.35 to -0.02, P = 0.031) and interpersonal and communication skills (mean difference: -0.41, 95% CI: -0.64 to -0.19, P < 0.001) in the generalized estimating equation logistic regression model. This study revealed that residents tended to overestimate themselves, emphasizing the need to improve the accuracy of Milestones self-assessment. The differences between self- and faculty-assessments were associated with residents' degrees and domains of competency.

Accuracy of Self-Assessment and Peer Assessment in Learning : A Systematic Literature Review

This study aims to explore the influence of self-assessment and peer assessment in learning, the factors affecting the accuracy of self-assessment and peer assessment, and efforts to enhance the accuracy of self-assessment and peer assessment. This research was a qualitative approach with a systematic literature review method. The data collected are from national and international scholarly journals related to self-assessment and peer assessment in learning, which were then analyzed using content analysis. According to the study's findings, using peer and self-evaluation to evaluate students' learning had a very beneficial effect. Thus, it should be put into practice. Additionally, a number of factors, such as self-confidence, emotional intelligence, self-perception of abilities, prior experience, feedback quality, assessment instrument clarity, assessor agreement, and trust in peer assessment, affect how accurately learners assess themselves and each other. Clear assessment criteria and procedures, which provide students direction for both their own and their peers' evaluations, can increase the accuracy of peer and self assessments. Good feedback is essential to improving both assessment techniques' accuracy. In addition, sufficient instruction for pupils is required to guarantee that evaluations are more precise, impartial, and beneficial.

Self-assessment: The role of the social environment

This study presents evidence on the role of the social environment in shaping the accuracy of self-assessment. We introduce a new measurement tool to elicit children’s accuracy of self-assessment. We use this tool to show that children from high SES families are more accurate in their self-assessment, compared to children from low SES families. We then exploit exogenous variation of participation in a mentoring program designed to enrich the social environment of children. The mentoring program has a causal positive effect on the accuracy of self-assessments and is most effective for children whose parents provide few interactive activities.

Metacomprehension in Multiple Document Reading

Abstract: Accurately assessing someone’s comprehension is central to learning from multiple documents. This study examines whether judgment accuracy can be increased if readers summarize texts before assessing their own comprehension. We asked 144 adults to read multiple documents on a historical controversy. They then produced Twitter-like summaries either (a) after a time delay or (b) immediately after reading; a control group (c) did not write summaries. Results showed that delaying writing summaries increased the absolute accuracy of self-assessments of undisputed historical content knowledge. We found negative effects for the accuracy of self-assessments of source–content links, intertextual conflicts, and integration performance in an essay. The relative accuracy was lower in the delayed summaries group than in the controls. In summary, the positive effect of writing summaries appears to be limited to the self-assessment of content knowledge, whereas it is reduced with respect to forms of knowledge specific to reading multiple texts.

A scoping review of self-monitoring in graduate medical education.

Physicians and physicians-in-training have repeatedly demonstrated poor accuracy of global self-assessments, which are assessments removed from the context of a specific task, regardless of any intervention. Self-monitoring, an in-the-moment self-awareness of one's performance, offers a promising alternative to global self-assessment. The purpose of this scoping review is to better understand the state of self-monitoring in graduate medical education. We performed a scoping review following Arksey and O'Malley's six steps: identifying a research question, identifying relevant studies, selecting included studies, charting the data, collating and summarising the results and consulting experts. Our search queried Ovid Medline, Web of Science, PsychINFO, Eric and EMBASE databases from 1 January 1999 to 12 October 2022. The literature search yielded 5363 unique articles. The authors identified 77 articles for inclusion. The search process helped create a framework to identify self-monitoring based on time and context dependence. More than 20 different terms were used to describe self-monitoring, and only 13 studies (17%) provided a definition for the equivalent term. Most research focused on post-performance self-judgements of a procedural skill (n= 31, 42%). Regardless of task, studies focused on self-judgement (n= 66, 86%) and measured the accuracy or impact on performance of self-monitoring (n= 41, 71%). Most self-monitoring was conducted post-task (n= 65, 84%). Self-monitoring is a time- and context-dependent phenomenon that seems promising as a research focus to improve clinical performance of trainees in graduate medical education and beyond. The landscape of current literature on self-monitoring is sparse and heterogeneous, suffering from a lack of theoretical underpinning, inconsistent terminology and insufficiently clear definitions.

Mindset and the desire for feedback during creative tasks

This mixed-methods research study sought to understand participant behaviors through a lens of mindset behaviors and to understand the accuracy of mindset self-assessments to actual performance during creative endeavors. During a makerspace period that took place for one hour over four days, sixty-two children were asked to create a product from a set of materials they were provided. They chose whether to work individually, with a partner, or in a small group. After each session, participants were asked whether they wanted feedback on their product. Researchers observed the participants while working in the makerspace for behaviors that were characteristic of a fixed or growth mindset, such as whether participants chose to receive feedback, whether they used feedback to improve their product, and how participants reacted to frustrations and failed attempts. Participants also received a mindset rating from instructors who worked with them over the course of the week on a Problem or Project-Based Learning experience. Scores were correlated with two mindset assessments participants completed at the beginning of the week, one adapted from Dweck (2006) and the MAP (Mindset Assessment Profile). The researcher scores of participants’ mindset and teacher scores of participants’ mindsets demonstrated a strong correlation. The two self-assessments used in the study did not show a strong correlation with performance as observed in the makerspace or in the classroom completing PBL-based classroom activities. Seven learning profiles emerged from the data based on participant attitudes (optimistic, approval-seeking, rule-breaking, perfectionistic) and behaviors (flexibility, playfulness, commitment to task). The mindset profiles may prove beneficial as educators develop activities to address mindsets in their classrooms, especially in the context of creative activities and projects.

Negative mood states as a correlate of cognitive performance and self-assessment of cognitive performance in bipolar disorder versus schizophrenia

IntroductionMood states have been reported to manifest a cross-sectional correlation with self-assessment accuracy across functional domains and psychiatric conditions. Ecological momentary assessment (EMA) provides a strategy to examine the momentary course and correlates of mood states. This study tested the association of moods assessed longitudinally with accuracy of immediate self-assessments of cognitive test performance in participants with schizophrenia and bipolar disorder. Methods240 well-diagnosed participants with schizophrenia and bipolar disorder completed a subset of tests from the MATRICS Consensus Cognitive Battery and an immediate self-assessment of cognitive performance. Differences between actual and self-reported performance were used to index the accuracy of self-assessment. Daily smartphone EMA, 3× per day for 30 days, sampled participants´ momentary moods (sad, happy, relaxed, anxious), aggregated into positive affect and negative affect (NA). ResultsBipolar participants had better cognitive performance, but both samples had equivalent mis-estimation. Repeated-measures analyses found that NA did not manifest significant variability over time either between or within participants in the two diagnostic groups. Within-group analyses found that higher average NA was associated with greater mis-estimation and poorer cognitive performance in participants with bipolar disorder, but not in those with schizophrenia. ConclusionNegative moods had a significant association with impairments in self-assessment of cognitive performance in participants with bipolar disorder. Our study did not confirm previous cross-sectional findings of more accurate self-assessment associated with greater NA in schizophrenia. These findings suggest that cross-sectional assessments, particularly self-reports, may lead to different results than aggregated data from longitudinal evaluations.

How to counteract biased self-assessments? An experimental investigation of reactions to social information

In a lab experiment, we investigate whether social information can improve the accuracy of self-assessments of relative performance. In particular, we compare the effectiveness of different types of social information: subjects either learn their close peers’ (i) average absolute performance, (ii) average self-assessment or (iii) average bias of self-assessments. Additionally, we explore the demand for the different types of social information. Our results suggest that social information can help debiasing subjects’ self-assessments, but not all types of information are equally effective. Only learning about the average bias of peers improves self-assessments. Subjects are, in general, willing to pay for social information but mostly prefer information about their peers’ absolute performance, which is not helpful. Nevertheless, self-selected information on peers’ average bias triggers a stronger reaction to the information.

Rubrics enhance accuracy and reduce cognitive load in self-assessment

Rubrics, which are designed to inform learners about assessment criteria and quality levels concerning a task assignment, are a widely used means to support learners in self-assessing their task performance. Usually, rubrics enhance task performance. Surprisingly, however, relatively little is known about the underlying mechanisms via which rubrics exert their beneficial effects. Although it is frequently suggested that the effects of rubrics on task performance are due to the fact that they support learners in accurately self-assessing their level of performance, which, in turn, paves the way for effective regulation, the empirical evidence that supports this notion is relatively scarce. Tightly controlled experimental studies, which allow isolating the effects of rubrics on the accuracy of self-assessments, are scarce. The present study was designed to close this gap. Specifically, in an experiment with N = 93 high school students we tested whether providing learners with a rubric in self-assessing the quality of self-written scientific abstracts would enhance judgment accuracy. As main results, we found that the rubric group showed higher absolute accuracy and lower bias. Furthermore, the rubric reduced cognitive load in self-assessing. These findings support the notion that one mechanism via which rubrics unfold their efficacy is by enhancing judgment accuracy.

Variability in the Accuracy of Self-Assessments among Low, Moderate, and High Performing Students in University Education.

Variability in the Accuracy of Self-Assessments among Low, Moderate, and High Performing Students in University Education.

How to counteract biased self-assessments? An experimental investigation of reactions to social information

In a lab experiment, we investigate whether social information can improve the accuracy of self-assessments of relative performance. In particular, we compare the effectiveness of different types of social information: subjects either learn their close peers' (i) average absolute performance, (ii) average self-assessment or (iii) average bias of selfassessments. Additionally, we explore the demand for the different types of social information. Our results suggest that social information can help debiasing subjects' selfassessments, but not all types of information are equally effective. Only learning about the average bias of peers' self-assessments improves own self-assessments. Subjects are, in general, willing to pay for social information but mostly prefer information about their peers' absolute performance, which is the least helpful type of social information. Consequently, endogenous choice of social information does not further improve self-assessments.

Examining the accuracy of residents' self-assessments and faculty assessment behaviours in anesthesiology.

BackgroundResidents’ accurate self-assessment and clinical judgment are essential for optimizing their clinical skills development. Evidence from the medical literature suggests that residents generally do poorly at self-assessing their performance, often due to factors relating to learners’ personal backgrounds, cultures, the specific contexts of the learning environment and rater bias or inaccuracies. We evaluated the accuracy of anesthesiology residents’ self-assessed Global Entrustment scores and determined whether differences between faculty and resident scores varied by resident seniority, faculty leniency, and/or year of assessment.MethodsWe employed variance components modeling techniques and analyzed 329 pairs of faculty and self-assessed entrustment scores among 43 faculty assessors and 15 residents. Using faculty scores as the gold standard, we compared faculty scores with residents’ scores (xi(faculty)–xi(resident)), and determined residents’ accuracy, including over- and under-confidence.ResultsThe results indicate that residents were respectively over- and under-confident in 10.9% and 54.4% of the assessments but more consistent in their individual self-assessments (rho = 0.70) than faculty assessors. Faculty scores were significantly higher (α = 0.396; z = 4.39; p < 0.001) than residents’ self-assessed scores. Being a lenient/dovish (β = 0.121, z = 3.16, p < 0.01) and a neutral (β = 0.137, z = 3.57, p < 0.001) faculty assessor predicted a higher likelihood of resident under-confidence. Senior residents were significantly less likely to be under-confident compared to junior residents (β = -0.182, z =-2.45, p < 0.05). The accuracy of self-assessments did not significantly vary during the two years of the study period.ConclusionsThe majority of residents’ self-assessments were inaccurate. Our findings may help identify the sources of such inaccuracies.

A155 VIDEO INTERVENTIONS TO IMPROVE SELF-ASSESSMENT ACCURACY IN GASTROINTESTINAL ENDOSCOPY: A SYSTEMATIC REVIEW

Abstract Background Physicians generally have inaccurate self-assessment of their performance. One type of proposed intervention to improve this inaccuracy is the use of video-based feedback. The overall impact of this intervention on self-assessment accuracy in gastrointestinal endoscopy is unclear. Aims To systematically review current literature to determine if video-based interventions can effectively improve self-assessment accuracy in endoscopy. Methods We searched the following electronic databases from inception to 2019: Ovid MEDLINE; Ovid EMBASE, the Cochrane Register of Controlled Trials (CENTRAL); Education Resources Information Center (ERIC); Education Source on EBSCO; and Canadian Business Current Affairs EBM Reviews. Specifically, we searched for terms related to self-assessment, self-report, self-efficacy, video recording, and physician. Studies were included if they met the following criteria: physicians at any level of training and/or practice; studies that used an experimental design; compared self-rated assessments with external assessments of procedural skills in endoscopy; and at least one arm of the study involved a video-based intervention. Results Our search yielded 755 articles, of which 2 met all inclusion criteria. One study explored the use of three feedback interventions (practice only with no video; observation of their own video performance; observation of expert video performance) among general surgery residents performing flexible endoscopy, which found that only participants who watched expert video performances had improved accuracy of self-assessments. The other study investigated the use of three video interventions (video of own performance; video of expert performance; video of both own and expert performances) among novice endoscopists performing esophagoduodenoscopy (EGD), which found that the video of expert performance significantly improved self-assessment accuracy compared to the group with both videos. Conclusions The current data tentatively support the use of video review of expert performance to improve self-assessment accuracy in gastrointestinal endoscopy. A meta-analysis is planned to quantitatively assess the overall impact. Funding Agencies None

Longitudinal effects of utilising photography on the accuracy of ergonomic self-assessments amongst dental hygiene students.

The purpose of this study was to determine the longitudinal effects of feedback involving photography and self-assessment on the accuracy of ergonomic self-assessments and ergonomic scores amongst dental hygiene students. The study involved an experimental design. Data collection took place over 4months, divided into weeks 1-4 and after 3months. At weeks 1 through 4 and 3months, all participants completed ergonomic self-evaluations, using a Modified-Dental Operator Posture Assessment Instrument. At week 1, week 4 and 3months, all participants were photographed, and in weeks 2 and 3, only participants in the training group were photographed and used those photographs to complete ergonomic self-assessments. All participants' photographs were evaluated by three faculty raters. A mixed-design ANOVA of Kappa coefficient values between student and rater scores measured the effects of the feedback with photography on the accuracy of student self-assessments. A main effect for group was not found to be significant (F(1,30)=2.90, P=.101). A mixed-design ANOVA measured the effects of the feedback with photography on the ergonomic scores. No significant main effects were found (F(1,120)=1.55, P=.216). Although self-assessment utilising photographs resulted in initial improvements in the accuracy of ergonomic self-assessments and ergonomic scores after 4weeks, both ergonomic accuracy and scores significantly decreased after 3months. Improvement in ergonomic accuracy and scores may be sustained by the continual facilitated use of photographs and self-assessment by dental and dental hygiene educators.

Influence of Self-Assessment on Dental Students' Performance on Technical Assessments in Periodontics.

Self-assessments were introduced into a third-year clinical periodontics course to enable dental students to self-assess and receive faculty feedback prior to technical assessment. The aims of this study were to determine if there was an improvement in the pass rate and average grade on the technical assessments following implementation of self-assessments and to analyze the accuracy of the student self-assessments by comparing them with faculty assessments. Scores from 106 third-year dental students for four periodontal procedures were included in the analysis. Faculty and student self-assessment scores, along with self-assessment and final technical assessments scores, were compared, as were the average number of self-assessment attempts for the various procedures and the pass rates. The average number of self-assessments per student was 6.5 compared to 4.2 for the technical assessments. There was no significant difference between students' self-assessment scores and faculty scores (average difference=0.005; p=0.7894). However, there was a significant increase in overall scores between students' self-assessments and technical assessments (average difference: 0.30 out of 5; p=0.0001). There was also a significant difference in pass rate between self-assessments and technical assessments (92.3% and 99.7%, respectively). The individual technical assessment with the greatest improvement was for scaling and root planing (average difference=0.47; p=0.0001), which also had significantly more self-assessments (2.3 vs. 1.2-1.7, p=0.0001). These results suggest that self-assessment and the subsequent faculty feedback provided students with information that enabled them to improve their performance on technical assessments.

Self-assessment of ergonomics amongst dental students utilising photography: RCT.

Dental professionals are at high risk of musculoskeletal disorders (MSDs) due to static working positions for extended periods of time. Musculoskeletal pain has been identified as early as during their entry-level dental education. The purpose of this study was to determine whether feedback involving photography and self-assessment would improve ergonomic scores and the accuracy of ergonomic self-assessments amongst dental students. The study involved a randomised control design of 135 dental students. At weeks 1 and 4, participants were photographed, and at weeks 1 through 4, participants completed ergonomic self-evaluations, using a Modified-Dental Operator Posture Assessment Instrument (M-DOPAI). During weeks 2 and 3, participants in the training group were photographed and used those photographs to complete ergonomic self-assessments. All participants' pre-training and post-training photographs were evaluated for ergonomic scores by two raters. A mixed-design ANOVA of ergonomic scores revealed that ergonomic scores improved for all students who received the ergonomics training (F(1,254)=17.41, P<.001). In addition, a mixed-design ANOVA of kappa coefficient values between student and rater scores revealed that the accuracy of self-assessments improved for all students who received the ergonomics training (F(1,127)=6.33, P<.05). The use of photographs and self-assessment provides dental and dental hygiene educators with a pragmatic method to improve self-assessment skills, increase student awareness of any postural deviations from ideal and improve musculoskeletal health.

Using Worker Self-Assessments for Competence-Based Pre-Selection in Crowdsourcing Microtasks

Paid crowdsourcing platforms have evolved into remarkable marketplaces where requesters can tap into human intelligence to serve a multitude of purposes, and the workforce can benefit through monetary returns for investing their efforts. In this work, we focus on individual crowd worker competencies. By drawing from self-assessment theories in psychology, we show that crowd workers often lack awareness about their true level of competence. Due to this, although workers intend to maintain a high reputation, they tend to participate in tasks that are beyond their competence. We reveal the diversity of individual worker competencies, and make a case for competence-based pre-selection in crowdsourcing marketplaces. We show the implications of flawed self-assessments on real-world microtasks, and propose a novel worker pre-selection method that considers accuracy of worker self-assessments. We evaluated our method in a sentiment analysis task and observed an improvement in the accuracy by over 15%, when compared to traditional performance-based worker pre-selection. Similarly, our proposed method resulted in an improvement in accuracy of nearly 6% in an image validation task. Our results show that requesters in crowdsourcing platforms can benefit by considering worker self-assessments in addition to their performance for pre-selection.

The accuracy of student self-assessments of English-Chinese bidirectional interpretation: a longitudinal quantitative study

The accuracy of self-assessment has long been examined empirically in higher education research, producing a substantial body of literature that casts light on numerous potential moderators. However, despite the growing popularity of self-assessment in interpreter training and education, very limited evidence-based research has been initiated to shed insight into the formative assessment of language interpretation. This longitudinal study was therefore conducted to investigate to what extent student self-assessments of English-Chinese interpretation are accurate and how the accuracy level would change over time. Major findings are: (1) in general self-assessment accuracy improved over time for both interpreting directions; (2) regarding the three scoring dimensions of information completeness, fluency of delivery and target language quality, there was greater self-assessment accuracy for English-to-Chinese interpretation than in the other direction; (3) while information completeness was most accurately self-assessed in English-to-Chinese direction, the pattern was reversed in the opposite direction; and (4) the students tended to over-score the three quality dimensions for Chinese-to-English interpretation at each time point. These results are discussed in the light of the accuracy issue, its longitudinal trend and potential factors affecting the self-assessment of bidirectional interpretation.