Assessor Agreement Research Articles

Discriminative Ability of Dye Injected Into a Meat Model to Determine Accuracy of Ultrasound-Guided Injection

Introduction The utility of using meat models for ultrasound-guided regional anesthesia simulation training has been well established. Feedback is considered the most important element of successful simulation-based education, and simulation offers an opportunity for evaluation. The objective of this study was to establish the discriminative ability of dye injected into a meat model to determine whether injectate is properly placed in the perineural (PN) space, thus providing an additional tool for learner feedback and evaluation. Methods Meat models containing a beef tendon (simulating a nerve) were injected with dye in one of 3 locations: PN, intraneural, and intramuscular. Blinded assessors then independently interpreted the dye staining on the models, marked the interpreted injection location, ease of interpretation, and whether staining was present on the beef tendon. Results Thirty meat models were injected with dye and independently assessed. Determining the location of injection was deemed to be easy or very easy in 72% of the models. Assessors correctly identified PN, intraneural, and intramuscular injections 100%, 95%, and 85% of the time, respectively. Assessor agreement was 87%. Conclusions The location of dye injected into a meat model, simulating a peripheral nerve blockade, can be accurately and reliably scored to provide feedback to learners. This technique offers a novel means of providing feedback to trainees and assessing block success in ultrasound-guided regional anesthesia simulation.

Accuracy of Self-Assessment and Peer Assessment in Learning : A Systematic Literature Review

This study aims to explore the influence of self-assessment and peer assessment in learning, the factors affecting the accuracy of self-assessment and peer assessment, and efforts to enhance the accuracy of self-assessment and peer assessment. This research was a qualitative approach with a systematic literature review method. The data collected are from national and international scholarly journals related to self-assessment and peer assessment in learning, which were then analyzed using content analysis. According to the study's findings, using peer and self-evaluation to evaluate students' learning had a very beneficial effect. Thus, it should be put into practice. Additionally, a number of factors, such as self-confidence, emotional intelligence, self-perception of abilities, prior experience, feedback quality, assessment instrument clarity, assessor agreement, and trust in peer assessment, affect how accurately learners assess themselves and each other. Clear assessment criteria and procedures, which provide students direction for both their own and their peers' evaluations, can increase the accuracy of peer and self assessments. Good feedback is essential to improving both assessment techniques' accuracy. In addition, sufficient instruction for pupils is required to guarantee that evaluations are more precise, impartial, and beneficial.

Developing an Instrument to Evaluate the Quality of Dementia Websites.

In today's digital era, health information, especially for conditions like dementia, is crucial. This study aims to develop an instrument, demenTia wEbsite measSurement insTrument (TEST), through four steps: identifying existing instruments, determining criteria, selecting and revising measurement statements, and validating the instrument from March to August 2020. Five health informatics experts used the content validity ratio (CVR) test for validation. Thirteen evaluators compared Fleiss Kappa and intraclass correlation coefficient (ICC) values across four dementia websites using TEST and another tool, DISCERN. TEST consists of seven criteria and 25 measurement statements focusing on content quality (relevance, credibility, currency) and user experience (accessibility, interactivity, attractiveness, privacy). CVR = 1 confirmed all statements as essential. The TEST demonstrated stronger consistency and assessor agreement compared to DISCERN, measured by Fleiss Kappa and ICC. Overall, it is a robust tool for reliable and user-friendly dementia resources, ensuring health holistic information accessibility.

Technical assessment in minimally invasive complete mesocolic excision: Is the complete mesocolic excision competency assessment tool valid and reliable?

The complete mesocolic excision competency assessment tool (CMECAT) is a novel tool designed to assess technical skills in minimally invasive complete mesocolic excision (CME) surgery. The aim of this study was to assess construct validity and reliability of CMECAT in a clinical context. Colorectal surgeons were asked to submit video recorded laparoscopic CME resections for independent assessment of their technical abilities. The videos were grouped by surgeons' training level, and four established CME experts were recruited as CMECAT assessors. Extended reliability analysis (G-theory) was applied to describe assessor agreement. A total of 19 videos and 72 assessments were included in the analysis. Overall, technical skills assessed by CMECAT improved with increased training level: the experts scored significantly better than the untrained surgeons (3.3 vs. 2.5 points; p < 0.01). On right-sided resections, significantly higher scores were reported with increased training level for all categories and sections, while for left-sided resections, the variance across groups was smaller and significantly higher scores were only reported for oncological safety describing items. Overall, assessor agreement was high (G-coefficient: 0.81). This study confirms that CMECAT can be applied to video recorded CME cases for technical skill assessment. Further, it can reliably assess technical performance in right sided CME surgery, where construct validity has now been established. More videos are required to evaluate its validity on left colonic CME. In the future, we hope CMECAT can improve feedback during CME training, serve as a tool in certification processes and contribute to distinguishing CME from conventional surgery in future research.

Validation of a method for the dry preservation and rehydration of Anopheles gambiae sensu lato for parity analysis to assess the impact of vector control measures in the field

BackgroundAs the control of malaria remains heavily dependent on vector management interventions, it is important to understand the impact of these on mosquito populations. Age-grading is a valuable tool for this; however, logistical challenges in remote, resource-poor areas make current methodologies difficult to incorporate into clinical trials and routine surveillance. Our aim was to validate a methodology that could be easily implemented in such settings. Using dried mosquito specimens instead of freshly killed ones, we validated the commonly used ovarian tracheation technique for assessing population age structure.MethodsLaboratory-reared Anopheles coluzzii mosquitoes with known parity status were dry preserved in silica gel for up to 12 weeks and rehydrated prior to parity assessment. The results were compared to parity results for freshly killed mosquitoes from the same colony. Preserved, field-caught Anopheles gambiae sensu lato (s.l.) from Guinea-Bissau were assessed by three different assessors blinded to each other’s scores. An overall index of agreement was calculated using inter-rater reliability of all assessor pairings. The impact of preservation time was investigated using a one-way ANOVA to look for differences in assessor agreement over three time periods.ResultsThe parity status was correctly identified for 90% of dry preserved and rehydrated insectary-reared An. coluzzii and for 98% of freshly killed insectary-reared An. coluzzii. The inter-rater reliability was highest (0.94) for freshly killed An. coluzzii. The results for all time points showed excellent strength of agreement between assessors. For field-caught An. gambiae s.l., the overall index of agreement between all three assessors was 0.86 (95% confidence interval 0.78–0.93), indicating almost perfect agreement. There was no significant difference between assessor agreement between time frames.ConclusionsDry preserving and rehydrating Anopheles mosquitoes provides an alternative to using freshly killed mosquitoes to assess the efficacy of a control intervention in remote settings where it is logistically difficult to dissect fresh specimens. This method also provides the flexibility required for parity assessment to be done on larger scales over bigger areas.Graphical

Impairment due to Contextual Factors: External Assessment and Self-Assessment of Rehabilitants in Psychosomatic Rehabilitation

In addition to an existing external assessment tool for the ICF-based assessment of the impairment of rehabilitation patients due to environmental and personal contextual factors, a self-assessment tool was developed and used for rehabilitation patients in a psychosomatic rehabilitation clinic. The aim was to compare the assessment of the impairment by contextual factors rated by the external assessment and by the rehabilitation patients on the other hand. Furthermore, the aim of the study was to obtain initial indications of the applicability of the newly designed self-evaluation instrument. A sample of N=103 rehabilitation patients was examined via both the external and the self-evaluation tools. A comparative evaluation of the assessment of the external assessor and the rehabilitants' assessments was carried out using measures of assessor agreement (Cohen's Kappa, percentual agreement). The results were set in relation to the data of the routine assessment of the rehabilitation facility (Eta-Coefficient). The external assessor and the rehabilitants get to different results in their assessment of the extent of impairment by individual contextual factors. For eight of the 15 context factors, there was only a moderate degree of agreement between the assessors (Cohen's Kappa). The percentage of non-agreement between rating of the external assessor and self-assessment lies between approx. 20% and up to approx. 40% of the cases for the majority of the context factors. Whether or not there is an impairment across all contextual factors (overall measure) was answered non-concordantly in just under a third of the cases. Mostly moderate correlations were found between the context factors classified as particularly relevant in the self-evaluation tool and scales of psychometric survey (e. g. BDI-II). This result provides a first indication of the convergent validity of the self-report tool. More attention should be paid to a supplementary assessment of impairment by contextual factors via the rehabilitation patient's self-assessment in addition to the external assessment. There is a need for further research with regard to the development of the self-assessment tool presented.

Dualisme Makna Akta Pengakuan Hutang

Introduction: Deed of debt recognition as an authentic deed can be made in two forms, namely as a debt agreement or equivalent to a credit agreement and as an assessor agreement of a credit agreement deed that is grosse deed. Both types of debt acknowledgment deeds are still used in public life, the benefits and functions of each deed are different, because the position is different, the impact of the deed is also different. &#x0D; Purposes of the Research: The purpose of this research is to see the differences in the actualization of the deed of recognition of debt in practice and to examine the position of the deed of debt recognition and the contribution of each deed in the legal relationship of the parties. &#x0D; Methods of the Research: This research was conducted in a normative juridical manner, namely by examining the existing secondary data in the field of law as library data which is focused on examining the application of rules or norms in positive law so that the data used by the author is secondary data. &#x0D; Results Originality of the Research: The dualism of interpreting the meaning of debt recognition basically has the same hope, namely the existence of a legal relationship in debts and receivables, the position deed of debt recognition as an authentic deed has a formal function and the function of evidence, the strength of the debt recognition deed provides perfect evidentiary power. The position of debt recognition as a grosse deed should not be merely a scrap of paper because it requires a follow-up to the imposition of collateral through a separate deed of guarantee, then the debt recognition as an assessor agreement becomes an intermediary deed which seems futile. The dualism of the meaning of this deed of recognition of debt would not occur again and provide a harmonious understanding. if you really want to emphasize it, you should use the term 'debt agreement or credit agreement' which is domiciled as the main agreement, without a grosse deed followed by the binding of special guarantees, both individual and material guarantees for the interests and legal protection of the parties.

Holistic vs. Analytic Evaluation in Writing Test of Eighth Grade Students

The purpose of this study is to explore the efficacy of holistic and analytic scoring rubrics when applied to the context of writing examinations. Throughout this research, a technique was utilized to compare two grading models based on (a) assessor agreement (reliability) and (b) reasons for the grades given (validity). Teachers were randomly allocated to one of two conditions, where they graded the same student performance using either an analytical or a holistic approach. The design is experimental. Agreement and rank correlation between grades have been compared. A comparison of the students' performance on a writing assignment is made in this study using both holistic and analytic scoring rubrics. The information for this study was gathered from twenty-four students in eighth grade at SMP N 5 Kecamatan Kapur IX. The authors conducted an analysis of the performance of the students on the two rubrics, using psychometric statistics, in order to precisely determine the strengths and weaknesses of the students, and to place them along a continuum of foreign language writing ability. Citation In contrast, analytic scoring rubrics placed examinees on a scale of writing competency that was more specifically defined. As a result, analytic scoring rubrics were more dependable than holistic scoring rubrics when it came to gauging writing for the purpose of achieving goals.

Influence of Clinical Instructor Experience on Assessing Doctor of Physical Therapist Student Clinical Performance: A Mixed-Methods Study

Introduction. Formative and summative feedback are essential for physical therapist students' professional growth. Achieving objectivity in student clinical assessment has been a longstanding concern for health care educators and ambiguity surrounds what is meant by competent clinical practice. According to a sociocultural perspective, clinical performance is socially constructed and based on an individual's perception of what is taking place, which may explain differences identified in the literature between novice and experienced assessors of student clinical performance. However, it is not known whether novice clinical instructors (CIs) differ from experienced CIs in their assessment practices of student physical therapists. The purposes of this study were to examine whether novice and experienced CIs' assessment practice differ and explore the reasons for these differences (if any). Methods. Novice and experienced CIs for physical therapist students' 3 full-time clinical education experiences served as participants for this study. A mixed-methods research design was used. In the quantitative phase, novice and experienced CI-assigned midterm and final student ratings of the Clinical Performance Instrument (CPI) were compared. In the qualitative phase, semi-structured interviews, document review, and participant observation were conducted. Results. Experienced CIs awarded higher ratings on the CPI than novice CIs, but ratings on only a few of the performance criteria were significantly different. Qualitatively, both novice and experienced CIs were similar in terms of the importance assigned to professional behavior and safety. However, novice CIs were more literal and concrete and experienced CIs were more flexible and varied. Three CI roles (CI as student, CI as clinician, and CI as assessor), CI self-confidence, reflection, context, assessor approach, and assessor agreement emerged as strong themes throughout. Discussion and Conclusion. The low number of statistically significant differences between novice and experienced CIs' awarding CPI ratings suggests that there were few true quantitative disparities between the assessors. Both study cohorts' backgrounds as former students and as current clinicians as well as their stage of CI professional development seem to have influenced their approach to clinical assessment.

Evaluating Family Medicine Resident Narrative Comments Using the RIME Scheme.

BackgroundIn 2013, the Accreditation Council on Graduate Medical Education (ACGME) launched the Next Accreditation System, which required explicit documentation of trainee competence in six domains. To document narrative comments, the University of North Carolina Family Medicine Residency Program developed a mobile application to document real time observations.ObjectiveThe objective of this work was to assess if the Reporter, Interpreter, Manager, Expert (RIME) framework could be applied to the narrative comments in order to convey a degree of competency.MethodsFrom August to December 2020, 7 individuals analyzed narrative comments of four family medicine residents. The narrative comments were collected from July to December 2019. Each individual applied the RIME framework to the comments and the team met to discuss. Comments where 5/7 individuals agreed were not further discussed. All other comments were discussed until consensus was achieved.Results102 unique comments were assessed. Of those comments, 25 (25.5%) met threshold for assessor agreement after independent review. Group discussion about discrepancies led to consensus about the appropriate classification for 92 (90.2%). General comments on performance were difficult to fit into the RIME framework.ConclusionsApplication of the RIME framework to narrative comments may add insight into trainee progress. Further faculty development is needed to ensure comments have discrete elements needed to apply the RIME framework and contribute to overall evaluation of competence.

Wound photography for evaluation of surgical site infection and wound healing after lower limb trauma.

Deep surgical site infection (SSI) is common after lower limb fracture. We compared the diagnosis of deep SSI using alternative methods of data collection and examined the agreement of clinical photography and in-person clinical assessment by the Centers for Disease Control and Prevention (CDC) criteria after lower limb fracture surgery. Data from two large, UK-based multicentre randomized controlled major trauma trials investigating SSI and wound healing after surgical repair of open lower limb fractures that could not be primarily closed (UK WOLLF), and surgical incisions for fractures that were primarily closed (UK WHiST), were examined. Trial interventions were standard wound care management and negative pressure wound therapy after initial surgical debridement. Wound outcomes were collected from 30 days to six weeks. We compared the level of agreement between wound photography and clinical assessment of CDC-defined SSI. We are also assessed the level of agreement between blinded independent assessors of the photographs. Rates of CDC-defined deep SSI were 7.6% (35/460) after open fracture and 6.3% (95/1519) after closed incisional repair. Photographs were obtained for 77% and 73% of WOLLF and WHiST cohorts respectively (all participants n = 1,478). Agreement between photographic-SSI and CDC-SSI was fair for open fracture wounds (83%; k = 0.27 (95% confidence interval (CI) 0.14 to 0.42)) and for closed incisional wounds (88%; k = 0.29 (95% CI 0.20 to 0.37)) although the rate of photographically detected deep SSIs was twice as high as CDC-SSI (12% vs 6%). Agreement between different assessors for photographic-SSI (WOLLF 88%, k = 0.63 (95% CI 0.52 to 0.72); WHiST 89%; k = 0.61 (95% CI 0.54 to 0.69)); and wound healing was good (WOLLF 90%; k = 0.80 (95% CI 0.73 to 0.86); WHiST 87%; k = 0.57 (95% CI 0.50 to 0.64)). Although wound photography was feasible within the research context and inter-rater assessor agreement substantial, digital photographs used in isolation overestimated deep SSI rates, when compared to CDC criteria. Wound photography should not replace clinical assessment in pragmatic trials but may be useful for screening purposes where surgical infection outcomes are paramount. Cite this article: Bone Joint J2021;103-B(12):1802-1808.

The Reliability and Accuracy of Palpation, Radiography, and Sonography for the Detection of Keel Bone Damage.

Simple SummaryDue to the recent increase in reports of the prevalence of keel bone damage in laying hens, this has become a topic of welfare concern. Keel bone damage is often in the form of a fracture, and therefore may compromise the hen’s welfare. Detecting keel bone damage in live hens has been problematic, as the bone must either be palpated, which is a measure poor in accuracy, or dissected, in which case the hen must be sacrificed. However, use of imaging technology is increasing in experimental studies. We set out to determine if training with feedback on accuracy could improve palpation accuracy, as well as to determine the accuracy of portable radiography and sonography to detect keel bone damage. Even with feedback, palpation remained an inaccurate method, while both radiography and sonography showed high accuracy for detecting fractures. These two techniques show promise in detecting keel bone fractures in live birds.Palpation is the most popular method of measuring keel bone damage on live birds, although it has been criticized for being subjective and inaccurate. The goals of this study were to examine intra- and inter-rater reliability when trained with feedback of accuracy, as well as determine the accuracy of portable radiography and sonography. Four evaluators palpated 50 103-week old Lohmann LSL-lite hens immediately following euthanasia. Of those birds, 34 were then radiographed, sonographed, and all 50 were re-palpated. Lastly, the keels were dissected and scored. The presence of deviations (DEV), fractures (FR), and tip fractures (TFR) was scored for each method. Reliability of palpation was analyzed using Cronbach’s Alpha (intra) and Fleiss’ Kappa (inter) tests. Radiography and Sonography scores were further compared with dissection scores to determine sensitivity and specificity. Initial inter-observer reliability was 0.39 DEV, 0.53 FR, and 0.12 TFR, with similar scores for the second round of palpation. Scores for intra-observer reliability ranged from 0.58–0.79 DEV, 0.66–0.90 FR, and 0.37–0.87 TFR. A high prevalence of TFR, but low assessor agreement, warrants the development of specialized training for the palpation of this area. Both radiography and sonography showed relatively high sensitivity for FR and TFR, but low for DEV. On the other hand, specificity was generally high across all damage types. Even with feedback, palpation reliability was poor. However, portable radiography and sonography show promise for detecting keel fractures.

Measuring the Validity and Reliability of Research Instruments

This paper discussed how the applying of Rasch Model in validity and reliability of research instruments. Three sets of research instruments were developed in this study. The Felder-Solomon Index of Learning Styles (ILS) is essential to find out the learning style abilities of learners. Students’ Perception in Cognitive Dimension (SPCD) was developed to identify student perception toward their cognitive abilities, and Students’ Cognitive Mastery Achievement Test (CMAT) is used to measure student mastery in a particular subject. The study aims to produce empirical evidence of validity and reliability using the Rasch Model. A small survey was conducted on 28 vocational college students enrolled in the Building Construction course. The ILS consists of four constructs, whereas the SPCD and CMAT validate based on three constructs. The value of reliability was based on Cronbach alpha with appropriate values range. The construct validity was analyzed based on the Rasch model with infit and outfit mean square (MNSQ) value. Three experts in the building construction subject examined the content validity of SPCD and CMAT. Assessor agreement can be calculated as percent-agreement. Percent-agreement statistics can be calculated and explained easily. In summary, Rasch Model is suitable to apply in instrument validation process because the concept of item response theory.

Reliable Assessors of Infant Cranial Asymmetry in Child Health Care.

Introduction:Acquired cranial asymmetry is prevalent in infants today and largely attributed to the supine sleep position recommended for infant safety. There is a risk of permanent cranial asymmetry, so prevention and early detection are important. A prevention project was initiated in Sweden, and an intervention was planned. The aim of this study was to evaluate reliability of assessors judging infant cranial asymmetry in order to evaluate if they could be considered reliable interchangeable assessors in the planned intervention. Materials and Methodology:Five assessors were taught how to assess infant cranial asymmetry using illustrated severity assessments. They were intra-rater and inter-rater reliability tested by taking a photograph test-retest and an infant test. Agreement matrices were devised to illustrate assessor agreement based on both type and degree of cranial asymmetry. Agreement based on degree of asymmetry was analyzed by calculating AC2 using quadratic weights. Results were adjusted to arrive at the perceived genuine agreement and interpreted according to Landis and Koch’s strength of agreement intervals. Results: In the photograph test, mean percentage of perfect intra-rater agreement was 73. Adjusted mean intra-rater AC2 was 0.69 [0.63; 0.76], and adjusted inter-rater AC2s were 0.72 [0.64; 0.81] and 0.71 [0.63; 0.79]. In the infant test, the adjusted inter-rater AC2 was 0.73 [0.60; 0.87]. Results indicate substantial strength of assessor agreement.Conclusion: Assessors were reliable and interchangeable. In a larger clinical context, results indicate that educating child health care nurses to assess infant cranial asymmetry can be used for early detection.

Standardised training program in spinal ultrasound for epidural insertion: protocol driven versus non-protocol driven teaching approach.

Spinal ultrasonography provides guidance for epidural insertion in obstetric patients. The primary objective of the study was to develop a training program in spinal ultrasound for anaesthetists and to determine its effect on the skill acquisition of anaesthetists with no prior spinal ultrasound experience. Eighteen anaesthetists underwent two structured workshops (one week apart), each followed by a practice session and videorecorded assessments. Participants were randomised to a protocol-driven or non-protocol driven spinal ultrasound teaching program. Two experts rated each individual's performance using a global rating scale (GRS), checklist and image quality scale. The primary outcome was the mean difference in GRS score between the two workshops, analysed using linear mixed models. Intraclass correlation coefficients were calculated to assess agreement between assessors' ratings. A total of 108 ultrasound scans were performed on five pregnant volunteers during the assessment periods. After adjusting for confounders, GRS scores increased on all three rating scales at the second workshop, this increase being 6.01 points (95% confidence interval 4.56 to 7.46, P<0.001) from a mean score of 28.4 (95% confidence interval 24.8 to 32.0). There was no significant difference in the scores between the two teaching groups (difference in GRS scores=1.36 points, 95% confidence interval -0.77 to 3.50, P=0.211). Intraclass correlation coefficients showed substantial assessor agreement for all three assessment methods (range 0.59 to 0.89). The results demonstrate that programmed spinal ultrasound training sessions involving practice with guidance and feedback from an expert, whether protocol-based or non-protocol based, lead to improved performance.

A comparison of grades awarded by peer assessment, faculty and a digital scanning device in a pre‐clinical operative skills course

The aim of this study was to compare the grades awarded by two experienced assessors with peer-assessment marks and measurements from a digital scanning device (Prepassistant; KaVo, Biberach, Germany), for full gold crown preparations completed in a pre-clinical operative skills course on typodont teeth. Seventy-eight preparations on typodont teeth were randomised and assessed by all three methods. Agreement was measured using weighted kappa statistics, and mean rank scores given by the Friedman test. The highest agreement was seen between the experienced assessors (0.38), closely followed by peer assessment and experienced assessor agreement (0.36, 0.29). Despite this, the results indicate poor levels of agreement. No agreement was seen between any of the assessment methods when compared to the digital scanning device. The findings of this study could be related to the difficulty of calculating a single grade from the output of the device, in addition to the inability of the machine to assess all the factors necessary for an acceptable preparation. From this study, it can be concluded that this device is not suitable for calculating grades when used in isolation. Further research could explore the role of the Prepassistant in providing student feedback, its potential to enhance the learning experience and the subsequent effect on performance.

Resident Self-Other Assessor Agreement

To review the literature on self-assessment in the context of resident performance and to determine the correlation between self-assessment across competencies in high- and low-performing residents and assessments performed by raters from a variety of professional roles (peers, nurses, and faculty). Retrospective analysis of prospectively collected anonymous self-assessment and multiprofessional (360) performance assessments by competency and overall. University-based academic general surgical program. Sixty-two residents rotating in general surgery. Mean difference for each self-assessment dyad (self-peer, self-nurse, and self-attending physician) by resident performance quartile, adjusted for measurement error, correlation coefficients, and summed differences across all competencies. Irrespective of self-other dyad, residents asked to rate their global performance overestimated their skills. Residents in the upper quartile underestimated their specific skills while those in the lowest-performing quartile overestimated their abilities when compared with faculty, peers, and especially nurse raters. Moreover, overestimation was greatest in competencies related to interpersonal skills, communication, teamwork, and professionalism. Rater, level of performance, and the competency being assessed all influence the comparison of the resident's self-assessment and those of other raters. Self-assessment of competencies related to behavior may be inaccurate when compared with raters from various professions. Residents in the lowest-performing quartile are least able to identify their weakness. These data have important implications for residents, program directors, and the public and suggest that strategies that help the lowest-performing residents recognize areas in need of improvement are needed.

Good questions, good answers: construct alignment improves the performance of workplace-based assessment scales

Assessment in the workplace is important, but many evaluations have shown that assessor agreement and discrimination are poor. Training discussions suggest that assessors find conventional scales invalid. We evaluate scales constructed to reflect developing clinical sophistication and independence in parallel with conventional scales. A valid scale should reduce assessor disagreement and increase assessor discrimination. We compare conventional and construct-aligned scales used in parallel to assess approximately 2000 medical trainees by each of three methods of workplace-based assessment (WBA): the mini-clinical evaluation exercise (mini-CEX); the acute care assessment tool (ACAT), and the case-based discussion (CBD). We evaluate how scores reflect assessor disagreement (V(j) and V(j*p) ) and assessor discrimination (V(p) ), and we model reliability using generalisability theory. In all three cases the conventional scale gave a performance similar to that in previous evaluations, but the construct-aligned scales substantially reduced assessor disagreement and substantially increased assessor discrimination. Reliability modelling shows that, using the new scales, the number of assessors required to achieve a generalisability coefficient ≥0.70 fell from six to three for the mini-CEX, from eight to three for the CBD, from 10 to nine for 'on-take' ACAT, and from 30 to 12 for 'post-take' ACAT. The results indicate that construct-aligned scales have greater utility, both because they are more reliable and because that reliability provides evidence of greater validity. There is also a wider implication: the disappointing reliability of existing WBA methods may reflect not assessors' differing assessments of performance, but, rather, different interpretations of poorly aligned scales. Scales aligned to the expertise of clinician-assessors and the developing independence of trainees may improve confidence in WBA.

Overlapping Systematic Reviews of Anterior Cruciate Ligament Reconstruction Comparing Hamstring Autograft with Bone-Patellar Tendon-Bone Autograft

Background: Systematic reviews published on the same topic during a similar period of time (i.e., overlapping reviews) on anterior cruciate ligament reconstruction complicate the choice between bone-patellar tendon-bone or hamstring tendon autograft. We aimed to evaluate reasons for differences among the overlapping systematic reviews and to assess the quality of reporting and internal validity. Methods: We performed a search of MEDLINE, the Cochrane Database of Systematic Reviews, and EMBASE to identify systematic reviews in which bone-patellar tendon-bone graft was compared with hamstring tendon autograft for reconstruction of the anterior cruciate ligament. We evaluated cross-citations among the overlapping reviews and the authors' rationale for repeating the review. The quality of reporting was assessed with the Quality of Reporting of Meta-analyses (QUOROM) statement, and the internal validity was assessed with the Oxman and Guyatt index for methodological quality by at least two assessors. Assessor agreement was evaluated with intraclass correlation coefficients. We evaluated the sensitivity analysis that had been performed in the reviews. Results: We identified eleven overlapping systematic reviews. Three reviews favored the patellar tendon graft for stability, and one favored the hamstring graft. Six reviews favored the hamstring graft to prevent anterior knee pain, and the rest were inconclusive. Only six reviews cited previously published systematic reviews on the same topic, and only two of these reviews cited all available systematic reviews that were available at that time. The quality of reporting ranged from 5 to 18 (median, 12; maximum score, 18). The internal validity ranged from 1 to 7 (median, 2; maximum score, 7). Reviewers reached almost perfect agreement (intraclass correlation coefficients, 0.83 and 0.94). Formal sensitivity analysis was utilized infrequently. The highest-quality review favored hamstring grafts to prevent anterior knee pain and showed weak evidence that bone-patellar tendon-bone grafts yielded better stability. Conclusions: When overlapping or discordant systematic reviews are encountered, each review must be appraised on the basis of its methodological quality before it can be used to guide clinical decision-making or policy making. The currently available best evidence, derived from a methodologically sound meta-analysis, suggests that hamstring tendon autografts are superior for preventing anterior knee pain, and there is limited evidence that bone-patellar tendon-bone autografts provide better stability.

Overlapping Systematic Reviews of Anterior Cruciate Ligament Reconstruction Comparing Hamstring Autograft with Bone-Patellar Tendon-Bone Autograft

Systematic reviews published on the same topic during a similar period of time (i.e., overlapping reviews) on anterior cruciate ligament reconstruction complicate the choice between bone-patellar tendon-bone or hamstring tendon autograft. We aimed to evaluate reasons for differences among the overlapping systematic reviews and to assess the quality of reporting and internal validity. We performed a search of MEDLINE, the Cochrane Database of Systematic Reviews, and EMBASE to identify systematic reviews in which bone-patellar tendon-bone graft was compared with hamstring tendon autograft for reconstruction of the anterior cruciate ligament. We evaluated cross-citations among the overlapping reviews and the authors' rationale for repeating the review. The quality of reporting was assessed with the Quality of Reporting of Meta-analyses (QUOROM) statement, and the internal validity was assessed with the Oxman and Guyatt index for methodological quality by at least two assessors. Assessor agreement was evaluated with intraclass correlation coefficients. We evaluated the sensitivity analysis that had been performed in the reviews. We identified eleven overlapping systematic reviews. Three reviews favored the patellar tendon graft for stability, and one favored the hamstring graft. Six reviews favored the hamstring graft to prevent anterior knee pain, and the rest were inconclusive. Only six reviews cited previously published systematic reviews on the same topic, and only two of these reviews cited all available systematic reviews that were available at that time. The quality of reporting ranged from 5 to 18 (median, 12; maximum score, 18). The internal validity ranged from 1 to 7 (median, 2; maximum score, 7). Reviewers reached almost perfect agreement (intraclass correlation coefficients, 0.83 and 0.94). Formal sensitivity analysis was utilized infrequently. The highest-quality review favored hamstring grafts to prevent anterior knee pain and showed weak evidence that bone-patellar tendon-bone grafts yielded better stability. When overlapping or discordant systematic reviews are encountered, each review must be appraised on the basis of its methodological quality before it can be used to guide clinical decision-making or policy making. The currently available best evidence, derived from a methodologically sound meta-analysis, suggests that hamstring tendon autografts are superior for preventing anterior knee pain, and there is limited evidence that bone-patellar tendon-bone autografts provide better stability.