Assessment Of Learner Performance Research Articles

Teaching and Evaluating the Virtual Physical Exam in Telemedicine.

With the rapid adoption of telemedicine since the COVID-19 pandemic, it has become imperative to teach and evaluate health professional trainees on skills important to conducting effective virtual visits. We developed a simulation-based workshop with (1) readings, (2) a lecture covering online communication and the virtual head and neck exam, (3) a telemedicine simulation with a standardized patient observed by faculty, (4) personalized feedback from faculty, and (5) a group debrief session. We created an evaluation rubric based on three of 20 Association of American Medical Colleges (AAMC) telemedicine competencies to assess learner performance during the simulations. Students (medical and physician assistant students; n = 50), and internal medicine residents (n = 20) completed this workshop in 2023. At least 90% of trainees across the two groups were rated as approaching entrustment or entrustable in each competency. This workshop is an example of a scalable telemedicine curriculum that can be used to teach and evaluate learners in the virtual physical exam across the training continuum.

Virtual simulations for neonatal education

Virtual simulation (VS) education involves the use of virtual reality, augmented reality, mixed reality and screen-based platforms, collectively referred to as extended reality, to provide education and assessment. This novel simulation modality supports experiential learning and increases access to practice opportunities, supplementing manikin-based simulation. VS has been used successfully for neonatal resuscitation training in high and low resource settings. Virtual simulators can be used to objectively assess learner performance in neonatal resuscitation knowledge and skills. When implementing VS for neonatal resuscitation training, key considerations include matching learning objectives with suitable technology, pre-session preparation, supporting learners, and debriefing. Additional research is needed to evaluate the impact of VS applications on clinical practice and patient outcomes.

"It Changed How I Think"-Impact of Programmatic Assessment Upon Practicing Physicians: A Qualitative Study.

Programmatic assessment is a systematic approach used to document and assess learner performance. It offers learners frequent formative feedback from a variety of contexts and uses both high- and low-stakes assessments to determine student progress. Existing research has explored learner and faculty perceptions of programmatic assessment, reporting favorable impact on faculty understanding of the importance of assessment stakes and feedback to learners while students report the ability to establish and navigate towards goals and reflect on their performance. The Cleveland Clinic Lerner College of Medicine (CCLCM) of Case Western Reserve University adopted programmatic assessment methods at its inception. With more than 18years' experience with programmatic assessment and a portfolio-based assessment system, CCLCM is well-positioned to explore its graduates' perceptions of their programmatic assessment experiences during and after medical school. In 2020, the investigators interviewed 26 of the 339 physician graduates. Participants were purposefully sampled to represent multiple class cohorts (2009-2019), clinical specialties, and practice locations. The investigators analyzed interview transcripts using thematic analysis informed by the frameworks of self-determination theory and professional identity formation. The authors identified themes and support each with participant quotes from the interviews. Based on findings, the investigators compiled a series of recommendations for other institutions who have already or plan to incorporate elements of programmatic assessment into their curricula. The authors concluded by discussing future directions for research and additional avenues of inquiry.

The application of simulators in dental medicine students’ training

The simulation training of dental students has developed rapidly in recent years. The development of new technologies and virtual reality are an indispensable part of the education of dental students. The introduction of simulators in the education of dental students supports the harmonization and integration of knowledge from the pre-clinical and clinical education of the students. The opportunity to train on simulators in the field of dentistry helps future dentists to put their theoretical knowledge into practice and gain confidence to work with patients. Simulation operators are included in the students’ pre-clinical exercises. In recent years, computer-based simulation and virtual reality-based simulation have become an indispensable part of the pre-clinical training of dental students. Nowadays, there is a wide range of dental simulator models with different features and functions. Some of them are: DentSim, Haptic Technology, Moog Simodont Dental Trainer, HapTEL, the Geneva System, Robotic Patients for Virtual Dental Patient Simulation, Virtual Reality Dental Training System, PerioSim, the VirDenT system, the Forsslund System, DSE TM Expert Dental Simulation Units—KaVo Dental. In addition to student training, these simulators can also be used to assess learner performance or quality control various teaching methods. The development of new technologies marks successes in the development and improvement of the simulators applied in the training of dental medicine students. The virtual patient application provides many opportunities to recreate clinical situations in virtual reality and will make learning even more rewarding and interesting for dental students.

Competency-based medical education across the continuum: How well aligned are medical school EPAs to residency milestones?

Introduction Competency-based medical education (CBME) provides a framework for describing learner progression throughout training. However, specific approaches to CBME implementation vary widely across educational settings. Alignment between various methods used across the continuum is critical to support transitions and assess learner performance. The purpose of this study was to investigate alignment between CBME frameworks used in undergraduate medical education (UME) and graduate medical education (GME) settings using the US context as a model. Method The authors analyzed content from the core entrustable professional activities for entering residency (Core EPAs; UME model) and residency milestones (GME model). From that analysis, they performed a series of cross-walk activities to investigate alignment between frameworks. After independent review, authors discussed findings until consensus was reached. Results Some alignment was found for activities associated with history taking, physical examination, differential diagnosis, patient safety, and interprofessional care; however, there were far more examples of misalignment. Conclusions These findings highlight challenges creating alignment of assessment frameworks across the continuum of training. The importance of these findings includes implications for assessment and persistence of the educational gap across UME and GME. The authors provide four next steps to improve upon the continuum of education.

Learner Handover: Who Is It Really For?

Learner handover is the sharing of information about learners between faculty supervisors. Learner handover can support longitudinal assessment in rotation-based systems, but there are concerns that the practice could bias future assessments or stigmatize struggling learners. Because successful implementation relies on an understanding of existing practices and beliefs, the purpose of this study was to explore how faculty perceive and enact learner handover in the workplace. Using constructivist grounded theory, 23 semistructured interviews were conducted with faculty from 2 Canadian universities between August and December 2018. Participants were asked to describe their learner handover practices, including learner handover delivered or received about resident and student trainees either within or between clinical rotations. The authors probed to understand why faculty used learner handover and their perceptions of its benefits and risks. Learner handover occurs both formally and informally and serves multiple purposes for learners and faculty. While participants reported that learner handover was motivated by both learner benefit and patient safety, they primarily described motivations focused on their own needs. Learner handover was used to improve faculty efficiency by focusing teaching and feedback and was perceived as a "self-defense mechanism" when faculty were uncertain about a learner's competence and trustworthiness. Informal learner handover also served social or therapeutic purposes when faculty used these conversations to gossip, vent, or manage insecurities about their assessment of learner performance. Because of its multiple, sometimes unsanctioned purposes, participants recommended being reflective about motivations behind learner handover conversations. Learners are not the only potential beneficiaries of learner handover; faculty use learner handover to lessen insecurities surrounding entrustment and assessment of learners and to openly share their frustrations. The latter created tensions for faculty needing to share stresses but wanting to act professionally. Formal education policies regarding learner handover should consider faculty perspectives.

EDUCATIONAL EVALUATION AND ANALYSIS OF STATISTICAL TECHNIQUES

In education evaluation plays an important role in the teaching-learning process. In Evaluation analysis it contributes to formulation of objectives. Structuring of learning experiences and assessment of learner performance. It is very useful to bring improvement in teaching and learning process. It provides accountability to the society, parents, and to the education Systems. Educational and Statistical Evaluation scrutiny provides the accurate results as approved judgments In evaluation process good evaluation always coordinated by Validity, Reliability, practicability, fairness, and Usefulness. In Statistical Analysis process we can identify the Quantitative Data, Continuous and Discrete Data, Qualitative, primary and Secondary Data. In Educational Evaluation Statistical Techniques plays a vital role Standard graphic forms will always provide evaluation confirmation in easy understanding method in the form of Graphical representations.

Integrating Socio-Critical and Problem Oriented Approach to Assess Learner Performance in Acids and Bases in Secondary School Chemistry

The main purpose of this study was to explore the effect of integrating socio-critical and problem oriented approach on performance of grade 11 learners in acids and bases. It also investigated the learner’s attitudes towards chemistry and the benefits and challenges faced by teachers in contextualizing the teaching and learning of chemistry. The design of this study was mixed method design involving quantitative and qualitative research methods together (Creswell, 2008; Gray, 2009). The study was conducted in Luanshya district of the Copperbelt Province in Zambia at three selected senior secondary schools (using rotary). These schools are Luanshya Central Secondary School. Luanshya Girls Secondary School and Mpelembe Secondary school. The sample consisted of all 16 chemistry teachers from the selected schools and 98 learners from two intact grade 11 classes randomly selected from two core schools of the three schools involved. These are Luanshya Central Secondary School and Mpelembe Secondary School. A non-randomised quasi-experimental pre-test, post-test experimental and control group counterbalanced was used. Chemistry achievement test and a five likert scale chemistry attitude questionnaire was used for quantitative data collection. Qualitative data was collected using audio recorded structured interviews. The experimental group was taught using integrating socio-critical and problem oriented approach while the control group was taught using conventional methods (discussion) method. An independent t-test at alpha 0.5 was conducted to analyse the results of pre-test and post-test. The significant difference in post-test was p=0.003 with mean score difference of 39.7 at t-value of 8.542. Results from the chemistry attitude test revealed that learners had positive attitudes towards chemistry having learner using socio-critical and problem oriented approach. Also qualitative data from interviews was analysed using content analysis technique on excel windows 7. The results showed that teacher’s challenges ranged from topic difficulty to lack of apparatus to use in lessons. Therefore, integrating socio-critical and problem oriented approach in the teaching of acids and bases at secondary school had a positive effect on learner performance. Both groups showed a significant difference with socio-critical and problem oriented approach than with conventional method. Keywords: socio-critical and problem oriented approach, performance, attitudes, conventional method DOI : 10.7176/EJBM/11-23-08 Publication date : August 31 st 2019

Masters of Morphology: Explicit Multisensory Structured Metacognitive Language Strategies to Foster Adolescent Learners’ Content Vocabulary

As students move through the grades, content area learning becomes increasingly significant for reading, writing, and speaking tasks. One research-evidenced approach to actively engaging adolescent learners in acquiring, remembering, and using academic vocabulary in reading, writing and speaking tasks across disciplines is multisensory structured metacognitive language (MSML) instruction. In this article, a variety of such MSML strategies are shared with specific examples relating to content disciplines. They help to strengthen teachers’ confidence in implementing research-evidenced prefix-root-suffix vocabulary learning strategies with their adolescent students. Selective examples of how to assess learner performance are provided in such a way that they can be adapted for different academic contexts.

UPDATES FROM THE GEAR-UP PROJECT: PERFORMANCE OF MULTIDISCIPLINARY LEARNERS ON LATE LIFE DEPRESSION AND COGNITIVE IMPAIRMENT SIMULATIONS

UPDATES FROM THE GEAR-UP PROJECT: PERFORMANCE OF MULTIDISCIPLINARY LEARNERS ON LATE LIFE DEPRESSION AND COGNITIVE IMPAIRMENT SIMULATIONS

Pediatric Program Director Minimum Milestone Expectations Before Allowing Supervision of Others and Unsupervised Practice

BackgroundThe Accreditation Council for Graduate Medical Education requires semiannual Milestone reporting on all residents. Milestone expectations of performance are unknown. ObjectiveTo determine pediatric program director (PD) minimum Milestone expectations for residents before being ready to supervise and before being ready to graduate. MethodsMixed methods survey of pediatric PDs on their programs' Milestone expectations before residents are ready to supervise and before they are ready to graduate, and in what ways PDs use Milestones to make supervision and graduation decisions. If programs had no established Milestone expectations, PDs indicated expectations they considered for use in their program. Mean minimum Milestone level expectations were adjusted for program size, region, and clustering of Milestone expectations by program were calculated for before supervise and before graduate. Free-text questions were analyzed using thematic analysis. ResultsThe response rate was 56.8% (113 of 199). Most programs had no required minimum Milestone level before residents are ready to supervise (80%; 76 of 95) or ready to graduate (84%; 80 of 95). For readiness to supervise, minimum Milestone expectations PDs considered establishing for their program were highest for humanism (2.46; 95% confidence interval [CI], 2.21–2.71) and professionalization (2.37; 95% CI, 2.15–2.60). Minimum Milestone expectations for graduates were highest for help-seeking (3.14; 95% CI, 2.83–3.46). Main themes included the use of Milestones in combination with other information to assess learner performance and Milestones are not equally weighted when making advancement decisions. ConclusionsMost PDs have not established program minimum Milestones, but would vary such expectations according to competency.

The Assessment of Competency in Thoracic Sonography (ACTS) scale: validation of a tool for point-of-care ultrasound

BackgroundThe rapid adoption of point-of-care ultrasound (POCUS) has created a need to develop assessment tools to ensure that learners can competently use these technologies. In this study, the authors developed and tested a rating scale to assess the quality of point-of-care thoracic ultrasound studies performed by novices. In Phase 1, the Assessment of Competency in Thoracic Sonography (ACTS) scale was developed based on structured interviews with subject matter experts. The tool was then piloted on a small series of ultrasound studies in Phase 2. In Phase 3 the tool was applied to a sample of 150 POCUS studies performed by ten learners; performance was then assessed by two independent raters.ResultsEvidence for the content validity of the ACTS scale was provided by a consensus exercise wherein experts agreed on the general principles and specific items that make up the scale. The tool demonstrated reasonable inter-rater reliability despite minimal requirements for evaluator training and displayed evidence of good internal structure, with related scale items correlating well with each other. Analysis of the aggregate learning curves suggested a rapid early improvement in learner performance with slower improvement after approximately 25–30 studies.ConclusionsThe ACTS scale provides a straightforward means to assess learner performance. Our results support the conclusion that the tool is an effective means of making valid judgments regarding competency in point-of-care thoracic ultrasound, and that the majority of learner improvement occurs during their first 25–30 practice studies.

Outcomes from extensive training in critical care echocardiography: Identifying the optimal number of practice studies required to achieve competency

PurposeOptimal instruction and assessment of critical care ultrasound (CCUS) skills requires an assessment tool to measure learner competency and changes over time. In this study, a previously published tool was used to monitor the development of critical care echocardiography (CCE) competencies, the attainment of performance plateaus, and the extent to which previous experience influenced learning. Materials and methodsA group of experts used the Rapid Assessment of Competency in Echocardiography (RACE) scale to rate a large pool of CCE studies performed by novices in a longitudinal design. A total of 380 studies performed by twelve learners were assessed; each study was independently rated by two experts. ResultsLearners demonstrated improvement in mean RACE scores over time, with peak performance occurring early in training and a performance plateau thereafter. Learners with little experience received the greatest benefit from training, with an average performance plateau reached at the twentieth study. ConclusionsSupporting earlier results, the RACE scale provided a straightforward means to assess learner performance with minimal requirements for evaluator training. The results of the present study suggest that novices experience the greatest gains in competency during their first twenty practice studies, a threshold which should serve to guide training initiatives.

Rapid Cycle Deliberate Practice: Application to Neonatal Resuscitation.

This curriculum includes two simulation cases for neonatal resuscitation training using the rapid cycle deliberate practice (RCDP) technique. RCDP is a simulation-based curriculum that presents participants with rounds of increasing difficulty in rapid repetition, interspersing brief, direct feedback within the simulation. In contrast, traditional debriefing focuses on learning after the scenario is complete. Traditional debriefing usually utilizes advocacy-inquiry debriefing but allows less opportunity for practice. Each case provides a neonatal resuscitation scenario (respiratory failure secondary to perinatal compromise and cardiac arrest secondary to placental abruption) for a term newborn in the delivery room. The curriculum utilizes high-fidelity neonatal mannequins with learner teams of three to six multidisciplinary teammates who participate in scenarios of increasing difficulty to revive a neonate. Learners can include a spectrum from beginning to advanced neonatal resuscitation providers. Learners are expected to perform the appropriate steps per the neonatal resuscitation program algorithm in addition to exhibiting effective crisis resource management skills. Immediate assessment of learner performance and feedback within the RCDP model is more directive, which allows for rapid resumption of practice. The instructor may also choose to pause and back up or to pause and restart, depending on the correction needed. Neonatal resuscitation program teaching utilizes a neonatal resuscitation performance evaluation, which may be used to guide opportunities for feedback within RCDP.

The Rapid Assessment of Competency in Echocardiography Scale: Validation of a Tool for Point-of-Care Ultrasound.

Increased use of point-of-care ultrasound (US) requires the development of assessment tools that measure the competency of learners. In this study, we developed and tested a tool to assess the quality of point-of-care cardiac US studies performed by novices. In phase 1, the Rapid Assessment of Competency in Echocardiography (RACE) scale was developed on the basis of structured interviews with subject matter experts; the tool was then piloted on a small series of US studies in phase 2. In phase 3, the tool was applied to a sample of 154 point-of-care US studies performed by 12 learners; each study was independently rated by 2 experts, with quantitative analysis subsequently performed. Evidence of the content validity of the RACE scale was supported by a consensus exercise, wherein experts agreed on the assessment dimensions and specific items that made up the RACE scale. The tool showed good inter-rater reliability. An analysis of inter-item correlations provided support for the internal structure of the scale, and the tool was able to discriminate between learners early in their point-of-care US learning and those who were more advanced in their training. The RACE scale provides a straightforward means to assess learner performance with minimal requirements for evaluator training. Our results support the conclusion that the tool is an effective means of making valid judgments regarding competency in point-of-care cardiac US.

The Pediatrics Milestones Assessment Pilot: Development of Workplace-Based Assessment Content, Instruments, and Processes.

To report on the development of content and user feedback regarding the assessment process and utility of the workplace-based assessment instruments of the Pediatrics Milestones Assessment Pilot (PMAP). One multisource feedback instrument and two structured clinical observation instruments were developed and refined by experts in pediatrics and assessment to provide evidence for nine competencies based on the Pediatrics Milestones (PMs) and chosen to inform residency program faculty decisions about learners' readiness to serve as pediatric interns in the inpatient setting. During the 2012-2013 PMAP study, 18 U.S. pediatric residency programs enrolled interns and subinterns. Faculty, residents, nurses, and other observers used the instruments to assess learner performance through direct observation during a one-month rotation. At the end of the rotation, data were aggregated for each learner, milestone levels were assigned using a milestone classification form, and feedback was provided to learners. Learners and site leads were surveyed and/or interviewed about their experience as participants. Across the sites, 2,338 instruments assessing 239 learners were completed by 630 unique observers. Regarding end-of-rotation feedback, 93% of learners (128/137) agreed the assessments and feedback "helped me understand how those with whom I work perceive my performance," and 85% (117/137) agreed they were "useful for constructing future goals or identifying a developmental path." Site leads identified several benefits and challenges to the assessment process. PM-based instruments used in workplace-based assessment provide a meaningful and acceptable approach to collecting evidence of learner competency development. Learners valued feedback provided by PM-based assessment.

Rating the Quality of Entrustable Professional Activities: Content Validation and Associations with the Clinical Context.

Entrustable professional activities (EPAs) have been developed to assess resident physicians with respect to Accreditation Council for Graduate Medical Education (ACGME) competencies and milestones. Although the feasibility of using EPAs has been reported, we are unaware of previous validation studies on EPAs and potential associations between EPA quality scores and characteristics of educational programs. Our aim was to validate an instrument for assessing the quality of EPAs for assessment of internal medicine residents, and to examine associations between EPA quality scores and features of rotations. This was a prospective content validation study to design an instrument to measure the quality of EPAs that were written for assessing internal medicine residents. Residency leadership at Mayo Clinic, Rochester participated in this study. This included the Program Director, Associate program directors and individual rotation directors. The authors reviewed salient literature. Items were developed to reflect domains of EPAs useful for assessment. The instrument underwent further testing and refinement. Each participating rotation director created EPAs that they felt would be meaningful to assess learner performance in their area. These 229 EPAs were then assessed with the QUEPA instrument to rate the quality of each EPA. Performance characteristics of the QUEPA are reported. Quality ratings of EPAs were compared to the primary ACGME competency, inpatient versus outpatient setting and specialty type. QUEPA tool scores demonstrated excellent reliability (ICC range 0.72 to 0.94). Higher ratings were given to inpatient versus outpatient (3.88, 3.66; p = 0.03) focused EPAs. Medical knowledge EPAs scored significantly lower than EPAs assessing other competencies (3.34, 4.00; p < 0.0001). The QUEPA tool is supported by good validity evidence and may help in rating the quality of EPAs developed by individual programs. Programs should take care when writing EPAs for the outpatient setting or to assess medical knowledge, as these tended to be rated lower.

Role of item analysis in post validation of multiple choice questions in formative assessment of medical students

Abstract Background:Multiple choice questions (MCQ) are used in the assessment of students in various fields. By this method of assessment it is possible to cover a wide range of topics in less amount of time. However the reliability of the test depends on the quality of the MCQ. The MCQ can be evaluated based on the Difficulty Index (DIFI), Discriminatory Index (Dl) and Distracter Efficiency (DE). Objectives:To evaluate the MCQs based on the Difficulty Index (DIFI), Discriminatory Index (Dl) &amp; Distracter Efficiency (DE) and develop a valid pool of questions.Also to assess learner performance and discriminate between students of higher and lower abilities. Materials and Methods:A total of 120 students were assessed based on multiple choice questions in pathology. The number of items were 20 and the number of distracters were 60. Data was entered and analyzed in MS Excel 2007 and simple proportions, mean and standard deviations were calculated. Results:Mean and standard deviations for DIFI, Dl and DE were 57.8 ± 17.4%, 0.27 ± 0.17 and 84.98 ± 20.2 % respectively. Out of the 20 items, 11 items had good level of DIFI (31 - 60%), eight (8) items were considered easy (DIFI &amp;gt;61%) and one (1) item was considered difficult (DIFI &amp;lt; 30). Mean Dl in present study was 0.27 ± 0.17. Analysis of the Dl showed good discrimination power in eighteen (18) of the items. Out of the 60 distracters, nine (9) were non -functional distracters (NFD) and were seen in eight items. Out of these, seven items had one NFD each and one item had two NFD. Conclusions:The study emphasizes on the importance of use of item analysis in construction of good quality MCQs and also in the evaluation of learner performance.

A Fuzzy Logic Approach to Assess Web Learner's Joint Skills

Skill assessment is an important but complicated task in the entire web based teaching and learning process. The learner's performance assessment has a strong influence on learners' approaches to learn and their learning outcomes like professional acceptability on desired skills. Most educators focus either on assessing a learner's technical skill set or non- technical skill set, individually, rather than focusing on both the aspects. This paper bridges the gap by applying fuzzy logic approach to analyze a learner's joint skills incorporating both skills-set. An already proven e-commerce website's evaluation technique has been chosen and applied in two situations of learner's skill assessment through case studies namely: technical skills evaluation, and non-technical skills evaluation. Experiments show that the learner's success depends on both sets of skill attributes. This work then proposed a novel method for skill assessment considering two (instead of one) sets of skill attributes invoking parallel or joint application of the technique. This new technique has also been analysed through a case study. Index Terms—E-learning, fuzzy logic, joint-skill, skill assessment, competencies.

The Osteopathic Clinical Practice Assessment – a pilot study to develop a new workplace-based assessment tool

BackgroundIt is widely recognised that multiple forms of assessment are required in order to make decisions about learner competency and fitness-to-practice. One assessment tool that is gaining popularity in the Australian allied health professions is the global clinical competency assessment. These tools are designed to assess learner performance across a range of areas required for professional practice, with the assessment typically performed at the end of a ‘block’ clinical placement. There is no literature on their use in osteopathy longitudinal placements. ObjectiveTo explore the applicability of an adaptation of the Assessment of Physiotherapy Practice in an osteopathic context. The study presents pilot data and discussion about the adaptation of a workplace-based assessment tool – the Osteopathic Clinical Practice Assessment (OCPA). DesignEvaluation of the OCPA using data from multiple administrations. SettingOsteopathic on-campus, student-led teaching clinic at one Australian university. MethodsClinical Educators completed the OCPA for learners during weeks 5 and 12 of a 12 week semester. Descriptive statistics were generated for the data. The Wilcoxon signed rank test was used to investigate differences between the week 5 and 12 ratings. Cronbach's alpha was used to assess the internal consistency and Spearman's coefficient used to assess the relationship between the total OCPA score and global rating. ParticipantsThe OCPA was completed by 12 Clinical Educators assessing 31 learners in year 4 of the program. ResultsLearners were rated, on average, at the ‘expected level’ at each of the two time points. This was consistent for both the OCPA items and the global rating contained on the OCPA. Cronbach's alpha was 0.822 and Spearman's coefficient was 0.59 suggesting a ‘large’ relationship between the total OCPA score and the global rating. ConclusionsThere is a clear need for the OCPA to be used in conjunction with other assessment tools in order to make learner progress and fitness-to-practice decisions. Data suggest if learners are assessed at two or more different time points (across a longitudinal placement) changes need to be considered in the rating scale used on the OCPA in order to more clearly demonstrate learner progression. The OCPA could be used as a formative assessment tool in osteopathy with an adjustment to the current rating scale. In its current format caution should be applied if it is to be used as a summative tool as we do not have data supporting its use for this purpose. At present we cannot make any further arguments about the ability of the OCPA results to be extrapolated and make reliable and defensible decisions as there are no data correlating it with other performance assessments nor data supporting its reliability. This will be the subject of future research.