Simulation-based Assessment Research Articles

Content Validity for a 3D-Printed Model for Simulation-Based Training of Basic Epistaxis Management

Introduction Hands-on training of basic epistaxis management is often minimal during pregraduate medical training. The objective was to develop and gather structured content validity evidence for a 3D-printed model for simulation-based training and assessment of technical skills in basic epistaxis management. Methods A simulator for epistaxis management training was built based on publicly available 3D-print files with addition of tubing to mimic anterior bleeding. Ten otorhinolaryngologists evaluated the model after performing nasal cavity inspection and insertion and insufflation of a nasal tamponade device. Content validity evidence was collected following Messick's framework. Results Two content experts contributed to define features in the iterative building process. In the structured evaluation of the model, experienced clinicians found the appearance of the outer nose and the resistance when insufflating a nasal tamponade device into the nasal cavity satisfying (mean score of 3.9 and 4.7 out of 5, respectively), whereas the nasal cavity was found to be a bit too spacious compared with typical real-life conditions. Conclusions A 3D-printed simulator for simulation-based training of basic epistaxis management was successfully built. Content validity was gathered, and overall, content experts found the model to adequately represent the technical skills challenges for training novices such as medical students and junior doctors.

S-RAF: A Simulation-Based Robustness Assessment Framework for Responsible Autonomous Driving

As artificial intelligence (AI) technology advances, ensuring the robustness and safety of AI-driven systems has become paramount. However, varying perceptions of robustness among AI developers create misaligned evaluation metrics, complicating the assessment and certification of safety-critical and complex AI systems such as autonomous driving (AD) agents. To address this challenge, we introduce Simulation-Based Robustness Assessment Framework (S-RAF) for autonomous driving. S-RAF leverages the CARLA Driving simulator to rigorously assess AD agents across diverse conditions, including faulty sensors, environmental changes, and complex traffic situations. By quantifying robustness and its relationship with other safety-critical factors, such as carbon emissions, S-RAF aids developers and stakeholders in building safe and responsible driving agents, and streamlining safety certification processes. Furthermore, S-RAF offers significant advantages, such as reduced testing costs, and the ability to explore edge cases that may be unsafe to test in the real world.

Simulation-Based Education of Health Workers in Low- and Middle-Income Countries: A Systematic Review.

Simulation-based education (SBE) is increasingly used to improve clinician competency and patient care and has been identified as a priority by the World Health Organization for low- and middle-income countries (LMICs). The primary aim of this review was to investigate the global distribution and effectiveness of SBE for health workers in LMICs. The secondary aim was to determine the learning focus, simulation modalities, and additional evaluation conducted in included studies. A systematic review was conducted following the Preferred Reporting Items for Systematic Reviews and Meta Analysis guidelines, searching Ovid (Medline, Embase, and Emcare) and the Cochrane Library from January 1, 2002, to March 14, 2022. Primary research studies reporting evaluation at Level 4 of The Kirkpatrick model were included. Studies on simulation-based assessment and validation were excluded. Quality and risk-of-bias assessments were conducted using appropriate tools. Narrative synthesis and descriptive statistics were used to present the results. A total of 97 studies were included. Of these, 54 were in sub-Saharan Africa (56%). Forty-seven studies focused on neonatology (48%), 29 on obstetrics (30%), and 16 on acute care (16%). Forty-nine used mannequins (51%), 46 used scenario-based simulation (47%), and 21 used synthetic part-task trainers (22%), with some studies using more than 1 modality. Sixty studies focused on educational programs (62%), while 37 used SBE as an adjunct to broader interventions and quality improvement initiatives (38%). Most studies that assessed for statistical significance demonstrated at least partial improvement in Level 4 outcomes (75%, n=81). SBE has been widely applied to improve outcomes in a variety of contexts across LMICs. Modalities of simulation are typically low-technology versions. However, there is a lack of standardized reporting of educational activities, particularly relating to essential features of SBE. Further research is required to determine which approaches are effective in specific contexts.

Simulation-based assessment in the context of paramedic education: A scoping review.

Simulation is a widespread modality in the field of medical education. Within the paramedic sphere, simulation is valuable in providing exposure to high-acuity, low occurrence incidents encountered rarely in practice, affording unique educational opportunities. Recognising this importance, this scoping review seeks to establish the contemporaneous evidence base for the use of simulation-based assessment in the context of paramedic education, systematically map the research done in this area and consider the implications for future educational programmes. A scoping review of peer, and non-peer reviewed literature across a broad range of medical literature databases for both published and grey material utilising previously published search filters for the paramedic field. The review was conducted aligned to the PRISMA Extension for Scoping Reviews checklist. Studies were selected based on relevance to the research question. Twenty four unique papers were identified, filtered via the application of inclusion and exclusion criteria to five included papers. The application of forward snowballing methodology revealed three additional papers included for appraisal. Thematic analysis of the eight papers revealed the domains of assessment acceptability and assessment validity as key considerations for the design and use of simulation-based assessment in the field. Simulation-based assessment has a role in paramedic education; additional research is necessary to empirically establish the validity and reliability of the modality in the field.

Neutron phase filtering for separating phase- and attenuation signal in aluminium and anodic aluminium oxide

Neutron imaging has gained significant importance as a material characterisation technique and is particularly useful to visualise hydrogenous materials in objects opaque to other radiations. Fields of application include investigations of hydrogen in metals as well as metal corrosion, thanks to the fact that neutrons can penetrate metals better than e.g. X-rays and are highly sensitive to hydrogen. However, at interfaces refraction effects sometimes obscure the attenuation image, which is used for hydrogen quantification. Refraction, a differential phase effect, diverts the neutron beam away from the interface in the image leading to intensity gain and intensity loss regions, which are superimposed to the attenuation image, thus obscuring the interface region and hindering quantitative analyses of e.g. hydrogen content in the vicinity of the interface. For corresponding effects in X-ray imaging, a phase filter approach was developed and is generally based on transport-of-intensity considerations. Here, we compare such an approach, that has been adapted to neutrons, with another simulation-based assessment using the ray-tracing software McStas. The latter appears superior and promising for future extensions which enable fitting forward models via simulations in order to separate phase and attenuation effects and thus pave the way for overcoming quantitative limitations at refracting interfaces.

A simulation-based techno-economic-safety-social-environmental decision-making framework for prioritizing plastic waste treatment processes

Chemical recycling technologies hold promising potential in converting mixed plastic waste into energy and chemicals. This study proposed a simulation-based quantitative life cycle sustainability assessment framework for the potential plastic waste valorization processes. Sustainability metrics including environment, economy, safety, society, and energy were quantified and comprehensively investigated. Based on the Bayesian best-worst weighting and vector-based ranking methods, the normalized and weighted life cycle sustainability scores were used to prioritize the best process. Gasification-based technology was found superior than incineration-based technology, especially in the economic, techno-energetic, and social aspects. However, incinerated-based process with direct CO2 emissions performed better in the environmental aspects compared with the process integrating with carbon capture and utilization unit. Additional environmental burdens were brought to the system when incorporating the carbon capture and methanol synthesis process. Increased system complexity might bring unexpected environmental burdens, and sometimes even outweighed the benefits. Various weights were assigned to the multi-criteria decision-making model, and the gasification-based process consistently ranked first across all scenarios.

A dynamic resilience management framework for deep-tier supply networks

The unprecedented supply chain disruptions caused by COVID-19 has had severe operational and financial consequences to firms across industries and continents. While tactical reactionary strategies can help, firms are in need of proactive management approaches to design more resilient supply chain networks in the first place. Firms are looking for an effective framework to design and monitor supply networks, mitigate disruption consequences, and manage resilience under different scenarios. We propose a framework to manage the resilience of deep-tier automotive supply networks by integrating a simulation-based resilience assessment scheme for effectiveness with an efficient optimization-based framework to find optimal strategies for handling regular disruption events. The framework promotes network analysis techniques combined with discrete-event simulation informed by secondary data sources and global supply risk databases for improving resilience management. We validate the effectiveness of the proposed framework using a real-world global automotive original equipment manufacturer case study. Our results demonstrate that the proposed dynamic framework relying on deep-tier visibility can optimize resilience strategies through all key performance indicators. The results show an average of 35% and 40% reductions in back-ordered cost and shipment delays, respectively, with a marginal growth in holding cost when the proposed framework is implemented with deep-tier visibility.

Simulation-based assessment of railhead repair welding process parameters

Simulation-based assessment of railhead repair welding process parameters

Active Sampling: A Machine-Learning-Assisted Framework for Finite Population Inference with Optimal Subsamples

Data subsampling has become widely recognized as a tool to overcome computational and economic bottlenecks in analyzing massive datasets. We contribute to the development of adaptive design for estimation of finite population characteristics, using active learning and adaptive importance sampling. We propose an active sampling strategy that iterates between estimation and data collection with optimal subsamples, guided by machine learning predictions on yet unseen data. The method is illustrated on virtual simulation-based safety assessment of advanced driver assistance systems. Substantial performance improvements are demonstrated compared to traditional sampling methods.

Neuromechanisms of simulation-based arthroscopic skills assessment: a fNIRS study.

The neural mechanisms underlying differences in the performance of simulated arthroscopic skills across various skill levels remain unclear. Our primary objective is to investigate the learning mechanisms of simulated arthroscopic skills using functional near-infrared spectroscopy (fNIRS). We recruited 27 participants, divided into three groups: novices (n = 9), intermediates (n = 9), and experts (n = 9). Participants completed seven arthroscopic tasks on a simulator, including diagnostic navigation, triangulation, grasping stars, diagnostic exploration, meniscectomy, synovial membrane cleaning, and loose body removal. All tasks were videotaped and assessed via the simulator system and the Arthroscopic Surgical Skill Evaluation Tool (ASSET), while cortical activation data were collected using fNIRS. Simulator scores and ASSET scores were analyzed to identify different level of performance of all participants. Brain region activation and functional connectivity (FC) of different types of participants were analyzed from fNIRS data. Both the expert and intermediate groups scored significantly higher than the novice group (p < 0.001). There were significant differences in ASSET scores between experts and intermediates, experts and novices, and intermediates and novices (p = 0.0047, p < 0.0001, p < 0.0001), with the trend being experts > intermediates > novices. The intermediate group exhibited significantly greater activation in the left primary motor cortex (LPMC) and left prefrontal cortex (LPFC) compared to the novice group (p = 0.0152, p = 0.0021). Compared to experts, the intermediate group demonstrated significantly increased FC between the presupplementary motor area (preSMA) and the right prefrontal cortex (RPFC; p < 0.001). Additionally, the intermediate group showed significantly increased FC between the preSMA and LPFC, RPFC and LPFC, and LPMC and LPFC compared to novices (p = 0.0077, p = 0.0285, p = 0.0446). Cortical activation and functional connectivity reveal varying levels of activation intensity in the PFC, PMC, and preSMA among novices, intermediates, and experts. The intermediate group exhibited the highest activation intensity.

A Simulation-Based Assessment of Levetiracetam Concentrations Following Fixed and Weight-Based Loading Doses: A Meta-Regression and Pharmacokinetic Modeling Analysis.

Current recommendations for refractory status epilepticus (SE) unresponsive to benzodiazepines suggest a loading dose of levetiracetam (LEV) of 60 mg/kg to a maximum of 4500 mg. LEV therapeutic drug monitoring can help guide therapy and is garnering increasing attention. The objective of this study is to simulate the probability of target attainment (PTA) of fixed dose and weight-based loading doses of LEV with respect to established therapeutic target concentrations. Meta-regression of the current literature was performed to evaluate the relationship between intravenous LEV loading dose and seizure cessation in refractory SE patients. A previously published pharmacokinetic model was used to simulate the PTA capacity of competing single intravenous dosing schemes (fixed vs weight-based dosing) to achieve maximum (Cpeak) and 12-h (C12h) plasma concentrations that exceed 12 mg/L. The meta-regression indicated that dosage was not a statistically significant modulator of seizure control at dosages between 20 and 60 mg/kg. Stochastic simulations showed all dosing schemes achieved plasma Cpeak >12 mg/L, but C12h levels were <12 mg/L in subjects over 60 kg with a fixed dose ≤2000 mg or in subjects <60 kg with a weight-based dose <30 mg/kg. Dosages of 40 and 60 mg/kg provided ≥90% PTAs across all weights. Using a weight-based loading dose of 40 mg/kg, up to a suggested maximum of 4500 mg, improves the likelihood of achieving a sustained therapeutic drug concentration after the initial LEV dose, whereas fixed <3000 mg may not achieve the desired concentration before maintenance dosing.

Augmenting intensive care unit nursing practice with generative AI: A formative study of diagnostic synergies using simulation-based clinical cases.

As generative artificial intelligence (GenAI) tools continue advancing, rigorous evaluations are needed to understand their capabilities relative to experienced clinicians and nurses. The aim of this study was to objectively compare the diagnostic accuracy and response formats of ICU nurses versus various GenAI models, with a qualitative interpretation of the quantitative results. This formative study utilized four written clinical scenarios representative of real ICU patient cases to simulate diagnostic challenges. The scenarios were developed by expert nurses and underwent validation against current literature. Seventy-four ICU nurses participated in a simulation-based assessment involving four written clinical scenarios. Simultaneously, we asked ChatGPT-4 and Claude-2.0 to provide initial assessments and treatment recommendations for the same scenarios. The responses from ChatGPT-4 and Claude-2.0 were then scored by certified ICU nurses for accuracy, completeness and response. Nurses consistently achieved higher diagnostic accuracy than AI across open-ended scenarios, though certain models matched or exceeded human performance on standardized cases. Reaction times also diverged substantially. Qualitative response format differences emerged such as concision versus verbosity. Variations in GenAI models system performance across cases highlighted generalizability challenges. While GenAI demonstrated valuable skills, experienced nurses outperformed in open-ended domains requiring holistic judgement. Continued development to strengthen generalized decision-making abilities is warranted before autonomous clinical integration. Response format interfaces should consider leveraging distinct strengths. Rigorous mixed methods research involving diverse stakeholders can help iteratively inform safe, beneficial human-GenAI partnerships centred on experience-guided care augmentation. This mixed-methods simulation study provides formative insights into optimizing collaborative models of GenAI and nursing knowledge to support patient assessment and decision-making in intensive care. The findings can help guide development of explainable GenAI decision support tailored for critical care environments. Patients or public were not involved in the design and implementation of the study or the analysis and interpretation of the data.

Sample size considerations for single-arm clinical trials with time-to-event endpoint using the gamma distribution

BackgroundTime-to-event (TTE) endpoints are evaluated as the primary endpoint in single-arm clinical trials; however, limited options are available in statistical software for sample size calculation. In single-arm trials with TTE endpoints, the non-parametric log-rank test is commonly used. Parametric options for single-arm design assume survival times follow exponential distribution or Weibull distribution. MethodsThe exponential- or Weibull-distributed survival time assumption does not always reflect hazard pattern of real-life diseases. We therefore propose gamma distribution as an alternative parametric option for designing single-arm studies with TTE endpoints. We outline a sample size calculation approach using gamma distribution with a known shape parameter and explain how to extract the gamma shape estimate from previously published resources. In addition, we conduct simulations to assess the accuracy of the extracted gamma shape parameter and to explore the impact on sample size calculation when survival time distribution is misspecified. ResultsOur simulations show that if a previously published study (sample sizes ≥ 60 and censoring proportions ≤ 20 %) reported median and inter-quartile range of survival time, we can obtain a reasonably accurate gamma shape estimate, and use it to design new studies. When true survival time is Weibull-distributed, sample size calculation could be underestimated or overestimated depending on the hazard shape. ConclusionsWe show how to use gamma distribution in designing a single-arm trial, thereby offering more options beyond the exponential and Weibull. We provide a simulation-based assessment to ensure an accurate estimation of the gamma shape and recommend caution to avoid misspecification of the underlying distribution.

Extracting valuable metals from zinc sulfide concentrate: A comprehensive simulation-based life cycle assessment study of oxidative pressure leaching

Considering the increasing environmental consciousness and the imperative for sustainable development, conventional zinc smelting operations employing the roast-leaching-electrowinning process face growing environmental pressures. Consequently, the oxidative pressure leaching process has emerged as an alternative processing route. This study focuses on recovery of major metals from ZnS concentrate, building a comprehensive model to simulate the oxidative pressure leaching process. The process model built demonstrated that zinc, sulfur, indium, gallium, cadmium, cobalt, germanium, and copper could be recovered effectively, the main chemical inputs being sulfuric acid, oxygen, and lime, while zinc electrowinning constituting over 90 % of the total electricity consumption. The results were used to compile a life cycle inventory, culminating in a gate-to-gate assessment of the environmental impacts of the process. The global warming potential for the treatment of 1 t of ZnS concentrate is approximately 2,000 kg CO2 eq. The primary environmental impacts in the process are from chemical inputs and electricity consumption. Based on these findings, recommendations for sustainable production practices are proposed. These include systematic recovery and reuse of electrolytic off-gas and roasting heat, greater utilization of renewable energy, enhanced iron removal processes, and exploration of alternative chemicals for Ga and Ge recovery.

A Simple Strategy for Identifying Conserved Features across Non-independent Omics Studies.

False discovery is an ever-present concern in omics research, especially for burgeoning technologies with unvetted specificity of their biomolecular measurements, as such unknowns obscure the ability to characterize biologically informative features from studies performed with any single platform. Accordingly, performing replication studies of the same samples using different omics platforms is a viable strategy for identifying high-confidence molecular associations that are conserved across studies. However, an important caveat of replication studies that include the same samples is that they are inherently non-independent, leading to overestimating conservation if studies are treated otherwise. Strategies for accounting for such inter-study dependencies have been proposed for meta-analysis methods devised to increase statistical power to detect molecular associations in one or more studies. Still, they are not immediately suited for identifying conserved molecular associations across multiple studies. Here, we present a unifying strategy for performing inter-study conservation analysis as an alternative to meta-analysis strategies for aggregating summary statistical results of shared features across complementary studies while accounting for inter-study dependency. This method, which we call "adjusted maximum p-value" (AdjMaxP), is easy to implement with inter-study dependency and conservation estimated directly from the p-values from each study's molecular feature-level association testing results. Through simulation-based assessment, we demonstrate AdjMaxP's improved performance for accurately identifying conserved features over a related meta-analysis strategy for non-independent studies. AdjMaxP offers an easily implementable strategy for improving the precision of analyses for biomarker discovery from cross-platform omics study designs, thereby facilitating the adoption of such protocols for robust inference from emerging omics technologies.

Using Simulation-Based Assessment for Event Management Students

This research project explores the effectiveness of simulation-based assessments (SBAs) as an alternative to practical assessments for event management (EM) students, considering the 2020 global pandemic's impact on higher education. SBAs mimic real-world scenarios, enabling innovative and authentic ways for assessors to determine how well candidates can put their knowledge, skills, and talents to use. For example, they give students an opportunity to instil and reinforce ethical principles and values essential for success in the workplace. The study aims to investigate the relationship between SBAs and graduate attributes, via a qualitative approach, to investigate and analyse students' lived experiences. Ten final-year EM students were selected purposively, with inclusion criteria. Semi-structured interviews were conducted using an original set of selected questions, and thematic analysis was used for data analysis. The study's findings suggest that if used correctly, SBAs can be an effective tool for developing students' graduate attributes, preparing them for employment. The article concludes on how to maximise the value of SBAs in preparing students for the workforce, especially when practical assessments are not feasible

Development and validation of a simulation-based assessment of operative competence for higher specialist trainees in general surgery

BackgroundSimulation is increasingly being explored as an assessment modality. This study sought to develop and collate validity evidence for a novel simulation-based assessment of operative competence. We describe the approach to assessment design, development, pilot testing, and validity investigation.MethodsEight procedural stations were generated using both virtual reality and bio-hybrid models. Content was identified from a previously conducted Delphi consensus study of trainers. Trainee performance was scored using an equally weighted Objective Structured Assessment of Technical Skills (OSATS) tool and a modified Procedure-Based Assessment (PBA) tool. Validity evidence was analyzed in accordance with Messick’s validity framework. Both ‘junior’ (ST2–ST4) and ‘senior’ trainees (ST 5–ST8) were included to allow for comparative analysis.ResultsThirteen trainees were assessed by ten assessors across eight stations. Inter-station reliability was high (α = 0.81), and inter-rater reliability was acceptable (inter-class correlation coefficient 0.77). A significant difference in mean station score was observed between junior and senior trainees (44.82 vs 58.18, p = .004), while overall mean scores were moderately correlated with increasing training year (rs = .74, p = .004, Kendall’s tau-b .57, p = 0.009). A pass-fail score generated using borderline regression methodology resulted in all ‘senior’ trainees passing and 4/6 of junior trainees failing the assessment.ConclusionThis study reports validity evidence for a novel simulation-based assessment, designed to assess the operative competence of higher specialist trainees in general surgery.Graphical abstract

RescuePY: Simulation-Based Rescue Response Impact Assessment

Mobility in metropolitan regions is changing. The distribution of space in cities, the design of transport modes, and the organization of mobility are being re-thought. However, no matter the changes and innovations on the way to a more sustainable future, essential constants must be upheld: In the event of minor, regionally limited emergencies, medical assistance must reach those in need quickly. When dealing with large-scale emergencies, the ability to evacuate the area promptly must be ensured. The impact analysis of mobility innovations on emergency services within urban areas so far has been based purely on empirical observations using existing data. Currently, it is only possible to analyze what-if considerations in a limited way. Nevertheless, due to the increasingly rapid changes in mobility, a comprehensive and interlinked analysis will be necessary. This is the key contribution of rescuePY: rescuePY is a simulation suite based on the mesoscopic and microscopic simulation environment hybridPY. It allows holistic and microscopic transport modeling of rescue infrastructure to quantify the impact of the mobility transition towards higher sustainability on the performance of rescue services. The main features of this software are: Rescue system assessment for strategic, long-term planning Mobility-influence studies for operative, mid-term planning Activity-based urban evacuation modeling The capabilities of rescuePY are demonstrated by two applications: a simulation- based, mesoscopic system analysis of emergency services in Munich compared to real-world data and microscopic modeling of emergency vehicles (EMVs) in different road architectures. Ongoing developments aim to improve the evaluation methodology for the aggregated impact analysis of mobility innovations on rescue response services.

Simulation-based personal fatality risk assessment due to the fragmentation hazard

In the military and chemical industry, modeling the fragmentation hazard field is of great significance in conducting the fatality risk assessment and calculating the safety distance for person. Although the ballistic methodology achieves the simulation of fragmentation flight trajectory, the acquisition of accurate initial projection data of fragmentation is a challenge. By coupling continuum-discontinuum element method and particle discrete element method, the fragmentation power algorithm is established, which achieves the integrated simulation of the initial projection data of fragmentation and the flight process of fragmentation. First, continuum-discontinuum element method is adopted to simulate the detonation products-driven fragmentation acceleration process by introducing the explosive detonation model, which achieves the acquisition of initial projection data of fragmentation. Then, the particle discrete element method is adopted to simulate the flight trajectory and power data of fragmentation. Based on the numerical and experimental result, the accuracy of fragmentation power algorithm is verified. In conjunction with the personal vulnerability model, a systematic numerical simulation framework is established to conduct the simulation-based personal fatality risk assessment when the metal-cased munition detonates accidentally, and the results indicate that the personal fatality risk due to the fragmentation becomes severer with the increase of munition curvature.

807 Reliability of Simulation-Based Assessment of Surgical Technical Skills

Abstract Aim To examine the reliability of surgical trainees’ assessment data from a structured simulation training and assessment programme. Method The General Surgical Skills Programme is a simulation-based surgical training programme integrated within the clinical training pathway for surgical trainees across London. It is tailored to each trainee’s level of training and subspecialty. The ‘Junior’ pathway involves nine training sessions and one assessment session performing three procedures (Inguinal hernia [IH], laparoscopic cholecystectomy [LC] and small bowel anastomoses [SBA]). Within the scope of the Programme, skills assessment data collection was initiated in 2012. Data collected includes demographics and training data (grade/subspecialty/case numbers). Each procedure is assessed by two consultant surgeons using two procedure specific assessment tools, a visual analogue scale (VAS) and competency assessment tool (CAT). Reliability analyses were performed using SPSS for ‘Junior’ data over a four-year period. Results In total, 355 surgical trainees (Male = 199, Female = 154, Missing = 2) were assessed from 2014 to 2017 (2014 n=105; 2015 n=92; 2016 n=89; 2017 n=69). Both assessment tools showed satisfactory inter-item reliability (IH-CAT α=.794-.903, LC-CAT α=.837–910, IH-VAS α=.851-.991, LC-VAS α=.831-.986, SBA-VAS α=.778-.984). Inter-rater reliability, ICC (1,2), was good to poor across all tools. (IH-CAT .345-.510, IH-VAS .243-.600, LC-CAT .271-.766, LC-VAS .157-.678, SBA-VAS .225-.278). Conclusions The assessment tools used within the Programme both demonstrate satisfactory inter-item reliability. Inter-rater reliability is variable, highlighting a potential utility for assessor training within the Programme.