Objective Assessment Of Skills Research Articles

Skills Acquisition and Engagement with the Use of a Structured Journal Club Curriculum.

Journal clubs are a common element of many residency programs. However, there is little data showing that they improve residents' ability to interpret and use the scientific literature. The authors developed a structured journal club curriculum intended to improve scientific literacy skills among residents and increase acceptability and participation in meaningful discussion of high-quality articles among both residents and facilitators. This study examined the impact of the curriculum on objective skills, subjective skills, and engagement. Psychiatry residents from three academic medical centers completed 10-12 virtual journal clubs during academic year 2022-2023. At the start and end of that year, residents completed a knowledge assessment and a self-rating of scientific literacy skills; at the end of the year, residents self-rated participation in, preparation for, and attitudes towards journal club sessions and assessment of scientific literacy skills. Twenty-seven residents completed self-reports at both time points. Performance on the objective skills assessment did not improve over the course of the year. Subjectively, residents reported increased understanding of common statistical techniques, ability to determine if techniques used in an article are appropriate, and the extent to which information learned from articles informs clinical practice. In the sample of 62 resident engagement surveys at follow-up, the journal club curriculum was well-received. Use of a structured journal club curriculum was associated with improvements in self-rated scientific literacy skills among residents and had high acceptability. More work is needed to understand the impact of teaching methodology on journal club success.

Hypospadias Reconstruction Training: Development of an Ex-Vivo Model for Objective Evaluation of Surgical Skills

To objectively evaluate technical skill acquisition in hypospadias repair procedures during surgical training using noninvasive wearable sensor technology. We combined subjective video evaluations with objective electromyography (EMG) measurements in a hands-on hypospadias training course. Surgeons wore wireless EMG and accelerometer sensors on their dominant hand while performing tasks on ex-vivo cadaveric calf penises. The study focused on 4 skills as follows: urethral mobilization, dorsal inlay graft harvest/implantation, meatal-based flap urethroplasty, and dorsal plication. Machine learning techniques analyzed muscle activation patterns and attributes for assessing surgical precision. The course included 18 participants (10 female, 8 males; average age 40.18±8.46 years) categorized as novice (n=10, <3 years' experience), intermediate (n=5, 3-5 years), and expert (n=3, >5 years). Video evaluations did not reveal significant differences due to short-term training. However, EMG measurements showed significant reductions in average EMG power, total time, dominant frequency, and cumulative muscle workload after training. Additionally, the mean power spectral density of the EMG signal decreased notably post-training. This study presents a structured approach for hypospadias training and highlights the effectiveness of wearable sensor technology for objective skill assessment. While video evaluations did not detect significant changes, EMG data provided measurable differences in skill acquisition, suggesting that wearable sensors could enhance objective evaluations of surgical proficiency in residency programs.

428 Artificial Intelligence Training Versus In-person Expert Training in Teaching Simulated Tumor Resection Skills - A Cross-Over Randomized Controlled Trial

INTRODUCTION: Neurosurgical simulations equipped with artificial intelligence systems provide an objective quantitative assessment of surgical technical skills and tailored intelligent feedback. These systems allow for repeated practice of complex skills such as subpial brain tumor resection. However, more work is needed to assess the utility of intelligent systems in teaching tumor resection skills, compared to traditional human instruction. METHODS: Twenty-five trainees who are currently enrolled in four Canadian medical schools participated in two training sessions, during which they completed a simulated subpial tumor resection five times. Participants were randomly assigned to two feedback groups: (1) real-time intelligent instruction, and (2) in-person human instruction. They were then assigned to the other feedback group in the second (cross-over) session. A composite-score was given by the intelligent system to assess technical skills during each task. RESULTS: Trainees who received real-time intelligent instruction significantly improved their composite-score in the first and second training sessions (p = .017, p = .005, respectively). Trainees who received in-person human instruction in the first training session had no statistically significant changes in their composite-score from the first to the last task repetition (p = .119). The composite-score decreased significantly in the second training session with in-person human instruction (p = .004). CONCLUSIONS: Real-time intelligent instruction provides an effective way of teaching simulated brain tumor resection skills. Artificial intelligence systems may be a useful addition to current surgical training curricula, providing objective, and tailored assessment and teaching of surgical technical skills in risk-free realistically simulated patient cases. Educators can benefit from these systems to guide their supervision, while trainees can use them for self-guided skill acquisition. Further research may assess the generalizability of these findings to other surgical procedures.

An objective skill assessment framework for microsurgical anastomosis based on ALI scores

IntroductionThe current assessment and standardization of microsurgical skills are subjective, posing challenges in reliable skill evaluation. We aim to address these limitations by developing a quantitative and objective framework for accurately assessing and enhancing microsurgical anastomosis skills among surgical trainees. We hypothesize that this framework can differentiate the proficiency levels of microsurgeons, aligning with subjective assessments based on the ALI score.MethodsWe select relevant performance metrics from the literature on laparoscopic skill assessment and human motor control studies, focusing on time, instrument kinematics, and tactile information. This information is measured and estimated by a set of sensors, including cameras, a motion capture system, and tactile sensors. The recorded data is analyzed offline using our proposed evaluation framework. Our study involves 12 participants of different ages (35.42±9.78\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$35.42\\pm 9.78$$\\end{document} years) and genders (nine males and three females), including six novice and six intermediate subjects, who perform surgical anastomosis procedures on a chicken leg model.ResultsWe show that the proposed set of objective and quantitative metrics to assess skill proficiency aligns with subjective evaluations, particularly the ALI score method, and can effectively differentiate novices from more proficient microsurgeons. Furthermore, we find statistically significant disparities, where microsurgeons with intermediate level of skill proficiency surpassed novices in both task speed, reduced idle time, and smoother, briefer hand displacements.ConclusionThe framework enables accurate skill assessment and provides objective feedback for improving microsurgical anastomosis skills among surgical trainees. By overcoming the subjectivity and limitations of current assessment methods, our approach contributes to the advancement of surgical education and the development of aspiring microsurgeons. Furthermore, our framework emerges to precisely distinguish and classify proficiency levels (novice and intermediate) exhibited by microsurgeons.

Enhancing proficiency in ascites puncture through procedural simulation: an audit type study investigating medium-term skill retention.

Medical simulation has become an essential teaching method for all health professionals. It not only allows to acquire technical and non-technical knowledge, but also helps the maintenance of acquired knowledge in the medium and long term. Ascites puncture is part of the basic technical procedures learned by medical students during their internship. To evaluate the role of simulation-based learning of ascites puncture on the improvement of theoretical knowledge and maintenance of skills at 3 months. We conducted an audit type study with two cycles of data collection at the simulation center at the Faculty of Medicine of Sousse between November 2020 and June 2021. We included learners in their third year of medical studies who had a hospital internship in the gastroenterology department at Sahloul Hospital in Sousse. All learners attended the initial simulation session on ascites fluid puncture. Thereafter, they were free to accept or refuse participation in the evaluation session that was scheduled after 3 months, depending on their availability. Forty learners participated in the procedural simulation of the ascites fluid puncture technique. Thirty-four (85%) were female and six (5%) were male. In our study, we showed that following procedural simulation training of ascites puncture, there was a significant improvement in the theoretical knowledge of the learners (p < 0.000). Objective assessment of technical skills after 3 months showed the benefit of performance maintenance (p < 0.000). Our study confirmed the benefit of simulation-based learning on the improvement of theoretical knowledge and the maintenance of technical performance in the medium term.

Development and validation of an objective assessment of surgical skill in arthroscopic management of meniscal tear: A pilot study

Development and validation of an objective assessment of surgical skill in arthroscopic management of meniscal tear: A pilot study

Measuring hand movement for suturing skill assessment: A simulation-based study

BackgroundTo maximize patient safety, surgical skills education is increasingly adopting simulation-based curricula for formative skills assessment and training. However, many standardized assessment tools rely on human raters for performance assessment, which is resource-intensive and subjective. Simulators that provide automated and objective metrics from sensor data can address this limitation. We present an instrumented bench suturing simulator, patterned after the clock face radial suturing model from the Fundamentals of Vascular Surgery, for automated and objective assessment of open suturing skills. MethodsFor this study, 97 participants (35 attending surgeons, 32 residents, and 30 novices) were recruited at national vascular conferences. Automated hand motion metrics, especially focusing on rotational motion analysis, were developed from the inertial measurement unit attached to participants’ hands, and the proposed suite of metrics was used to differentiate between the skill levels of the 3 groups. ResultsAttendings’ and residents’ performances were found to be significantly different from novices for all metrics. Moreover, most of our novel metrics could successfully distinguish between finer skill differences between attending and resident groups. In contrast, traditional operative skill metrics, such as time and path length, were unable to distinguish attendings from residents. ConclusionThis study provides evidence for the effectiveness of rotational motion analysis in assessing suturing skills. The suite of inertial measurement unit-based hand motion metrics introduced in this study allows for the incorporation of hand movement data for suturing skill assessment.

Assessing Parental Competence and Self-Ratings in Management of Pediatric Type 1 Diabetes and Emergency Glucagon Administration-An Exploratory Observational Study.

Parents of pediatric patients with type I diabetes require competence in hypoglycemia management and skills in glucagon administration to deal with potentially life-threatening severe hypoglycemia. We aimed to compare parents' subjective self-ratings to an objective expert assessment of competences and skills in dealing with severe hypoglycemia. We interviewed 140 participants to assess their subjective self-ratings. The objective expert assessments used a standardized clinical case scenario of severe hypoglycemia and a practical demonstration of glucagon administration. The participants self-rated their competence in hypoglycemia management as good (5) or very good (6), and their skills in administering glucagon as acceptable (3) [Scale: very poor (1) to very good (6)]. In the standardized clinical case scenario, 1.4% (2/140) of participants named all relevant steps of severe hypoglycemia management. In the practical demonstration of glucagon administration, 92.9% (130/140) of participants committed at least one drug handling error; 52.1% (73/140) committed at least one drug handling error rated with high clinical risk. We found discrepancies regarding participants' subjective self-ratings compared to their performance in the respective objective expert assessments. These discrepancies indicate a lack of error awareness and the need for intervention studies to improve competence in hypoglycemia management and glucagon administration.

COMPARING THE EFFICIENCY OF A REAL-TIME ARTIFICIAL INTELLIGENCE INSTRUCTOR TO HUMAN EXPERT INSTRUCTORS IN SIMULATED SURGICAL TECHNICAL SKILLS TRAINING– A RANDOMIZED CONTROLLED TRIAL

Abstract BTFC travel award recipient Artificial intelligence systems provide risk-free training on realistically simulated patient cases and objective assessment of surgical technical skills. This randomized controlled study compared a real-time intelligent tutoring system in technical skills learning with human expert instructor-mediated training. METHODS: Ninety-eight medical students performed six simulated brain tumor resections. Participants were randomly allocated into (1)no-real-time feedback, (2)real-time intelligent instruction, and (3)in- person human instruction. All students performed the first repetition without receiving feedback (baseline). Group-1 received visual feedback only after each procedure based on expert benchmarks. Group-2 was instructed by the intelligent system in real-time. After each task, the students were shown their error-video clips generated by this system alongside expert-level demonstrations on how to improve. Group-3 was instructed by human instructors during the tasks. After each task, instructors summarized the areas of improvement and demonstrated correction techniques. Participant performance was scored by the intelligent system and also by blinded experts using OSATS scores. The performance score was compared within groups and between groups to compare learning. RESULTS: Compared to baseline performance, Group-2 and Group-3 significantly improved in the performance score by the third and second repetition, respectively (p&amp;lt;0.01, p=0.01). The between-groups comparison demonstrated that Group-2 scored significantly higher than Group-3 in the fifth repetition (p&amp;lt;0.01). Group-2 achieved significantly higher OSATS scores than Group-1 in the sixth task. CONCLUSIONS: Artificial intelligence may facilitate trainee learning by providing equally or more efficient learning when compared to human instruction. These systems may aid in developing competency-based standardized curricula in surgical training.

FAST skill assessment from kinematics data using convolutional neural networks

FAST is a point of care ultrasound study that evaluates for the presence of free fluid, typically hemoperitoneum in trauma patients. FAST is an essential skill for Emergency Physicians. Thus, it requires objective evaluation tools that can reduce the necessity of direct observation for proficiency assessment. In this work, we use deep neural networks to automatically assess operators' FAST skills. We propose a deep convolutional neural network for FAST proficiency assessment based on motion data. Prior work has shown that operators demonstrate different domain-specific dexterity metrics that can distinguish novices, intermediates, and experts. Therefore, we augment our dataset with this domain knowledge and employ fine-tuning to improve the model's classification capabilities. Our model, however, does not require specific points of interest (POIs) to be defined for scanning. The results show that the proposed deep convolutional neural network can classify FAST proficiency with 87.5% accuracy and 0.884, 0.886, 0.247 sensitivity for novices, intermediates, and experts, respectively. It demonstrates the potential of using kinematics data as an input in FAST skill assessment tasks. We also show that the proposed domain-specific features and region fine-tuning increase the model's classification accuracy and sensitivity. Variations in probe motion at different learning stages can be derived from kinematics data. These variations can be used for automatic and objective skill assessment without prior identification of clinical POIs. The proposed approach can improve the quality and objectivity of FAST proficiency evaluation. Furthermore, skill assessment combining ultrasound images and kinematics data can provide a more rigorous and diversified evaluation than using ultrasound images alone.

Relating process and outcome metrics for meaningful and interpretable cannulation skill assessment: A machine learning paradigm

Background and Objectives: The quality of healthcare delivery depends directly on the skills of clinicians. For patients on hemodialysis, medical errors or injuries caused during cannulation can lead to adverse outcomes, including potential death. To promote objective skill assessment and effective training, we present a machine learning approach, which utilizes a highly-sensorized cannulation simulator and a set of objective process and outcome metrics. Methods: In this study, 52 clinicians were recruited to perform a set of pre-defined cannulation tasks on the simulator. Based on data collected by sensors during their task performance, the feature space was then constructed based on force, motion, and infrared sensor data. Following this, three machine learning models– support vector machine (SVM), support vector regression (SVR), and elastic net (EN)– were constructed to relate the feature space to objective outcome metrics. Our models utilize classification based on the conventional skill classification labels as well as a new method that represents skill on a continuum. Results: With less than 5% of trials misplaced by two classes, the SVM model was effective in predicting skill based on the feature space. In addition, the SVR model effectively places both skill and outcome on a fine-grained continuum (versus discrete divisions) that is representative of reality. As importantly, the elastic net model enabled the identification of a set of process metrics that highly impact outcomes of the cannulation task, including smoothness of motion, needle angles, and pinch forces. Conclusions: The proposed cannulation simulator, paired with machine learning assessment, demonstrates definite advantages over current cannulation training practices. The methods presented here can be adopted to drastically increase the effectiveness of skill assessment and training, thereby potentially improving clinical outcomes of hemodialysis treatment.

Objective assessment of intraoperative skills for robot-assisted partial nephrectomy (RAPN)

RAPN training usually takes place in-vivo and methods vary across countries/institutions. No common system exists to objectively assess trainee capacity to perform RAPN at predetermined performance levels prior to in-vivo practice. The identification of objective performance metrics for RAPN training is a crucial starting point to improve training and surgical outcomes. The authors sought to examine the reliability, construct and discriminative validity of objective intraoperative performance metrics which best characterize the optimal and suboptimal performance of a reference approach for training novice RAPN surgeons. Seven Novice and 9 Experienced RAPN surgeons video recorded one or two independently performed RAPN procedures in the human. The videos were anonymized and two experienced urology surgeons were trained to reliably score RAPN performance, using previously developed metrics. The assessors were blinded to the performing surgeon, hospital and surgeon group. They independently scored surgeon RAPN performance. Novice and Experienced group performance scores were compared for procedure steps completed and errors made. Each group was divided at the median for Total Errors score, and subgroup scores (i.e., Novice HiErrs and LoErrs, Experienced HiErrs and LoErrs) were compared. The mean inter-rater reliability (IRR) for scoring was 0.95 (range 0.84–1). Compared with Novices, Experienced RAPN surgeons made 69% fewer procedural Total Errors. This difference was accentuated when the LoErr Expert RAPN surgeon’s performance was compared with the HiErrs Novice RAPN surgeon’s performance with an observed 170% fewer Total Errors. GEARS showed poor reliability (Mean IRR = 0.44; range 0.0–0.8), for scoring RAPN surgical performance. The RAPN procedure metrics reliably distinguish Novice and Experienced surgeon performances. They further differentiated performance levels within a group with similar experiences. Reliable and valid metrics will underpin quality-assured novice RAPN surgical training.

Next in Surgical Data Science: Autonomous Non-Technical Skill Assessment in Minimally Invasive Surgery Training.

It is well understood that surgical skills largely define patient outcomes both in Minimally Invasive Surgery (MIS) and Robot-Assisted MIS (RAMIS). Non-technical surgical skills, including stress and distraction resilience, decision-making and situation awareness also contribute significantly. Autonomous, technologically supported objective skill assessment can be efficient tools to improve patient outcomes without the need to involve expert surgeon reviewers. However, autonomous non-technical skill assessments are unstandardized and open for more research. Recently, Surgical Data Science (SDS) has become able to improve the quality of interventional healthcare with big data and data processing techniques (capture, organization, analysis and modeling of data). SDS techniques can also help to achieve autonomous non-technical surgical skill assessments. An MIS training experiment is introduced to autonomously assess non-technical skills and to analyse the workload based on sensory data (video image and force) and a self-rating questionnaire (SURG-TLX). A sensorized surgical skill training phantom and adjacent training workflow were designed to simulate a complicated Laparoscopic Cholecystectomy task; the dissection of the cholecyst's peritonial layer and the safe clip application on the cystic artery in an uncomfortable environment. A total of 20 training sessions were recorded from 7 subjects (3 non-medicals, 2 residents, 1 expert surgeon and 1 expert MIS surgeon). Workload and learning curves were studied via SURG-TLX. For autonomous non-technical skill assessment, video image data with tracked instruments based on Channel and Spatial Reliability Tracker (CSRT) and force data were utilized. An autonomous time series classification was achieved by a Fully Convolutional Neural Network (FCN), where the class labels were provided by SURG-TLX. With unpaired t-tests, significant differences were found between the two groups (medical professionals and control) in certain workload components (mental demands, physical demands, and situational stress, p&lt;0.0001, 95% confidence interval, p&lt;0.05 for task complexity). With paired t-tests, the learning curves of the trials were also studied; the task complexity resulted in a significant difference between the first and the second trials. Autonomous non-technical skill classification was based on the FCN by applying the tool trajectories and force data as input. This resulted in a high accuracy (85%) on temporal demands classification based on the z component of the used forces and 75% accuracy for classifying mental demands/situational stress with the x component of the used forces validated with Leave One Out Cross-Validation. Non-technical skills and workload components can be classified autonomously based on measured training data. SDS can be effective via automated non-technical skill assessment.

A High-Fidelity Artificial Urological System for the Quantitative Assessment of Endoscopic Skills.

Minimally-invasive surgery is rapidly growing and has become a standard approach for many operations. However, it requires intensive practice to achieve competency. The current training often relies on animal organ models or physical organ phantoms, which do not offer realistic surgical scenes or useful real-time feedback for surgeons to improve their skills. Furthermore, the objective quantitative assessment of endoscopic skills is also lacking. Here, we report a high-fidelity artificial urological system that allows realistic simulation of endourological procedures and offers a quantitative assessment of the surgical performance. The physical organ model was fabricated by 3D printing and two-step polymer molding with the use of human CT data. The system resembles the human upper urinary tract with a high-resolution anatomical shape and vascular patterns. During surgical simulation, endoscopic videos are acquired and analyzed to quantitatively evaluate performance skills by a customized computer algorithm. Experimental results show significant differences in the performance between professional surgeons and trainees. The surgical simulator offers a unique chance to train endourological procedures in a realistic and safe environment, and it may also lead to a quantitative standard to evaluate endoscopic skills.

Prediction of laparoscopic skills: objective learning curve analysis

IntroductionPrediction of proficiency of laparoscopic skills is essential to establish personalized training programs. Objective assessment of laparoscopic skills has been validated in a laparoscopic box trainer with force, motion and time recognition. The aim of this study is to investigate whether acquiring proficiency of laparoscopic skills can be predicted based on performance in such a training box.MethodsSurgical residents in their first year of training performed six different tasks in the Lapron box trainer. Force, motion and time data, three objective measures of tissue manipulation and instrument handling, were collected and analyzed for the six different tasks. Linear regression tests were used to predict the learning curve and the number of repetitions required to reach proficiency.ResultsA total of 6010 practice sessions performed by 42 trainees from 13 Dutch hospitals were assessed and included for analysis. Proficiency level was determined as a mean result of seven experts performing 42 trials. Learning curve graphs and prediction models for each task were calculated. A significant relationship between force, motion and time during six different tasks and prediction of proficiency was present in 17 out of 18 analyses.ConclusionThe learning curve of proficiency of laparoscopic skills can accurately be predicted after three repetitions of six tasks in a training box with force, path length and time recognition. This will facilitate personalized training programs in laparoscopic surgery.

The use of objective assessments in the evaluation of technical skills in cardiothoracic surgery: a systematic review.

Summary OBJECTIVESWith reductions in training time and intraoperative exposure, there is a need for objective assessments to measure trainee progression. This systematic review focuses on the evaluation of trainee technical skill performance using objective assessments in cardiothoracic surgery and its incorporation into training curricula.METHODSDatabases (EBSCOHOST, Scopus and Web of Science) and reference lists of relevant articles for studies that incorporated objective assessment of technical skills of trainees/residents in cardiothoracic surgery were included. Data extraction included task performed; assessment setting and tool used; number/level of assessors; study outcome and whether the assessments were incorporated into training curricula. The methodological rigour of the studies was scored using the Medical Education Research Study Quality Instrument (MERSQI).RESULTSFifty-four studies were included for quantitative synthesis. Six were randomized-controlled trials. Cardiac surgery was the most common speciality utilizing objective assessment methods with coronary anastomosis the most frequently tested task. Likert-based assessment tools were most commonly used (61%). Eighty-five per cent of studies were simulation-based with the rest being intraoperative. Expert surgeons were primarily used for objective assessments (78%) with 46% using blinding. Thirty (56%) studies explored objective changes in technical performance with 97% demonstrating improvement. The other studies were primarily validating assessment tools. Thirty-nine per cent of studies had established these assessment tools into training curricula. The mean ± standard deviation MERSQI score for all studies was 13.6 ± 1.5 demonstrating high validity.CONCLUSIONSDespite validated technical skill assessment tools being available and demonstrating trainee improvement, their regular adoption into training curricula is lacking. There is a need to incorporate these assessments to increase the efficiency and transparency of training programmes for cardiothoracic surgeons.

American College of Surgeons Objective Assessment of Skills in Surgery (ACS OASIS): A Formative Assessment of Junior Residents’ Technical Skills

The objective assessment of technical skills of junior residents is essential in implementing competency-based training and providing specific feedback regarding areas for improvement. An innovative assessment that can be easily implemented by training programs nationwide has been developed by expert surgeon educators under the aegis of the American College of Surgeons (ACS) Division of Education. This assessment, ACS Objective Assessment of Skills in Surgery (ACS OASIS) uses eight stations to address technical skills important for junior residents within the domains of laparoscopic appendectomy, excision of lipoma, central line placement, laparoscopic cholecystectomy, trocar placement, exploratory laparotomy, repair of enterotomy, and tube thoracostomy. The purpose of this study was to implement ACS OASIS at a number of sites to study its psychometric rigor. The ACS OASIS was pre-piloted at two programs to establish feasibility and to gather information regarding implementation. Each skills station was 12 minutes long, and the faculty completed a checklist with 5 to 15 items, and a global assessment scale. The study was then repeated at three pilot sites and included 29 junior residents who were assessed by a total of 44 faculty. Psychometric data for the stations and checklists were collected and analyzed. The pre-pilot sites were Geisinger and University of Tennessee Knoxville.Data were gathered from pilot sites that included Wellspan Health, Duke University, and University of California Los Angeles. The mean checklist score for all learners was 76% (IQR of 66%-85%). The average global rating was 3.36 on a 5-point scale with a standard deviation of 0.56. The overall cut score derived using the borderline group method was at 68% with 34% of performances requiring remediation. Using this criterion, the average number of stations that were completed by each learner without need for remediation was five.The station discrimination index ranged from 0.27 to 0.65 (all above the threshold of 0.25), demonstrating solid psychometric characteristics at the station level. The internal-consistency reliability was 0.76 with SEM of 5.8%. The inter-rater reliability (intraclass correlation) was high at 0.73 with general agreement of 79% between the two raters. The station discrimination was at 0.45 (range of 0.27 to 0.65) indicating a high level of differentiation between high and low performers. Using the generalizability theory, the G-coefficient reliability was at 0.72 with the reliability projection flattening after 8 stations. Overall, 75% to 82% the faculty and learners rated ACS OASIS as realistic and beneficial. ACS OASIS is a psychometrically sound technical skills assessment tool that can provide useful information for feedback to junior residents and support efforts to remediate gaps in performance.

Identifying and Assessing the Required I4.0 Skills for Manufacturing Companies’ Workforce

Nowadays, the diffusion of digital and industry 4.0 (I4.0) technologies is affecting the manufacturing sector with a twofold effect. While on one side it represents the boost fastening the competitive advantage of companies, on the other hand it is often accompanied by several challenges that companies need to face. Among all, companies are required to invest in technologies to empower their production activities on the shopfloor without lagging behind their workforce in order to undertake a linear, aware, and structured path toward digitization. The extant literature presents some research conducted to support companies toward digitization, and they usually rely on maturity models in this intention. Nevertheless, few studies included the assessment of workforce skills and competencies in the overall assessment, and in this case, they provide a high level perspective of the investigation, mainly based on check lists which may limit the objectivity of the assessment, and usually they do not customize the assessment based on companies’ requirements. Therefore, considering the importance to balance investments in technologies with those in the workforce to move toward the same direction, this contribution aims to develop a structured, customizable, and objective skill assessment model. With this intention, it has been first clarified the set of job profiles required in I4.0, together with the needed related skills based on the extant literature findings; second, it has been identified the set of key criteria to be considered while performing the assessment of the workforce; third, it has been defined the method to be integrated in the maturity model to enable the initial setting of the weights of the criteria identified according to the company needs; and fourth, based on these findings, it has been developed the assessment model. The developed model facilitates the elaboration of the proper workforce improvement plans to be put in practice to support the improvement of the skills of the whole workforce based on company’s needs.

The Technique of Processing the Results of Skills Domain Assessment in the Field of Tahfidz at Islamic Boarding Schools

The phenomenon of memorizing the Qur'an in Indonesia can be seen from the establishments of many tahfidz houses, Al-Qur'an huts or other names with the vision of creating a Qur'anic generation. However, with the large number of enthusiasts of the tahfidz program, it raises doubt about the quality of its graduates due to the tendency of institutions to prioritize quantity over quality,which cause the lack of supervision due to the imbalance between supervisors and tahfidz candidates. The purpose of this study is to analyze the specific strategy applied by one of the Tahfidz Islamic Boarding Schools about how they evaluate their students so that they can score a tahfidz who is not only mumtaz 30 juz but also have mutqin (strongly stick and correct) memorization. This research is located at Pondok Pesantren Al-Munawwir, Madrasah Huffadh Complex II, Krapyak Yogyakarta. Data collection techniques were documentation techniques, interviews, and direct observation. Data analysis used were data reduction, data presentation, and drawing conclusions or verification. The finding in this study is that the assessment is carried out every semester. This Qur'an memorization assessment is included in the assessment of abstract skills. The assessment mechanism is carried out by listening to the memorization of students who read their memorization in one time while observing the aspects that are assessed based on the assessment guidelines that have been formulated. This assessment technique is included in the type of performance test with the type of identification test. The technique for processing the assessment results is applied using a non-test instrument in the form of a rubric and using a rating scale. The aspects assessed include Adabiah, Muhafazah, and recitation. The weakness of the implementation of Imtihan is that there is no interpretation or judgment from the evaluation results for students who do not meet the assessment criteria or students who read their memorization incompletely. The evaluation model is an illustration of how to formulate an objective skill assessment instrument and determine the right aspects of the target without ignoring the principles of evaluation even though there are still some weaknesses in it.

Surgesture: a novel instrument based on surgical actions for objective skill assessment

Due to varied surgical skills and the lack of an efficient rating system, we developed Surgesture based on elementary functional surgical gestures performed by surgeons, which could serve as objective metrics to evaluate surgical performance in laparoscopic cholecystectomy (LC). We defined 14 LC basic Surgestures. Four surgeons annotated Surgestures among LC videos performed by experts and novices. The counts, durations, average action time, and dissection/exposure ratio (D/E ratio) of LC Surgestures were compared. The phase of mobilizing hepatocystic triangle (MHT) was extracted for skill assessment by three professors using a modified Global Operative Assessment of Laparoscopic Skills (mGOALS). The novice operation time was significantly longer than the expert operation time (58.12 ± 19.23min vs. 26.66 ± 8.00min, P < 0.001), particularly during MHT phase. Novices had significantly more Surgestures than experts in both hands (P < 0.05). The left hand and inefficient Surgesture of novices were dramatically more than those of experts (P < 0.05). The experts demonstrated a significantly higher D/E ratio of duration than novices (0.79 ± 0.37 vs. 2.84 ± 1.98, P < 0.001). The counts and time pattern map of LC Surgestures during MHT demonstrated that novices tended to complete LC with more types of Surgestures and spent more time exposing the surgical scene. The performance metrics of LC Surgesture had significant but weak associations with each aspect of mGOALS. The newly constructed Surgestures could serve as accessible and quantifiable metrics for demonstrating the operative pattern and distinguishing surgeons with various skills. The association between Surgestures and Global Rating Scale laid the foundation for establishing a bridge to automated objective surgical skill evaluation.