Task-specific Metrics Research Articles

Validity of task-specific metrics for assessment in perineal proctectomy.

Perineal proctectomy is a complex procedure that requires advanced skills. Currently, there are no simulators for training in this procedure. As part of our objective of developing a virtual reality simulator, our goal was to develop and validate task-specific metrics for the assessment of performance for this procedure. We conducted a three-phase study to establish task-specific metrics, obtain expert consensus on the appropriateness of the developed metrics, and establish the discriminant validity of the developed metrics. In phase I, we utilized hierarchical task analysis to formulate the metrics. In phase II, a survey involving expert colorectal surgeons determined the significance of the developed metrics. Phase III was aimed at establishing the discriminant validity for novices (PGY1-3) and experts (PGY4-5 and faculty). They performed a perineal proctectomy on a rectal prolapse model. Video recordings were independently assessed by two raters using global ratings and task-specific metrics for the procedure. Total scores for both metrics were computed and analyzed using the Kruskal-Wallis test. A Mann-Whitney U test with Benjamini-Hochberg correction was used to evaluate between-group differences. Spearman's rank correlation coefficient was computed to assess the correlation between global and task-specific scores. In phase II, a total of 23 colorectal surgeons were recruited and consensus was obtained on all the task-specific metrics. In phase III, participants (n = 22) included novices (n = 15) and experts (n = 7). There was a strong positive correlation between the global and task-specific scores (rs = 0.86; P < 0.001). Significant between-group differences were detected for both global (χ2 = 15.38; P < 0.001; df = 2) and task-specific (χ2 = 11.38; P = 0.003; df = 2) scores. Using a biotissue rectal prolapse model, this study documented high IRR and significant discriminant validity evidence in support of video-based assessment using task-specific metrics.

Towards automated field ballast condition evaluation: Field validation of the ballast scanning vehicle capabilities

Ballast degradation can lead to adverse effects such as inadequate drainage, track settlement and reduced lateral stability, which could compromise track safety, daily functionality, and long-term maintenance. Field inspection of ballast for monitoring degradation and functional performance is a challenging task. Current state-of-the-practice methods for evaluating ballast primarily depend on subjective visual inspection, labor-intensive sampling, laboratory sieve analyses or Ground Penetrating Radar (GPR) technology. These methods fall short in providing an in-depth assessment of ballast, specifically in determining the degradation level and aggregate size and shape characteristics at various depths. In this regard, this research developed an innovative ballast investigation platform, the Ballast Scanning Vehicle (BSV), to automate the processes of acquiring detailed ballast inspection data. The BSV utilizes a deep learning-based pipeline for image segmentation to evaluate task-specific metrics such as coarse aggregate gradation, Fouling Index (FI), and continuous track FI depth profiles. This paper provides a detailed overview of the BSV’s functions as well as the different modules of the deep learning-based pipeline. Validation of the BSV’s capabilities was conducted at the Transportation Technology Center (TTC) and is discussed in detail. Based on the field results, the BSV is capable of providing accurate and near real-time evaluation of in-service ballast conditions, serves as a robust means for inspecting long sections of track, and can be used to investigate persistent trouble-spots related to track performance.

Text-to-OverpassQL: A Natural Language Interface for Complex Geodata Querying of OpenStreetMap

Abstract We present Text-to-OverpassQL, a task designed to facilitate a natural language interface for querying geodata from OpenStreetMap (OSM). The Overpass Query Language (OverpassQL) allows users to formulate complex database queries and is widely adopted in the OSM ecosystem. Generating Overpass queries from natural language input serves multiple use-cases. It enables novice users to utilize OverpassQL without prior knowledge, assists experienced users with crafting advanced queries, and enables tool-augmented large language models to access information stored in the OSM database. In order to assess the performance of current sequence generation models on this task, we propose OverpassNL,1 a dataset of 8,352 queries with corresponding natural language inputs. We further introduce task specific evaluation metrics and ground the evaluation of the Text-to-OverpassQL task by executing the queries against the OSM database. We establish strong baselines by finetuning sequence-to-sequence models and adapting large language models with in-context examples. The detailed evaluation reveals strengths and weaknesses of the considered learning strategies, laying the foundations for further research into the Text-to-OverpassQL task.

Recursive label attention network for nested named entity recognition

Nested named entity recognition (NER) has been mainly studied by recognizing inner entities first, then outer entities, or vice versa, depending on the nesting level. However, the previous models on nested NER do not deal with the data sparseness problems of deep nested entities (i.e., insufficiency of training data for deeper nested entities) and the error propagation problems between nested entities (i.e., accumulation of errors caused by inner entities or outer entities). To alleviate the data sparseness problems, we propose a recursive label attention network. In contrast to previous models, the proposed model explicitly reflects the nested level and effectively re-uses lower level label information through level-reflected label embeddings. To counteract error propagation in the recursive architecture, we propose an inner entity pretraining strategy in which the proposed model is sequentially trained from lower to higher levels. In experiments, we use task-specific metrics for nested NER, unlike the evaluation metrics used in previous studies. The experimental results show that the proposed model is particularly effective in cases where entities of the same type were nested. In addition, it showed high F1-scores even when named entities are deeply nested. This fact reveals that our proposed model is effective for nest NER.

Validity and reliability evidence support task-specific metrics for laparoscopic fundoplication.

Laparoscopic hiatal hernia repair (LHHR) is a complex operation requiring advanced surgical training. Surgical simulation offers a potential solution for learning complex operations without the need for high surgical volume. Our goal is to develop a virtual reality (VR) simulator for LHHR; however, data supporting task-specific metrics for this procedure are lacking. The purpose of this study was to develop and assess validity and reliability evidence of task-specific metrics for the fundoplication phase of LHHR. In phase I, structured interviews with expert foregut surgeons were conducted to develop task-specific metrics (TSM). In phase II, participants with varying levels of surgical expertise performed a laparoscopic Nissen fundoplication procedure on a porcine stomach explant. Video recordings were independently assessed by two blinded graders using global and TSM. An intraclass correlation coefficient (ICC) was used to assess interrater reliability (IRR). Performance scores were compared using a Kruskal-Wallis test. Spearman's rank correlation was used to evaluate the association between global and TSM. Phase I of the study consisted of 12 interviews with expert foregut surgeons. Phase II engaged 31 surgery residents, a fellow, and 6 attendings in the simulation. Phase II results showed high IRR for both global (ICC = 0.84, p < 0.001) and TSM (ICC = 0.75, p < 0.001). Significant between-group differences were detected for both global (χ2 = 24.01, p < 0.001) and TSM (χ2 = 18.4, p < 0.001). Post hoc analysis showed significant differences in performance between the three groups for both metrics (p < 0.05). There was a strong positive correlation between the global and TSM (rs = 0.86, p < 0.001). We developed task-specific metrics for LHHR and using a fundoplication model, we documented significant reliability and validity evidence. We anticipate that these LHHR task-specific metrics will be useful in our planned VR simulator.

Model-based deep CNN-regularized reconstruction for digital breast tomosynthesis with a task-based CNN image assessment approach

Objective. Digital breast tomosynthesis (DBT) is a quasi-three-dimensional breast imaging modality that improves breast cancer screening and diagnosis because it reduces fibroglandular tissue overlap compared with 2D mammography. However, DBT suffers from noise and blur problems that can lower the detectability of subtle signs of cancers such as microcalcifications (MCs). Our goal is to improve the image quality of DBT in terms of image noise and MC conspicuity. Approach. We proposed a model-based deep convolutional neural network (deep CNN or DCNN) regularized reconstruction (MDR) for DBT. It combined a model-based iterative reconstruction (MBIR) method that models the detector blur and correlated noise of the DBT system and the learning-based DCNN denoiser using the regularization-by-denoising framework. To facilitate the task-based image quality assessment, we also proposed two DCNN tools for image evaluation: a noise estimator (CNN-NE) trained to estimate the root-mean-square (RMS) noise of the images, and an MC classifier (CNN-MC) as a DCNN model observer to evaluate the detectability of clustered MCs in human subject DBTs. Main results. We demonstrated the efficacies of CNN-NE and CNN-MC on a set of physical phantom DBTs. The MDR method achieved low RMS noise and the highest detection area under the receiver operating characteristic curve (AUC) rankings evaluated by CNN-NE and CNN-MC among the reconstruction methods studied on an independent test set of human subject DBTs. Significance. The CNN-NE and CNN-MC may serve as a cost-effective surrogate for human observers to provide task-specific metrics for image quality comparisons. The proposed reconstruction method shows the promise of combining physics-based MBIR and learning-based DCNNs for DBT image reconstruction, which may potentially lead to lower dose and higher sensitivity and specificity for MC detection in breast cancer screening and diagnosis.

Leveraging Modality-Specific Representations for Audio-Visual Speech Recognition via Reinforcement Learning

Audio-visual speech recognition (AVSR) has gained remarkable success for ameliorating the noise-robustness of speech recognition. Mainstream methods focus on fusing audio and visual inputs to obtain modality-invariant representations. However, such representations are prone to over-reliance on audio modality as it is much easier to recognize than video modality in clean conditions. As a result, the AVSR model underestimates the importance of visual stream in face of noise corruption. To this end, we leverage visual modality-specific representations to provide stable complementary information for the AVSR task. Specifically, we propose a reinforcement learning (RL) based framework called MSRL, where the agent dynamically harmonizes modality-invariant and modality-specific representations in the auto-regressive decoding process. We customize a reward function directly related to task-specific metrics (i.e., word error rate), which encourages the MSRL to effectively explore the optimal integration strategy. Experimental results on the LRS3 dataset show that the proposed method achieves state-of-the-art in both clean and various noisy conditions. Furthermore, we demonstrate the better generality of MSRL system than other baselines when test set contains unseen noises.

Automatic performance evaluation of the intracorporeal suture exercise.

This work uses deep learning algorithms to provide automated feedback on the suture with intracorporeal knot exercise in the fundamentals of laparoscopic surgery simulator. Different metrics were designed to provide informative feedback to the user on how to complete the task more efficiently. The automation of the feedback will allow students to practice at any time without the supervision of experts. Five residents and five senior surgeons participated in the study. Object detection, image classification, and semantic segmentation deep learning algorithms were used to collect statistics on the practitioner's performance. Three task-specific metrics were defined. The metrics refer to the way the practitioner holds the needle before the insertion to the Penrose drain, and the amount of movement of the Penrose drain during the needle's insertion. Good agreement between the human labeling and the different algorithms' performance and metric values was achieved. The difference between the scores of the senior surgeons and the surgical residents was statistically significant for one of the metrics. We developed a system that provides performance metrics of the intracorporeal suture exercise. These metrics can help surgical residents practice independently and receive informative feedback on how they entered the needle into the Penrose.

317 Pilot Analysis of Surgeon Instrument Utilization Signatures Based on Shannon Entropy for Surgeon Performance Assessment in a Cadaveric Carotid Artery Injury Control Simulation

INTRODUCTION: Automated video-based feedback for a surgeon’s technical skills would improve patient care and facilitate trainee education, but current feedback methods require the development of task-specific metrics. A generalizable, automated method of surgeon skills assessment with surgeon tool-usage signatures could expedite the development and validation of video-based feedback methods. METHODS: Annotated images from the publicly-available SOCAL dataset were used to train an off-the-shelf computer vision model (YOLOv4) which detected surgical instruments. Shannon entropy of unique instruments combinations was calculated from each surgical trial. Logistic regression was used to predict trial success using Shannon entropy. RESULTS: Surgeon signatures based on Shannon entropy were created for each trial and each surgeon. Instrument usage patterns demonstrated differences between successful and unsuccessful trials. Shannon entropy of instrument combinations demonstrated significant correlation with trial success (p &lt; 0.001) and predicted success with 97% average precision and 78% accuracy using computer vision detections. Unsuccessful trials displayed a rapid initial peak in entropy which then declined over time. In contrast, successful trials demonstrated slower progression of tool usage with gradually increasing use of a critical, final maneuver. CONCLUSIONS: Surgeon signatures based on Shannon entropy predicted task success with acceptable accuracy and revealed patterns between successful and unsuccessful trials. Shannon entropy offers a generalizable and summative signal about surgeon performance, regardless of the task, and can predict outcomes based only on the probabilities of unique instrument combinations. Future efforts will describe the instrument movement signatures.

Entropy and a sub-group of geometric measures of paths predict the navigability of an environment

Despite extensive research on navigation, it remains unclear which features of an environment predict how difficult it will be to navigate. We analysed 478,170 trajectories from 10,626 participants who navigated 45 virtual environments in the research app-based game Sea Hero Quest. Virtual environments were designed to vary in a range of properties such as their layout, number of goals, visibility (varying fog) and map condition. We calculated 58 spatial measures grouped into four families: task-specific metrics, space syntax configurational metrics, space syntax geometric metrics, and general geometric metrics. We used Lasso, a variable selection method, to select the most predictive measures of navigation difficulty. Geometric features such as entropy, area of navigable space, number of rings and closeness centrality of path networks were among the most significant factors determining the navigational difficulty. By contrast a range of other measures did not predict difficulty, including measures of intelligibility. Unsurprisingly, other task-specific features (e.g. number of destinations) and fog also predicted navigation difficulty. These findings have implications for the study of spatial behaviour in ecological settings, as well as predicting human movements in different settings, such as complex buildings and transport networks and may aid the design of more navigable environments.

A meta-learning approach to personalized blood glucose prediction in type 1 diabetes

Accurate blood glucose prediction is a critical element in modern artificial pancreas systems. Recently, many deep learning-based models have been proposed for glucose prediction, showing encouraging results in population modeling. However, due to the large amount of data required for training deep learning-based models, few studies have successfully addressed personalized modeling, which is critical to ensure safe policies in a closed-loop scheme given the high inter-patient variability. To address this concern, we propose a meta-learning-based technique for accurate personalized modeling that requires minimal data volume to personalize from its population version, needs few training iterations, and has a low risk of over-fitting. Results using the UVA/Padova simulator show that the proposed technique generalizes better and outperforms other approaches in standard and task-specific metrics, particularly for longer prediction horizons and higher degrees of distributional shifts.

SARN: Shifted Attention Regression Network for 3D Hand Pose Estimation

Hand pose estimation (HPE) plays an important role during the functional assessment of the hand and in potential rehabilitation. It is a challenge to predict the pose of the hand conveniently and accurately during functional tasks, and this limits the application of HPE. In this paper, we propose a novel architecture of a shifted attention regression network (SARN) to perform HPE. Given a depth image, SARN first predicts the spatial relationships between points in the depth image and a group of hand keypoints that determine the pose of the hand. Then, SARN uses these spatial relationships to infer the 3D position of each hand keypoint. To verify the effectiveness of the proposed method, we conducted experiments on three open-source datasets of 3D hand poses: NYU, ICVL, and MSRA. The proposed method achieved state-of-the-art performance with 7.32 mm, 5.91 mm, and 7.17 mm of mean error at the hand keypoints, i.e., mean Euclidean distance between the predicted and ground-truth hand keypoint positions. Additionally, to test the feasibility of SARN in hand movement recognition, a hand movement dataset of 26K depth images from 17 healthy subjects was constructed based on the finger tapping test, an important component of neurological exams administered to Parkinson's patients. Each image was annotated with the tips of the index finger and the thumb. For this dataset, the proposed method achieved a mean error of 2.99 mm at the hand keypoints and comparable performance on three task-specific metrics: the distance, velocity, and acceleration of the relative movement of the two fingertips. Results on the open-source datasets demonstrated the effectiveness of the proposed method, and results on our finger tapping dataset validated its potential for applications in functional task characterization.

Facial Landmark, Head Pose, and Occlusion Analysis Using Multitask Stacked Hourglass

In this study, we proposed a multitask network architecture for three attributes, landmark, head pose, and occlusion, from a face image. A 2-stacked hourglass with three task-specific heads is the proposed network architecture. We also designed three auxiliary components for the network. First is the feature pyramid fusion module, which plays a crucial role in facilitating contextual information from various receptive fields. Second is the interlevel occlusion-aware fusion module, which explicitly fuses intermediate occlusion prediction between subnetworks. The third is the gimbal-lock-free head pose head, which outputs a rotation matrix from a 6D rotation representation. We conducted an ablative study of these auxiliary components to determine their impacts on the network. Additionally, we introduced the landmark heatmap scaling approach to avoid falling local minima. We trained the proposed network with a 300W-LP dataset for landmark and head pose and a C-CM dataset for occlusion. Then, we fine-tuned the network using the 300W or WFLW dataset, instead of the 300W-LP dataset for the landmark task. This 2-stage training method contributes to enhancing the landmark detection accuracy and that of other tasks. In the experiments, we assessed the proposed network using eight test datasets and task-specific metrics.

Development and Validation of Task-Specific Metrics for the Assessment of Linear Stapler-Based Small Bowel Anastomosis.

Task-specific metrics facilitate the assessment of surgeon performance. This 3-phased study was designed to (1) develop task-specific metrics for stapled small bowel anastomosis, (2) obtain expert consensus on the appropriateness of the developed metrics, and (3) establish its discriminant validity. In Phase I, a hierarchical task analysis was used to develop the metrics. In Phase II, a survey of expert colorectal surgeons established the importance of the developed metrics. In Phase III, to establish discriminant validity, surgical trainees and surgeons, divided into novice and experienced groups, constructed a side-to-side anastomosis on porcine small bowel using a linear cutting stapler. The participants' performances were videotaped and rated by 2 independent observers. Partial least squares regression was used to compute the weights for the task-specific metrics to obtain weighted total score. In Phase II, a total of 45 colorectal surgeons were surveyed: 28 with more than 15 years, 13 with 5 to 15 years, and 4 with less than 5 years of experience. The consensus was obtained on all the task-specific metrics in the more experienced groups. In Phase III, 20 subjects participated equally in both groups. The experienced group performed better than the novice group regardless of the rating scale used: global rating scale (p = 0.009) and the task-specific metrics (p = 0.012). After partial least squares regression, the weighted task-specific metric score continued to show that the experienced group performed better (p < 0.001). Task-specific metric items were developed based on expert consensus and showed good discriminant validity compared with a global rating scale between experienced and novice operators. These items can be used for evaluating technical skills in a stapled small bowel anastomosis model.

Objective metrics for hand-sewn bowel anastomoses can differentiate novice from expert surgeons.

Assessing performance automatically in a virtual reality trainer or from recorded videos is advantageous but needs validated objective metrics. The purpose of this study is to obtain expert consensus and validate task-specific metrics developed for assessing performance in double-layered end-to-end anastomosis. Subjects were recruited into expert (PGY 4-5, colorectal surgery residents, and attendings) and novice (PGY 1-3) groups. Weighted average scores of experts for each metric item, completion time, and the total scores computed using global and task-specific metrics were computed for assessment. A total of 43 expert surgeons rated our task-specific metric items with weighted averages ranging from 3.33 to 4.5 on a 5-point Likert scale. A total of 20 subjects (10 novices and 10 experts) participated in validation study. The novice group completed the task significantly more slowly than the experienced group (37.67 ± 7.09 vs 25.47 ± 7.82min, p = 0.001). In addition, both the global rating scale (23.47 ± 4.28 vs 28.3 ± 3.85, p = 0.016) and the task-specific metrics showed a significant difference in performance between the two groups (38.77 ± 2.83 vs 42.58 ± 4.56 p = 0.027) following partial least-squares (PLS) regression. Furthermore, PLS regression showed that only two metric items (Stay suture tension and Tool handling) could reliably differentiate the performance between the groups (20.41 ± 2.42 vs 24.28 ± 4.09 vs, p = 0.037). Our study shows that our task-specific metrics have significant discriminant validity and can be used to evaluate the technical skills for this procedure.

Task-Oriented Multi-User Semantic Communications

While semantic communications have shown the potential in the case of single-modal single-users, its applications to the multi-user scenario remain limited. In this paper, we investigate deep learning (DL) based multi-user semantic communication systems for transmitting single-modal data and multimodal data, respectively. We adopt three intelligent tasks, including, image retrieval, machine translation, and visual question answering (VQA) as the transmission goal of semantic communication systems. We propose a Transformer based framework to unify the structure of transmitters for different tasks. For the single-modal multi-user system, we propose two Transformer based models, named, DeepSC-IR and DeepSC-MT, to perform image retrieval and machine translation, respectively. In this case, DeepSC-IR is trained to optimize the distance in embedding space between images and DeepSC-MT is trained to minimize the semantic errors by recovering the semantic meaning of sentences. For the multimodal multi-user system, we develop a Transformer enabled model, named, DeepSC-VQA, for the VQA task by extracting text-image information at the transmitters and fusing it at the receiver. In particular, a novel layer-wise Transformer is designed to help fuse multimodal data by adding connection between each of the encoder and decoder layers. Numerical results show that the proposed models are superior to traditional communications in terms of the robustness to channels, computational complexity, transmission delay, and the task-execution performance at various task-specific metrics.

“Bring me sunshine, bring me (physical) strength”: The case of dementia. Designing and implementing a virtual reality system for physical training during the COVID-19 pandemic

People living with Dementia (PwD) are amongst the most vulnerable populations in society, often depending on caregivers for their quality of life (QoL). Emerging research confirms the need for technological solutions to support non-pharmacological interventions that can enhance the QoL of PwD. This paper posits that Virtual Reality (VR) is a useful technology for boosting the physical training of PwD. In a study with PwD, we compared the conventional physical training PwD receive at a nursing home, against Semi Immersive VR (SIVR) and Fully Immersive VR (FIVR) training paradigms. We recorded the emotional behaviour, task-specific metrics, and level of independence of PwD during the training. The presence and usability of the systefm by both medical staff and PwD was also assessed. Results indicated that FIVR can improve the training of PwD leading to more accurate execution of the exercises while preventing external distractions. Beyond the findings, this article discusses the opportunities, challenges, along with the feasibility and acceptability of VR, to facilitate physical training for PwD who reside in restricted health care environments.

Validation of Task-Specific Metrics for the Assessment of Linear Stapler Based Small Bowel Anastomosis

Validation of Task-Specific Metrics for the Assessment of Linear Stapler Based Small Bowel Anastomosis

FIXMYPOSE: Pose Correctional Captioning and Retrieval

Interest in physical therapy and individual exercises such as yoga/dance has increased alongside the well-being trend, and people globally enjoy such exercises at home/office via video streaming platforms. However, such exercises are hard to follow without expert guidance. Even if experts can help, it is almost impossible to give personalized feedback to every trainee remotely. Thus, automated pose correction systems are required more than ever, and we introduce a new captioning dataset named FixMyPose to address this need. We collect natural language descriptions of correcting a “current” pose to look like a “target” pose. To support a multilingual setup, we collect descriptions in both English and Hindi. The collected descriptions have interesting linguistic properties such as egocentric relations to the environment objects, analogous references, etc., requiring an understanding of spatial relations and commonsense knowledge about postures. Further, to avoid ML biases, we maintain a balance across characters with diverse demographics, who perform a variety of movements in several interior environments (e.g., homes, offices). From our FixMyPose dataset, we introduce two tasks: the pose-correctional-captioning task and its reverse, the target-pose-retrieval task. During the correctional-captioning task, models must generate the descriptions of how to move from the current to the target pose image, whereas in the retrieval task, models should select the correct target pose given the initial pose and the correctional description. We present strong cross-attention baseline models (uni/multimodal, RL, multilingual) and also show that our baselines are competitive with other models when evaluated on other image-difference datasets. We also propose new task-specific metrics (object-match, body-part-match, direction-match) and conduct human evaluation for more reliable evaluation, and we demonstrate a large human-model performance gap suggesting room for promising future work. Finally, to verify the sim-to-real transfer of our FixMyPose dataset, we collect a set of real images and show promising performance on these images. Data and code are available: https://fixmypose-unc.github.io.

Development and validation of non-guided bladder-neck and neurovascular-bundle dissection modules of the RobotiX-Mentor® full-procedure robotic-assisted radical prostatectomy virtual reality simulation.

BackgroundFull‐procedure virtual reality (VR) simulator training in robotic‐assisted radical prostatectomy (RARP) is a new tool in surgical education.MethodsDescription of the development of a VR RARP simulation model, (RobotiX‐Mentor®) including non‐guided bladder neck (ngBND) and neurovascular bundle dissection (ngNVBD) modules, and assessment of face, content, and construct validation of the ngBND and ngNVBD modules by robotic surgeons with different experience levels.ResultsSimulator and ngBND/ngNVBD modules were rated highly by all surgeons for realism and usability as training tool. In the ngBND‐task construct, validation was not achieved in task‐specific performance metrics. In the ngNVBD, task‐specific performance of the expert/intermediately experienced surgeons was significantly better than that of novices.ConclusionsWe proved face and content validity of simulator and both modules, and construct validity for generic metrics of the ngBND module and for generic and task‐specific metrics of the ngNVBD module.