Established Performance Metrics Research Articles

Diving into a pool of data: Using principal component analysis to optimize performance prediction in women’s short-course swimming

ABSTRACT This study aimed to optimise performance prediction in short-course swimming through Principal Component Analyses (PCA) and multiple regression. All women’s freestyle races at the European Short-Course Swimming Championships were analysed. Established performance metrics were obtained including start, free-swimming, and turn performance metrics. PCA were conducted to reduce redundant variables, and a multiple linear regression was performed where the criterion was swimming time. A practical tool, the Potential Predictor, was developed from regression equations to facilitate performance prediction. Bland and Altman analyses with 95% limits of agreement (95% LOA) were used to assess agreement between predicted and actual swimming performance. There was a very strong agreement between predicted and actual swimming performance. The mean bias for all race distances was less than 0.1s with wider LOAs for the 800 m (95% LOA −7.6 to + 7.7s) but tighter LOAs for the other races (95% LOAs −0.6 to + 0.6s). Free-Swimming Speed (FSS) and turn performance were identified as Key Performance Indicators (KPIs) in the longer distance races (200 m, 400 m, 800 m). Start performance emerged as a KPI in sprint races (50 m and 100 m). The successful implementation of PCA and multiple regression provides coaches with a valuable tool to uncover individual potential and empowers data-driven decision-making in athlete training.

Improving Breast Cancer Detection with Convolutional Neural Networks and Modified ResNet Architecture.

The pathogenesis of breast cancer is characterized by dysregulated cell proliferation, leading to the formation of a neoplastic mass. Conventional methodologies for analyzing carcinomatous distal areas within whole-slide images (WSIs) tissue regions may lack comprehensive insights. This study aims to introduce an innovative methodology based on convolutional neural networks (CNN), specifically employing a CNN Modified ResNet architecture for breast cancer detection. The research seeks to address the limitations of existing approaches and provide a robust solution for the comprehensive analysis of tissue regions. The dataset utilized in this study comprises approximately 275,000 RGB image patches, each standardized at 50x50 pixels. The CNN Modified ResNet architecture is implemented, and a comparative evaluation against diverse architectures is conducted. Rigorous validation tests employing established performance metrics are carried out to assess the proposed methodology. The proposed architecture achieves a notable 89% accuracy in breast cancer detection, surpassing alternative methods by 2%. The results signify the efficacy and superiority of the CNN Modified ResNet model in analyzing carcinomatous distal areas within WSIs tissue regions. In conclusion, this study demonstrates the potential of the CNN Modified ResNet architecture as an effective tool for breast cancer detection. The enhanced accuracy and comprehensive analysis capabilities make it a promising approach for advancing the understanding of neoplastic masses in WSIs tissue regions. Further research and validation could solidify its role in clinical applications and diagnostic procedures.

Multiple Branch Deep Neural Network and Feature Fusion Model for Underwater Image Enhancement

Underwater imaging faces formidable challenges due to light absorption, scattering, and restricted visibility, demanding sophisticated enhancement methods. This research presents a novel Multiple Branch Deep Neural Network and Feature Fusion (MB-DNN-FF) Model tailored specifically for Underwater Images Enhancement (UWIE), with a focus on its application to the UFO120 dataset. Driven by the limitations of existing techniques, our model harnesses the potential of Deep Learning (DL) and Feature Fusion (FF) to effectively address the intricate complexities present in underwater environments. The innovative architecture incorporates multiple branches, each strategically designed to tackle distinct challenges such as contrast degradation, limited visibility, and color distortion. Central to our model is feature fusion, a critical aspect that harmoniously integrates information from diverse branches, thereby enhancing overall image quality. The training and optimization processes are thoroughly detailed, encompassing unique strategies and loss functions fine-tuned for the nuances of UWIE, with emphasis on the characteristics of the UFO120 dataset. Experimental evaluations leverage the comprehensive UFO120 dataset, employing established performance metrics to quantitatively assess the efficacy of our model. Results exhibit substantial improvements over baseline models and state-of-the-art methods, showcasing the effectiveness and versatility of the proposed multiple branch model, particularly on the UFO120 dataset. The discussion interprets these findings in the context of underwater imaging challenges, highlighting the model's effectiveness and outlining potential applications. This research contributes a valuable asset to the area of UWIE, offering a nuanced and potent solution through the integration of multiple branches and feature fusion, validated on the UFO120 dataset. The conclusions underscore the significance of the MB-DNN-FF in advancing the state-of-the-art in UWIE, opening avenues for future research and practical applications in marine sciences, surveillance, and exploration.

Development of an image-based Random Forest classifier for prediction of surgery duration of laparoscopic sigmoid resections

PurposeSigmoid diverticulitis is a disease with a high socioeconomic burden, accounting for a high number of left-sided colonic resections worldwide. Modern surgical scheduling relies on accurate prediction of operation times to enhance patient care and optimize healthcare resources. This study aims to develop a predictive model for surgery duration in laparoscopic sigmoid resections, based on preoperative CT biometric and demographic patient data.MethodsThis retrospective single-center cohort study included 85 patients who underwent laparoscopic sigmoid resection for diverticular disease. Potentially relevant procedure-specific anatomical parameters recommended by a surgical expert were measured in preoperative CT imaging. After random split into training and test set (75% / 25%) multiclass logistic regression was performed and a Random Forest classifier was trained on CT imaging parameters, patient age, and sex in the training cohort to predict categorized surgery duration. The models were evaluated in the test cohort using established performance metrics including receiver operating characteristics area under the curve (AUROC).ResultsThe Random Forest model achieved a good average AUROC of 0.78. It allowed a very good prediction of long (AUROC = 0.89; specificity 0.71; sensitivity 1.0) and short (AUROC = 0.81; specificity 0.77; sensitivity 0.56) procedures. It clearly outperformed the multiclass logistic regression model (AUROC: average = 0.33; short = 0.31; long = 0.22).ConclusionA Random Forest classifier trained on demographic and CT imaging biometric patient data could predict procedure duration outliers of laparoscopic sigmoid resections. Pending validation in a multicenter study, this approach could potentially improve procedure scheduling in visceral surgery and be scaled to other procedures.

Model Performance Evaluation: VGG19 and Dense201 for Fresh Meat Detection

To guarantee consumer safety and meet quality expectations, accurate detection of meat quality is a critical component of the food industry. The objective of this research endeavor is to assess and contrast the fresh meat detection capabilities of two distinct artificial neural network architectures, denoted as Dense201 and VGG19. Automated systems that can identify vital qualities in fresh meat, including color, texture, and cleanliness, have become feasible due to the development of image processing technology. For this reason, however, there are still few direct comparisons between various architectures of artificial neural networks, particularly VGG19 and Dense201. Comparing and contrasting the performance of both models in identifying the quality of meat from visual images, this study attempts to fill this void. Utilizing a vast dataset containing a variety of fresh meats exhibiting substantial visible variations constituted the research methodology. The assessment was conducted by examining the efficacy of both models in determining the quality of meat using established performance metrics, including accuracy, precision, recall, and F1-score. Regarding the detection of fresh meat, it is anticipated that the findings of this study will offer a comprehensive understanding of the benefits and drawbacks associated with every artificial neural network architecture. Contributing to a greater comprehension of the application of precise and efficient meat detection technology, this study also furnishes the food industry with a foundation for determining which model best meets the requirements of meat quality detection on a larger production scale.

Assessment of the Correlation between Energy Rating Labeling Regulations and Performance Metrics for Residential Air Conditioning Units: Case Study Variable Type Air Conditioners

To comply with the Paris Agreement, many countries have implemented mandatory energy certification policies in the HVAC sector. These policies have encouraged the deployment of more energy efficient technologies such as variable or inverter driven technologies, which have been developed based on established seasonal performance metrics. Therefore, this paper presents an experimental evaluation of a variable-type air conditioner based on seasonal cooling metrics and performance ratings using various energy labeling programs from multiple regions around the world. The results of this work demonstrate that the energy efficiency rating of an air conditioner is significantly influenced by the distribution of the reference outdoor temperature in the labeling program issued by each region, and not just by the adoption of a technical standard. These results are crucial for the development of public policies aimed at designing better energy efficiency and labeling programs. In particular, the findings are especially relevant for decision making by governments, as the adoption of a technical standard can be simplified, and greater harmonization of performance metrics used globally can lead to greater energy savings and mitigate the effects of global warming.

Conformal efficiency as a metric for comparative model assessment befitting federated learning

In a drug discovery setting, pharmaceutical companies own substantial but confidential datasets. The MELLODDY project developed a privacy-preserving federated machine learning solution and deployed it at an unprecedented scale. Each partner built models for their own private assays that benefitted from a shared representation. Established predictive performance metrics such as AUC ROC or AUC PR are constrained to unseen labeled chemical space and cannot gage performance gains in unlabeled chemical space. Federated learning indirectly extends labeled space, but in a privacy-preserving context, a partner cannot use this label extension for performance assessment. Metrics that estimate uncertainty on a prediction can be calculated even where no label is known. Practically, the chemical space covered with predictions above an uncertainty threshold, reflects the applicability domain of a model. After establishing a link to established performance metrics, we propose the efficiency from the conformal prediction framework (‘conformal efficiency’) as a proxy to the applicability domain size. A documented extension of the applicability domain would qualify as a tangible benefit from federated learning. In interim assessments, MELLODDY partners reported a median increase in conformal efficiency of the federated over the single-partner model of 5.5% (with increases up to 9.7%). Subject to distributional conditions, that efficiency increase can be directly interpreted as the expected increase in conformal i.e. low uncertainty predictions. In conclusion, we present the first indication that privacy-preserving federated machine learning across massive drug-discovery datasets from ten pharma partners indeed extends the applicability domain of property prediction models.

Pedestrian Behavior Prediction for Automated Driving: Requirements, Metrics, and Relevant Features

Automated vehicles require a comprehensive understanding of traffic situations to ensure safe and anticipatory driving. In this context, the prediction of pedestrians is particularly challenging as pedestrian behavior can be influenced by multiple factors. In this paper, we thoroughly analyze the requirements on pedestrian behavior prediction for automated driving via a system-level approach. To this end we investigate real-world pedestrian-vehicle interactions with human drivers. Based on human driving behavior we then derive appropriate reaction patterns of an automated vehicle and determine requirements for the prediction of pedestrians. This includes a novel metric tailored to measure prediction performance from a system-level perspective. The proposed metric is evaluated on a large-scale dataset comprising thousands of real-world pedestrian-vehicle interactions. We furthermore conduct an ablation study to evaluate the importance of different contextual cues and compare these results to ones obtained using established performance metrics for pedestrian prediction. Our results highlight the importance of a system-level approach to pedestrian behavior prediction.

Optimum weighted multimodal medical image fusion using particle swarm optimization

Medical image fusion is a technique that maps information from multiple image modalities into a single image. In past, several multimodal fusion approaches were developed by various researchers, even so, adaptive fusion and robustness are challenging tasks in clinical diagnosis. In this paper, we developed an optimum weighted average fusion (OWAF) for multimodal medical image fusion to improve the multimodal mapping performance. In our approach, conventional discrete wavelet transform (DWT) is used for the decomposition of input multiple modalities into various subgroups. The resultant energy bands were weighted using optimum weights, attained using well known particle swarm optimization algorithm (PSO). Our proposed approach was tested over MRI-SPECT, MRI-PET and MRI-CT image fusion and we used 20 sets of MR/SPECT, 20 sets of MR/PET and 18 sets of MR/CT for our method validation. The quantitative evaluation were performed using established fusion performance metrics such as structural similarity index measure (SSIM), root mean square error (RMSE), peak signal to noise ratio (PSNR), entropy, mutual information (MI), and Edge-based similarity metric(QABF). Robustness of our proposed fusion approach is tested over gaussian and speckle noise in all the input modalities. The computational performance of our OAWF with PSO algorithm is tested in terms of computational time. The OWAF showed superior performance than existing conventional fusion approaches in terms of information mapping, edge quality and structural similarity in MR/PET, MR/SPECT and MR/CT images in both normal and noisy fusion backgrounds.

Automatically Designing State-of-the-Art Multi- and Many-Objective Evolutionary Algorithms.

A recent comparison of well-established multiobjective evolutionary algorithms (MOEAs) has helped better identify the current state-of-the-art by considering (i) parameter tuning through automatic configuration, (ii) a wide range of different setups, and (iii) various performance metrics. Here, we automatically devise MOEAs with verified state-of-the-art performance for multi- and many-objective continuous optimization. Our work is based on two main considerations. The first is that high-performing algorithms can be obtained from a configurable algorithmic framework in an automated way. The second is that multiple performance metrics may be required to guide this automatic design process. In the first part of this work, we extend our previously proposed algorithmic framework, increasing the number of MOEAs, underlying evolutionary algorithms, and search paradigms that it comprises. These components can be combined following a general MOEA template, and an automatic configuration method is used to instantiate high-performing MOEA designs that optimize a given performance metric and present state-of-the-art performance. In the second part, we propose a multiobjective formulation for the automatic MOEA design, which proves critical for the context of many-objective optimization due to the disagreement of established performance metrics. Our proposed formulation leads to an automatically designed MOEA that presents state-of-the-art performance according to a set of metrics, rather than a single one.

New mammography screening performance metrics based on the entire screening episode.

Established mammography screening performance metrics use the initial screening mammography assessment because they were developed for radiologist performance auditing, yet these metrics are frequently used to inform health policy and screening decision making. The authors have developed new performance metrics based on the final assessment that consider the entire screening episode, including diagnostic workup. The authors used data from 2,512,577 screening episodes during 2005-2017 at 146 facilities in the United States participating in the Breast Cancer Surveillance Consortium. Screening performance metrics based on the final assessment of the screening episode were compared with conventional metrics defined with the initial assessment. Results were also stratified by breast density and breast cancer risk. The cancer detection rates were similar for the final assessment (4.1 per 1000; 95% confidence interval [CI], 3.8-4.3 per 1000) and the initial assessment (4.1 per 1000; 95% CI, 3.9-4.3 per 1000). The interval cancer rate was 12% higher when it was based on the final assessment (0.77 per 1000; 95% CI, 0.71-0.83 per 1000) versus the initial assessment (0.69 per 1000; 95% CI, 0.64-0.74 per 1000), and this resulted in a modest difference in sensitivity (84.1% [95% CI, 83.0%-85.1%] vs 85.7% [95% CI, 84.8%-86.6%], respectively). Absolute differences in the interval cancer rate between final and initial assessments increased with breast density and breast cancer risk (eg, a difference of 0.29 per 1000 for women with extremely dense breasts and a 5-year risk >2.49%). Established screening performance metrics underestimate the interval cancer rate of a mammography screening episode, particularly for women with dense breasts or an elevated breast cancer risk. Women, clinicians, policymakers, and researchers should use final-assessment performance metrics to support informed screening decisions.

How do you measure up? Methods to assess linkage quality

ABSTRACT
 ObjectivesRecord linkage is a powerful technique which transforms discrete episode data into longitudinal person-based records. These records enable the construction and analysis of complex pathways of health and disease progression, and service use. Achieving high linkage quality is essential for ensuring the quality and integrity of research based on linked data. The methods used to assess linkage quality will depend on the volume and characteristics of the datasets involved, the processes used for linkage and the additional information available for quality assessment. This paper proposes and evaluates two methods to routinely assess linkage quality.
 ApproachLinkage units currently use a range of methods to measure, monitor and improve linkage quality; however, no common approach or standards exist. There is an urgent need to develop “best practices” in evaluating, reporting and benchmarking linkage quality. In assessing linkage quality, of primary interest is in knowing the number of true matches and non-matches identified as links and non-links. Any misclassification of matches within these groups introduces linkage errors. We present efforts to develop sharable methods to measure linkage quality in Australia. This includes a sampling-based method to estimate both precision (accuracy) and recall (sensitivity) following record linkage and a benchmarking method - a transparent and transportable methodology to benchmark the quality of linkages across different operational environments.
 ResultsThe sampling-based method achieved estimates of linkage quality that were very close to actual linkage quality metrics. This method presents as a feasible means of accurately estimating matching quality and refining linkages in population level linkage studies. The benchmarking method provides a systematic approach to estimating linkage quality with a set of open and shareable datasets and a set of well-defined, established performance metrics. The method provides an opportunity to benchmark the linkage quality of different record linkage operations. Both methods have the potential to assess the inter-rater reliability of clerical reviews.
 ConclusionsBoth methods produce reliable estimates of linkage quality enabling the exchange of information within and between linkage communities. It is important that researchers can assess risk in studies using record linkage techniques. Understanding the impact of linkage quality on research outputs highlights a need for standard methods to routinely measure linkage quality. These two methods provide a good start to the quality process, but it is important to identify standards and good practices in all parts of the linkage process (pre-processing, standardising activities, linkage, grouping and extracting).

Impact of a Regional Pharmacy Call Center on Telephone Access Metrics Within the Veterans Health Administration.

To establish a cost-effective centralized pharmacy call center to serve the patients of Veterans Integrated Service Network (VISN) 11 that would meet established performance metrics. A pilot project began in August 2011 with the Indianapolis VA Medical Center (VAMC) and the Health Resource Center (HRC) in Topeka, Kansas. The Indianapolis VAMC used a first-call resolution business model consisting of pharmacy technicians receiving tier 1 phone calls that could be escalated to a tier 2 line that consisted of lead technicians and pharmacists, while the HRC utilized general telephone agents that would transfer unresolved calls to the primary facility. Pre- and post-VISN 11 Pharmacy Call Center performance metrics were compared for each of the 7 facilities in the network with the goals being monthly average abandoned call rate less than 5% and average speed to answer less than 30 seconds. Cost per call was also compared. The average abandoned call rate for the network during the year prior to VISN 11 Pharmacy Call Center implementation (August 2010-July 2011) was 15.66% and decreased to 3% in July 2014. The average abandoned call rate decreased for each individual facility. In fiscal year 2014, the VISN 11 Pharmacy Call Center was operating at a cost of $4.35 per call while providing more services than the HRC, resulting in less workload being transferred back to the individual facilities. A centralized VISN pharmacy call center is a reasonable alternative to individual facility call centers or the HRC.

PD19-04 VALIDATION OF LAPAROSCOPIC TRAINING CURRICULUM: THE BASIC LAPAROSCOPIC UROLOGIC SKILLS (BLUS) INITIATIVE

You have accessJournal of UrologyTechnology & Instruments: Surgical Education & Skills Assessment III1 Apr 2015PD19-04 VALIDATION OF LAPAROSCOPIC TRAINING CURRICULUM: THE BASIC LAPAROSCOPIC UROLOGIC SKILLS (BLUS) INITIATIVE Timothy Kowalewski, Robert Sweet, Ashleigh Menhadji, Timothy Averch, Geoffrey Box, Timothy Brand, Michael Fearrandino, Jihad Kaouk, Bodo Knudsen, Jamie Landman, Benjamin Lee, Bradley Schwartz, Elspeth McDougall, and Thomas Lendvay Timothy KowalewskiTimothy Kowalewski More articles by this author , Robert SweetRobert Sweet More articles by this author , Ashleigh MenhadjiAshleigh Menhadji More articles by this author , Timothy AverchTimothy Averch More articles by this author , Geoffrey BoxGeoffrey Box More articles by this author , Timothy BrandTimothy Brand More articles by this author , Michael FearrandinoMichael Fearrandino More articles by this author , Jihad KaoukJihad Kaouk More articles by this author , Bodo KnudsenBodo Knudsen More articles by this author , Jamie LandmanJamie Landman More articles by this author , Benjamin LeeBenjamin Lee More articles by this author , Bradley SchwartzBradley Schwartz More articles by this author , Elspeth McDougallElspeth McDougall More articles by this author , and Thomas LendvayThomas Lendvay More articles by this author View All Author Informationhttps://doi.org/10.1016/j.juro.2015.02.703AboutPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissionsReprints ShareFacebookTwitterLinked InEmail INTRODUCTION AND OBJECTIVES Standard basic laparoscopic training is required for urologic trainees to ensure proficiency prior to graduation from residency. The American Urological Association created the Basic Laparoscopic Urologic Skills (BLUS) skills tasks as an objective assessment tool to determine laparoscopic skills proficiency. We sought to establish construct validity of the psychomotor skills portion of this curriculum using objective performance metrics and structured assessment by video review. METHODS After IRB approval, eight US medical teaching institutions contributed 117 subjects of varying skill from medical student to full-time faculty urologist. All subjects performed four dry-lab laparoscopic tasks: Peg Transfer, Cutting, Intracorporeal Suturing, and Clip Applying. The Electronic Data Generation and Evaluation (EDGE) laparoscopic platform (Simulab Corp, Seattle WA) recorded synchronized video and tool motion metrics for all tasks along with proctor-entered task errors. Skill levels were assigned by both demographically-derived status (training level) and blinded video review by five faculty urologists employing the Global Objective Assessment of Laparoscopic Skills (GOALS) instrument for a representative subset of the database (12 Suturing, 12 Peg Transfer; maximum, median, and minimum task times). We employed Pearson's correlation to evaluate agreement between metrics and ground truth. RESULTS Using the structured assessment tool – GOALS – provided skill discrimination. Demographically-derived assignments were less correlated with skill. Task time, path length, and motion smoothness (jerk cost) correlated with ground truth structured video assessment (-0.69, -0.69 -0.79 respectively) for Peg Transfer and (-0.86, -0.91 -0.84 respectively) for Suturing (p<0.01). Inter-rater reliability of the five experienced faculty GOALS reviewers was >0.9. CONCLUSIONS The BLUS Peg Transfer and Intracorporeal Suturing skill tasks showed good construct validity based on a consensus of established objective performance metrics and blinded video review by experienced faculty. Training level or case experience alone may be inadequate to assign proficiency levels, thus more objective quantification of skill is required for advancement strategies in training centers. Table 1. Correlation of GOALS scores provided by five expert laparoscopists and EDGE objective performance metrics. Expert GOALS Assessments vs. Tool Metrics Peg Transfer Task Suturing Task N=24, Pearsons R n=12 n=12 R (p-value) R (p-value) EDGE Overall Score 0.95 (0.00) 0.95 (0.00) Task Time -0.91 (0.00) -0.95 (0.00) Tool Path Length -0.95 (0.00) -0.81 (0.00) Jerk Cost -0.63 (0.03) -0.82 (0.00) Movement Count -0.94 (0.00) -0.72 (0.01) Grasp Force NS NS Max Grasp Force NS NS Economy of Motion 0.62 (0.03) NS Errors -0.68 (0.02) 0.62 (0.03) © 2015 by American Urological Association Education and Research, Inc.FiguresReferencesRelatedDetails Volume 193Issue 4SApril 2015Page: e392-e393 Advertisement Copyright & Permissions© 2015 by American Urological Association Education and Research, Inc.MetricsAuthor Information Timothy Kowalewski More articles by this author Robert Sweet More articles by this author Ashleigh Menhadji More articles by this author Timothy Averch More articles by this author Geoffrey Box More articles by this author Timothy Brand More articles by this author Michael Fearrandino More articles by this author Jihad Kaouk More articles by this author Bodo Knudsen More articles by this author Jamie Landman More articles by this author Benjamin Lee More articles by this author Bradley Schwartz More articles by this author Elspeth McDougall More articles by this author Thomas Lendvay More articles by this author Expand All Advertisement Advertisement PDF downloadLoading ...

A View From the Corner Command Post

A View From the Corner Command Post