Performance assessment methods and metrics for passive daytime radiative cooling materials

Constructing hybrid infrastructure: Exploring the potential ecological, social, and economic benefits of integrating municipal infrastructure into constructed environments

A human-robot collaborative system in the form of a power and skill assist robotic system was developed where a human and a robot could collaborate to perform object manipulation for targeted assembly tasks in automotive manufacturing. We assumed such assembly tasks as the representative assembly tasks in automotive manufacturing. We reflected human's weight perception in the dynamics and control of the power and skill assist system following a psychophysical method using a reinforcement learning scheme. We recruited 20 human subjects who separately performed assembly tasks with the system in human-robot collaboration (HRC). We then observed the collaborative assembly tasks, conducted extensive literature reviews, reviewed our previous and ongoing related works and brainstormed with the subjects and other relevant researchers, and then proposed HRC performance assessment metrics and methods for collaborative automotive manufacturing. The proposed metrics comprised of assessment criteria and methods related to both human-robot interaction (HRI) and manufacturing performance. We then verified the proposed performance metrics in pilot studies in the laboratory environment using the same collaborative system and subjects. The verification results proved the effectiveness of the assessment metrics and methods in terms of usability, practicability and reliability. We then proposed to apply classification and regression type machine learning approaches under supervised and reinforcement learning setups to learn different classes and decision-making rules respectively regarding HRC performance. The proposed performance metrics and methods can serve as the preliminary efforts towards developing comprehensive assessment metrics for HRC in general and for human-robot collaborative automotive manufacturing in particular.

Targeting on the verification and validation of prognostic and health management functional design, the performance assessment metrics and methods are rare. Aiming at establishing an application-oriented metrics system model, a set of testability, diagnosis, prognosis and efficiency algorithm performance metrics for the PHM system is described on the basis of performance requirements. The analytic hierarchy process is used to compare and decide PHM performance metrics system, and the weights are computed. The fuzzy comprehensive assessment is used to generalize the specified metrics. A comprehensive assessment method on performance metrics is proposed by combining the above two methods and the assessment of PHM system function is achieved.

Analytical studies on automatic bug triaging approach have the main objective to recommend appropriate developer for bug report with reduced bug tossing length, time and effort in bug resolution. In bug triaging process, if the first recommended developer cannot fix a bug, it is tossed to another developer and the tossing process is continued till the bug gets assigned and resolved. Existing approaches to the best of our knowledge have not considered developer's contributions and performance assessment metrics for bug triaging process. In this paper, we proposed a novel and improved two phase bug triager that involves a developer profile creation and assignment phases. In this, developer profile is built by using individual contributions (IC) and performance assessment (PA) metrics. Contribution and performance of a developer in pre-fixed bug reports are analysed to calculate a developer's weighted score. This score indicates the level of expertise to fix and resolve a newly reported bug. This approach is tested on two open source projects - Eclipse and Mozilla. Empirical results show that proposed approach has achieved a significantly higher F-score up to 90% for both projects and has effectively reduced bug tossing length up to 11.8% as compared to existing approaches.

In this paper, the dimension estimation of memory polynomial based power amplifiers' behavioral models is investigated. Comprehensive study of state of the art performance assessment metrics is carried out. To ensure the generality of the conclusions, the study is experimentally validated using three power amplifiers prototypes designed using various transistors technologies and amplifier topologies. The results show that the performance assessment metrics are equally reliable for the estimation of the device under test nonlinearity order. However, they are inconsistent when used to estimate its memory depth. This limitation is circumvented by applying memoryless post-compensation technique during the memory depth estimation. The proposed approach demonstrates that the dimension estimation can be made objective and independent of the performance assessment metric.

Maintaining trauma-specific surgical skills is an ongoing challenge for surgical training programs. An objective assessment of surgical skills is needed. We hypothesized that a validated surgical performance assessment tool could detect differences following a training intervention. We developed surgical performance assessment metrics based on discussion with expert trauma surgeons, video review of 10 experts and 10 novice surgeons performing three vascular exposure procedures and lower extremity fasciotomy on cadavers, and validated the metrics with interrater reliability testing by five reviewers blinded to level of expertise and a consensus conference. We tested these performance metrics in 12 surgical residents (Year 3-7) before and 2 weeks after vascular exposure skills training in the Advanced Surgical Skills for Exposure in Trauma (ASSET) course. Performance was assessed in three areas as follows: knowledge (anatomic, management), procedure steps, and technical skills. Time to completion of procedures was recorded, and these metrics were combined into a single performance score, the Trauma Readiness Index (TRI). Wilcoxon matched-pairs signed-ranks test compared pretraining/posttraining effects. Mean time to complete procedures decreased by 4.3 minutes (from 13.4 minutes to 9.1 minutes). The performance component most improved by the 1-day skills training was procedure steps, completion of which increased by 21%. Technical skill scores improved by 12%. Overall knowledge improved by 3%, with 18% improvement in anatomic knowledge. TRI increased significantly from 50% to 64% with ASSET training. Interrater reliability of the surgical performance assessment metrics was validated with single intraclass correlation coefficient of 0.7 to 0.98. A trauma-relevant surgical performance assessment detected improvements in specific procedure steps and anatomic knowledge taught during a 1-day course, quantified by the TRI. ASSET training reduced time to complete vascular control by one third. Future applications include assessing specific skills in a larger surgeon cohort, assessing military surgical readiness, and quantifying skill degradation with time since training.

Organizations have to manage human resources effectively, as these are fundamental to their success. Indeed, it is widely recognized that human resources have a direct influence in the performance of organizations. Therefore, organizational success is highly dependent on an adequate management of human resources. In this context, the performance assessment of people is crucial, as it is an important process for implementing efficient and effective motivational and rewarding systems. However, in the case of information systems projects, there is not much research work focused on human resources performance evaluation. This chapter aims to contribute to fill this gap by reviewing several approaches and methods for performance assessment, which can be applied to information systems projects. The presented approaches and methods are focused on personality, behaviors, comparison, and outcomes/results.

Cancer remains a formidable global health challenge, claiming millions of lives annually. Timely and accurate cancer diagnosis is imperative. While numerous reviews have explored cancer classification using machine learning and deep learning techniques, scant literature focuses on traditional ML methods. In this manuscript, we undertake a comprehensive review of colorectal and gastric cancer detection specifically employing traditional ML classifiers. This review emphasizes the mathematical underpinnings of cancer detection, encompassing preprocessing techniques, feature extraction, machine learning classifiers, and performance assessment metrics. We provide mathematical formulations for these key components. Our analysis is limited to peer-reviewed articles published between 2017 and 2023, exclusively considering medical imaging datasets. Benchmark and publicly available imaging datasets for colorectal and gastric cancers are presented. This review synthesizes findings from 20 articles on colorectal cancer and 16 on gastric cancer, culminating in a total of 36 research articles. A significant focus is placed on mathematical formulations for commonly used preprocessing techniques, features, ML classifiers, and assessment metrics. Crucially, we introduce our optimized methodology for the detection of both colorectal and gastric cancers. Our performance metrics analysis reveals remarkable results: 100% accuracy in both cancer types, but with the lowest sensitivity recorded at 43.1% for gastric cancer.

Morphing attacks involve generating a single artificial facial photograph that represents two distinct qualities and utilizing it as a reference photograph on a document. The high quality of the morph raises the question of how vulnerable facial recognition systems are to morph attacks. Morphing Attack Detection (MAD) systems have aroused a lot of interest in recent years, owing to the freely available digital alteration tools that criminals can employ to perform face morphing attacks. There is, however, little research that critically reviews the methodology and performance metrics used to evaluate MAD systems. The goal of this study is to find MAD methodologies, feature extraction techniques, and performance assessment metrics that can help MAD systems become more robust. To fulfill this study's goal, a Systematic Literature Review was done. A manual search of 9 well-known databases yielded 2089 papers. Based on the study topic, 33 primary studies were eventually considered. A novel taxonomy of the strategies utilized in MAD for feature extraction is one of the research's contributions. The study also discovered that (1) single and differential image-based approaches are the commonly used approaches for MAD; (2) texture and keypoint feature extraction methods are more widely used than other feature extraction techniques; and (3) Bona-fide Presentation Classification Error Rate and Attack Presentation Classification Error Rate are the commonly used performance metrics for evaluating MAD systems. This paper addresses open issues and includes additional pertinent information on MAD, making it a valuable resource for researchers developing and evaluating MAD systems.KeywordsFace morphingMorphing attack detectionSystematic literature reviewFeature extraction techniquesPerformance metrics

Although various algorithms have widely been studied for bankruptcy and credit risk prediction, conclusions regarding the best performing method are divergent when using different performance assessment metrics. As a solution to this problem, the present paper suggests the employment of two well-known multiple-criteria decision-making (MCDM) techniques by integrating their preference scores, which can constitute a valuable tool for decision-makers and analysts to choose the prediction model(s) more properly. Thus, selection of the most suitable algorithm will be designed as an MCDM problem that consists of a finite number of performance metrics (criteria) and a finite number of classifiers (alternatives). An experimental study will be performed to provide a more comprehensive assessment regarding the behavior of ten classifiers over credit data evaluated with seven different measures, whereas the Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) and Preference Ranking Organization METHod for Enrichment of Evaluations (PROMETHEE) techniques will be applied to rank the classifiers. The results demonstrate that evaluating the performance with a unique measure may lead to wrong conclusions, while the MCDM methods may give rise to a more consistent analysis. Furthermore, the use of MCDM methods allows the analysts to weight the significance of each performance metric based on the intrinsic characteristics of a given credit granting decision problem.

Assessment of Individual Operator Performance Using a Risk-Adjustment Model for Percutaneous Coronary Interventions

Background Current methods of surgeon performance assessment are unstructured, labour-intensive and subjective. Machine learning could provide rapid, automated and subjective assessment. We design a custom deep learning model to output performance metrics from videos of LSG and validate these against OSATS and clinical outcomes. Methods A novel dataset of 3210 images from videos of LSG was annotated to train an instrument segmentation framework based on Mask R-CNN, a state-of-the-art deep learning algorithm. Spatial outputs of detected instruments were used to determine workflow through a Markov chain and secondly to determine instrument metrics for assessment of surgical performance. Metrics including time spent in stage and instrument time, distance, smoothness, concentration, and efficiency were then validated against OSATS ratings from 2 independent scorers and clinical outcomes from 35 LSGs. Results Grasper and ligasure efficiency and smoothness were higher in videos rated within the top quartile by OSATS score when compared to those in the bottom quartile and when comparing consultants to trainees. Videos in the top quartile for grasper smoothness had higher total OSATS scores (17.31±2.27) compared to those in the lowest quartile (15.69±2.93) and across all 5 individual OSATS domains in addition to higher percentage of total weight loss at 3, 6, and 12 months. Conclusions This study demonstrates how vision based deep learning can be used for surgeon performance assessment. Future work will aim to validate this against a large number of videos, expand upon the range of metrics produced, and look towards the steps needed for clinical translation.

With government, industry, and public pressure to decarbonize the building sector through reducing embodied and operational emissions, there have been a wide range of innovative materials used in building envelope retrofits. Although these innovative materials, such as super insulating materials, bio-based insulation, and many others, are assessed on thermal performance and code requirements before use in retrofits, there is no unified standard assessment metric for hygrothermal performance of innovative materials in building envelope retrofits. This paper performs a rapid review of the available literature from January 2013 to March 2025 on hygrothermal performance assessment metrics used in retrofits. Using rapid review methods to search for records in Scopus, Web of Science, and Google Scholar, fifty-nine publications were selected for bibliometric and qualitative analysis. Most selected publications include discussions and analysis of relative humidity in the wall assembly post retrofit, moisture content, and mould index within the envelope. There is a research gap in publications considering hygrothermal damage functions such as freeze–thaw index, relative humidity and temperature (RHT) index, or condensation prediction. There is also a research gap in country and climate studies and analyses of in situ retrofits with innovative materials, and occupant comfort post retrofit.

Satellite radar interferometry (InSAR) is an emerging technique to monitor the stability and health of line-infrastructure assets, such as railways, dams, and pipelines. However, InSAR is an opportunistic approach as the location and occurrence of its measurements (coherent scatterers) cannot be guaranteed, and the quality of the InSAR products is not uniform. This is a problem for operational asset managers, who are used to surveying techniques that provide results with uniform quality at predefined locations. Therefore, advanced integrated products and generic performance assessment metrics are necessary. Here, we propose several new monitoring products and quality metrics for a-priori and a-posteriori performance assessment using multisensor InSAR. These products and metrics are demonstrated on a 125 km railway line-infrastructure asset in the Netherlands.

In Machine Learning, the datasets used to build models are one of the main factors limiting what these models can achieve and how good their predictive performance is. Machine Learning applications for cyber-security or computer security are numerous including cyber threat mitigation and security infrastructure enhancement through pattern recognition, real-time attack detection, and in-depth penetration testing. Therefore, for these applications in particular, the datasets used to build the models must be carefully thought to be representative of real-world data. However, because of the scarcity of labelled data and the cost of manually labelling positive examples, there is a growing corpus of literature utilizing Semi-Supervised Learning with cyber-security data repositories. In this work, we provide a comprehensive overview of publicly available data repositories and datasets used for building computer security or cyber-security systems based on Semi-Supervised Learning, where only a few labels are necessary or available for building strong models. We highlight the strengths and limitations of the data repositories and sets and provide an analysis of the performance assessment metrics used to evaluate the built models. Finally, we discuss open challenges and provide future research directions for using cyber-security datasets and evaluating models built upon them.