Predefined Performance Metrics Research Articles

Technical skill assessment in minimally invasive surgery using artificial intelligence: a systematic review.

Technical skill assessment in surgery relies on expert opinion. Therefore, it is time-consuming, costly, and often lacks objectivity. Analysis of intraoperative data by artificial intelligence (AI) has the potential for automated technical skill assessment. The aim of this systematic review was to analyze the performance, external validity, and generalizability of AI models for technical skill assessment in minimally invasive surgery. A systematic search of Medline, Embase, Web of Science, and IEEE Xplore was performed to identify original articles reporting the use of AI in the assessment of technical skill in minimally invasive surgery. Risk of bias (RoB) and quality of theincluded studies were analyzed according to Quality Assessment of Diagnostic Accuracy Studies criteria and the modified Joanna Briggs Institute checklists, respectively. Findings were reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement. In total, 1958 articles were identified, 50 articles met eligibility criteria and were analyzed. Motion data extracted from surgical videos (n = 25) or kinematic data from robotic systems or sensors (n = 22) were the most frequent input data for AI. Most studies used deep learning (n = 34) and predicted technical skills using an ordinal assessment scale (n = 36) with good accuracies in simulated settings. However, all proposed models were in development stage, only 4 studies were externally validated and 8 showed a low RoB. AI showed good performance in technical skill assessment in minimally invasive surgery. However, models often lacked external validity and generalizability. Therefore, models should be benchmarked using predefined performance metrics and tested in clinical implementation studies.

Adoption of information and digital technologies for sustainable smart manufacturing systems for industry 4.0 in small, medium, and micro enterprises (SMMEs)

An enterprise reference model in a conventional system paradigm guides users when selecting manufacturing elements, configuring the required elements into a manufacturing system, modeling the system options for evaluative processes, and comparing the system solutions with the predefined performance metrics. At present, digital innovation has a close connection with firms' sustainability. Sustainability and digital innovation are two essential components of the circular economy. In this study, an integrated decision-making framework called the q-ROF-MEREC-RS-DNMA is developed. In this approach, the q-ROF-MEREC-RS method is applied to compute the subjective and objective weights of criteria to the adoption of information and digital technologies for sustainable smart manufacturing systems for Industry 4.0 in small, medium, and micro enterprises (SMMEs), and the q-ROF-DNMA model is used to assess the preferences of industries over different criteria to the adoption of information and digital technologies for sustainable smart manufacturing systems for Industry 4.0 in SMMEs. An empirical case study to evaluate the main criteria for the adoption of information and digital technologies for sustainable smart manufacturing systems for Industry 4.0 in small, medium, and micro enterprises (SMMEs) is taken. Also, comparison and sensitivity investigation are made to show the superiority of the developed framework.

A monitoring and control test bench for assessing the upgraded low voltage power supplies for the ATLAS Tile Calorimeter Phase-II Upgrade front-end electronics

The ATLAS detector is set to undergo a substantial upgrade termed the “Phase-II” upgrade during the Long-Shutdown in preparation for the start of operation of the High Luminosity Large Hadron Collider (HL-LHC). This paper describes the development and implementation of the Phase-II upgrade low voltage power supply transformer-coupled buck converter (Brick) test bench. A large scale production of approximately 2048 finger low voltage power supplies, with an identical output voltage, is set to be undertaken in the year 2022. This production will culminate in the complete replacement of the LVPS Brick V7.5.0 currently installed within the Tile Calorimeter. Due to the location of the Bricks within the inner barrel of the TileCal their reliability is of the utmost importance. Therefore, the implementation of performance screening will be implemented and will involve the testing and certification of all Bricks produced. A custom test bench has been developed in order to facilitate the above-mentioned testing. The test bench described herein quantifies and logs the performance of an upgrade Brick allowing for comparison with predefined performance metrics before being approved and subsequently installed on-detector.

Artificial intelligence for detection of microsatellite instability in colorectal cancer—a multicentric analysis of a pre-screening tool for clinical application

BackgroundMicrosatellite instability (MSI)/mismatch repair deficiency (dMMR) is a key genetic feature which should be tested in every patient with colorectal cancer (CRC) according to medical guidelines. Artificial intelligence (AI) methods can detect MSI/dMMR directly in routine pathology slides, but the test performance has not been systematically investigated with predefined test thresholds.MethodWe trained and validated AI-based MSI/dMMR detectors and evaluated predefined performance metrics using nine patient cohorts of 8343 patients across different countries and ethnicities.ResultsClassifiers achieved clinical-grade performance, yielding an area under the receiver operating curve (AUROC) of up to 0.96 without using any manual annotations. Subsequently, we show that the AI system can be applied as a rule-out test: by using cohort-specific thresholds, on average 52.73% of tumors in each surgical cohort [total number of MSI/dMMR = 1020, microsatellite stable (MSS)/ proficient mismatch repair (pMMR) = 7323 patients] could be identified as MSS/pMMR with a fixed sensitivity at 95%. In an additional cohort of N = 1530 (MSI/dMMR = 211, MSS/pMMR = 1319) endoscopy biopsy samples, the system achieved an AUROC of 0.89, and the cohort-specific threshold ruled out 44.12% of tumors with a fixed sensitivity at 95%. As a more robust alternative to cohort-specific thresholds, we showed that with a fixed threshold of 0.25 for all the cohorts, we can rule-out 25.51% in surgical specimens and 6.10% in biopsies.InterpretationWhen applied in a clinical setting, this means that the AI system can rule out MSI/dMMR in a quarter (with global thresholds) or half of all CRC patients (with local fine-tuning), thereby reducing cost and turnaround time for molecular profiling.

Recognizing the Physical Activity of Hospitalized Older People From Wearable Sensors Data Using IoT

The IoT is a new concept that provides a world where smart, connected, embedded systems operate, giving rise to the amount of data from different sources that are considered to have highly useful and valuable information. Data mining would play a critical role in creating smarter IoT. Traditional care of an elderly person is a difficult and complex task. The need to have a caregiver with the elderly person almost all the time drains the human and financial resources of the health care system. The emergence of Artificial intelligence has allowed the conception of technical assistance where it helps and reduces the time spent by the caregiver with the elderly person. This work aims to focus on analyzing techniques that are used for prediction purposes of falls in the elderly. We examine the applicability of three classification algorithms for IoT data. These algorithms are analyzed and a comparative study is undertaken to find the classifier that performs the best analysis on the dataset using a set of predefined performance metrics to compare the results of each classifier.

Evaluation of the ultimate eccentric load of rectangular CFSTs using advanced neural network modeling

In this paper an Artificial Neural Network (ANN) model is developed for the prediction of the ultimate compressive load of rectangular Concrete Filled Steel Tube (CFST) columns, taking into account load eccentricity. To this end, an experimental database of CFST specimens from the literature has been compiled, totaling 1224 individual tests, both under concentric and under eccentric loading. Except for eccentricity, other parameters taken into consideration include the cross section width, height and thickness, the steel yield limit, the concrete strength and the column length. Both short and long specimens were evaluated. The architecture of the proposed ANN model was optimally selected, according to predefined performance metrics. The developed model was then compared against available design codes. It was found that its accuracy was significantly improved while maintaining a stable numerical behavior. The explicit equation that describes mathematically the ANN is offered in the paper, for easier implementation and evaluation purposes.

A machine learning approach to galaxy properties: joint redshift–stellar mass probability distributions with Random Forest

ABSTRACTWe demonstrate that highly accurate joint redshift–stellar mass probability distribution functions (PDFs) can be obtained using the Random Forest (RF) machine learning (ML) algorithm, even with few photometric bands available. As an example, we use the Dark Energy Survey (DES), combined with the COSMOS2015 catalogue for redshifts and stellar masses. We build two ML models: one containing deep photometry in the griz bands, and the second reflecting the photometric scatter present in the main DES survey, with carefully constructed representative training data in each case. We validate our joint PDFs for 10 699 test galaxies by utilizing the copula probability integral transform and the Kendall distribution function, and their univariate counterparts to validate the marginals. Benchmarked against a basic set-up of the template-fitting code bagpipes, our ML-based method outperforms template fitting on all of our predefined performance metrics. In addition to accuracy, the RF is extremely fast, able to compute joint PDFs for a million galaxies in just under 6 min with consumer computer hardware. Such speed enables PDFs to be derived in real time within analysis codes, solving potential storage issues. As part of this work we have developed galpro1, a highly intuitive and efficient python package to rapidly generate multivariate PDFs on-the-fly. galpro is documented and available for researchers to use in their cosmology and galaxy evolution studies.

Identifying Outcome Measures for Migraine Value-Based Contracting Using the Delphi Method.

To identify meaningful migraine outcome measures among key stakeholders to inform value-based contracts for migraine medications. Value-based contracts linking medication payments to predefined performance metrics aim to promote value through aligned incentives and shared risk between manufacturers and payers. The emergence of new and expensive pharmaceuticals for migraine presents an opportunity for value-based contract development. However, uncertainty remains around which outcomes are most meaningful to all migraine stakeholders. This study utilized a Delphi survey to incorporate views from 82 stakeholders, including patients (n=21), providers (n=23), payers (n=10), employers (n=18), and pharmaceutical company representatives (n=10). A list of 15 migraine-related outcomes was created from a literature review and subject matter expert consultation. Stakeholders reported on the value of these outcomes through a 5-point Likert scale and selection of their top 3 most meaningful outcomes. All participants except patients and employers also used a 5-point Likert scale to rate the feasibility of collecting each outcome measure. Consensus was defined as ≥75% agreement on the importance and feasibility of an outcome (Likert scores ≥4/5 or selection of an outcome as most meaningful). After 2 rounds, consensus was achieved for importance of 9 outcomes on the Likert scale. "Decrease in migraine frequency" reached 100% agreement (82/82), followed by "increased ability to resume normal activities" (96%, 79/82). When asked to choose the 3 most meaningful outcomes, stakeholders selected "decrease in migraine frequency" (88%, 72/82) followed by "decrease in migraine severity" (80%, 66/82). The 2 measures rated as most feasibly collected were "decrease in emergency department/urgent care visits" (95%, 40/42) and "decrease in migraine frequency" (90%, 38/42). There were statistically significant differences between non-patient and patient stakeholders in selection of "decrease in emergency department/urgent care visits" [20% (12/61) vs 0% (0/21), P=.031]; and employer and patient stakeholders in selection of "decrease in work days missed" [44% (8/18) vs 5% (1/21), P=.006] and "decrease in emergency department/urgent care visits" [22% (4/18) vs 0% (0/21), P=.037] as most meaningful outcomes. The measures "decrease in migraine frequency" followed by "decrease in migraine severity" were identified as top priority migraine outcome measures.

Organisational learning through knowledge management systems: a case study on improvement of customer support processes

Customer support processes and customer relationship management act as enablers of organisational learning by providing valuable inputs to quality improvement and innovative processes in the existence of appropriate knowledge management systems and problem-solving mechanisms. This single case study explores the impact of effective utilisation of knowledge management systems and six sigma-based problem-solving practices in customer support processes on organisational learning by using the findings from practices of a telecommunication company which is a subsidiary of a MNC in Turkey. Based on the interviews, observations and data provided from the company, customer support processes and organisational structure of customer support teams of the company are analysed, and performance metrics/indicators of the processes are identified. The impacts of three new systems, knowledge management system, a problem-solving system, that is based on six sigma methodology, and a 'skill-based resource allocation system' on performance of customer support processes, are explored by analysing the changes in the predefined performance metrics. Findings reveal that a knowledge management system that is integrated with six sigma methodology improved the effectiveness of customer support processes by providing shorter problem resolution time, reuse of the previous solutions and reduced engineer efforts. Appropriate organisational structuring, effective integration between the systems and acceptance of the systems by the users played major roles in the success of the new systems.

Metabolic engineering with multi-objective optimization of kinetic models

Kinetic models have a great potential for metabolic engineering applications. They can be used for testing which genetic and regulatory modifications can increase the production of metabolites of interest, while simultaneously monitoring other key functions of the host organism. This work presents a methodology for increasing productivity in biotechnological processes exploiting dynamic models. It uses multi-objective dynamic optimization to identify the combination of targets (enzymatic modifications) and the degree of up- or down-regulation that must be performed in order to optimize a set of pre-defined performance metrics subject to process constraints. The capabilities of the approach are demonstrated on a realistic and computationally challenging application: a large-scale metabolic model of Chinese Hamster Ovary cells (CHO), which are used for antibody production in a fed-batch process. The proposed methodology manages to provide a sustained and robust growth in CHO cells, increasing productivity while simultaneously increasing biomass production, product titer, and keeping the concentrations of lactate and ammonia at low values. The approach presented here can be used for optimizing metabolic models by finding the best combination of targets and their optimal level of up/down-regulation. Furthermore, it can accommodate additional trade-offs and constraints with great flexibility.

Towards cross‐layer approaches to coping with misbehavior in mobile ad hoc networks: an anatomy of reputation systems

AbstractIn mobile ad hoc networks (MANETs), the nodes need to cooperate each other to establish multi‐hop routes for out‐of‐range wireless communication. However, some of them may not always behave normally, either behaving selfishly for saving computational resource or maliciously for compromising communication protocols. Regardless of intents, such misbehavior would lead to the degradation of network performance. It is therefore important to design appropriate mechanisms to ensure that network performance could be maintained at an acceptable level in the presence of misbehaving nodes. But the open nature of MANETs makes such designs challenging. Reputation system has been widely recognized as an effective approach, which associates the behavior of nodes with its reputation, which is calculated by specifying and quantifying the observations of interest with respect to predefined performance metrics. More interestingly, the observations can be obtained and integrated from multiple layers, facilitating cross‐layer analysis. This paper intends to take a deep look into several well‐studied reputation systems and examine their operational characteristics in terms of modeling approaches and redemption techniques, with an objective to identify their capabilities in terms of misbehavior detection coverage and blind spots. Furthermore, such an anatomy allows us to better understand the failure curses of the deployment and operation of reputation systems in MANETs, so as to improve their performance by adopting effective countermeasures.Copyright © 2014 John Wiley & Sons, Ltd.

Understanding performance issues in IP networks

This paper considers the techniques available for assessing whether a network is capable of complying with predefined performance metrics, which are likely to be expressed in terms of throughput, delay and packet loss. It describes how dimensioning was traditionally done and the challenges offered by new insights into data arrival patterns. It demonstrates how modern quality of service management techniques can be used to manage performance, but can complicate prediction. Finally it will illustrate how semi-empirical statistical techniques offer some resolution.