Suitable Performance Metrics Research Articles

Graph neural networks are promising for phenotypic virtual screening on cancer cell lines.

Artificial intelligence is increasingly driving early drug design, offering novel approaches to virtual screening. Phenotypic virtual screening (PVS) aims to predict how cancer cell lines respond to different compounds by focusing on observable characteristics rather than specific molecular targets. Some studies have suggested that deep learning may not be the best approach for PVS. However, these studies are limited by the small number of tested molecules as well as not employing suitable performance metrics and dissimilar-molecules splits better mimicking the challenging chemical diversity of real-world screening libraries. Here we prepared 60 datasets, each containing approximately 30 000-50 000 molecules tested for their growth inhibitory activities on one of the NCI-60 cancer cell lines. We conducted multiple performance evaluations of each of the five machine learning algorithms for PVS on these 60 problem instances. To provide even a more comprehensive evaluation, we used two model validation types: the random split and the dissimilar-molecules split. Overall, about 14 440 training runs aczross datasets were carried out per algorithm. The models were primarily evaluated using hit rate, a more suitable metric in VS contexts. The results show that all models are more challenged by test molecules that are substantially different from those in the training data. In both validation types, the D-MPNN algorithm, a graph-based deep neural network, was found to be the most suitable for building predictive models for this PVS problem.

Improvement of an order-to-cash business process by deploying lean six sigma tools: a case study

PurposeThis study is focused to support an ongoing development project of the case company's current state and the challenges of the order-to-cash (O2C) process. The O2C process is the most visible process to the customer, and therefore, its punctual and fluent order management is vital. It is observed that the high degree of manual work in the O2C process causes mistakes, delays and rework in the process. The purpose of this article is therefore to analyze the case company's current state of the O2C process as well as to identify the areas of development in this process by deploying the means of Lean Six Sigma tools such as value stream mapping (VSM).Design/methodology/approach The study was conducted as a mix of quantitative and qualitative analysis. Based on both the quantitative and qualitative data, a workshop on VSM was organized to analyze the current state of the O2C process of a case company, engaged in the energy and environment sector in Finland.Findings The results found that excessive manual work was highly connected to inadequate or incorrect data in pricing and invoicing activities, which resulted in canceled invoices. Canceled invoices are visible to the customer and have a negative impact on the customer experience. This study found that by improving the performance of the O2C process activities and improving communication among the internal and external stakeholders, the whole O2C process can perform more effectively and provide better customer value.Originality/value The O2C process is the most visible process to the customer and therefore its punctual and fluent order management is vital. To ensure that the O2C process is operating as desired, suitable process performance metrics need to be aligned and followed. The results gathered from the case company's data, questionnaire interviews, and the VSM workshop are all highlighted in this study. The main practical and managerial implications were to understand the real-time O2C process performance, which is necessary to ensure strong performance and enhance continuous improvement of the O2C process that leads to operational excellence and commercial competitiveness of the studied case company.

Reducing False Negative Intrusions Rates of Ensemble Machine Learning Model based on Imbalanced Multiclass Datasets

In spite of the efforts to improve the efficiency of intrusion detection systems based on machine learning algorithms, these systems still need more. The false negative (FN) prediction outcome is of a major priority among other outcomes, when attacks are considered as normal by classifiers. FN outputs are highly a concern issue, especially for multiclass classification, where minor classes have less instances in imbalanced datasets. In this work, three types of well-known imbalanced multiclass classification datasets are used with ensemble machine learning classifiers. The datasets: KDD99, UNSW_NB15, and CICIDS2017 are balanced using different combination of oversampling and under-sampling techniques to improve false negative rates. Suitable performance metrics have been used to obtain significant outputs improvements in all three datasets types using Random Forest classifier. Achieved accuracies are 99.9852% for KDD99, 83.5451% for UNSW_NB15 and 99.8613% for CICIDS2017. The outcomes of the work using the mentioned datasets have been compared with state-of-the-art related works and the results show a clear improvement in false negative rates.

Guest Editorial Special Issue on Benchmarking Sampling-Based Optimization Heuristics: Methodology and Software

Benchmarking provides an essential ground base for adequately assessing and comparing evolutionary computation methods and other optimization algorithms. It allows us to gain insights into strengths and weaknesses of different existing techniques, and consequently design more efficient optimization approaches. The need for good benchmarking practices opens up a broad range of complementary research questions, arising as a byproduct of challenges encountered when optimization methods are assessed. From the selection of representative benchmark problem instances, different algorithms, and suitable performance metrics, over efficient experimentation, to a sound evaluation of the benchmark data, these research questions lie at the core of establishing a well-designed and standardized benchmarking procedure.

Biometric multi‐factor authentication: On the usability of the FingerPIN scheme

AbstractFingerprint‐based authentication has been successfully adopted in a wide range of applications, including law enforcement and immigration, due to its numerous advantages over traditional password‐based authentication. Despite the usability and accuracy of this technology, some significant concerns still exist, which can potentially hinder its further adoption. For instance, a fingerprint is permanently associated with an individual and, once stolen, cannot be replaced, thus compromising biometric‐based authentication. To mitigate this concern, we previously designed a multi‐factor authentication approach that integrates Type‐1 and Type‐3 authentication factors into a fingerprint‐based personal identification number (PIN). To authenticate, a subject is required to present a sequence of fingerprints corresponding to the digits of the PIN, based on a predefined secret mapping between digits and fingers. We conducted a preliminary vulnerability analysis and demonstrated that this approach is robust to the compromise of one or more of the subject's fingerprints. The objective of the work presented in this paper is to identify any usability issues for this FingerPIN scheme, collect qualitative and quantitative data through a user study, and determine the participants' satisfaction with the authentication mechanism. We carried out systematic usability tests, designed suitable performance metrics for assessing authentication usability on an initial cohort of 100 individuals, and performed a comparative analysis of the FingerPIN scheme against traditional sequential multi‐factor authentication schemes.

A novel methodology to quantify hydraulic conveyance through an ice-impacted lake-outlet system

River ice can pose considerable challenges to hydropower operations in cold regions. Relative to open-water conditions, the presence of ice in various forms contributes to lower hydraulic efficiency. At the outlet of Lake Winnipeg, maintaining winter conveyance through the channels of the Outlet Lakes Area (OLA) is of paramount importance for downstream energy production on the Nelson River. The complex hydraulic conditions of the OLA are attributed to various factors, including the influence of hydraulic system operation, flow routing through lakes and rivers with non-uniform attributes, and the presence of dynamic ice processes throughout winter. In this study, a methodology was designed to quantify the impacts of ice on the OLA hydraulics using a combination of numerical modelling and statistical techniques. This approach includes adaptation and configuration of a two-dimensional river ice model, selection of suitable performance metrics, and derivation of hydraulic datasets for model simulations using historical observations and statistical relationships. Numerical modelling simulations feature treatments for predominant ice processes in the OLA, including freeze-up jamming, thermal growth and decay of a stable ice cover, and under cover transport and deposition of brash ice. The methodology was evaluated using historical winters 1996–1997 to 2020–2021, with most results falling within an acceptable performance range, even in response to varying hydro-meteorological conditions. Instances of error exceeding accepted thresholds were attributed mainly to factors outside of the study, including uncertainty in discharge estimation and the effects of mid-winter forebay operation. Influence of Lake Winnipeg water levels on winter hydraulics was identified as significant, as higher upstream Lake Winnipeg water levels can mitigate ice impacts by providing more hydraulic head for conveyance. As expected, ice restriction severity also varies in accordance with heat fluxes. Targeted areas for improvement of the methodology and future work are discussed, including enhancing the simulation of the thawing (‘pre-breakup’) period and field verification of flow split quantities during winter. The methodology presented provides an estimate of ice-impacted hydraulic conveyance which serves to benefit hydropower operators, regional water resources studies, and further investigations of river ice impacts that can be attributed to future climate change.

Recommendations for machine learning benchmarks in neuroimaging

The field of neuroimaging has embraced methods from machine learning in a variety of ways. Although an increasing number of initiatives have published open-access neuroimaging datasets, specifically designed benchmarks are rare in the field. In this article, we first describe how benchmarks in computer science and biomedical imaging have fostered methodological progress in machine learning. Second, we identify the special characteristics of neuroimaging data and outline what researchers have to ensure when establishing a neuroimaging benchmark, how datasets should be composed and how adequate evaluation criteria can be chosen. Based on lessons learned from machine learning benchmarks, we argue for an extended evaluation procedure that, next to applying suitable performance metrics, focuses on scientifically relevant aspects such as explainability, robustness, uncertainty, computational efficiency and code quality. Lastly, we envision a collaborative neuroimaging benchmarking platform that combines the discussed aspects in a collaborative and agile framework, allowing researchers across disciplines to work together on the key predictive problems of the field of neuroimaging and psychiatry.

Cooperative Communication and Mobility for Securing URLLC of Future Wireless Networks

Ultra-reliable low-latency communication (uRLLC) emerges as one of the three service categories in fifth-generation (5G) wireless networks, with a potential to support multiple advanced services for latency-sensitive communications. The inherent openness of the wireless environment makes security in 5G unprecedentedly urgent to be addressed. Accordingly, the physical layer security (PLS) technique exploits the randomness characteristics of the wireless channel to fulfill security and confidentiality of information transmission. Different from many current works, which ignored the significance of security in uRLLC, this paper studies secrecy improvement strategy design in uRLLC scenarios. First, we propose suitable performance metrics for evaluating PLS of uRLLC and make a compromise between reliability, latency, and security, while maximizing the network throughput. Next, we identify the key challenging issues for achieving PLS in the context of uRLLC, and discuss the role of cooperative communication and mobility in terms of possible solutions. To the best of our knowledge, this is the first paper to make a trade-off among reliability, security and network throughput. Simulations validate the theoretical analyses.

Required parameters for modelling heterogeneous geographically dispersed manufacturing systems

COVID-19 and global crises/events are driving governments to rethink their national manufacturing strategies. The drastic change of societal conditions has exposed our reliance on a constrained set of production practices. Furthermore, the future manufacturing landscape indicates - supply chain crises, trade agreements and natural disasters - a high level of volatility which requires a response that is far from being achieved.While these emergent challenges have called the efficacy of established practices into question, new manufacturing technologies, such as Additive Manufacturing (AM), present the capability to provide a solution. One proposal is agent-based brokering of AM which could be a method for tackling local, regional, national, and international production needs. However, to achieve the reality of brokered AM, it is imperative that the diversity of AM capability is considered. Diversity that existing homogeneous modelling of AM and manufacturing systems rarely consider or capture. This paper conceptualizes the reality of AM systems and elucidates parameters that are necessary for successful modelling and subsequent co-ordination. Having presented the required parameters the paper continues to discuss requisite levels of abstraction, suitable performance metrics and the role of humans in agent-based manufacturing systems.

COST-SENSITIVE MULTI-CLASS ADABOOST FOR UNDERSTANDING DRIVING BEHAVIOR BASED ON TELEMATICS

ABSTRACTUsing telematics technology, insurers are able to capture a wide range of data to better decode driver behavior, such as distance traveled and how drivers brake, accelerate, or make turns. Such additional information also helps insurers improve risk assessments for usage-based insurance, a recent industry innovation. In this article, we explore the integration of telematics information into a classification model to determine driver heterogeneity. For motor insurance during a policy year, we typically observe a large proportion of drivers with zero accidents, a lower proportion with exactly one accident, and a far lower proportion with two or more accidents. We here introduce a cost-sensitive multi-class adaptive boosting (AdaBoost) algorithm we call SAMME.C2 to handle such class imbalances. We calibrate the algorithm using empirical data collected from a telematics program in Canada and demonstrate an improved assessment of driving behavior using telematics compared with traditional risk variables. Using suitable performance metrics, we show that our algorithm outperforms other learning models designed to handle class imbalances.

Evapotranspiration estimation using SEBAL algorithm integrated with remote sensing and experimental methods

ABSTRACT Evapotranspiration is one of the most important elements of the hydrological cycle. Estimation of evapotranspiration is imperative for effective forest, irrigation, rangeland and water resources management as well as to increase yields and for better crop management. This study aims to evaluate the effectiveness of the Surface Energy Balance Algorithm for Land (SEBAL) in estimating evapotranspiration and crop coefficient of corn in the Mediterranean region of Adana province, Turkey. The Landsat 8 satellite images from March to September 2018 were used to acquire the coefficients of the respective bands. Then, the net radiation flux on the earth’s surface and the earth’s heat flux is obtained using incoming-outgoing radiation fluxes from albedo, surface emissivity coefficients, land surface temperature, and plant indicators. Next, the sensible heat flux is calculated by determining the hot and cold pixels under consideration via the atmospheric stability conditions. Finally, evapotranspiration maps are plotted. The crop coefficient of corn is also estimated with the respected maps being plotted. To validate the outcomes from the SEBAL algorithm, experimental methods were employed to calculate the evapotranspiration values and evaluated using suitable performance metrics. The results showed that the SEBAL generated evapotranspiration values are in high agreement with the FAO Penman-Monteith method registering the highest correlation (R = 0.91) and the lowest error (RMSE = 1.14). In addition, the SEBAL method registered the highest correlation values of 0.89, 0.87 and 0.68 with Turk, Makkink and Hargreaves experimental methods, respectively. Moreover, the crop coefficients estimated using SEBAL also manifested an acceptable correlation with all methods. The highest correlation value registered was with the FAO Penman-Monteith method (R = 0.98). The outcomes show that since the performance of the SEBAL algorithm in estimating the actual evapotranspiration and crop coefficient using Landsat 8 satellite images is acceptable, the SEBAL algorithm could be a very convenient method. Moreover, it could easily be assimilated into farming management systems and precision agriculture for better decision-making and higher yield.

A review on multimodal online educational engagement detection system using facial expression, eye movement and speech recognition

During this lockdown period, an online educational engagement system plays a vital role to enrich the knowledge of learners in various fields without interrupting their learning process. Online educational engagement systems include all the activities of a learner like listening, reading, writing, and so on. While participating in these activities, a participant may show various levels of engagements like fully engaged, partially engaged and completely not engaged. The participation of online learners has to be identified for an effective learning process. The existing literature could be classified depending upon the learners’ participation as automatic, semi-automatic and manual. Further it could be sub categorised based on the data types used to identify the engagement system. In this paper, a review on computer based automatic online educational engagement detection systems is presented. Several educational engagement methods are applied for computer based online engagement detection systems. In these systems examining a participant’s presence and attention with the modalities of facial expression, eye movement and speech are found to be a challenging task. In this work, it is also identified that there are few challenges like preparation and usage of proper datasets, identifying suitable performance metrics for different tasks involved and providing recommendations for future enhancement of online educational engagement detection by combining the modalities of facial expression, eye movement and speech are still unattended. Though there are several research gaps involved, an online educational engagement system will help the learners to engage themselves in a productive way of learning and getting evaluated efficiently and effectively during the lockdown period of pandemic disease COVID-19 without interrupting their learning process and gaining knowledge. © 2021 Karadeniz Technical University. All rights reserved.

Phase I monitoring of individual normal data: Design and implementation

Control charts play an important role in Phase I or retrospective studies, which are essential to establish stability and generate reference data necessary for calculation of prospective (Phase II) control limits for process monitoring. The Shewhart-type Phase I chart for individual normally distributed data is one of the most popular monitoring methods. However, the control limits (charting constants) available in the literature do not account for either the effect of parameter estimation or use an appropriate chart performance metric. This can increase the rate of false alarms to an unacceptable level. In this paper, we consider the Shewhart-type Phase I chart for individual data for two common estimators of standard deviation and a suitable chart performance metric. We derive the formulas and tabulate the corrected (or adjusted) charting constants. Effects of violations of assumptions are examined in terms of in-control (IC) robustness to normality and presence of autocorrelation, via simulations. Some out-of-control (OOC) performance analysis is also presented. An illustration using some data is provided along with a summary and some recommendations. The accompanying R program may be used to calculate the corrected charting constants in practice on demand.

An implementation of a fault-tolerant database system using the actor model

Fault-tolerant systems are an important discussion subject in our world of interconnected devices. One of the major failure points of every distributed infrastructure is the database. A data migration or an overload of one of the servers could lead to a cascade of failures and service downtime for the users. NoSQL databases sacrifice some of the consistency provided by traditional SQL databases while privileging availability and partition tolerance. This paper presents the design and implementation of a distributed in-memory database that is based on the actor model. The benefits of the actor model and development using functional languages are detailed, and suitable performance metrics are presented. A case study is also performed, showcasing the system’s capacity to quickly recover from the loss of one of its machines and maintain functionality.

A low‐power delay stage ring VCO based on wrap‐gate CNTFET technology for X‐band satellite communication applications

SummaryOver the past few years, with lower power consumption, reasonable layout area, and the ease of integration with standard circuit design technologies compared to the other counterparts, delay stage ring voltage‐controlled oscillators (VCOs) have been in the limelight of microelectronics scientists. However, few efforts have focused on representing high‐performance delay stage ring VCOs in the deep nanometric regime. In this regard, by virtue of outstanding electrical properties of carbon nanotube wrap‐gate transistors, this work aims to propose a carbon nanotube field‐effect transistor (CNTFET)–based delay stage ring VCO. After performing rigorous simulations, the proposed ring VCO which has been designed by 10‐nm gate‐all‐around (GAA) CNTFET technology shows suitable electrical performance metrics. The simulation results demonstrate that the proposed GAA‐CNTFET‐based ring VCO consumes 85.176 μW at with a 6.12‐ to 10.42‐GHz frequency tuning range. At the worst‐case noise conditions, the proposed design presents ‐90.747 dBc/Hz phase noise at 1 MHz offset frequency. With occupying 1.414 μm2 physical area, the proposed VCO is appropriate for the ultracompact nanoscale radio frequency apparatus. Our simulation results accentuate that with further improvements and commercializing the fabrication techniques for CNTFET transistors, the proposed GAA‐CNTFET‐based VCO can be considered as a potential candidate for X‐band satellite communication applications.

Photovoltaic system monitoring for high latitude locations

Reliable monitoring of PV systems is essential to establish efficient maintenance routines that minimize the levelized cost of electricity. The existing solutions for affordable monitoring of commercial PV systems are however inadequate for climates where snow and highly varying weather result in unstable performance metrics. The aim of this work is to decrease this instability to enable more reliable monitoring solutions for PV systems installed in these climates.Different performance metrics have been tested on Norwegian installations with a total installed capacity of 3.3 MW: (i) comparison of specific yield, (ii) temperature corrected performance ratio, and (iii) power performance index based on both physical modelling and machine learning. The most influential effects leading to instability are identified as snow, low light, curtailment, and systematic irradiance differences over the system. The standard deviation of all the performance metrics is reduced when filters targeting these four effects are applied. Compared to general low irradiance or clear sky filtering, a greater reduction in the variation of the metrics is achieved, and more data remains in the useful dataset. The most suitable performance metrics are comparison of specific yield and performance index based on machine learning modelling.The analysis highlights two paths to accomplish increased reliability of PV monitoring systems without increased hardware costs. First, better reliability can be achieved by selecting a suitable performance metric. Second, the variability of the performance metric can be reduced by utilizing filters that specifically target the origin of the variability instead of using standard literature thresholds.

A Review on Lossless Compression Techniques for Wireless Capsule Endoscopic Data.

The videos produced during wireless capsule endoscopy have larger data size causing difficulty in transmission with limited bandwidth. The constraint on wireless capsule endoscopy hinders the performance of the compression module. The objectives of this paper are as follows: (i) to conduct an extensive review of the lossless compression techniques and (ii) to find out the limitations of the existing system and the possibilities for improvement. The literature review was conducted with a focus on the compression schemes satisfying minimum computational complexity, less power dissipation and low memory requirements for hardware implementation. A thorough study of various lossless compression techniques was conducted under two perspectives, i.e., techniques applied to Bayer CFA and RGB images. The detail of the various stages of wireless capsule endoscopy compression was investigated to have a better understanding. The suitable performance metrics for evaluating the compression techniques were listed from various literature studies. In addition to the Gastrolab database, WEO clinical endoscopy atlas and Gastrointestinal atlas were found to be better alternatives for experimentation. Pre-processing operations, especially new subsampling patterns need to be given more focus to exploit the redundancies in the images. Investigations showed that encoder module can be modified to bring more improvement towards compression. The real-time endoscopy still exists as a promising area for exploration. This review presents a research update on the details of wireless capsule endoscopy compression together with the findings as an eye-opener and guidance for further research.

Dental Age Estimation Based on X-ray images

Chronological age estimation using panoramic dental X-ray images is an essential task in forensic sciences. Various statistical approaches have proposed by considering the teeth and mandible. However, building automated dental age estimation based on machine learning techniques needs more research efforts. In this paper, an automated dental age estimation is proposed using transfer learning. In the proposed approach, features are extracted using two deep neural networks namely, AlexNet and ResNet. Several classifiers are proposed to perform the classification task including decision tree, k-nearest neighbor, linear discriminant, and support vector machine. The proposed approach is evaluated using a number of suitable performance metrics using a dataset that contains 1429 dental X-ray images. The obtained results show that the proposed approach has a promising performance.

A Regression Approach to Certain Information Transmission Problems

A general information transmission model, under independent and identically distributed Gaussian codebook and nearest neighbor decoding rule with processed channel output, is investigated using the performance metric of generalized mutual information. When the encoder and the decoder know the statistical channel model, it is found that the optimal channel output processing function is the conditional expectation operator, thus hinting a potential role of regression, a classical topic in machine learning, for this model. Without utilizing the statistical channel model, a problem formulation inspired by machine learning principles is established, with suitable performance metrics introduced. A data-driven inference algorithm is proposed to solve the problem, and the effectiveness of the algorithm is validated via numerical experiments. Extensions to more general information transmission models are also discussed.

Performance modeling and evaluation of transportation systems using analytical recursive decomposition algorithm for cyclone mitigation

Emergencies place demands on communication techniques that are frequently different from the demands of non-emergency circumstances. Message prioritization, computerization of correspondence, quick message conveyance and different facilities are regularly needed by every emergency circumstance. Lacking crisis correspondence capabilities can have results that are inconvenient and devastating at worst. Ensuring that such systems have inherent redundancies for geo-location message focusing on using multiple map providers like Google, OpenStreetMap, etc. will need to turn into the standard. Furthermore, utilizing standard push notification gateways, and taking into consideration different operating modes for the emergency alert system, contingent on information system accessibility, will help to guarantee the framework’s utility and effectiveness. The objective of this paper is to develop a risk assessment model in the case of natural calamities such as a cyclone and create an effective evacuation model for residents of the area by calculating the effective performance of the usual means of transportation using suitable performance metrics.