Proposed Evaluation Metrics Research Articles

Can we trust explainable artificial intelligence in wind power forecasting?

Advanced artificial intelligence (AI) models typically achieve high accuracy in wind power forecasting, but their internal mechanisms lack interpretability, which undermines user confidence in forecast value and strategy execution. To this end, this paper aims to investigate the interpretability of AI models, which is crucial but usually overlooked in wind power forecasting. Specifically, four model-agnostic explainable artificial intelligence (XAI) techniques (i.e., Shapley additive explanations, permutation feature importance, partial dependence plot, and local interpretable model-agnostic explanations) are tailored to provide global and instance interpretability for AI models in wind power forecasting. Then, several metrics are proposed to evaluate the trustworthiness of interpretations provided by XAI techniques. Simulation results demonstrate that the proposed XAI techniques can not only identify important features from wind power datasets, but also enable the understanding of the contribution of each feature to the forecast power output for a specific sample. Furthermore, the proposed evaluation metrics aid users in comprehensively assessing the trustworthiness of XAI techniques in wind power forecasting, enabling them to judiciously select suitable XAI techniques for their AI models.

Practical solutions in fully homomorphic encryption: a survey analyzing existing acceleration methods

Fully homomorphic encryption (FHE) has experienced significant development and continuous breakthroughs in theory, enabling its widespread application in various fields, like outsourcing computation and secure multi-party computing, in order to preserve privacy. Nonetheless, the application of FHE is constrained by its substantial computing overhead and storage cost. Researchers have proposed practical acceleration solutions to address these issues. This paper aims to provide a comprehensive survey for systematically comparing and analyzing the strengths and weaknesses of FHE acceleration schemes, which is currently lacking in the literature. The relevant researches conducted between 2019 and 2022 are investigated. We first provide a comprehensive summary of the latest research findings on accelerating FHE, aiming to offer valuable insights for researchers interested in FHE acceleration. Secondly, we classify existing acceleration schemes from algorithmic and hardware perspectives. We also propose evaluation metrics and conduct a detailed comparison of various methods. Finally, our study presents the future research directions of FHE acceleration, and also offers both guidance and support for practical application and theoretical research in this field.

A time series anomaly detection method based on series-parallel transformers with spatial and temporal association discrepancies

Multivariate time series anomaly detection methods can discover malfunctions in a complex system by detecting anomalies in the monitoring data. Multivariate time series have complex temporal and spatial correlations. Existing methods that explicitly model both correlations only extract semantic information sequentially in a single temporal-spatial or spatial-temporal order. However, different orders have a significant influence on the method's performance. In addition, a static graph structure trained on normal data has difficulties capturing the change of correlations between features when anomalies occur, limiting the method's performance. This paper proposes the Spatial-Temporal Anomaly Transformer (STAT), which extracts semantic information comprehensively by combining both spatial-temporal and temporal-spatial orders. STAT uses the Temporal Anomaly Transformer and the Spatial Anomaly Transformer (SAT) to explicitly compute temporal and spatial association discrepancies to conduct anomaly detection. Since the topological structure is usually unknown, SAT adopts a trainable hypersphere to learn the prior-association between features on normal data and computes the distance between the attention weights and the center of the hypersphere as the spatial association discrepancy. AUC and two recently proposed evaluation metrics are used to compare STAT with various typical methods on public data sets, proving that our method can achieve state-of-the-art performance.

Shape-Aware Monocular 3D Object Detection

The 3D object detection is the key issue in the autonomous driving system. This issue is particularly challenging when the detection only relies on a single perspective camera. The anchor-free and keypoint-based models receive increasing attention recently due to their effectiveness and simplicity. However, most of these methods are vulnerable to the occlusion and truncation of objects. In this paper, a single-stage monocular 3D object detection model is proposed. An instance-segmentation head is integrated into the model training, which allows the model to be aware of the visible shape of a target object. Therefore, the detection largely avoids interference from irrelevant regions surrounding the target objects. In addition, we also reveal that the popular IoU-based evaluation metrics, which were originally designed for evaluating stereo or LiDAR-based detection methods, are insensitive to the improvement achieved by the monocular 3D object detection algorithms. A novel evaluation metric, namely average depth similarity (ADS) is proposed for the monocular 3D object detection models. Our method outperforms the comparison baseline in terms of both the popular and the proposed evaluation metrics while maintaining real-time efficiency.

Improving Readability for Automatic Speech Recognition Transcription

Modern Automatic Speech Recognition (ASR) systems can achieve high performance in terms of recognition accuracy. However, a perfectly accurate transcript still can be challenging to read due to grammatical errors, disfluency, and other noises common in spoken communication. These readable issues introduced by speakers and ASR systems will impair the performance of downstream tasks and the understanding of human readers. In this work, we present a task called ASR post-processing for readability (APR) and formulate it as a sequence-to-sequence text generation problem. The APR task aims to transform the noisy ASR output into a readable text for humans and downstream tasks while maintaining the semantic meaning of speakers. We further study the APR task from the benchmark dataset, evaluation metrics, and baseline models: First, to address the lack of task-specific data, we propose a method to construct a dataset for the APR task by using the data collected for grammatical error correction. Second, we utilize metrics adapted or borrowed from similar tasks to evaluate model performance on the APR task. Lastly, we use several typical or adapted pre-trained models as the baseline models for the APR task. Furthermore, we fine-tune the baseline models on the constructed dataset and compare their performance with a traditional pipeline method in terms of proposed evaluation metrics. Experimental results show that all the fine-tuned baseline models perform better than the traditional pipeline method, and our adapted RoBERTa model outperforms the pipeline method by 4.95 and 6.63 BLEU points on two test sets, respectively. The human evaluation and case study further reveal the ability of the proposed model to improve the readability of ASR transcripts.

Multi-modal pedestrian detection with misalignment based on modal-wise regression and multi-modal IoU

Multi-modal pedestrian detection, which integrates visible and thermal sensors, has been developed to overcome many limitations of visible-modal pedestrian detection, such as poor illumination, cluttered background, and occlusion. By adopting the combination of multiple modalities, we can efficiently detect pedestrians even with poor visibility. Nevertheless, the critical assumption of multi-modal pedestrian detection is that multi-modal images are perfectly aligned. In general, however, this assumption often becomes invalid in real-world situations. Viewpoints of the different modal sensors are usually different. Then, the positions of pedestrians on the different modal images have disparities. We proposed a multi-modal faster-RCNN specifically designed to handle misalignment between two modalities. The faster-RCNN consists of a region proposal network (RPN) and a detector. We introduce position regressors for both modalities in the RPN and the detector. Intersection over union (IoU) is one of the useful metrics for object detection but is defined only for a single-modal image. We extend it into multi-modal IoU to evaluate the preciseness of both modalities. Our experimental results with the proposed evaluation metrics demonstrate that the proposed method has comparable performance with state-of-the-art methods and outperforms them for data with significant misalignment.

Systematic assessment of cyber-physical security of lane keeping control system for autonomous vehicles

In this article, a systematic assessment of cyber-physical security is developed for the lane-keeping control (LKC) system of autonomous vehicles (AVs), which has not been attempted before. A novel impact analysis methodology for typical cyber-attacks is proposed with new evaluation metrics from the viewpoint of safety and performance and innovative index-based resilience and security criteria. Specifically, a security criterion is investigated by tracking performance, comfort and vehicle stability to evaluate the safety and security of the vehicle LKC system. Hardware-in-the-Loop (HiL) experimental results show that the proposed evaluation metrics are effective in analyzing the impact of cyber-attacks on the commercial LKC system of Dongfeng Motor comprehensively. Finally, the conclusion provides guidelines for attack detection, diagnosis, and countermeasures for AVs.

The application of intelligent control technology for the evaluation of temperature segregation in asphalt mixture paving

The index of temperature segregation in asphalt mixture paving is one of the key indicators of quality control of asphalt pavement construction. Intelligent control techniques have also been recently used as a novel technology in controlling the temperature segregation of asphalt mixture for paving. Hence, the present study aims at developing an intelligent control system of asphalt mixture paving. Accordingly, to monitor and visualize the quality of the real-time paving, a temperature cloud was mapped for the asphalt mixture paving. Furthermore, a comprehensive evaluation method of paving temperature segregation was developed based on two indices, i.e., temperature segregation index and low-temperature area index. In addition, the accuracy of the developed system in controlling the paving temperature, thickness and segregation degree was practically confirmed. Moreover, the influence of paving temperature segregation on the asphalt pavement performance (compactness and smoothness) were practically explored. The obtained results corroborated the real-time effectiveness of the smart system in controlling the entire process of asphalt mixture paving. Also, the cloud map enabled the visual displaying of the asphalt pavement temperature segregation in real-time. Consequently, the quality of asphalt pavement construction was improved. Hence, the proposed evaluation metric was capable of accurately assess and predict the state and performance of the asphalt mixture paving temperature segregation.

A Novel 5G-NR Resources Partitioning Framework Through Real-Time User-Provider Traffic Demand Analysis

Network slicing (NS) is a key enabler of the 5G and beyond network architectures, allowing multiple dedicated logical networks with selected functionality to be executed on top of a common infrastructure. At the radio access network, novel flexibility paradigms and dynamic response to wireless channel variations are necessary for any NS solution. In this framework, this article proposes a novel real-time NS management framework for the 5G New-Radio, where the slice resource management is achieved through a joint dynamic evaluation of served users’ quality of service and tenants’ service level agreement. The design of the proposed framework within the standardized 5G architecture is discussed, together with the integration of the dynamic resource assignation procedure within the slice life cycle workflow. A novel mathematical model is proposed for the dynamic resource assignation within the slice life cycle workflow. The solution’s accuracy is tested by means of computer simulations, and found to be satisfactory according to the proposed evaluation metrics. Finally, the practicality of the optimal radio slicing configuration extracted from the proposed model is applied to our experimental platform, and the performance of a real case scenario is comprehensively evaluated.

Evaluating the Privacy Policy of Android Apps: A Privacy Policy Compliance Study for Popular Apps in China and Europe

Recently, with the increase in the market share of the Android system and the sharp increase in the number of Android mobile apps, many countries and regions have successively launched laws and regulations related to data security. The EU’s GDPR and China’s Information Security Technology-Personal Information Security Specification are two of the most important bills, affecting vast areas and large populations. Both regulations impose requirements on privacy policy specifications for Android apps. With these requirements, however, apps’ privacy policies have become larger. Researchers have conducted studies on whether the actual privacy behavior of apps conforms to their privacy policy description but have not focused on compliance with the privacy policy itself. In this paper, we propose evaluation metrics for privacy policy compliance and evaluate popular apps by analyzing privacy policies and apps. We applied our method to 1,000 apps from the Google Play Store in Europe and 1,000 apps from the Tencent Appstore in China. We detected a number of app privacy policy noncompliance issues and discovered a number of privacy issues with third-party services and third-party libraries.

Electromagnetic Interference Effects of Continuous Waves on Memristors: A Simulation Study.

As two-terminal passive fundamental circuit elements with memory characteristics, memristors are promising devices for applications such as neuromorphic systems, in-memory computing, and tunable RF/microwave circuits. The increasingly complex electromagnetic interference (EMI) environment threatens the reliability of memristor systems. However, various EMI signals’ effects on memristors are still unclear. This paper selects continuous waves (CWs) as EMI signals. It provides a deeper insight into the interference effect of CWs on the memristor driven by a sinusoidal excitation voltage, as well as a method for investigating the EMI effect of memristors. The optimal memristor model is obtained by the exhaustive traversing of the possible model parameters, and the interference effect of CWs on memristors is quantified based on this model and the proposed evaluation metrics. Simulation results indicate that CW interference may affect the switching time, dynamic range, nonlinearity, symmetry, time to the boundary, and variation of memristance. The specific interference effect depends on the operating mode of the memristor, the amplitude, and the frequency of the CW. This research provides a foundation for evaluating EMI effects and designing electromagnetic protection for memristive neuromorphic systems.

TRAVEL: Traversable Ground and Above-Ground Object Segmentation Using Graph Representation of 3D LiDAR Scans

Perception of traversable regions and objects of interest from a 3D point cloud is one of the critical tasks in autonomous navigation. A ground vehicle needs to look for traversable terrains that are explorable by wheels. Then, to make safe navigation decisions, the segmentation of objects positioned on those terrains has to be followed up. However, over-segmentation and under-segmentation can negatively influence such navigation decisions. To that end, we propose <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">TRAVEL</i> , which performs traversable ground detection and object clustering simultaneously using the graph representation of a 3D point cloud. To segment the traversable ground, a point cloud is encoded into a graph structure, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">tri-grid field</i> , which treats each tri-grid as a node. Then, the traversable regions are searched and redefined by examining local convexity and concavity of edges that connect nodes. On the other hand, our above-ground object segmentation employs a graph structure by representing a group of horizontally neighboring 3D points in a spherical-projection space as a node and vertical/horizontal relationship between nodes as an edge. Fully leveraging the node-edge structure, the above-ground segmentation ensures real-time operation and mitigates over-segmentation. Through experiments using simulations, urban scenes, and our own datasets, we have demonstrated that our proposed traversable ground segmentation algorithm outperforms other state-of-the-art methods in terms of the conventional metrics and that our newly proposed evaluation metrics are meaningful for assessing the above-ground segmentation. We make the code and our own dataset available to public at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/url-kaist/TRAVEL</uri> .

Unmasking the Mask – Evaluating Social Biases in Masked Language Models

Masked Language Models (MLMs) have shown superior performances in numerous downstream Natural Language Processing (NLP) tasks. Unfortunately, MLMs also demonstrate significantly worrying levels of social biases. We show that the previously proposed evaluation metrics for quantifying the social biases in MLMs are problematic due to the following reasons: (1) prediction accuracy of the masked tokens itself tend to be low in some MLMs, which leads to unreliable evaluation metrics, and (2) in most downstream NLP tasks, masks are not used; therefore prediction of the mask is not directly related to them, and (3) high-frequency words in the training data are masked more often, introducing noise due to this selection bias in the test cases. Therefore, we propose All Unmasked Likelihood (AUL), a bias evaluation measure that predicts all tokens in a test case given the MLM embedding of the unmasked input and AUL with Attention weights (AULA) to evaluate tokens based on their importance in a sentence. Our experimental results show that the proposed bias evaluation measures accurately detect different types of biases in MLMs, and unlike AUL and AULA, previously proposed measures for MLMs systematically overestimate the measured biases and are heavily influenced by the unmasked tokens in the context.

Deep-attack over the deep reinforcement learning

Recent adversarial attack developments have made reinforcement learning more vulnerable, and different approaches exist to deploy attacks against it, where the key is how to choose the right timing of the attack. Some work tries to design an attack evaluation function to select critical points that will be attacked if the value is greater than a certain threshold. This approach makes it difficult to find the right place to deploy an attack without considering the long-term impact. In addition, there is a lack of appropriate indicators of assessment during attacks. To make the attacks more intelligent as well as to remedy the existing problems, we propose the reinforcement learning-based attacking framework by considering the effectiveness and stealthy spontaneously, while we also propose a new metric to evaluate the performance of the attack model in these two aspects. Experimental results show the effectiveness of our proposed model and the goodness of our proposed evaluation metric. Furthermore, we validate the transferability of the model, and also its robustness under the adversarial training.

Developing metrics to assess the effectiveness of cybersecurity awareness program

Abstract Cybersecurity awareness (CSA) is not just about knowing, but also transforming things learned into practice. It is a continuous process that needs to be adjusted in subsequent iterations to improve its usability as well as sustainability. This is possible only if a CSA program is reviewed and evaluated timely. Review and evaluation of an awareness program offer an insight into the program's effectiveness on the audience and organization, an invaluable piece of information for the continuous improvement of the program. Further, it provides the information required by the management and sponsor to decide on whether to invest in the program or not. Despite these advantages, there does not exist a common understanding of what factors to measure and how to measure them during the evaluation process. As a result, we have proposed evaluation metrics for the purpose. In order to do so, we performed a literature review of 32 papers mainly to extract the following data: (i) what factors did the paper measure, and (ii) how did it measure the factors? Next, we adapted the European Literacy Policy Network's four indicators (i.e. impact, sustainability, accessibility, and monitoring) for awareness evaluation to make it appropriate for evaluating a CSA program. We believe that measuring all four indicators will contribute to making the evaluation process systematic, complete, and replicable. More importantly, it will help to produce more inclusive, accurate, and usable results for the future enhancement of the program.

A Survey of Deep Learning Architectures for Privacy-Preserving Machine Learning With Fully Homomorphic Encryption

Outsourced computation for neural networks allows users access to state-of-the-art models without investing in specialized hardware and know-how. The problem is that the users lose control over potentially privacy-sensitive data. With homomorphic encryption (HE), a third party can perform computation on encrypted data without revealing its content. In this paper, we reviewed scientific articles and publications in the particular area of Deep Learning Architectures for Privacy-Preserving Machine Learning (PPML) with Fully HE. We analyzed the changes to neural network models and architectures to make them compatible with HE and how these changes impact performance. Next, we find numerous challenges to HE-based privacy-preserving deep learning, such as computational overhead, usability, and limitations posed by the encryption schemes. Furthermore, we discuss potential solutions to the HE PPML challenges. Finally, we propose evaluation metrics that allow for a better and more meaningful comparison of PPML solutions.

A heuristic for two-echelon urban distribution systems

The negative impacts of urban freight transport have fostered the design of new distribution systems for inner city deliveries. One of the proposed city logistics solutions is the establishment of two-echelon distribution systems, where freight originating from the periphery of the city is transferred at intermediate locations (satellites) from larger vehicles (urban freighters) to smaller more environmentally friendly vehicles (city freighters), deemed more suited to operate in the city centre. In this paper, we address a variant of the two-echelon (capacitated) vehicle routing problem (2E-CVRP) arising in the context of city logistics encompassing characteristics such as a heterogeneous fleet of city freighters, time windows, vehicles synchronisation at satellites, direct deliveries, the possibility of freight transfers at customers, and multiple trips by city freighters. We propose a heuristic solution method for this problem based on a Variable Neighbourhood Search (VNS) heuristic, as well as a set of auxiliary evaluation metrics designed to support the decision making process. Preliminary results show the flexibility of the proposed heuristic to account for different problem characteristics, as well as the impacts of considering different best insertion criteria for the initial solution construction. Additionally, the proposed evaluation metrics have allowed for an improved assessment of the trade-offs between different city logistic solutions, and may be considered when developing decision support systems.

Optimized lighting energy consumption for non-visual effects: A case study in office spaces based on field test and simulation

Recently, the non-visual effects of the light environment have received increasing attention. Some standards have proposed evaluation metrics for non-visual effects in office spaces to ensure the health and well-being of occupants. Non-visual metrics are new requirements for indoor light environments, whose impacts on lighting adjustment and ultimate lighting energy performance are currently unknown. Therefore, this study aimed to present a workflow to obtain optimal light outputs considering non-visual lighting requirements, in addition to traditional visual requirements. This workflow was implemented to calculate the annual lighting energy consumption under different light correlated color temperatures (CCTs), daylight conditions, and standard requirements for an actual office scenario. As deduced from the results, adding non-visual requirements may significantly increase the lighting energy consumption by 57 %. Furthermore, daylight and light with a higher CCT are beneficial for non-visual energy-saving. The additional non-visual requirements will not increase the lighting energy consumption under favorable daylight conditions. The 5500 K light can save up to 20 % of the annual lighting power consumption than the 4000 K light. The presented workflow and conclusions contribute to engineering office environments with enhanced lighting, better circadian stimulus, and improved office lighting energy efficiency.

Experimental Evaluation of Individualized Treatment Rules

The increasing availability of individual-level data has led to numerous applications of individualized (or personalized) treatment rules (ITRs). Policy makers often wish to empirically evaluate ITRs and compare their relative performance before implementing them in a target population. We propose a new evaluation metric, the population average prescriptive effect (PAPE). The PAPE compares the performance of ITR with that of non-individualized treatment rule, which randomly treats the same proportion of units. Averaging the PAPE over a range of budget constraints yields our second evaluation metric, the area under the prescriptive effect curve (AUPEC). The AUPEC represents an overall performance measure for evaluation, like the area under the receiver and operating characteristic curve (AUROC) does for classification, and is a generalization of the QINI coefficient used in uplift modeling. We use Neyman’s repeated sampling framework to estimate the PAPE and AUPEC and derive their exact finite-sample variances based on random sampling of units and random assignment of treatment. We extend our methodology to a common setting, in which the same experimental data are used to both estimate and evaluate ITRs. In this case, our variance calculation incorporates the additional uncertainty due to random splits of data used for cross-validation. The proposed evaluation metrics can be estimated without requiring modeling assumptions, asymptotic approximation, or resampling methods. As a result, it is applicable to any ITR including those based on complex machine learning algorithms. The open-source software package is available for implementing the proposed methodology. Supplementary materials for this article are available online.

Virtual reality safety training using deep EEG-net and physiology data

Virtual reality (VR) safety training systems can enhance safety awareness while supporting health assessment in various work conditions. This paper proposes a novel VR system for construction safety training, which augments an individual’s functioning in VR via a brain–computer interface of electroencephalography (EEG) and physiology data such as blood pressure and heart rate. The use of VR aims to support high levels of interactions and immersion. Crucially, we apply novel clipping training algorithms to improve the performance of a deep EEG neural network, including batch normalization and ELU activation functions for real-time assessment. It significantly improves the system performance in time efficiency while maintaining high accuracy of over 80% on the testing datasets. For assessing workers’ competence under various construction environments, the risk assessment metrics are developed based on a statistical model and workers’ EEG data. One hundred and seventeen construction workers in Shanghai took part in the study. Nine of the participants’ EEG is identified with highly abnormal levels by the proposed evaluation metric. They have undergone further medical examinations, and among them, six are diagnosed with high-risk health conditions. It proves that our system plays a significant role in understanding workers’ physical condition, enhancing safety awareness, and reducing accidents.