Detection Performance Metrics Research Articles

Insights into Modern Intrusion Detection Strategies for Internet of Things Ecosystems

The swift explosion of Internet of Things (IoT) devices has brought about a new era of interconnectivity and ease of use while simultaneously presenting significant security concerns. Intrusion Detection Systems (IDS) play a critical role in the protection of IoT ecosystems against a wide range of cyber threats. Despite research advancements, challenges persist in improving IDS detection accuracy, reducing false positives (FPs), and identifying new types of attacks. This paper presents a comprehensive analysis of recent developments in IoT, shedding light on detection methodologies, threat types, performance metrics, datasets, challenges, and future directions. We systematically analyze the existing literature from 2016 to 2023, focusing on both machine learning (ML) and non-ML IDS strategies involving signature, anomaly, specification, and hybrid models to counteract IoT-specific threats. The findings include the deployment models from edge to cloud computing and evaluating IDS performance based on measures such as accuracy, FP rates, and computational costs, utilizing various IoT benchmark datasets. The study also explores methods to enhance IDS accuracy and efficiency, including feature engineering, optimization, and cutting-edge solutions such as cryptographic and blockchain technologies. Equally, it identifies key challenges such as the resource-constrained nature of IoT devices, scalability, and privacy issues and proposes future research directions to enhance IoT-based IDS and overall ecosystem security.

A transfer learning enabled approach for ocular disease detection and classification.

Ocular diseases pose significant challenges in timely diagnosis and effective treatment. Deep learning has emerged as a promising technique in medical image analysis, offering potential solutions for accurately detecting and classifying ocular diseases. In this research, we propose Ocular Net, a novel deep learning model for detecting and classifying ocular diseases, including Cataracts, Diabetic, Uveitis, and Glaucoma, using a large dataset of ocular images. The study utilized an image dataset comprising 6200 images of both eyes of patients. Specifically, 70% of these images (4000 images) were allocated for model training, while the remaining 30% (2200 images) were designated for testing purposes. The dataset contains images of five categories that include four diseases, and one normal category. The proposed model uses transfer learning, average pooling layers, Clipped Relu, Leaky Relu and various other layers to accurately detect the ocular diseases from images. Our approach involves training a novel Ocular Net model on diverse ocular images and evaluating its accuracy and performance metrics for disease detection. We also employ data augmentation techniques to improve model performance and mitigate overfitting. The proposed model is tested on different training and testing ratios with varied parameters. Additionally, we compare the performance of the Ocular Net with previous methods based on various evaluation parameters, assessing its potential for enhancing the accuracy and efficiency of ocular disease diagnosis. The results demonstrate that Ocular Net achieves 98.89% accuracy and 0.12% loss value in detecting and classifying ocular diseases by outperforming existing methods.

Polarization insensitive multiband metamaterial absorber for bio-plastic sensing application

This article introduces a microwave multiband metamaterial absorber (MA) with high potential for bio-plastic sensing applications. The innovative design is realized through a compact unit cell dimension of 1.33λο × 1.33λο × 0.22λο, employing a state-of-the-art split-ring resonator structure. Strikingly, this absorber demonstrates polarization insensitivity and can be characterized by five distinct absorption peaks at 18.69 GHz, 23.34 GHz, 25.72 GHz, 32.94 GHz, and 35.40 GHz. The corresponding absorption rates are exceptionally high, ranging from 92.82% to 99.95%. To elucidate the underlying absorption mechanism, detailed analyses of the electric field, magnetic field, surface current distribution, and input impedance are provided, showcasing the absorber's effective MA structure. The designed MA sensor exhibits outstanding performance metrics for bio-plastic detection, including an average quality factor (Q-factor) of 210 and sensitivity of 0.888 GHz/RIU (refractive index unit), marking a significant advancement in the field. Additionally, the sensor's linearity, output variation, and sensitivity are considerably improved for bio-plastic sensing. The research confirms the proposed MA absorber's functionality by comparing measured results and simulation outcomes, indicating a high degree of accuracy and reliability. The potential applications of this MA are manifold, extending to microwave absorption and enhanced plastic sensing techniques. Its compact and thin design facilitates seamless integration with various sensor equipment, broadening the scope of its utility. Furthermore, the article delves into the relevance of this research to several Sustainable Development Goals (SDGs), particularly highlighting its alignment with SDG 9 (Industry, Innovation, and Infrastructure), SDG 12 (Responsible Consumption and Production), and SDG 14 (Life Below Water). Thus, the innovative MA design signifies a technological leap in the field of metamaterial absorbers and contributes to the pursuit of sustainability and environmental protection.

A Novel IDS System based on Hedge Algebras to Detect DDOS Attack in IoT Systems

In recent years, we have experienced IoT solutions' rapid and beneficial development throughout all aspects of life. In addition to the apparent advantages, the increased number and variety of devices have resulted in more security issues. The DDOS attack, which originates from a broad range of sources and is a significant challenge for IoT systems, is one of the most prevalent but devastating attacks. IoT devices are typically simple and have few computing resources, which puts them at risk of being infected and attackers. IDS intrusion detection systems are considered superior protection against DDOS attacks. Therefore, the IDS system attracts many researchers and implements intelligent techniques such as machine learning and fuzzy logic to detect these DDOS attacks quickly and precisely. Along with the approach of intelligent computation, this study presents a novel technique for detecting DDOS attacks based on hedge algebra, which has never been implemented on IDS systems. We use the PSO swarm optimization algorithm to optimize the proposed model's parameters for optimized performance. Our experiment on the IoT-23 dataset shows that the proposed model's accuracy and performance metrics for DDOS attack detection are better than those proposed by other previous authors.

Enhancing Anomaly Detection Models for Industrial Applications through SVM-Based False Positive Classification

Unsupervised anomaly detection models are crucial for the efficiency of industrial applications. However, frequent false alarms hinder the widespread adoption of unsupervised anomaly detection, especially in fault detection tasks. To this end, our research delves into the dependence of false alarms on the baseline anomaly detector by analyzing the high-response regions in anomaly maps. We introduce an SVM-based false positive classifier as a post-processing module, which identifies false alarms from positive predictions at the object level. Moreover, we devise a sample synthesis strategy that generates synthetic false positives from the trained baseline detector while producing synthetic defect patch features from fuzzy domain knowledge. Following comprehensive evaluations, we showcase substantial performance enhancements in two advanced out-of-distribution anomaly detection models, Cflow and Fastflow, across image and pixel-level anomaly detection performance metrics. Substantive improvements are observed in two distinct industrial applications, with notable instances of elevating the image-level F1-score from 46.15% to 78.26% in optimal scenarios and boosting pixel-level AUROC from 72.36% to 94.74%.

Active RIS Assisted Spectrum Sharing: Able to Achieve Energy-Efficient Notable Detection Performance Gains

Efficient radio resource utilization is crucially important for future wireless communication systems. The advanced reconfigurable intelligent surface (RIS) has recently been incorporated into the spectrum sharing mechanism, allowing for both spectrum efficiency (SE) and energy efficiency (EE). Spectrum sensing is the cornerstone of spectrum sharing. Nevertheless, due to the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">doubling fading</i> effect, the path loss experienced from the reflection link is thousands of times that from the direct link. Consequently, the detection performance gains obtained through passive RIS are negligible, especially in typical communication scenarios where the direct link is not weak. To address this issue, this paper proposes an active RIS assisted spectrum sensing mechanism that can notably improve the detection performance gains. Multiple active RISs are introduced and the closed-form expressions of the detection performance metrics are derived in this paper. More importantly, due to the extra energy consumption in amplifying reflected signals, it's necessary to investigate the EE maximization problem for the active RIS assisted mechanism by jointly optimizing the detection parameter and the reflecting precoding. Alternating optimization (AO) algorithm and quadratic transform (QT) method are utilized to decouple the multidimensional <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">fractional programming</i> (FP) problem. Then, the optimal detection parameter is obtained with a closed-form solution based on the convex optimization theory. The phase shift and amplification coefficient of each active element are optimized through the sequential rotation (SR) method and an improved differential evolutionary (DE) algorithm, respectively. Simulation results show that the proposed active-RIS mechanism could outperform the conventional passive-RIS mechanism in achieving notably enhanced detection performance gains and higher EE by optimizing the reflecting precoding.

Marine Oil Spill Detection from Low-Quality SAR Remote Sensing Images

Oil spills pose a significant threat to the marine ecological environment. The intelligent interpretation of synthetic aperture radar (SAR) remote sensing images serves as a crucial approach to marine oil spill detection, offering the potential for real-time, continuous, and accurate monitoring. This study makes valuable contributions to the field of marine oil spill detection based on low-quality SAR images, focusing on the following key aspects: (1) We thoroughly analyze the Deep SAR Oil Spill dataset, known as the SOS dataset, a prominent resource in the domain of marine oil spill detection from low-quality SAR images, and rectify identified issues to ensure its reliability. (2) By identifying and rectifying errors in the original literature that presented the SOS dataset, and reproducing the experiments to provide accurate results, benchmark performance metrics for marine oil spill detection with low-quality SAR remote sensing images are established. (3) We propose three progressive deep learning-based marine oil spill detection methods (a direct detection method based on Transformer and UNet, a detection method based on FFDNet and TransUNet with denoising before detection, and a detection method based on integrated multi-model learning) and the performance advantages of the proposed methods are verified by comparing them with semantic segmentation models such as UNet, SegNet, and DeepLabV3+. (4) We introduce a feasible, highly robust and easily scalable system architecture approach that effectively addresses practical engineering applications. This paper is an important addition to the research on marine oil spill detection from low-quality SAR images, and the proposed experimental method and performance details can provide a reference for related research.

Gastric polyp detection module based on improved attentional feature fusion

Gastric cancer is a deadly disease and gastric polyps are at high risk of becoming cancerous. Therefore, the timely detection of gastric polyp is of great importance which can reduce the incidence of gastric cancer effectively. At present, the object detection method based on deep learning is widely used in medical images. However, as the contrast between the background and the polyps is not strong in gastroscopic image, it is difficult to distinguish various sizes of polyps from the background. In this paper, to improve the detection performance metrics of endoscopic gastric polyps, we propose an improved attentional feature fusion module. First, in order to enhance the contrast between the background and the polyps, we propose an attention module that enables the network to make full use of the target location information, it can suppress the interference of the background information and highlight the effective features. Therefore, on the basis of accurate positioning, it can focus on detecting whether the current location is the gastric polyp or background. Then, it is combined with our feature fusion module to form a new attentional feature fusion model that can mitigate the effects caused by semantic differences in the processing of feature fusion, using multi-scale fusion information to obtain more accurate attention weights and improve the detection performance of polyps of different sizes. In this work, we conduct experiments on our own dataset of gastric polyps. Experimental results show that the proposed attentional feature fusion module is better than the common feature fusion module and can improve the situation where polyps are missed or misdetected.

Investigation on jamming detection in WSN using optimal decision rule

A jamming attack is a special case of a Denial of Service (DoS) attack that completely blocks the data transmission in Wireless Sensor Networks (WSNs). When sensor nodes are distributed in the field, numerous attacks, such as collision, black hole, selective forwarding, jamming, etc., caused by the presence of malicious nodes have the potential to cause network damage. Jamming is a highly risky attack that completely blocks data transmission within the wireless network. The existing technique for detecting jamming attacks are based on predetermined hopping-sequence, cryptographic, or random frequency hopping techniques. However, these mechanisms are more complex and frequently have energy constraints and high overhead. A novel jamming detection method based on a statistical approach that provides high network performance measures is proposed. It is a technique that uses energy-based clustering with a Received Signal Strength Indicator (RSSI). The selection of thresholds used for the detection of jamming is analyzed. The proposed approach employs three detection performance metrics for investigating the jamming attack, namely, Packet to Delivery Ratio (PDR), ENERGY, and RSSI. The jamming node is identified using the Optimal Decision Rule (ODR), which is determined by the hypothesis rule. If the hypothesis is not satisfied, then jamming exists; otherwise, there is no jamming. The novel technique is implemented using a Network Simulator, and various performance metrics such as PDR, Energy consumption, Network throughput, Routing overhead, network, and node lifetime are evaluated to conclude that the statistical approach outperforms the timestamp and IEWMA approaches.

High-Dynamic-Range Tone Mapping in Intelligent Automotive Systems

Intelligent driver assistance systems are becoming increasingly popular in modern passenger vehicles. A crucial component of intelligent vehicles is the ability to detect vulnerable road users (VRUs) for an early and safe response. However, standard imaging sensors perform poorly in conditions of strong illumination contrast, such as approaching a tunnel or at night, due to their dynamic range limitations. In this paper, we focus on the use of high-dynamic-range (HDR) imaging sensors in vehicle perception systems and the subsequent need for tone mapping of the acquired data into a standard 8-bit representation. To our knowledge, no previous studies have evaluated the impact of tone mapping on object detection performance. We investigate the potential for optimizing HDR tone mapping to achieve a natural image appearance while facilitating object detection of state-of-the-art detectors designed for standard dynamic range (SDR) images. Our proposed approach relies on a lightweight convolutional neural network (CNN) that tone maps HDR video frames into a standard 8-bit representation. We introduce a novel training approach called detection-informed tone mapping (DI-TM) and evaluate its performance with respect to its effectiveness and robustness in various scene conditions, as well as its performance relative to an existing state-of-the-art tone mapping method. The results show that the proposed DI-TM method achieves the best results in terms of detection performance metrics in challenging dynamic range conditions, while both methods perform well in typical, non-challenging conditions. In challenging conditions, our method improves the detection F2 score by 13%. Compared to SDR images, the increase in F2 score is 49%.

Template‐Assisted Fabrication of Flexible Perovskite Scintillators for X‐Ray Detection and Imaging

AbstractMetal halide perovskites have recently emerged as exceptional scintillator materials for ionizing radiation detection devices. Their chemical composition consists of elements with high atomic numbers, leading them to have a high attenuation coefficient. Their high attenuation coefficient, in combination with their excellent optoelectronic properties, versatile chemical tunability, and facile and low‐cost fabrication processes, makes them the ideal scintillator material. However, existing perovskite‐based scintillators suffer from poor material stability, especially in humid atmospheres. Moreover, current perovskite films have morphologies that have been optimized for photovoltaics, which results in the relatively long charge carrier lifetimes, a property that is detrimental for fast scintillation. Furthermore, existing reports of perovskite‐based scintillators have shown limited spatial resolution due to poor light transmittance that arises from light scattering from large aggregates of perovskite grains. To address these issues, this work introduces a template‐assisted in situ polymerization‐based process to prepare perovskite/polymer composite scintillators that simultaneously reduces afterglow effects, improves perovskite stability, and is industrially scalable. The optimized perovskite scintillators are then incorporated into X‐ray detection devices to investigate detection performance and device stability. Compared with existing commercial scintillators, the perovskite scintillators show superior detection performance metrics for imaging, indicating the great potential of perovskites for next‐generation, large‐area, and flexible scintillation screens.

Advanced Face Mask Detection Model Using Hybrid Dilation Convolution Based Method

A face-mask object detection model incorporating hybrid dilation convolutional network termed ResNet Hybrid-dilation-convolution Face-mask-detector (RHF) is proposed in this paper. Furthermore, a lightweight face-mask dataset named Light Masked Face Dataset (LMFD) and a medium-sized face-mask dataset named Masked Face Dataset (MFD) with data augmentation methods applied is also constructed in this paper. The hybrid dilation convolutional network is able to expand the perception of the convolutional kernel without concern about the discontinuity of image information during the convolution process. For the given two datasets being constructed above, the trained models are significantly optimized in terms of detection performance, training time, and other related metrics. By using the MFD dataset of 55,905 images, the RHF model requires roughly 10 hours less training time compared to ResNet50 with better detection results with mAP of 93.45%.

Spatial Information-Theoretic Optimal LPI Radar Waveform Design.

In this paper, the design of low probability of intercept (LPI) radar waveforms considers not only the performance of passive interception systems (PISs), but also radar detection and resolution performance. Waveform design is an important considerations for the LPI ability of radar. Since information theory has a powerful performance-bound description ability from the perspective of information flow, LPI waveforms are designed in this paper within the constraints of the detection performance metrics of radar and PISs, both of which are measured by the Kullback-Leibler divergence, and the resolution performance metric, which is measured by joint entropy. The designed optimization model of LPI waveforms can be solved using the sequential quadratic programming (SQP) method. Simulation results verify that the designed LPI waveforms not only have satisfactory target-detecting and resolution performance, but also have a superior low interception performance against PISs.

A new approach for malaria diagnosis in thick blood smear images

This paper presents a new approach for detecting malaria parasites in full images of thick blood smear using pixel classifiers for obtaining foreground objects and delimiting parasite-stained objects. For both processes, the HSV components were used as input variables of the following pixel classifiers: multilayer perceptron and a decision tree. The obtained patches were classified using a deep neural network with 34 layers, trained from scratch. The image dataset used was divided into sets with different parasite sizes. This enables characterizing performance metrics (accuracy, sensitivity, specificity, precision, and F1-score) for parasite detection with varying parasite sizes. The best metric values were obtained in images with large parasite sizes. For image sets 1 and 2, with large parasite sizes, precision rates of 91.71% and 93.14% were obtained. For image sets 3 and 4, with small parasite sizes, precision rates of 76.58% and 71.58% were obtained. As shown by the literature review, these results are comparable to others previously published. Nevertheless, a rigorous comparison could not be done, as different works use different datasets.

Asymmetric Identification Model for Human-Robot Contacts via Supervised Learning

Human-robot interaction (HRI) occupies an essential role in the flourishing market for intelligent robots for a wide range of asymmetric personal and entertainment applications, ranging from assisting older people and the severely disabled to the entertainment robots at amusement parks. Improving the way humans and machines interact can help democratize robotics. With machine and deep learning techniques, robots will more easily adapt to new tasks, conditions, and environments. In this paper, we develop, implement, and evaluate the performance of the machine-learning-based HRI model in a collaborative environment. Specifically, we examine five supervised machine learning models viz. the ensemble of bagging trees (EBT) model, the k-nearest neighbor (kNN) model, the logistic regression kernel (LRK), the fine decision trees (FDT), and the subspace discriminator (SDC). The proposed models have been evaluated on an ample and modern contact detection dataset (CDD 2021). CDD 2021 is gathered from a real-world robot arm, Franka Emika Panda, when it was executing repetitive asymmetric movements. Typical performance assessment factors are applied to assess the model effectiveness in terms of detection accuracy, sensitivity, specificity, speed, and error ratios. Our experiential evaluation shows that the ensemble technique provides higher performance with a lower error ratio compared with other developed supervised models. Therefore, this paper proposes an ensemble-based bagged trees (EBT) detection model for classifying physical human–robot contact into three asymmetric types of contacts, including noncontact, incidental, and intentional. Our experimental results exhibit outstanding contact detection performance metrics scoring 97.1%, 96.9%, and 97.1% for detection accuracy, precision, and sensitivity, respectively. Besides, a low prediction overhead has been observed for the contact detection model, requiring a 102 µS to provide the correct detection state. Hence, the developed scheme can be efficiently adopted through the application requiring physical human–robot contact to give fast accurate detection to the contacts between the human arm and the robot arm.

On the performance metrics for cyber-physical attack detection in smart grid

Supervisory Control and Data Acquisition (SCADA) systems play an important role in Smart Grid. Though the rapid evolution provides numerous advantages it is one of the most desired targets for malicious attackers. So far security measures deployed for SCADA systems detect cyber-attacks, however, the performance metrics are not up to the mark. In this paper, we have deployed an intrusion detection system to detect cyber-physical attacks in the SCADA system concatenating the Convolutional Neural Network and Gated Recurrent Unit as a collective approach. Extensive experiments are conducted using a benchmark dataset to validate the performance of the proposed intrusion detection model in a smart metering environment. Parameters such as accuracy, precision, and false-positive rate are compared with existing deep learning models. The proposed concatenated approach attains 98.84% detection accuracy which is much better than existing techniques.

A review of natural scene text detection methods

Natural scene text detection has an important role to play in getting textual information from natural scenes. With the continuous development of deep learning, natural scene text detection methods are emerging and achieving better results on detection tasks. In this paper, analysis, and summary of the current stage of deep learning-based text algorithms for natural scenes, can be divided into two types: region of the proposal and semantic segmentation, and the content of these two series of related algorithms is described. Secondly, a publicly available dataset and detection performance metrics for scene text detection are presented. Ultimately, the research in scene text detection is summarized and looked forward to in the hope of providing new research directions for subsequent algorithms.

Optimal Parallel Sequential Change Detection Under Generalized Performance Measures

This paper considers the detection of change points in parallel data streams, a problem widely encountered when analyzing large-scale real-time streaming data. Each stream may have its own change point, at which its data has a distributional change. With sequentially observed data, a decision maker needs to declare whether changes have already occurred to the streams at each time point.Once a stream is declared to have changed, it is deactivated permanently so that its future data will no longer be collected. This is a compound decision problem in the sense that the decision maker may want to optimize certain compound performance metrics that concern all the streams as a whole. Thus, the decisions are not independent for different streams. Our contribution is three-fold. First, we propose a general framework for compound performance metrics that includes the ones considered in the existing works as special cases and introduces new ones that connect closely with the performance metrics for single-stream sequential change detection and large-scale hypothesis testing. Second, data-driven decision procedures are developed under this framework. Finally, optimality results are established for the proposed decision procedures. The proposed methods and theory are evaluated by simulation studies and a case study.

Performance metrics for marine mammal signal detection and classification.

Automatic algorithms for the detection and classification of sound are essential to the analysis of acoustic datasets with long duration. Metrics are needed to assess the performance characteristics of these algorithms. Four metrics for performance evaluation are discussed here: receiver-operating-characteristic (ROC) curves, detection-error-trade-off (DET) curves, precision-recall (PR) curves, and cost curves. These metrics were applied to the generalized power law detector for blue whale D calls [Helble, Ierley, D'Spain, Roch, and Hildebrand (2012). J. Acoust. Soc. Am. 131(4), 2682-2699] and the click-clustering neural-net algorithm for Cuvier's beaked whale echolocation click detection [Frasier, Roch, Soldevilla, Wiggins, Garrison, and Hildebrand (2017). PLoS Comp. Biol. 13(12), e1005823] using data prepared for the 2015 Detection, Classification, Localization and Density Estimation Workshop. Detection class imbalance, particularly the situation of rare occurrence, is common for long-term passive acoustic monitoring datasets and is a factor in the performance of ROC and DET curves with regard to the impact of false positive detections. PR curves overcome this shortcoming when calculated for individual detections and do not rely on the reporting of true negatives. Cost curves provide additional insight on the effective operating range for the detector based on the a priori probability of occurrence. Use of more than a single metric is helpful in understanding the performance of a detection algorithm.

Hybrid feature selection approach to identify optimal features of profile metadata to detect social bots in Twitter

The last few years have revealed that social bots in social networks have become more sophisticated in design as they adapt their features to avoid detection systems. The deceptive nature of bots to mimic human users is due to the advancement of artificial intelligence and chatbots, where these bots learn and adjust very quickly. Therefore, finding the optimal features needed to detect them is an area for further investigation. In this paper, we propose a hybrid feature selection (FS) method to evaluate profile metadata features to find these optimal features, which are evaluated using random forest, naïve Bayes, support vector machines, and neural networks. We found that the cross-validation attribute evaluation performance was the best when compared to other FS methods. Our results show that the random forest classifier with six optimal features achieved the best score of 94.3% for the area under the curve. The results maintained overall 89% accuracy, 83.8% precision, and 83.3% recall for the bot class. We found that using four features: favorites_count, verified, statuses_count, and average_tweets_per_day, achieves good performance metrics for bot detection (84.1% precision, 81.2% recall).