Performance Evaluation Of Detection Research Articles

Drone Detection Performance Evaluation via Real Experiments with Additional Synthetic Darkness

Detecting drones is increasingly challenging, particularly when developing passive and low-cost defense systems capable of countering malicious attacks in environments with high levels of darkness and severe weather conditions. This research addresses the problem of drone detection under varying darkness levels by conducting an extensive study using deep learning models. Specifically, the study evaluates the performance of three advanced models: Yolov8, Vision Transformers (ViT), and Long Short-Term Memory (LSTM) networks. The primary focus is on how these models perform under synthetic darkness conditions, ranging from 20% to 80%, using a composite dataset (CONNECT-M) that simulates nighttime scenarios. The methodology involves applying transfer learning to enhance the base models, creating Yolov8-T, ViT-T, and LSTM-T variants. These models are then tested across multiple datasets with varying darkness levels. The results reveal that all models experience a decline in performance as darkness increases, as measured by Precision-Recall and ROC Curves. However, the transfer learning-enhanced models consistently outperform their original counterparts. Notably, Yolov8-T demonstrates the most robust performance, maintaining higher accuracy across all darkness levels. Despite the general decline in performance with increasing darkness, each model achieves an accuracy above 0.6 for data subjected to 60% or greater darkness. The findings highlight the challenges of drone detection under low-light conditions and emphasize the effectiveness of transfer learning in improving model resilience. The research suggests further exploration into multi-modal systems that combine audio and optical methods to enhance detection capabilities in diverse environmental settings.

Deep learning-based detection of primary bone tumors around the knee joint on radiographs: a multicenter study.

Most primary bone tumors are often found in the bone around the knee joint. However, the detection of primary bone tumors on radiographs can be challenging for the inexperienced or junior radiologist. This study aimed to develop a deep learning (DL) model for the detection of primary bone tumors around the knee joint on radiographs. From four tertiary referral centers, we recruited 687 patients diagnosed with bone tumors (including osteosarcoma, chondrosarcoma, giant cell tumor of bone, bone cyst, enchondroma, fibrous dysplasia, etc.; 417 males, 270 females; mean age 22.8±13.2 years) by postoperative pathology or clinical imaging/follow-up, and 1,988 participants with normal bone radiographs (1,152 males, 836 females; mean age 27.9±12.2 years). The dataset was split into a training set for model development, an internal independent and an external test set for model validation. The trained model located bone tumor lesions and then detected tumor patients. Receiver operating characteristic curves and Cohen's kappa coefficient were used for evaluating detection performance. We compared the model's detection performance with that of two junior radiologists in the internal test set using permutation tests. The DL model correctly localized 94.5% and 92.9% bone tumors on radiographs in the internal and external test set, respectively. An accuracy of 0.964/0.920, and an area under the receiver operating characteristic curve (AUC) of 0.981/0.990 in DL detection of bone tumor patients were for the internal and external test set, respectively. Cohen's kappa coefficient of the model in the internal test set was significantly higher than that of the two junior radiologists with 4 and 3 years of experience in musculoskeletal radiology (Model vs. Reader A, 0.927 vs. 0.777, P<0.001; Model vs. Reader B, 0.927 vs. 0.841, P=0.033). The DL model achieved good performance in detecting primary bone tumors around the knee joint. This model had better performance than those of junior radiologists, indicating the potential for the detection of bone tumors on radiographs.

YOLO-based object detection performance evaluation for automatic target aimbot in first-person shooter games

First-person shooter (FPS) focuses on first-person perspective action gameplay, with gunfights usually giving the player a choice of weapons, significantly impacting how the player approaches or strategies. General military-themed FPS games have realistic models with actual weapons’ shapes and characteristics. This type of game requires high aiming accuracy while using a mouse on a PC. However, not all players have a fast response time in knowing the surrounding situation. New players may need aid when targeting enemies in the FPS world. One popular yet underhanded method is injecting a program code using a dynamic-link library (DLL) to manipulate memory and asset data from the game. Instead of DLL, we promote a novel approach using the player’s real-time game screen, detecting the person without injecting program code into the game. The you only look once (YOLO) algorithm is used as an object detector model since it can process images in real time for up to 45 frames per second. The proposed object detection has an outstanding performance with 65% accuracy, 98% precision, and 61% recall of 51 tests for each game. YOLO’s fastest detection speed produces an average of 35 FPS on the YOLO tiny variant using a mixed precision (half) graphics processing unit (GPU).

Performance evaluation of lightweight network-based bot detection using mouse movements

Confronted with the challenge of malicious bot abuse disrupting daily life, bot detection has become a crucial field of research. Among the various methods proposed, a behavior-driven framework analyzing mouse movements stands out for its widespread applicability, adaptability, and non-intrusive nature. Traditionally, this area of research has relied on manually extracted features from mouse dynamics and classical machine learning techniques, which often struggle to identify complex behavior patterns. To address these limitations, our research turns to Convolutional Neural Networks (CNNs), which have proven effective in image processing, to explore their application in bot detection using mouse movement data. We employ various visual representation techniques to convert mouse movements into trajectory images suitable for CNN analysis. Our empirical findings show that among all the lightweight networks tested, EfficientNet_b3 not only achieves the highest accuracy, with an impressive precision of 99.82%, but also surpasses more complex models like ResNet50 in terms of detection accuracy and modeling speed. These results not only enhance the practical application of deep learning in bot detection but also offer valuable insights into more effective strategies for advancing bot detection technologies.

Closeby Habitable Exoplanet Survey (CHES). I. Astrometric Noise and Planetary Detection Efficiency Due to Stellar Spots and Faculae

The Closeby Habitable Exoplanet Survey (CHES) is dedicated to the astrometric exploration for habitable-zone Earth-like planets orbiting solar-type stars in close proximity, achieving unprecedented microarcsecond precision. Given the elevated precision, meticulous consideration of photocenter jitters induced by stellar activity becomes imperative. This study endeavors to model the stellar activity of solar-type stars, compute astrometric noise, and delineate the detection limits of habitable planets within the astrometric domain. Simulations were conducted for identified primary targets of CHES, involving the generation of simulated observed data for astrometry and photometry, accounting for the impact of stellar activity. Estimation of activity levels in our sample was achieved through chromospheric activity indices, revealing that over 90% of the stars exhibited photocenter jitters below 1 μas. Notably, certain proximate stars, such as α Cen A and B, displayed more discernible noise arising from stellar activity. Subsequent tests were performed to evaluate detection performance, unveiling that stellar activity tends to have a less pronounced impact on planetary detectability for the majority of the stars. Approximately 95% of the targets demonstrated a detection efficiency exceeding 80%. However, for several cold stars, e.g., HD 32450 and HD 21531, with the habitable zones close to the stars, a reduction in detection efficiency was observed. These findings offer invaluable insights into the intricate interplay between stellar activity and astrometric precision, significantly advancing our understanding in the search for habitable planets.

Evaluation and analysis of feature point detection methods based on vSLAM systems

Feature point detection is identified as a significant issue in the field of computer vision. Quantitative conclusions have been established regarding the performance evaluation of feature point detection using manually annotated datasets. However, these datasets, obtained through affine transformations with preset parameters on images, exhibit limitations in terms of variety, quantity, and challenges, differing from real-world application scenarios. In actual scenes, Vision Simultaneous Localization and Mapping (vSLAM) systems are extensively applied, and the precision of their localization and mapping is directly linked to the performance of feature points. To profoundly understand the performance of feature point detection in practical applications, vSLAM systems are chosen for evaluation in this study. More diverse and challenging datasets are utilized, encompassing real datasets covering variations in lighting, rotation, occlusion, and camera angle changes, as well as synthetic datasets reflecting complex conditions such as time, season, motion patterns, and environmental textures. Based on the evaluation results, the applicability of various feature point detection methods in different environments is thoroughly discussed, and the underlying principles are analyzed (Table 13). The conclusions drawn from this research provide references for the development of new feature point detection methods, the selection of such methods in vSLAM systems, and other related studies in the field of computer vision.

QRS Detector Performance Evaluation Aware of Temporal Accuracy and Presence of Noise.

Algorithms for QRS detection are fundamental in the ECG interpretive processing chain. They must meet several challenges, such as high reliability, high temporal accuracy, high immunity to noise, and low computational complexity. Unfortunately, the accuracy expressed by missed or redundant events statistics is often the only parameter used to evaluate the detector's performance. In this paper, we first notice that statistics of true positive detections rely on researchers' arbitrary selection of time tolerance between QRS detector output and the database reference. Next, we propose a multidimensional algorithm evaluation method and present its use on four example QRS detectors. The dimensions are (a) influence of detection temporal tolerance, tested for values between 8.33 and 164 ms; (b) noise immunity, tested with an ECG signal with an added muscular noise pattern and signal-to-noise ratio to the effect of "no added noise", 15, 7, 3 dB; and (c) influence of QRS morphology, tested on the six most frequently represented morphology types in the MIT-BIH Arrhythmia Database. The multidimensional evaluation, as proposed in this paper, allows an in-depth comparison of QRS detection algorithms removing the limitations of existing one-dimensional methods. The method enables the assessment of the QRS detection algorithms according to the medical device application area and corresponding requirements of temporal accuracy, immunity to noise, and QRS morphology types. The analysis shows also that, for some algorithms, adding muscular noise to the ECG signal improves algorithm accuracy results.

Performance evaluation of individual tree detection and segmentation algorithms using ALS data in Chir Pine (Pinus roxburghii) forest

The application of individual tree detection algorithms for assessing forest inventories and aiding decision-making in forestry has been a subject of research for more than two decades. Nevertheless, there is a notable research gap in the development of robust algorithms capable of automatically detecting trees of different species, ages, and varied crown sizes in dense forest environments. In this study, we conducted a comprehensive evaluation of six different individual tree detection (ITD) algorithms using airborne LiDAR data in Chir Pine (Pinus roxburghii) forests. This research represents one of the pioneering efforts in applying ITD algorithms to Chir Pine forests using ALS data. We categorized ITD routines into two groups: those reliant on local maxima as treetops and initial seeds for crown delineation, and those that do not require explicit treetop identification. To assess accuracy, we developed a special Individual Tree Matching (ITM) algorithm, enabling the matching of LiDAR-detected trees with 284 reference trees measured in the field. Our analysis involved various combinations of filtering fixed window sizes and adaptive window sizes applied to the point cloud, unsmoothed, and smoothed canopy height model (CHM). Our results highlighted the effectiveness of the 3 × 3m fixed window size method on unsmoothed CHM, achieving an overall F-score of 0.65 and a tree detection rate of 86%. Additionally, the Dalponte 2016 method proved superior for crown segmentation using identified treetops, consistently measuring mean crown radii within 0.5 m of reference field trees. Among the methods not relying on treetops, the adaptive mean shift algorithm (AMS3D) delivered strong performance, boasting an overall F-score of 0.67 and mean crown radii within 0.1 m. Our study revealed a high correlation between LiDAR-detected tree heights and field-measured tree heights across all evaluated methods. Overall, our findings underscore the potential of ITD algorithms in enhancing forest attribute measurement accuracy and facilitating climate-responsive forest management strategies.

MPG-LSD: A high-quality line segment detector based on multi-scale perceptual grouping

Existing methods that rely on a single Gaussian scale to detect line segments could yield poor line continuity and inferior orientation and position accuracy due to insufficient suppression of quantization noise inherent in digital images. To address this fundamental issue, a novel multi-scale perceptual grouping-based line segment detector (MPG-LSD) is proposed in this paper. Our multi-scale perceptual grouping is developed to identify and aggregate collinear pixels that have similar gradient orientations for producing line segment candidates, not only over the input image but also across a set of Gaussian scales. Multi-scale line segment refinement and validation are then further developed and implemented to produce the final detection result, which delivers high quality in terms of line continuity, orientation and position accuracy. To enrich the evaluation of line segment detection performance, a new dataset consisting of high-resolution and natural noise-corrupted images with line segment annotations is constructed. Extensive experimental results show that our proposed MPG-LSD can outperform the current state-of-the-arts by a large margin.

GNSS Radio Frequency Interference Monitoring from LEO Satellites: An In-Laboratory Prototype.

The disruptive effect of radio frequency interference (RFI) on global navigation satellite system (GNSS) signals is well known, and in the last four decades, many have been investigated as countermeasures. Recently, low-Earth orbit (LEO) satellites have been looked at as a good opportunity for GNSS RFI monitoring, and the last five years have seen the proliferation of many commercial and academic initiatives. In this context, this paper proposes a new spaceborne system to detect, classify, and localize terrestrial GNSS RFI signals, particularly jamming and spoofing, for civil use. This paper presents the implementation of the RFI detection software module to be hosted on a nanosatellite. The whole development work is described, including the selection of both the target platform and the algorithms, the implementation, the detection performance evaluation, and the computational load analysis. Two are the implemented RFI detectors: the chi-square goodness-of-fit (GoF) algorithm for non-GNSS-like interference, e.g., chirp jamming, and the snapshot acquisition for GNSS-like interference, e.g., spoofing. Preliminary testing results in the presence of jamming and spoofing signals reveal promising detection capability in terms of sensitivity and highlight room to optimize the computational load, particularly for the snapshot-acquisition-based RFI detector.

Studying Drowsiness Detection Performance While Driving Through Scalable Machine Learning Models Using Electroencephalography

Driver drowsiness is a significant concern and one of the leading causes of traffic accidents. Advances in cognitive neuroscience and computer science have enabled the detection of drivers’ drowsiness using Brain-Computer Interfaces (BCIs) and Machine Learning (ML). However, the literature lacks a comprehensive evaluation of drowsiness detection performance using a heterogeneous set of ML algorithms, being also necessary to study the performance of scalable ML models suitable for groups of subjects. To address these limitations, this work presents an intelligent framework employing BCIs and features based on electroencephalography for detecting drowsiness in driving scenarios. The SEED-VIG dataset is used to evaluate the best-performing models for individual subjects and groups. Results show that Random Forest (RF) outperformed other models used in the literature, such as Support Vector Machine (SVM), with a 78% f1-score for individual models. Regarding scalable models, RF reached a 79% f1-score, demonstrating the effectiveness of these approaches. This publication highlights the relevance of exploring a diverse set of ML algorithms and scalable approaches suitable for groups of subjects to improve drowsiness detection systems and ultimately reduce the number of accidents caused by driver fatigue. The lessons learned from this study show that not only SVM but also other models not sufficiently explored in the literature are relevant for drowsiness detection. Additionally, scalable approaches are effective in detecting drowsiness, even when new subjects are evaluated. Thus, the proposed framework presents a novel approach for detecting drowsiness in driving scenarios using BCIs and ML.

CHART: a novel system for detector evaluation against toxic chemical aerosols

Concern over the possibility of deliberate dispersion of chemical warfare agents and highly toxic pharmaceutical based agents as persistent aerosols has raised the need for experimental assessment of current and future defensive capabilities of armed forces and law enforcement agencies. Therefor we herewith present the design, realization and validation of the Chemical Hot Aerosol Research Tool (CHART) as a validated and safe experimental set-up for performance evaluation of chemical detection and identification equipment against chemical warfare agents and other highly toxic compounds. In the CHART liquid and solid compounds in solution or suspension are being dispersed as aerosols in a nebulization chamber. A broad dynamic particle size range can be generated, including particles known to be able to reach the lower respiratory tract. The aerosol generated is presented to the detection system-under-test while being monitored and characterized in real-time, using an optical particle counter and a time-of-flight aerosol analyzer, respectively. Additionally, the chemical composition of the aerosol is ex situ measured by analytical chemical methods. Evidently, in the design of the CHART significant emphasis was placed on laboratory safety and containment of toxic chemicals. The CHART presented in this paper has proven to be an indispensable experimental tool to study detectors and fieldable identification equipment against toxic chemical aerosols.

Ultra-Low Power Analog Folded Neural Network for Cardiovascular Health Monitoring.

Wearable sensors are increasingly used for continuous health monitoring, but their small size limits battery capacity, affecting user experience and monitoring capabilities. To overcome this, we introduce an ultra-low power analog Folded Neural Network (FNN) for physiological signal processing in a batteryless fashion. Our proposed FNN, by serializing computation, provides several benefits over traditional analog implementations, such as lower space, lower power consumption, and lower peak-to-average power ratio. We evaluate our method extensively using a dataset designed for ECG-based screening and diagnosis. Our analysis considers factors such as thermal noise, spatial requirements, and power consumption. Additionally, we evaluate detection performance, investigating various parameters of the proposed FNN. This evaluation provides insights into the optimal configuration for accurate anomaly detection. We observe a good trade-off for accuracy around 6 layers and a hidden size of 30 and further demonstrate that such architecture could be implemented in a wearable device and executed in a batteryless fashion.

RFI detection with spiking neural networks

Abstract Detecting and mitigating radio frequency interference (RFI) is critical for enabling and maximising the scientific output of radio telescopes. The emergence of machine learning (ML) methods capable of handling large datasets has led to their application in radio astronomy, particularly in RFI detection. Spiking neural networks (SNNs), inspired by biological systems, are well suited for processing spatio-temporal data. This study introduces the first exploratory application of SNNs to an astronomical data processing task, specifically RFI detection. We adapt the nearest latent neighbours (NLNs) algorithm and auto-encoder architecture proposed by previous authors to SNN execution by direct ANN2SNN conversion, enabling simplified downstream RFI detection by sampling the naturally varying latent space from the internal spiking neurons. Our subsequent evaluation aims to determine whether SNNs are viable for future RFI detection schemes. We evaluate detection performance with the simulated HERA telescope and hand-labelled LOFAR observation dataset the original authors provided. We additionally evaluate detection performance with a new MeerKAT-inspired simulation dataset that provides a technical challenge for machine-learnt RFI detection methods. This dataset focuses on satellite-based RFI, an increasingly important class of RFI and is an additional contribution. Our SNN approach remains competitive with the original NLN algorithm and AOFlagger in AUROC, AUPRC, and F1-scores for the HERA dataset but exhibits difficulty in the LOFAR and Tabascal datasets. However, our method maintains this accuracy while completely removing the compute and memory-intense latent sampling step found in NLN. This work demonstrates the viability of SNNs as a promising avenue for ML-based RFI detection in radio telescopes by establishing a minimal performance baseline on traditional and nascent satellite-based RFI sources and is the first work to our knowledge to apply SNNs in astronomy.

Anomaly Detection in Binary Time Series Data: An unsupervised Machine Learning Approach for Condition Monitoring

A key element of smart manufacturing is condition monitoring and heath controlling of production machines. In today's rapidly evolving landscape of industrial machinery and equipment, optimizing the operation of production lines is critical to ensure high productivity and product quality. Timely detection and prevention of faults in the production process plays a crucial role in minimizing downtime, reducing costs, and ensuring optimal performance. The scientific challenge here is that the increasing number of sensors and actuators with digital input and output signals in the production machines creates different patterns, which are difficult to evaluate using conventional statistical methods. Another difficulty is identifying the cause of the failure to be able to intervene rapidly and in a focused manner in the event of irregularities. For this reason, this research study presented a comprehensive analysis of anomaly detection in binary time series data using various machine learning models. The study included preprocessing of the dataset, normalizing the data, and evaluation of the anomaly detection performance of the different models. The accuracy, detection rate, and F1-score are used as evaluation measures. The execution time of each model is also analyzed. In addition, the identification of sensors that cause anomalies is investigated and the impact of false detections is discussed. Experimental results show the strengths and weaknesses of each model and provide valuable insights for selecting the appropriate anomaly detection approach. The Isolation Forest, Local Outlier Factor, DBSCAN, and kMeans models show high precision and detection, while the Autoencoder and Variational Autoencoder models show high precision but lower detection. The one-class Support Vector Machine model achieves balanced performance. AutoML shows excellent results in recognition rate but is not real-time capable. The results highlight the trade-offs between performance and computational efficiency and point to further research potential in real-time implementation.

Evaluation of maneuvering target detection performance based on measured data

Evaluation of maneuvering target detection performance based on measured data

Performance Evaluation of Retinal OCT Fluid Segmentation, Detection, and Generalization Over Variations of Data Sources

Performance Evaluation of Retinal OCT Fluid Segmentation, Detection, and Generalization Over Variations of Data Sources

Performance Evaluation of HBV-DNA Detection by ABI7500 Polmerase Chain Reactor and Analysis of Anti-interference Ability

Performance Evaluation of HBV-DNA Detection by ABI7500 Polmerase Chain Reactor and Analysis of Anti-interference Ability

Performance Evaluation of YOLOv8-Based Bib Number Detection in Media Streaming Race

Performance Evaluation of YOLOv8-Based Bib Number Detection in Media Streaming Race

Low-complexity detector performance evaluation for cell-free massive MIMO systems

Low-complexity detector performance evaluation for cell-free massive MIMO systems