Benchmarking Framework Research Articles

UNIQUE: A Framework for Uncertainty Quantification Benchmarking.

Machine learning (ML) models have become key in decision-making for many disciplines, including drug discovery and medicinal chemistry. ML models are generally evaluated prior to their usage in high-stakes decisions, such as compound synthesis or experimental testing. However, no ML model is robust or predictive in all real-world scenarios. Therefore, uncertainty quantification (UQ) in ML predictions has gained importance in recent years. Many investigations have focused on developing methodologies that provide accurate uncertainty estimates for ML-based predictions. Unfortunately, there is no UQ strategy that consistently provides robust estimates about model's applicability on new samples. Depending on the dataset, prediction task, and algorithm, accurate uncertainty estimations might be unfeasible to obtain. Moreover, the optimum UQ metric also varies across applications, and previous investigations have shown a lack of consistency across benchmarks. Herein, the UNIQUE (UNcertaInty QUantification bEnchmarking) framework is introduced to facilitate a comparison of UQ strategies in ML-based predictions. This Python library unifies the benchmarking of multiple UQ metrics, including the calculation of nonstandard UQ metrics (combining information from the dataset and model), and provides a comprehensive evaluation. In this framework, UQ metrics are evaluated for different application scenarios, e.g., eliminating the predictions with the lowest confidence or obtaining a reliable uncertainty estimate for an acquisition function. Taken together, this library will help to standardize UQ investigations and evaluate new methodologies.

Innovative Benchmarking of Offshore Wind O&M Strategies via AIS Data Analysis

Abstract By analyzing AIS data, this study provides independent insights into the diverse logistic concepts deployed by wind farm operators. The benchmarking framework developed herein facilitates the evaluation of logistic efficiency, resilience, and sustainability, thereby contributing to the identification of best practices in offshore wind operations and maintenance (O&M) logistics. The methodology encompasses data collection, preprocessing, and analysis phases. By providing a transparent and data-driven mechanism for benchmarking offshore wind O&M logistics, this study aids in the identification of inefficiencies and the formulation of strategies to enhance logistic operations. The results indicate significant variations in logistic efficiency across different offshore wind farms (OWF), suggesting that the adoption of best practices could lead to substantial improvements.

BG-LoA: A Benchmarking Framework for BIM/GIS Data Integration Based on Meta-Modelling Theory

Abstract. There are various studies on the integration of building information modelling (BIM) and geographic information system (GIS) that convert BIM data into GIS data. These studies used different technologies, data standards, data models, and data formats such as RDF (Resource Description Framework), graph, CityGML, and Shapefile to address the data incompatibility problem between BIM and GIS. This makes it difficult to compare these studies for the purpose of assessing research innovation, and sometimes, improper comparisons were carried out between studies. To solve this problem, this study adopted the metamodeling theory used in model-driven software engineering (MDSE) and extended its 4-layer framework of level of abstraction. The new benchmarking framework, referred to as level of abstraction for BIM/GIS data integration (BG-LoA), has 5 levels, including meta metamodel (M3), metamodel (M2), model (M1), reality (data) (M0), and reality (real world) (M00). The common types of IFC (Industry Foundation Classes) conversion were analysed using the new framework, including IFC-to-CityGML, IFC-to-CityJSON, IFC-to-Shapefile, IFC-to-GML, IFC-to-3DTiles, IFC-to-RDF, IFC-to-IfcGraph, and IFC-to-UML. The result shows that the new framework can show the difference between those conversion types, which makes it easier for future studies to benchmark their work and assess level of contribution to the body of knowledge.

A Performance Benchmark for the PostgreSQL and MySQL Databases

This study highlights the necessity for efficient database management in continuous authentication systems, which rely on large-scale behavioral biometric data such as keystroke patterns. A benchmarking framework was developed to evaluate the PostgreSQL and MySQL databases, minimizing repetitive coding through configurable functions and variables. The methodology involved experiments assessing select and insert queries under primary and complex conditions, simulating real-world scenarios. Our quantified results show PostgreSQL’s superior performance in select operations. In primary tests, PostgreSQL’s execution time for 1 million records ranged from 0.6 ms to 0.8 ms, while MySQL’s ranged from 9 ms to 12 ms, indicating that PostgreSQL is about 13 times faster. For select queries with a where clause, PostgreSQL required 0.09 ms to 0.13 ms compared to MySQL’s 0.9 ms to 1 ms, making it roughly 9 times more efficient. Insert operations were similar, with PostgreSQL at 0.0007 ms to 0.0014 ms and MySQL at 0.0010 ms to 0.0030 ms. In complex experiments with simultaneous operations, PostgreSQL maintained stable performance (0.7 ms to 0.9 ms for select queries during inserts), while MySQL’s performance degraded significantly (7 ms to 13 ms). These findings underscore PostgreSQL’s suitability for environments requiring low data latency and robust concurrent processing capabilities, making it ideal for continuous authentication systems.

Geometric neural network based on phase space for BCI-EEG decoding.

\textbf{Objective:} 
The integration of Deep Learning (DL) algorithms on brain signal analysis is still in its nascent stages compared to their success in fields like Computer Vision. This is particularly true for BCI, where the brain activity is decoded to control external devices without requiring muscle control.
Electroencephalography (EEG) is a widely adopted choice for designing BCI systems due to its non-invasive and cost-effective nature and excellent temporal resolution. Still, it comes at the expense of limited training data, poor signal-to-noise, and a large variability across and within-subject recordings. 
Finally, setting up a BCI system with many electrodes takes a long time, hindering the widespread adoption of reliable DL architectures in BCIs outside research laboratories. To improve adoption, we need to improve user comfort using, for instance, reliable algorithms that operate with few electrodes. 
\textbf{Approach:} Our research aims to develop a DL algorithm that delivers effective results with a limited number of electrodes. Taking advantage of the Augmented Covariance Method and the framework of SPDNet, we propose the \method{} architecture and analyze its performance and the interpretability of the results. The evaluation is conducted on 5-fold cross-validation, using only three electrodes positioned above the Motor Cortex. The methodology was tested on nearly 100 subjects from several open-source datasets using the Mother Of All BCI Benchmark (MOABB) framework. 
\textbf{Main results:} The results of our \method{} demonstrate that the augmented approach combined with the SPDNet significantly outperforms all the current state-of-the-art DL architecture in MI decoding. 
\textbf{Significance:} This new architecture is explainable and with a low number of trainable parameters.

An Open Source Hydroacoustic Benchmarking Framework for Geophonic Signal Detection

Passive hydroacoustic studies have underscored the efficiency and relevance of deploying autonomous hydrophones for the surveillance of underwater geophony. In particular, monitoring networks have been deployed for detecting SOFAR-propagating hydroacoustic waves generated by seismic events and locating their sources. The technique has been extended to study other hydroacoustic signals, such as P-waves from teleseismic events or impulsive waves generated by sea water-lava interactions. A significant challenge in this endeavor lies in the time required for the manual detection and annotation of these signals in long-term records. To address this issue, we tested the feasibility of implementing automated algorithms based on machine learning to detect and identify these various signals, and obtained satisfying classification and time picking accuracies. We incorporated those models in a benchmarking framework, proposing a training dataset, two evaluation datasets, two tasks to solve and the evaluations of the mentionned models on them. The goal of this framework is to foster the development of new models in the community, as it gives a clear way to evaluate them.

Selecting CMIP6 global climate models (GCMs) for Coordinated Regional Climate Downscaling Experiment (CORDEX) dynamical downscaling over Southeast Asia using a standardised benchmarking framework

Abstract. Downscaling global climate models (GCMs) provides crucial high-resolution data needed for informed decision-making at regional scales. However, there is no uniform approach to select the most suitable GCMs. Over Southeast Asia (SEA), observations are sparse and have large uncertainties, complicating GCM selection especially for rainfall. To guide this selection, we apply a standardised benchmarking framework to select CMIP6 GCMs for dynamical downscaling over SEA, addressing current observational limitations. This framework identifies fit-for-purpose models through a two-step process: (a) selecting models that meet minimum performance requirements in simulating the fundamental characteristics of rainfall (e.g. bias, spatial pattern, annual cycle and trend) and (b) selecting models from (a) to further assess whether key precipitation drivers (monsoon) and teleconnections from modes of variability are captured, i.e. the El Niño–Southern Oscillation (ENSO) and Indian Ocean Dipole (IOD). GCMs generally exhibit wet biases, particularly over the complex terrain of the Maritime Continent. Evaluations from the first step identify 19 out of 32 GCMs that meet our minimum performance expectations in simulating rainfall. These models also consistently capture atmospheric circulations and teleconnections with modes of variability over the region but overestimate their strength. Ultimately, we identify eight GCMs meeting our performance expectations. There are obvious, high-performing GCMs from allied modelling groups, highlighting the dependency of the subset of models identified from the framework. Therefore, further tests of model independence, data availability and future climate change spread are conducted, resulting in a final subset of two independent models that align with our a priori expectations for downscaling over the Coordinated Regional Climate Downscaling Experiment –Southeast Asia (CORDEX-SEA).

Benchmark experimental study on cavitating flow around Clark-Y 11.7 % hydrofoil at various angles of attack under controlled levels of dissolved air

The present research aims to study cavitating flow around a CLARK-Y 11.7 % hydrofoil with variable angles of attack (α) while maintaining controlled levels of dissolved air in the operating fluid, which is water. A series of experiments were conducted using a water tunnel facility, where the cavitation characteristics were measured and observed using sensors and high-speed imaging techniques. The variables studied in the present work are cavitation number (1 ≤ σ ≤ 2.2) and angle of attack (α = 4°, 6°, 8°, 10°, and 12°), with dissolved air levels (DAL) in the range of 9.3 ppm to 13.1 ppm. The dimensionless cavity length decreases significantly with increasing values of σ/α, indicating an inverse relationship where higher cavitation numbers or lower angles of attack result in shorter cavities. The cavity length follows a power-law scaling relationship, with the empirical equation Lmax/C=4.78×σ/α−0.76.Increasing the angle of attack transitions the cavitation nature from stable (Mode I) to dynamic (Mode II) and highly oscillating (Mode III). Larger cavities result in lower Strouhal numbers, which indicates reduced vortex shedding activity. The relationship between the Strouhal number and the normalized cavitation number σ/α is characterized by the power-law equation St=0.041×σ/α0.3. The pressure coefficient at the leading-edge increases with the angle of attack at low cavitation numbers, while higher cavitation numbers lead to greater pressure coefficient differences between the leading and trailing edges.The present study offers an extensive dataset and empirical correlations that may serve as a benchmark framework, which facilitates the validation of computational and experimental models of cavitating flow under similar conditions.

A Global Resilience Analysis-Based Benchmark Framework for Comparing Reliability Surrogate Measures of Water Distribution Systems

Various reliability surrogate measures have emerged over the last three decades to design water distribution systems. However, existing comparative studies cannot assess surrogate measures from the resilience perspective considering the dynamic absorption–recovery process imposed by pipe failures. In this work, we propose a novel benchmark framework based on the global resilience analysis to examine surrogate measures’ performance. Surrogate measures were compared via the stress–strain curve derived from the global resilience analysis under extended period simulation. In particular, we identify the comparable stress range to articulate the differences among surrogate measures and significantly reduce the computational burden. Then, we develop the normalized resilience score (NRS) to evaluate the quality of solutions to network design. Five well-known measures are compared for the multiobjective design of two benchmark networks. Results show that the Network Resilience Index achieves 2.5% to 10.1% better NRSs than the mean NRSs over five surrogate measures, implying that both nodal surplus energy and pipe diameter uniformity greatly impact the network system’s resilience. The uniformity of pipe diameters is more significant than the uniformity of flow rate. Our findings contribute to the design of new and better surrogate measures for network resilience evaluation.

A benchmark for Rey-Osterrieth complex figure test automatic scoring

The Rey–Osterrieth complex figure (ROCF) test is a neuropsychological task that can be useful for early detection of cognitive decline in the elderly population. Several computer vision systems have been proposed to automate this complex analysis task, but the lack of public benchmarks does not allow a fair comparison of these systems. To advance in that direction, we present a benchmarking framework for the automatic scoring of the ROCF test that provides: the ROCFD528 dataset, which is the first open dataset of ROCF line drawings; and experimental results obtained by several modern deep learning models, which can be used as a baseline for comparing new proposals. We evaluate different state-of-the-art convolutional neural networks (CNNs) under traditional and transfer learning paradigms. Experimental quantitative results (MAE = 3.448) indicate that a CNN specifically designed for sketches outperforms other state of the art CNN architectures when the number of examples available is limited. This benchmark can also be a paradigmatic example within the broad field of machine learning for the development of efficient and robust models for analyzing line drawings and sketches not only in classification but also in regression tasks.

Spatiotemporal Differentiation of Carbon Budget and Carbon Compensation Zoning Based on the Plan for Major Function-oriented Zones：A Case Study of Counties in the Yellow River Basin

Exploring the spatiotemporal differentiation of the carbon budget and clarifying the zoning of carbon compensation based on the perspective of the main function-oriented zones is of great significance for promoting regional low-carbon development and achieving the "dual carbon" goal. This study was conducted using 550 counties in the Yellow River Basin as the basic unit, and based on the concentration index to examine the spatial distribution characteristics of carbon emissions and carbon absorptions in the main function-oriented zones of the Yellow River Basin, a four-dimensional benchmark framework for carbon compensation zoning was constructed by total scale, economic contribution, ecological carrying capacity, and land development intensity. The SOM-K-means algorithm was used to establish a differentiated carbon compensation zoning scheme. The results showed that： ① The carbon emissions and carbon absorptions and their growth trends of the main function-oriented zones in the Yellow River Basin were in line with the positioning of the main functional area, and the significant space-time distribution characteristics of the concentration level of carbon emissions and carbon absorptions were shown. ② Significant regional differences exist in the four attributes of total carbon emissions scale, economic contribution, ecological carrying capacity, and land development intensity. The overall scale of carbon emissions showed a growth trend, with obvious comparative advantages in the midstream and downstream. The overall comparative advantage of economic contribution increased first and then decreased, with the midstream and downstream being the attribute advantage areas of economic contribution. The overall ecological carrying capacity had been improved, and the upstream had obvious advantages in ecological carrying properties. The advantageous areas for land development were mainly concentrated in the midstream and downstream； in particular, the comparative advantage index in the midstream was showing a rapid upward trend. ③ Moreover, there were 287 compensated areas, 78 balanced areas, and 185 payment areas in the Yellow River Basin. Eleven types of carbon compensation zones were finally formed by combining the comparative advantage index of four types of attributes with the plan for main function-oriented zones, and corresponding low-carbon development strategies were proposed for each type of zone.

Towards standarized benchmarks of LLMs in software modeling tasks: a conceptual framework

AbstractThe integration of Large Language Models (LLMs) in software modeling tasks presents both opportunities and challenges. This Expert Voice addresses a significant gap in the evaluation of these models, advocating for the need for standardized benchmarking frameworks. Recognizing the potential variability in prompt strategies, LLM outputs, and solution space, we propose a conceptual framework to assess their quality in software model generation. This framework aims to pave the way for standardization of the benchmarking process, ensuring consistent and objective evaluation of LLMs in software modeling. Our conceptual framework is illustrated using UML class diagrams as a running example.

Warm, moderate, or cool-liker? A benchmarking framework to characterize occupant overall thermal preferences based on large-scale thermostat data

Humans could exhibit distinct overall thermal preferences when exposed to identical indoor thermal environments, leading to preference groups such as "warm-likers" or "cool-likers", who consistently prefer warmer or cooler conditions than the average population, respectively. Currently, most thermal comfort modelling studies focus on capturing the instantaneous comfort states/preferences of occupants, while overlooking their overall thermal preferences. This paper proposes a benchmarking framework to identify and characterize overall thermal preferences based on the relationship between preferred setpoints and outdoor air temperatures, derived from the ECOBEE Donate Your Data program. Using descriptive statistics, we establish three temporally consistent overall preference groups, including warm-liker, moderate and cool-liker, along with a temporally chaotic preference group termed random. Our results demonstrate that warm-likers prefer temperature setpoints above 21.5 °C on heating days and 24–25 °C on cooling days, while cool-likers prefer setpoints below 19.6 °C on heating days and 22 °C on cooling days, indicating a significant impact of the seasonal transitions. We observed that around 50 % of users exhibit secondary overall preferences, implying that overall thermal preference could change over time. On average, overall thermal preference can be established in 10–16 setpoint adjustments. The study reveals varied responses to outdoor temperature changes among users: many maintain constant indoor temperature preferences, while a significant number adjust their indoor temperatures upwards by 0.1 °C–0.4 °C for each 1 °C rise in outdoor temperature. A smaller group prefers cooler indoor temperatures as it gets warmer outside, showing a unique negative adjustment trend of −0.1. We also found that climate interacts with the overall preference group, with warmer climates having more warm-likers and vice versa.

Diagnostic benchmarking of many-objective evolutionary algorithms for real-world problems

Despite progress in multiobjective evolutionary algorithms (MOEAs) research, their efficacy in real-world scenarios remains unclear. This article introduces a diagnostic benchmarking framework to evaluate MOEAs, comprising (1) flexible MOEA construction software, (2) performance evaluation metrics and (3) real-world applications for benchmarking, reflecting diverse mathematical challenges. Utilizing this framework, NSGA-II, NSGA-III, RVEA, MOEA/D and Borg MOEA were evaluated across four applications with three to ten objectives. Collectively, the four applications capture challenges such as stochastic objectives, severe constraints, nonlinearity and complex Pareto frontiers. The study demonstrates how MOEAs that have shown strong performance on standard test problems can struggle on real-world applications. The benchmarking framework and results have value for enhancing the design and use of MOEAs in real-world applications. Further, the results highlight the need to improve the adaptability and ease-of-use of MOEAs given the often ill-defined nature of real-world problem-solving.

MedFrenchmark, a Small Set for Benchmarking Generative LLMs in Medical French.

Generative Large Language Models (LLMs) have become ubiquitous in various fields, including healthcare and medicine. Consequently, there is growing interest in leveraging LLMs for medical applications, leading to the emergence of novel models daily. However, evaluation and benchmarking frameworks for LLMs are scarce, particularly those tailored for medical French. To address this gap, we introduce a minimal benchmark consisting of 114 open questions designed to assess the medical capabilities of LLMs in French. The proposed benchmark encompasses a wide range of medical domains, reflecting real-world clinical scenarios' complexity. A preliminary validation involved testing seven widely used LLMs with a parameter size of 7 billion. Results revealed significant variability in performance, emphasizing the importance of rigorous evaluation before deploying LLMs in medical settings. In conclusion, we present a novel and valuable resource for rapidly evaluating LLMs in medical French. By promoting greater accountability and standardization, this benchmark has the potential to enhance trustworthiness and utility in harnessing LLMs for medical applications.

SuperSIM: a comprehensive benchmarking framework for neural networks using superconductor Josephson devices

Abstract This paper introduces SuperSIM, a benchmarking framework tailored for neural networks using superconducting Josephson devices, specifically focusing on Adiabatic Quantum Flux Parametron (AQFP) based Processing-in-Memory (PIM) architectures. Our framework offers in-depth architecture-level simulations and performance assessments to enhance AQFP PIM chip development. It supports single and multi-bit PIM designs, various AQFP memory cell types, and diverse clocking methods. Additionally, it integrates circuit-level models for precise energy, delay, and area measurements, ensuring accurate performance evaluation. The framework includes application, device, and architectural layers for versatile configurations and cycle-accurate energy, latency, and area simulations. Experiments validate our framework, with case studies on algorithm and architecture-level features, examining data precision, crossbar size, operating frequency and clocking scheme impacts on computational accuracy, energy use, overall latency and hardware cost.

Optimization Applications as Quantum Performance Benchmarks

Combinatorial optimization is anticipated to be one of the primary use cases for quantum computation in the coming years. The Quantum Approximate Optimization Algorithm and Quantum Annealing can potentially demonstrate significant run-time performance benefits over current state-of-the-art solutions. Inspired by existing methods to characterize classical optimization algorithms, we analyze the solution quality obtained by solving Max-cut problems using gate-model quantum devices and a quantum annealing device. This is used to guide the development of an advanced benchmarking framework for quantum computers designed to evaluate the trade-off between run-time execution performance and the solution quality for iterative hybrid quantum-classical applications. The framework generates performance profiles through compelling visualizations that show performance progression as a function of time for various problem sizes and illustrates algorithm limitations uncovered by the benchmarking approach. As an illustration, we explore the factors that influence quantum computing system throughput, using results obtained through execution on various quantum simulators and quantum hardware systems.

A benchmarking framework for energy management systems with commercial hardware models

Energy Management Systems (EMS) for buildings are pivotal in leveraging flexibility from sector coupling in future power systems. Currently, most EMS are designed and evaluated using non-standardized and incompatible simulation models built within the specific EMS development cycle. Such evaluation techniques make it difficult to compare different EMS solutions according to well-defined and universal performance indicators. Therefore, open-access benchmark models of realistic building energy systems would be beneficial for wider research community. This article introduces the ProHMo benchmarking framework which provides experimentally validated commercial heating and cooling equipment models in various building energy system configurations. The building energy system models are available as openly accessible Functional Mock-Up Units (FMU) to allow for toolchain-independent benchmarking of EMS. The framework includes a Python code template that enables easy integration with different EMS interfaces. In a case study, we show the potential of the benchmarking framework by comparing a rule-based, optimization-based, and reinforcement learning-based EMS. The results show that the optimization-based EMS with perfect foresight achieves the lowest costs. Although the reinforcement learning-based EMS performs slightly poorer, it operates independently of forecasts, which makes it attractive for practical applications. The ProHMo benchmarking framework is designed to equip researchers with a robust framework for developing, evaluating, and comparing different EMS, particularly those focused on optimization and data-driven control methods.

Sensitivity of Australian Rainfall to Driving SST Data Sets in a Variable‐Resolution Global Atmospheric Model

AbstractIn this study, we employ the Conformal Cubic Atmospheric Model (CCAM), a variable‐resolution global atmospheric model, driven by two distinct sea surface temperature (SST) data sets: the 0.25° Optimum Interpolation Sea Surface Temperature (CCAM_OISST) version 2.1 and the 2° Extended Reconstruction SSTs Version 5 (CCAM_ERSST5). Model performance is assessed using a benchmarking framework, revealing good agreement between both simulations and the climatological rainfall spatial pattern, seasonality, and annual trends obtained from the Australian Gridded Climate Data (AGCD). Notably, wet biases are identified in both simulations, with CCAM_OISST displaying a more pronounced bias. Furthermore, we have examined CCAM’s ability to capture El Niño‐Southern Oscillation (ENSO) and Indian Ocean Dipole (IOD) correlations with rainfall during Austral spring (SON) utilizing a novel hit rate metric. Results indicate that only CCAM_OISST successfully replicates observed SON ENSO‐ and IOD‐rainfall correlations, achieving hit rates of 86.6% and 87.5%, respectively, compared to 52.7% and 41.8% for CCAM_ERSST5. Large SST differences are found surrounding the Australian coastline between OISST and ERSST5 (termed the “Coastal Effect”). Differences can be induced by the spatial interpolation error due to the discrepancy between model and driving SST. An additional CCAM experiment, employing OISST with SST masked by ERSST5 in 5° proximity to the Australian continent, underscores the “Coastal Effect” has a significant impact on IOD‐Australian rainfall simulations. In contrast, its influence on ENSO‐Australian rainfall is limited. Therefore, simulations of IOD‐Australian rainfall teleconnection are sensitive to local SST representation along coastlines, probably dependent on the spatial resolution of driving SST.

Evolutionary Constrained Multiobjective Optimization: Scalable High-Dimensional Constraint Benchmarks and Algorithm

Evolutionary constrained multiobjective optimization has received extensive attention and research in the past two decades, and a lot of benchmarks have been proposed to test the effect of the constrained multiobjective evolutionary algorithms (CMOEAs). Specially, the constraint functions are highly correlated with the objective values, which makes the features of constraints too monotonic and differ from the properties of the real-world problems. Accordingly, previous CMOEAs cannot solve real-world problems well, which generally involve decision space constraints with multi-modal/non-linear features. Therefore, we propose a new benchmark framework and design a suite of new test functions with scalable high-dimensional decision space constraints. To be specific, different high-dimensional constraint functions and mixed linkages in variables are considered to be close to realistic features. In this framework, several parameter interfaces are provided, so that users can easily adjust the parameters to obtain the variant functions and test the generalization performance of the algorithms. Different types of existing CMOEAs are employed to test the use of the proposed test functions, and the results show that they are easy to fall into local feasible regions. Therefore, we improve one evolutionary multitasking-based CMOEA to better handle these problems, in which a new search algorithm is designed to enhance the search abilities of populations. Compared with the existing CMOEAs, the proposed CMOEA presents better performance.