Benchmarking Approach Research Articles

A proximal policy optimization approach for food delivery problem with reassignment due to order cancellation

Unexpected cancellation of food delivery orders poses significant challenges to resource allocation planning and could lead to reduced revenue for service providers. This paper addresses this issue by developing an optimization framework that can reassign canceled orders to alternative customers to maximize net revenue and minimize resource wastage. The problem is formulated as a route-based Markov decision process, named the Dynamic Routing and Pricing Problem with Cancellation (DRPPC). A solution approach based on the proximal policy optimization strategy is introduced as a computationally effective way of solving the optimization problem using reinforcement learning techniques. Experimental results demonstrate that the proposed computational method outperforms selected benchmark approaches with higher revenue from various kinds of scenarios with uncertainties. This research advances the intersection of urban logistics and reinforcement learning, offering actionable strategies for enhanced operational resilience in food delivery service providers.

Improving Task-Agnostic Energy Shaping Control of Powered Exoskeletons with Task/Gait Classification.

Emerging task-agnostic control methods offer a promising avenue for versatile assistance in powered exoskeletons without explicit task detection, but typically come with a performance trade-off for specific tasks and/or users. One such approach employs data-driven optimization of an energy shaping controller to provide naturalistic assistance across essential daily tasks with passivity/stability guarantees. This study introduces a novel control method that merges energy shaping with a machine learning-based classifier to deliver optimal support accommodating diverse individual tasks and users. The classifier detects transitions between multiple tasks and gait patterns in order to employ a more optimal, task-agnostic controller based on the weighted sum of multiple optimized energy-shaping controllers. To demonstrate the efficacy of this integrated control strategy, an in-silico assessment is conducted over a range of gait patterns and tasks, including incline walking, stairs ascent/descent, and stand-to-sit transitions. The proposed method surpasses benchmark approaches in 5-fold cross-validation ( ), yielding 93.17 ± 7.39% cosine similarity and 77.92 ± 19.76% variance-accounted-for across tasks and users. These findings highlight the control approach's adaptability in aligning with human joint moments across various tasks.

Part feeding scheduling for mixed-model assembly lines with autonomous mobile robots: benefits of using real-time data

ABSTRACT Mixed-model assembly is increasingly widespread to meet customer requirements for customisation and short delivery times. Flexible part feeding systems are required to timely replenish assembly stations with materials, avoid station idle times, and limit inventory levels on the shop floor. Part feeding scheduling is a complex and dynamic problem, affected by processing time fluctuations, equipment failures, and variations of product mix. Although real-time data of factory processes and resources is widely available and can be exploited using a digital twin of the part feeding system, there is a lack of scientific evidence on the benefits of using real-time data in part feeding scheduling. This research addresses this gap by developing an agent-based simulation model of a part feeding system with a fleet of autonomous mobile robots (AMRs) and comparing a real-time dynamic part feeding scheduling approach with static benchmark approaches. Numerical results indicate that using real-time data improves the performance of the part feeding system and the assembly system significantly, and allows improving the trade-off between the AMR fleet size and the total storage capacity on the shop floor, resulting in lower investment costs for AMRs given a certain storage capacity or lower required storage capacity given an AMR fleet.

A systematic benchmarking framework for future assessments of soil health: An example from Denmark

Based on current evidence and established critical thresholds for soil degradation indicators, it is concerning that over 60–70% of European soils are unhealthy due to unsustainable management and the impact of climate change. Despite European and national efforts to improve soil health, significant gaps remain. The proposal for a Soil Monitoring and Resilience Law, to be implemented by the European Union, seeks to establish a framework for soil monitoring and promote sustainable management practices to achieve healthy soils by 2050. This requires extensive data collection and soil monitoring systems to accurately estimate soil health across Europe, considering the diversity of soil types, climates, and land uses. To establish a framework for soil monitoring, we must understand the site-specific status of soil and the ranges of soil health indicators across specific pedoclimatic regions. In our study, we evaluated the soil status in agricultural areas in Denmark using soil health indicators and a site-specific benchmarking approach. We compiled nationally representative datasets, combining point and model-informed data of soil parameters such as organic carbon content, bulk density, pH, electrical conductivity, clay-to-soil organiccarbon ratio, water erosion, and nitrogen leaching. By categorizing Danish agricultural soils into monitoring units based on textural classes, landscape elements, and wetland types, we calculated benchmarks for these indicators, considering different cropping systems. Our approach provided detailed point-based results and a spatially explicit overview of the status of soil health indicators in Denmark. We identified areas where soil deviates from the benchmarks of different indicators. Such deviations might indicate soil functions operating outside the normal range, posing potential threats to soil health. This proposed framework could support the establishment of a baseline for assessing the directionality of future changes in soil health. Moreover, it is adaptable for implementation by other countries to support assessments of soil health.

Ground-truth-free Deep Learning for 3-D Seismic Denoising and Reconstruction with Channel Attention Mechanism

Seismic denoising methods using supervised methods rely on a large number of high-quality paired training datasets to reach satisfactory performances. There are two ways to generate labels for network training: one is to simulate the synthetic data using the wave equation, and the other is to utilize denoised data obtained via conventional methods. However, using these labels will limit the networks' noise attenuation performance compared with using large volumes of noise-free data as labels. Here, we propose a ground-truth-free way for three-dimensional (3-D) seismic data processing. First, we use the 3-D patch scheme to divide the noisy seismic data into many fixed-size blocks and then flatten the obtained 3-D patches to expand the training set and capture more higher-order waveform characteristics from the input noisy data. Next, the obtained training dataset is sent into the proposed deep learning (DL) network, where the encoder blocks compress the feature map to extract the waveform features, and the decoder blocks reconstruct the denoised feature map. Notably, the convolutional bottleneck attention module (CBAM) and efficient channel attention (ECA) module are applied to guide the network to focus on signal fluctuation features with fewer network parameters. In addition, the concatenation mechanism is used to enable deep networks to reuse shallow-layer waveform features and mitigate overfitting during training. Finally, the unpatching scheme is used to reconstruct the denoised 3-D seismic data. Numerical experiments demonstrate that the proposed method outperforms benchmark approaches in terms of signal-to-noise ratio (SNR) improvement and useful signal preservation.

A discussion on benchmarking unconventional configurations with conventional aircraft: the box-wing study case

This article is intended to introduce an alternative approach to comparative analyses between innovative aeronautical technologies and established state-of-the-art references. Commonly, the tendency is to use a ‘like-for-like’ comparative approach with respect to current technologies -defined as reference benchmarks- that is, to evaluate the possible incremental improvements that can be achieved by introducing a specific technological innovation. However, when innovations that potentially introduce step improvements or new functions compared to the state of the art are evaluated, typically referred to as ‘breakthrough’, this approach may not be the most formally sound one, and it may introduce bias and misjudgements. In the field of aircraft design, using the same top-level requirements and figures of merit as those used for conventional aircraft to initialise and steer the design of unconventional configurations, could undermine the exploitation of their operating and functional potential. The soundness of the comparative approach is of paramount importance, especially in the very early stages of the development of disruptive technologies and unconventional aircraft configurations. In this paper, with the supporting example of the application of the box-wing configuration to medium-range transport aircraft, a general discussion is offered on the necessity of leaving aside the ‘like-for-like’ benchmark approach when investigating the potential of disruptive aircraft innovations. This argumentation does not only refer to the case study proposed as an example, but is generally extendable to aeronautical innovations that may introduce operating and functional novelties compared to current technologies.

Selecting and weighting dynamical models using data-driven approaches

Abstract. In geosciences, multi-model ensembles are helpful to explore the robustness of a range of results. To obtain a synthetic and improved representation of the studied dynamic system, the models are usually weighted. The simplest method, namely the model democracy, gives equal weights to all models, while more advanced approaches base weights on agreement with available observations. Here, we focus on determining weights for various versions of an idealized model of the Atlantic Meridional Overturning Circulation. This is done by assessing their performance against synthetic observations (generated from one of the model versions) within a data assimilation framework using the ensemble Kalman filter (EnKF). In contrast to traditional data assimilation, we implement data-driven forecasts using the analog method based on catalogs of short-term trajectories. This approach allows us to efficiently emulate the model's dynamics while keeping computational costs low. For each model version, we compute a local performance metric, known as the contextual model evidence, to compare observations and model forecasts. This metric, based on the innovation likelihood, is sensitive to differences in model dynamics and considers forecast and observation uncertainties. Finally, the weights are calculated using both model performance and model co-dependency and then evaluated on averages of long-term simulations. Results show good performance in identifying numerical simulations that best replicate observed short-term variations. Additionally, it outperforms benchmark approaches such as strategies based on model democracy or climatology when reconstructing missing distributions. These findings encourage the application of the proposed methodology to more complex datasets in the future, like climate simulations.

I-Health: SDN-Based Fog Architecture for IIoT Applications in Healthcare.

The Industrial Internet of Things (IIoT) has been introduced in an era of increasingly broad potentials in the medical industry. In recent years, IIoT-based healthcare applications have grown in popularity, with the majority of them relying on Wireless Body Area Network (WBAN) for flexibility. There have been a few recent works that have investigated SDN-based fog architecture for constructing smart healthcare systems. However, the best fog node from the fog layer must be identified and limit the transmission of unnecessary data. To address this issue, the Intelligent Software-defined Fog Architecture (i-Health) is developed in this work. Based on the prior data pattern of each patient, the controller will decide whether to send the data to the fog layer. Furthermore, we introduced the Fog Ranking Service (FRS) and Fog Probing Service (FPS) to select the best fog node. The performance comparison reveals that the proposed i-Health outperforms existing benchmark approaches.

C-SLA-MLO: Enhancing SLA Compliance in Industrial Wi-Fi through Cooperative Multilink Operation

In the dynamic landscape of Industry 4.0, robust wireless connectivity emerges as a critical enabler for Industrial Internet of Things (IIoT) applications. With the advent of Wi-Fi 7’s multilink operation (MLO), the demand for schedulers capable of meeting diverse Service Level Agreement (SLA) requirements within industrial environments becomes paramount. Addressing this need, this paper introduces Cooperative SLA-MLO (C-SLA-MLO), a novel distributed multilink scheduler that balances data across MLO channels to fulfill the SLA of STAs operating within the same BSS. We provide a simulation-based evaluation where we compare C-SLA-MLO with other benchmark approaches, in terms of SLA compliance when varying the SLA characteristics, the level of QoS, or background traffic, showcasing an average reduction of 90% in SLA deviation in the industrial use case. This study contributes practical insights for enhancing wireless connectivity in IIoT, offering potential benefits for industrial network optimization by exploiting the MLO feature brought by the new IEEE 802.11be.

Optimization of containerized application deployment in virtualized environments: a novel mathematical framework for resource-efficient and energy-aware server infrastructure

This study addresses the critical need for effective resource management through software container migration in cloud data centers. It emphasizes its role in avoiding resource shortages and reducing energy consumption in cloud environments. This study introduces a novel multi-objective integer linear programming (ILP) approach for software container replacements, complemented by a specialized algorithm designed to migrate software containers between over- and underutilized hosts to enhance efficiency compared to traditional optimization methods. The simulation results demonstrate the algorithm's effectiveness, validating its potential for optimizing energy usage and resource allocation in cloud environments. Statistical analyses confirm the proposed model's and algorithm's superiority over benchmark approaches, highlighting their potential for enhancing resource management in cloud computing systems.

End-to-end optimization of optical communication systems based on directly modulated lasers

The use of directly modulated lasers (DMLs) is attractive in low-power, cost-constrained short-reach optical links. However, their limited modulation bandwidth can induce waveform distortion, undermining their data throughput. Traditional distortion mitigation techniques have relied mainly on the separate training of transmitter-side pre-distortion and receiver-side equalization. This approach overlooks the potential gains obtained by simultaneous optimization of the transmitter (constellation and pulse shaping) and receiver (equalization and symbol demapping). Moreover, in the context of DML operation, the choice of laser-driving configuration parameters such as the bias current and peak-to-peak modulation current has a significant impact on system performance. We propose, to our knowledge, a novel end-to-end optimization approach for DML systems, incorporating the learning of bias and peak-to-peak modulation current to the optimization of constellation points, pulse shaping, and equalization. The simulation of the DML dynamics is based on the use of the laser rate equations at symbol rates between 15 and 25 Gbaud. The resulting output sequences from the rate equations are used to build a differentiable data-driven model, simplifying the calculation of gradients needed for end-to-end optimization. The proposed end-to-end approach is compared to three additional benchmark approaches: the uncompensated system without equalization, a receiver-side finite impulse response equalization approach, and an end-to-end approach with learnable pulse shape and nonlinear Volterra equalization but fixed bias and peak-to-peak modulation current. The numerical simulations on the four approaches show that the joint optimization of bias, peak-to-peak current, constellation points, pulse shaping, and equalization outperforms all other approaches throughout the tested symbol rates.

Generalized hop‐based approaches for identifying influential nodes in social networks

AbstractLocating a set of influential users within a social network, known as the Influence Maximization (IM) problem, can have significant implications for boosting the spread of positive information/news and curbing the spread of negative elements such as misinformation and disease. However, the traditional simulation‐based spread computations under conventional diffusion models render existing algorithms inefficient in finding optimal solutions. In recent years, hop and path‐based approaches have gained popularity, particularly under the cascade models to address the scalability issue. Nevertheless, these existing functions vary based on the considered hop‐distance and provide no guidance on capturing spread sizes beyond two‐hops. In this paper, we introduce Hop‐based Expected Influence Maximization (HEIM), an approach utilizing generalized functions to compute influence spread across varying hop‐distances in conventional diffusion models. We extend our investigation to the Linear Threshold (LT) model, in addition to the Independent Cascade (IC) and Weighted Cascade (WC) models, filling a gap in current literature. Our theoretical analysis shows that the proposed functions preserve both monotonicity and submodularity, and the proposed HEIM algorithm can achieve an approximation ratio of under a limited hop‐measures, whereas a multiplicative ‐approximation under global measures. Furthermore, we show that expected spread methods can serve as a better benchmark approach than existing simulation‐based methods. The performance of the HEIM algorithm is evaluated through experiments on three real‐world networks, and is compared to six other existing algorithms. Results demonstrate that the three‐hop based HEIM algorithm achieves superior solution quality, ranking first in statistical tests, and is notably faster than existing benchmark approaches. Conversely, the one‐hop‐based HEIM offers faster computation while still delivering competitive solutions, providing decision‐makers with flexibility based on application needs.

On the information content of explainable artificial intelligence for quantitative approaches in finance

AbstractWe simulate economic data to apply state-of-the-art machine learning algorithms and analyze the economic precision of competing concepts for model agnostic explainable artificial intelligence (XAI) techniques. Also, we assess empirical data and provide a discussion of the competing approaches in comparison with econometric benchmarks, when the data-generating process is unknown. The simulation assessment provides evidence that the applied XAI techniques provide similar economic information on relevant determinants when the data generating process is linear. We find that the adequate choice of XAI technique is crucial when the data generating process is unknown. In comparison to econometric benchmark models, the application of boosted regression trees in combination with Shapley values combines both a superior fit to the data and innovative interpretable insights into non-linear impact factors. Therefore it describes a promising alternative to the econometric benchmark approach.

Adaptive multibeam hopping in geo satellite networks with non-uniformly distributed ground users

This paper designs a novel low-complexity user-cluster grouping algorithm for adaptive beam hopping in geostationary satellite networks equipped with multibeam phased-array antennas. Each beam serves a cluster of users, and the challenge is to design a beam-hopping pattern where no beams are simultaneously serving nearby user clusters. We develop a line search procedure to identify near-optimum groupings for heterogeneous traffic demands. We provide a necessary condition to determine the boundaries of the line search space. Our approach employs exclusion regions around critical user clusters in congested areas, iterates a sequential congestion-based grouping algorithm, and applies a group-member-swapping procedure. It provides max-min fairness for ground users. Extensive numerical studies have shown that our user grouping algorithm produces near-optimum beam-hopping schedules with low outage probability. It achieves an improvement of up to 13dB in the worst-case signal-to-interference and noise ratio, and doubles the zero-outage data rate, compared to benchmark approaches.

Sequential metamodel‐based approaches to level‐set estimation under heteroscedasticity

AbstractThis paper proposes two sequential metamodel‐based methods for level‐set estimation (LSE) that leverage the uniform bound built on stochastic kriging: predictive variance reduction (PVR) and expected classification improvement (ECI). We show that PVR and ECI possess desirable theoretical performance guarantees and provide closed‐form expressions for their respective sequential sampling criteria to seek the next design point for performing simulation runs, allowing computationally efficient one‐iteration look‐ahead updates. To enhance understanding, we reveal the connection between PVR and ECI's sequential sampling criteria. Additionally, we propose integrating a budget allocation feature with PVR and ECI, which improves computational efficiency and potentially enhances robustness to the impacts of heteroscedasticity. Numerical studies demonstrate the superior performance of the proposed methods compared to state‐of‐the‐art benchmarking approaches when given a fixed simulation budget, highlighting their effectiveness in addressing LSE problems.

Are BSR Municipalities on Track for Energy Transition?

Climate neutrality targets and the growing decentralization of energy systems have substantially increased the role of municipalities in global energy transition. However, global shifts and national government demands have often left local public authorities unprepared to face numerous challenges related to local space planning, cost-effective integration, and decarbonization of electricity, heating, industry, and mobility. Therefore, there is a need to investigate the current state of municipal energy transition and analyze how municipalities face climate neutrality target achievement. This study conducts an integrated energy sustainability assessment to investigate the progress of the energy transition in six municipalities in the Baltic Sea region. A benchmarking approach is applied to compare the different levels of energy efficiency and decarbonization in the municipalities. The study reveals different energy consumption patterns of municipal buildings, which are influenced by various factors such as the type of building (educational, office, social facilities, etc.), the heat source (district heating or individual local heat source) and the energy efficiency management practices applied. In addition, a different trend in the installation of renewable energy capacity can be observed in municipalities in Latvia, Lithuania, Poland and Sweden. The study analyses the overall gap between the production and consumption of renewable energy to determine the storage potential and the role in the local energy transitions. The findings of this study highlight the key cornerstones in the current state of municipal energy transition, setting a foundation for better and more effective energy policy planning at national and local scales.

A Novel Benchmarking Approach to Assess Fire Safety of Liquid Electrolytes in Lithium‐Ion Batteries

AbstractEstablishing authoritative electrolyte safety assessment methods at laboratory levels is crucial for addressing conflicts from thermal runaway of lithium‐ion batteries. However, self‐extinguishing time (SET), as the most widely used evaluation method now, lacks benchmarks and heavily relies on the specific implementation of test procedures. This work systematically summarizes and investigates key test parameters. Based on repeatability and reliability, burning a glassfiber separator (Φ16 mm) with absorption of 0.1 g liquid electrolyte (LE) can be proposed as a unified method. The concept of self‐extinguishing efficiency (SEE) is further proposed with a new evaluation criterion of i) SEE≤70, Flammable; ii) 70<SEE<90, Flame Retarded; iii) 90≤SEE≤100, Nonflammable. The feasibility of the new protocol is verified by employment in evaluating the effects of 15 representative flame retardants (FRs) on combustion behaviors of LEs. Meanwhile, the underlying flame retardancy mechanism is revealed, indicating that preceding or simultaneous vaporization and pyrolysis of FRs in conjunction with the decomposition of LE promote mitigating electrolyte fire risk. In addition, machine learning is employed to aid in analyzing the significance of various features on SEE, providing further validation of the proposed mechanism. This work provides a benchmark for the safety evaluation of LEs, facilitating the advancement of fire‐safe batteries.

A Computationally Efficient Approach to Fully Bayesian Benchmarking

In small area estimation, it is sometimes necessary to use model-based methods to produce estimates in areas with little or no data. In official statistics, we often require that aggregates of small area estimates agree with national estimates for internal consistency purposes. Enforcing this agreement is referred to as benchmarking, and while methods currently exist to perform benchmarking, few are ideal for applications with non-normal outcomes and benchmarks with uncertainty. Fully Bayesian benchmarking is a theoretically appealing approach insofar as we can obtain posterior distributions conditional on a benchmarking constraint. However, existing implementations may be computationally prohibitive. In this paper, we critically review benchmarking methods in the context of small area estimation in low- and middle-income countries with binary outcomes and uncertain benchmarks, and propose a novel approach in which posterior samples of small area characteristics from an unbenchmarked model can be combined with a rejection sampler or Metropolis-Hastings algorithm to produce benchmarked posterior distributions in a computationally efficient way. To illustrate the flexibility and efficiency of our approach, we provide comparisons to an existing benchmarking approach in a simulation, and applications to HIV prevalence and under-5 mortality estimation. Code implementing our methodology is available in the R package stbench.

Evaluating feature attribution methods in the image domain

Feature attribution maps are a popular approach to highlight the most important pixels in an image for a given prediction of a model. Despite a recent growth in popularity and available methods, the objective evaluation of such attribution maps remains an open problem. Building on previous work in this domain, we investigate existing quality metrics and propose new variants of metrics for the evaluation of attribution maps. We confirm a recent finding that different quality metrics seem to measure different underlying properties of attribution maps, and extend this finding to a larger selection of attribution methods, quality metrics, and datasets. We also find that metric results on one dataset do not necessarily generalize to other datasets, and methods with desirable theoretical properties do not necessarily outperform computationally cheaper alternatives in practice. Based on these findings, we propose a general benchmarking approach to help guide the selection of attribution methods for a given use case. Implementations of attribution metrics and our experiments are available online (https://github.com/arnegevaert/benchmark-general-imaging).Graphical abstract

MieAI: a neural network for calculating optical properties of internally mixed aerosol in atmospheric models

Aerosols influence weather and climate by interacting with radiation through absorption and scattering. These effects heavily rely on the optical properties of aerosols, which are mainly governed by attributes such as morphology, size distribution, and chemical composition. These attributes undergo continuous changes due to chemical reactions and aerosol micro-physics, resulting in significant spatio-temporal variations. Most atmospheric models struggle to incorporate this variability because they use pre-calculated tables to handle aerosol optics. This offline approach often leads to substantial errors in estimating the radiative impacts of aerosols along with posing significant computational burdens. To address this challenge, we introduce a computationally efficient and robust machine learning approach called MieAI. It allows for relatively inexpensive calculation of the optical properties of internally mixed aerosols with a log-normal size distribution. Importantly, MieAI fully incorporates the variability in aerosol chemistry and microphysics. Our evaluation of MieAI against traditional Mie calculations, using number concentrations from the ICOsahedral Nonhydrostatic model with Aerosol and Reactive Trace gases (ICON-ART) simulations, demonstrates that MieAI exhibits excellent predictive accuracy for aerosol optical properties. MieAI achieves this with errors well within 10%, and it operates more than 1000 times faster than the benchmark approach of Mie calculations. Due to its generalized nature, the MieAI approach can be implemented in any chemistry transport model which represents aerosol size distribution in the form of log-normally distributed internally mixed modes. This advancement has the potential to replace frequently employed look-up tables and plays a substantial role in the ongoing attempts to reduce uncertainties in estimating aerosol radiative forcing.