Benchmark Environment Research Articles

TMIC-60. BRATS-PATH: ASSESSING HETEROGENEOUS HISTOPATHOLOGIC REGIONS IN GLIOBLASTOMA

Abstract Glioblastoma, the most common malignant primary adult brain tumor, poses significant diagnostic and treatment challenges due to its heterogeneous molecular and micro-environmental profiles. To this end, we organize the BraTS-Path challenge to provide a public benchmarking environment and a comprehensive dataset to develop and validate AI models for identifying distinct histopathologic glioblastoma sub-regions in H&E-stained digitized tissue sections. We identified 188 multi-institutional diagnostic slides of glioblastoma (IDH-wt, Gr.4) cases, from the TCGA-GBM and TCGA-LGG data collections, following their reclassification according to the 2021 WHO classification criteria. Sub-regions were selected according to distinctive morphology of histopathologic features and included aggressive tumor biology and areas consistent with potential treatment effect. Selected sub-region annotations included cellular tumor, geographic necrosis, cortical infiltration, pseudopalisading necrosis, microvascular proliferation, white matter penetration, regions dense with macrophages, leptomeningeal infiltration, and presence of lymphocytes. We obtained 107,340 patches of size 512x512 from the 9 sub-regions. A global network of board-certified expert neuropathologists defined and followed a systematic annotation protocol based on clinical definitions and only delineated sub-regions with high confidence, thus ensuring high-quality standardized data. Each tissue section was assigned to an annotator-approver pair, with the annotator delineating sub-regions and the approver ensuring the consistency of the annotations. By crowdsourcing annotations, the BraTS-Path challenge harnesses the collective expertise of clinical neuropathologists and fosters a collaborative environment to advance the neuro-oncology field. The anticipated developed algorithms are expected to integrate state-of-the-art computational methods, achieving high accuracy in identifying diverse histopathologic features and advancing clinical decision-making processes. The BraTS-Path challenge aims to bridge the gap between research and clinical practice by promoting the development of AI-driven tools for precise tumor characterization. This collaborative effort can significantly enhance our understanding of glioblastoma, improve diagnostic accuracy, and inform treatment strategies, thereby contributing to better patient outcomes.

Os noise mitigations for benchmarking web browser execution environment performance

Web browsers have almost since birth sought to host and run feature-rich, compute-intensive and complex applications, within their execution environments (EEs), thereby achieving performance akin to native desktop applications. Studies that have proposed performance improvements to these Web browser EEs together with their respective benchmarks have endeavoured to use proven methodologies with which they confirmed the scientific rigour of those benchmarks, but have often overlooked operating system (OS) noise and the negative impact that it has on benchmarks. This paper identifies the various types of OS noise that exist within a typical Intel-based computer system that utilises a Linux-based OS. While also reviewing what related studies have done to mitigate against them. A series of mitigations were then proposed for the Linux-based system that was then benchmarked using the PolyBench/C benchmarking suite. Whilst not all OS noise was mitigated against, our results confirm that the OS noise mitigations that were provided, significantly reduced OS noise. Further significance of this study is that a more controlled benchmarking environment is made available, due to focusing the CPU processing almost entirely on the benchmarks only, while also enhancing the precision and accuracy of the benchmarks. In this study, we lay down a foundation that will make future Web browser EE as well as any other type of computer software benchmarking more precise and accurate, thereby improving the scientific rigour of those benchmarks.

An Efficient Multi-Agent Policy Self-Play Learning Method Aiming at Seize-Control Scenarios

Aiming at the problem of multi-agent cooperative confrontation in seize-control scenarios, we design an efficient multi-agent policy self-play (EMAP-SP) learning method. First, a multi-agent centralized policy model is constructed to command the agents to perform tasks cooperatively. Considering that the policy being trained and its historical policies usually have poor exploration capability under incomplete information in self-play trainings, the intrinsic reward mechanism based on random network distillation (RND) is introduced in the self-play learning method. In addition, we propose a multi-step on-policy deep reinforcement learning (DRL) algorithm assisted by off-policy policy evaluation (MSOAO) to learn the best response policy in the self-play. Compared with DRL algorithms commonly used in complex decision problems, MSOAO has more efficient policy evaluation capability, and efficient policy evaluation further improves the policy learning capability. The effectiveness of EMAP-SP is fully verified in MiaoSuan wargame simulation system, and the evaluation results show that EMAP-SP can learn the cooperative policy of effectively defeating the Blue side’s knowledge-based policy under incomplete information. Moreover, the evaluations results in DRL benchmark environments also show that the best response policy learning algorithm MSOAO can promote the agent to learn approximately optimal policies.

Dynamic preference inference network: Improving sample efficiency for multi-objective reinforcement learning by preference estimation

Multi-objective reinforcement learning (MORL) addresses the challenge of optimizing policies in environments with multiple conflicting objectives. Traditional approaches often rely on scalar utility functions, which require predefined preference weights, limiting their adaptability and efficiency. To overcome this, we propose the Dynamic Preference Inference Network (DPIN), a novel method designed to enhance sample efficiency by dynamically estimating the trajectory decision preference of the agent. DPIN leverages a neural network to predict the most favorable preference distribution for each trajectory, enabling more effective policy updates and improving overall performance in complex MORL tasks. Extensive experiments in various benchmark environments demonstrate that DPIN significantly outperforms existing state-of-the-art methods, achieving higher scalarized returns and hypervolume. Our findings highlight DPIN’s ability to adapt to varying preferences, reduce sample complexity, and provide robust solutions in multi-objective settings.

A sequential multi-agent reinforcement learning framework for different action spaces

In many real-world decision-making tasks, multi-agent need to learn collaboration in a high-dimensional complex action space, rather than just a single discrete action space. Recently, value decomposition learning methods such as QMIX have emerged as a promising approaches for collaborative multi-agent tasks. However, most of the value decomposition algorithms can only be used in discrete action space, which would limit their practicability. To address the limitation, we propose a novel algorithm called Multi-Agent Sequential Q-Networks (MASQN), which can be applied to the multi-agent domains with continuous, multidiscrete or hybrid action spaces. The proposed algorithm is based on the structure of centralized training with decentralized execution (CTDE). The decentralized actors ensure adaptability to different action spaces by utilizing action space discretization and sequential models, and the centralized critic utilizes the value decomposition architecture to guide effective updates of the policy parameters for each agent. We also give the convergence of joint policy from the perspective of policy iteration, by combining it with the CTDE structure and the constraint of the Individual Global Max (IGM) condition. Finally, we evaluate the MASQN algorithm on two benchmark environments: MAMuJoCo and Hybrid Predator–Prey. The empirical results show that MASQN out performs the state-of-the-art performance on three different action spaces.

Regularities and Anomalies in Neon Matrix Shifts of Hydrogen-Bonded O-H Stretching Fundamentals.

O-H bond stretching vibrations in hydrogen-bonded complexes embedded into cryogenic neon matrices are subtly downshifted from cold gas phase reference wavenumbers. To the extent that this shift is systematic, it enables neon matrices as more universally applicable spectroscopic benchmark environments for quantum chemical predictions. Outliers are indicative of either an assignment problem in one of the two cryogenic experiments or they reveal interesting dynamics or structural effects on the complexes as a function of the environment. We compile 6 literature-known pairs of experimental data in jet and neon matrix expansions and realize a 6-fold expansion of that number through targeted matrix isolation and/or slit jet expansion spectroscopy presented in this work. In many cases, the neon matrix shift is less than the uncertainty of the currently best-performing blind quantum chemical predictions for the gas phase, but in specific cases, it may exceed the currently achievable theoretical accuracy. Some evidence for a positive correlation of the matrix shift with the hydrogen bond shift is found, similar to observations for helium nanodroplets. Outliers in particular for water acting as a donor are discussed, and in a few cases they call for a future reinvestigation. Substantial improvement in the correlation of the matrix shift with the hydrogen bond shift is achieved for ketone monohydrates by removing a vibrational resonance. New insights into nitrile hydration isomerism are obtained, and the linear OH stretching spectrum of the jet-cooled ammonia-water complex is presented for the first time. Vibrational spectroscopy in weakly perturbing solid rare gas quantum matrices for the benchmarking of gas phase theory and future explicit theoretical treatments of the quantum matrix environment to better understand the outliers are both encouraged.

Emergence of integrated behaviors through direct optimization for homeostasis

Homeostasis is a self-regulatory process, wherein an organism maintains a specific internal physiological state. Homeostatic reinforcement learning (RL) is a framework recently proposed in computational neuroscience to explain animal behavior. Homeostatic RL organizes the behaviors of autonomous embodied agents according to the demands of the internal dynamics of their bodies, coupled with the external environment. Thus, it provides a basis for real-world autonomous agents, such as robots, to continually acquire and learn integrated behaviors for survival. However, prior studies have generally explored problems pertaining to limited size, as the agent must handle observations of such coupled dynamics. To overcome this restriction, we developed an advanced method to realize scaled-up homeostatic RL using deep RL. Furthermore, several rewards for homeostasis have been proposed in the literature. We identified that the reward definition that uses the difference in drive function yields the best results. We created two benchmark environments for homeostasis and performed a behavioral analysis. The analysis showed that the trained agents in each environment changed their behavior based on their internal physiological states. Finally, we extended our method to address vision using deep convolutional neural networks. The analysis of a trained agent revealed that it has visual saliency rooted in the survival environment and internal representations resulting from multimodal input.

The quantum cartpole: A benchmark environment for non-linear reinforcement learning

Feedback-based control is the de-facto standard when it comes to controlling classical stochastic systems and processes. However, standard feedback-based control methods are challenged by quantum systems due to measurement induced backaction and partial observability. Here we remedy this by using weak quantum measurements and model-free reinforcement learning agents to perform quantum control. By comparing control algorithms with and without state estimators to stabilize a quantum particle in an unstable state near a local potential energy maximum, we show how a trade-off between state estimation and controllability arises. For the scenario where the classical analogue is highly nonlinear, the reinforcement learned controller has an advantage over the standard controller. Additionally, we demonstrate the feasibility of using transfer learning to develop a quantum control agent trained via reinforcement learning on a classical surrogate of the quantum control problem. Finally, we present results showing how the reinforcement learning control strategy differs from the classical controller in the non-linear scenarios.

Trust region policy optimization via entropy regularization for Kullback–Leibler divergence constraint

Trust region policy optimization (TRPO) is one of the landmark policy optimization algorithms in deep reinforcement learning. Its purpose is to maximize a surrogate objective based on an advantage function, subject to the limited Kullback–Leibler (KL) divergence of two consecutive policies. Although there have been many successful applications of this algorithm in the literature, the approach has often been criticized for suppressing the exploration ability of some application environments due to its strict divergence constraint. As such, most researchers prefer to use entropy regularization, which is added to the expected discounted rewards or the surrogate objectives. That said, there is much debate about whether there might be an alternative strategy for regularizing TRPOs. In this paper, we present just that. Our strategy is to regularize the KL divergence-based constraint via Shannon entropy. This approach enlarges the difference between two consecutive policies and thus derives a new TRPO scheme with entropy regularization for use with KL divergence constraint. Next, the surrogate objective and Shannon entropy are approximated linearly, while the KL divergence is expanded quadratically. An efficient conjugate gradient optimization procedure then solves two sets of linear equations, providing a detailed code-level implementation that can be used for a fair experimental comparison. Extensive experiments within eight benchmark environments demonstrate that our proposed method is superior to both the original TRPO and the entropy regularized objective TRPO. Further, theoretical and experimental analysis shows that three TRPO-like methods have an equal time complexity and a close computational burden.

Integrating human learning and reinforcement learning: A novel approach to agent training

Off-policy reinforcement learning (RL) algorithms are known for improving sample efficiency by employing prior experiences in experience replay memory. However, most existing off-policy RL algorithms are bottlenecked by the slow convergence speed and demand of a large number of interaction samples. In contrast, on-policy RL algorithms converge fast and continuously generate new samples. However, their success heavily depends on the accuracy of the generated samples. To address these challenges, a novel RL framework called HI-FER (Human-learning Inspired Frequent Experience Replay) is proposed by mimicking the process of human learning. HI-FER employs a parallelized experience replay and repetitive training framework to expedite the convergence rate of off-policy algorithms, which imitates the function of the human brain’s parallel information processing and repetitive learning. Additionally, a periodic network reset strategy and dynamic memory updating are leveraged by imitating the forgetting mechanism of humans to prevent overfitting triggered by repetitive updating on limited experiences. Extensive comparison experiments and ablation studies are performed on benchmark environments to evaluate the proposed method. The empirical results demonstrate that HI-FER outperforms the baselines in terms of sample efficiency on state-based (14% improvements) and image-based (51% improvements) tasks from DMControl. Project website and code: https://github.com/Arya87/HI-FER.

Quafu-RL: The cloud quantum computers based quantum reinforcement learning

With the rapid advancement of quantum computing, hybrid quantum–classical machine learning has shown numerous potential applications at the current stage, with expectations of being achievable in the noisy intermediate-scale quantum (NISQ) era. Quantum reinforcement learning, as an indispensable study, has recently demonstrated its ability to solve standard benchmark environments with formally provable theoretical advantages over classical counterparts. However, despite the progress of quantum processors and the emergence of quantum computing clouds, implementing quantum reinforcement learning algorithms utilizing parameterized quantum circuits (PQCs) on NISQ devices remains infrequent. In this work, we take the first step towards executing benchmark quantum reinforcement problems on real devices equipped with at most 136 qubits on the BAQIS Quafu quantum computing cloud. The experimental results demonstrate that the policy agents can successfully accomplish objectives under modified conditions in both the training and inference phases. Moreover, we design hardware-efficient PQC architectures in the quantum model using a multi-objective evolutionary algorithm and develop a learning algorithm that is adaptable to quantum devices. We hope that the Quafu-RL can be a guiding example to show how to realize machine learning tasks by taking advantage of quantum computers on the quantum cloud platform.

Interpretable Learned Emergent Communication for Human–Agent Teams

Learning interpretable communication is essential for multi-agent and human-agent teams (HATs). In multi-agent reinforcement learning for partially-observable environments, agents may convey information to others via learned communication, allowing the team to complete its task. Inspired by human languages, recent works study discrete (using only a finite set of tokens) and sparse (communicating only at some time-steps) communication. However, the utility of such communication in human-agent team experiments has not yet been investigated. In this work, we analyze the efficacy of sparse-discrete methods for producing emergent communication that enables high agent-only and human-agent team performance. We develop agent-only teams that communicate sparsely via our scheme of Enforcers that sufficiently constrain communication to any budget. Our results show no loss or minimal loss of performance in benchmark environments and tasks. In human-agent teams tested in benchmark environments, where agents have been modeled using the Enforcers, we find that a prototype-based method produces meaningful discrete tokens that enable human partners to learn agent communication faster and better than a one-hot baseline. Additional HAT experiments show that an appropriate sparsity level lowers the cognitive load of humans when communicating with teams of agents and leads to superior team performance.

On Neuroevolution of Multi-Input Compositional Pattern Producing Networks: A Case of Entertainment Computing, Edge Devices, and Smart Cities

This work presents a novel approach by utilizing Heterogeneous Activation Neural Networks (HA-NNs) to evolve the weights of Artificial Neural Networks (ANNs) for reinforcement learning in console and arcade computer games like Atari's Breakout and Sonic the Hedgehog. It is the first study to explore the potential of HA-NNs as potent ANNs in solving gaming-related reinforcement learning problems. Additionally, the proposed solution optimizes data transmission over networks for edge devices, marking a novel application of HA-NNs. The study achieved outstanding results, outperforming recent works in benchmark environments like CartPole-v1, Lunar Lander Continuous, and MountainCar-Continuous, with HA-NNs and ANNs evolved using the Neuroevolution of Augmenting Topologies (NEAT) algorithm. Notably, the key advancements include exceptional scores of 500 in CartPole-v1 and 98.2 in Mountain Car Continuous, demonstrating the efficacy of HA-NNs in reinforcement learning tasks. Beyond gaming, the research addresses the challenge of efficient data communication between edge devices, which has the potential to enhance performance in smart cities while reducing the load on edge devices and supporting seamless entertainment experiences with minimal commuting. This work pioneers the application of HA-NNs in reinforcement learning for computer games and introduces a novel approach for optimizing edge device communication, promising significant advancements in the fields of AI, neural networks, and smart city technologies.

Pioneering approaches for enhancing the speed of hierarchical LA by ordering the actions

Fixed Structure Stocastic Automata (FSSA), Variable Structure Learning Automata (VSSA), and their discretized versions have been significantly improved by utilizing inexpensive estimates of the actions' reward probabilities. These represent the fastest LA to date. However, the concept of ordering the actions has never been used within the field, and the reason for this is that there is no way to order the actions a priori. The recently-introduced Hierarchical Discrete Pursuit Automaton (HDPA) has an interesting concept of placing two-action LA along the nodes of a tree, implying that the leaves signify an underlying ordering. In this paper, we show that if estimates are available (as in the case of estimator algorithms), these can be used to place the actions at the leaf level to further enhance the convergence capabilities of the overall ensemble of the two-action LAs. This paper contains the design of this HDPA, the proof of this assertion, and the simulation results on benchmark Environments. Based on the results, we believe that it is the fastest and most accurate LA to date. Our position is that it will be very hard to beat its performance, since it has been incorporated all the salient features of the entire field of LA.

Pasta: A Case for Hybrid Homomorphic Encryption

The idea of hybrid homomorphic encryption (HHE) is to drastically reduce bandwidth requirements when using homomorphic encryption (HE) at the cost of more expensive computations in the encrypted domain. To this end, various dedicated schemes for symmetric encryption have already been proposed. However, it is still unclear if those ideas are already practically useful, because (1) no cost-benefit analysis was done for use cases and (2) very few implementations are publicly available. We address this situation in several ways. We build an open-source benchmarking r framework, we explore properties of the respective HHE proposals. It turns out that even medium-sized use cases are infeasible, especially when involving integer arithmetic. Next, we propose Pasta, a cipher thoroughly optimized for integer HHE use cases. Pasta is designed to minimize the multiplicative depth, while also leveraging the structure of two state-of-the-art integer HE schemes (BFV and BGV) to minimize the homomorphic evaluation latency. Using our new benchmarking environment, we extensively evaluate Pasta in SEAL and HElib and compare its properties to 8 existing ciphers in two use cases. Our evaluations show that Pasta outperforms its competitors for HHE both in terms of homomorphic evaluation time and noise consumption, showing its efficiency for applications in real-world HE use cases. Concretely, Pasta outperforms Agrasta by a factor of up to 82, Masta by a factor of up to 6 and Hera up to a factor of 11 when applied to the two use cases.

A Framework for Mapping DRL Algorithms With Prioritized Replay Buffer Onto Heterogeneous Platforms

Despite the recent success of Deep Reinforcement Learning (DRL) in self-driving cars, robotics and surveillance, training DRL agents takes tremendous amount of time and computation resources. In this paper, we aim to accelerate DRL with Prioritized Replay Buffer due to its state-of-the-art performance on various benchmarks. The computation primitives of DRL with Prioritized Replay Buffer include environment emulation, neural network inference, sampling from Prioritized Replay Buffer, updating Prioritized Replay Buffer and neural network training. The speed of running these primitives varies for various DRL algorithms such as Deep Q Network and Deep Deterministic Policy Gradient. This makes a fixed mapping of DRL algorithms inefficient. In this work, we propose a framework for mapping DRL algorithms onto heterogeneous platforms consisting of a multi-core CPU, a GPU and a FPGA. First, we develop specific accelerators for each primitive on CPU, FPGA and GPU. Second, we relax the data dependency between priority update and sampling performed in the Prioritized Replay Buffer. By doing so, the latency caused by data transfer between GPU, FPGA and CPU can be completely hidden without sacrificing the rewards achieved by agents learned using the target DRL algorithms. Finally, given a DRL algorithm specification, our design space exploration automatically chooses the optimal mapping of various primitives based on an analytical performance model. On widely used benchmark environments, our experimental results demonstrate up to 997.3× improvement in training throughput compared with baseline mappings on the same heterogeneous platform. Compared with the state-of-the-art distributed Reinforcement Learning framework RLlib, we achieve 1.06 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\times \sim$</tex-math></inline-formula> 1005× improvement in training throughput.

Federated Ensemble Model-Based Reinforcement Learning in Edge Computing

Federated learning (FL) is a privacy-preserving distributed machine learning paradigm that enables collaborative training among geographically distributed and heterogeneous devices without gathering their data. Extending FL beyond the supervised learning models, federated reinforcement learning (FRL) was proposed to handle sequential decision-making problems in edge computing systems. However, the existing FRL algorithms directly combine model-free RL with FL, thus often leading to high sample complexity and lacking theoretical guarantees. To address the challenges, we propose a novel FRL algorithm that effectively incorporates model-based RL and ensemble knowledge distillation into FL for the first time. Specifically, we utilise FL and knowledge distillation to create an ensemble of dynamics models for clients, and then train the policy by solely using the ensemble model without interacting with the environment. Furthermore, we theoretically prove that the monotonic improvement of the proposed algorithm is guaranteed. The extensive experimental results demonstrate that our algorithm obtains much higher sample efficiency compared to classic model-free FRL algorithms in the challenging continuous control benchmark environments under edge computing settings. The results also highlight the significant impact of heterogeneous client data and local model update steps on the performance of FRL, validating the insights obtained from our theoretical analysis.

Notre-Dame de Paris as a validation case to improve fire safety modelling in historic buildings

The analysis of the thermal damages in Notre-Dame de Paris is necessary to estimate the impact of the dramatic 2019 fire on the remaining structure prior to reconstruction. In doing so, the large amount of data being generated creates a benchmark environment to test the relevance of numerical fire models in the unconventional configuration of a medieval roof. While being an uncontrolled and complex configuration, it can provide insights regarding the relevance of numerical tools for fire risk assessment in historic buildings. Analysing the thermal degradation of the Lutetian limestone in a vault of the choir, experimental techniques are developed to track the in-depth maximum temperature profile reached during the fire. Numerical simulations of the fire development in the roof space then aim at replicating the observations through the evaluation of the heat flux impinging the vaults during the fire. These simulations are carried out using Fire Dynamic Simulator, which requires a large range of assumptions prior to any simulation regarding materials, geometry, meshing and scale. These assumptions are described and pave the way to a future sensitivity analysis to confront the upcoming outcomes of the simulations with the experimental observations.

Policy gradients using variational quantum circuits

Variational quantum circuits are being used as versatile quantum machine learning models. Some empirical results exhibit an advantage in supervised and generative learning tasks. However, when applied to reinforcement learning, less is known. In this work, we considered a variational quantum circuit composed of a low-depth hardware-efficient ansatz as the parameterized policy of a reinforcement learning agent. We show that an \U0001d716-approximation of the policy gradient can be obtained using a logarithmic number of samples concerning the total number of parameters. We empirically verify that such quantum models behave similarly to typical classical neural networks used in standard benchmarking environments and quantum control, using only a fraction of the parameters. Moreover, we study the barren plateau phenomenon in quantum policy gradients using the Fisher information matrix spectrum.

SentiMLBench: Benchmark Evaluation of Machine Learning Algorithms for Sentiment Analysis

Sentiment Analysis has been a topic of interest for researchers due to its increasing usage by Industry. To measure end-user sentiment., there is no clear verdict on which algorithms are better in real-time scenarios. A rigorous benchmark evaluation of various algorithms running across multiple datasets and different hardware architectures is required that can guide future researchers on potential advantages and limitations. In this paper, proposed SentiMLBench is a critical evaluation of key ML algorithms as standalone classifiers, a novel cascade feature selection (CFS) based ensemble technique in multiple benchmark environments each using a different twitter dataset and processing hardware. The best trained ensemble model with CFS enhancement surpasses current state-of-the-art models, according to experimental results. In a study, though ensemble model provides good accuracy, it falls short of neural networks accuracy by 2%. ML algorithms accuracy is poor as standalone classifiers across all three studies. The supremacy of neural networks is further stamped in study three where it outperforms other algorithms in accuracy by over 10%. Graphical processing unit provide speed and higher computational power at a fraction of a cost compared to a normal processor thereby providing critical architectural insights into developing a robust expert system for sentiment analysis.