Benchmark Tasks Research Articles

Efficient Robot Manipulation via Reinforcement Learning with Dynamic Movement Primitives-Based Policy

Reinforcement learning (RL) that autonomously explores optimal control policies has become a crucial direction for developing intelligent robots while Dynamic Movement Primitives (DMPs) serve as a powerful tool for efficiently expressing robot trajectories. This article explores an efficient integration of RL and DMP to enhance the learning efficiency and control performance of reinforcement learning in robot manipulation tasks by focusing on the forms of control actions and their smoothness. A novel approach, DDPG-DMP, is proposed to address the efficiency and feasibility issues in the current RL approaches that employ DMP to generate control actions. The proposed method naturally integrates a DMP-based policy into the actor–critic framework of the traditional RL approach Deep Deterministic Policy Gradient (DDPG) and derives the corresponding update formulas to learn the networks that properly decide the parameters of DMPs. A novel inverse controller is further introduced to adaptively learn the translation from observed states into various robot control signals through DMPs, eliminating the requirement for human prior knowledge. Evaluated on five robot arm control benchmark tasks, DDPG-DMP demonstrates significant advantages in control performance, learning efficiency, and smoothness of robot actions compared to related baselines, highlighting its potential in complex robot control applications.

Physical reservoir computing: a tutorial

AbstractThis tutorial covers physical reservoir computing from a computer science perspective. It first defines what it means for a physical system to compute, rather than merely evolve under the laws of physics. It describes the underlying computational model, the Echo State Network (ESN), and also some variants designed to make physical implementation easier. It explains why the ESN model is particularly suitable for direct physical implementation. It then discusses the issues around choosing a suitable material substrate, and interfacing the inputs and outputs. It describes how to characterise a physical reservoir in terms of benchmark tasks, and task-independent measures. It covers optimising configuration parameters, exploring the space of potential configurations, and simulating the physical reservoir. It ends with a look at the future of physical reservoir computing as devices get more powerful, and are integrated into larger systems.

A Novel Snow Leopard Optimization for High-Dimensional Feature Selection Problems

To address the limitations of traditional optimization methods in achieving high accuracy in high-dimensional problems, this paper introduces the snow leopard optimization (SLO) algorithm. SLO is a novel meta-heuristic approach inspired by the territorial behaviors of snow leopards. By emulating strategies such as territory delineation, neighborhood relocation, and dispute mechanisms, SLO achieves a balance between exploration and exploitation, to navigate vast and complex search spaces. The algorithm’s performance was evaluated using the CEC2017 benchmark and high-dimensional genetic data feature selection tasks, demonstrating SLO’s competitive advantage in solving high-dimensional optimization problems. In the CEC2017 experiments, SLO ranked first in the Friedman test, outperforming several well-known algorithms, including ETBBPSO, ARBBPSO, HCOA, AVOA, WOA, SSA, and HHO. The effective application of SLO in high-dimensional genetic data feature selection further highlights its adaptability and practical utility, marking significant progress in the field of high-dimensional optimization and feature selection.

Tree sequences as a general-purpose tool for population genetic inference.

As population genetics data increases in size new methods have been developed to store genetic information in efficient ways, such as tree sequences. These data structures are computationally and storage efficient, but are not interchangeable with existing data structures used for many population genetic inference methodologies such as the use of convolutional neural networks (CNNs) applied to population genetic alignments. To better utilize these new data structures we propose and implement a graph convolutional network (GCN) to directly learn from tree sequence topology and node data, allowing for the use of neural network applications without an intermediate step of converting tree sequences to population genetic alignment format. We then compare our approach to standard CNN approaches on a set of previously defined benchmarking tasks including recombination rate estimation, positive selection detection, introgression detection, and demographic model parameter inference. We show that tree sequences can be directly learned from using a GCN approach and can be used to perform well on these common population genetics inference tasks with accuracies roughly matching or even exceeding that of a CNN-based method. As tree sequences become more widely used in population genetics research we foresee developments and optimizations of this work to provide a foundation for population genetics inference moving forward.

Tree sequences as a general-purpose tool for population genetic inference.

As population genetics data increases in size new methods have been developed to store genetic information in efficient ways, such as tree sequences. These data structures are computationally and storage efficient, but are not interchangeable with existing data structures used for many population genetic inference methodologies such as the use of convolutional neural networks (CNNs) applied to population genetic alignments. To better utilize these new data structures we propose and implement a graph convolutional network (GCN) to directly learn from tree sequence topology and node data, allowing for the use of neural network applications without an intermediate step of converting tree sequences to population genetic alignment format. We then compare our approach to standard CNN approaches on a set of previously defined benchmarking tasks including recombination rate estimation, positive selection detection, introgression detection, and demographic model parameter inference. We show that tree sequences can be directly learned from using a GCN approach and can be used to perform well on these common population genetics inference tasks with accuracies roughly matching or even exceeding that of a CNN-based method. As tree sequences become more widely used in population genetics research we foresee developments and optimizations of this work to provide a foundation for population genetics inference moving forward.

A Policy Resonance Approach to Solve the Problem of Responsibility Diffusion in Multiagent Reinforcement Learning.

State-of-the-art (SOTA) multiagent reinforcement algorithms distinguish themselves in many ways from their single-agent equivalences. However, most of them still totally inherit the single-agent exploration-exploitation strategy. Naively inheriting this strategy from single-agent algorithms causes potential collaboration failures, in which the agents blindly follow mainstream behaviors and reject taking minority responsibility. We name this problem the responsibility diffusion (RD) as it shares similarities with the same-name social psychology effect. In this work, we start by theoretically analyzing the cause of this RD problem, which can be traced back to the exploration-exploitation dilemma of multiagent systems (especially large-scale multiagent systems). We address this RD problem by proposing a policy resonance (PR) approach which modifies the collaborative exploration strategy of agents by refactoring the joint agent policy while keeping individual policies approximately invariant. Next, we show that SOTA algorithms can equip this approach to promote the collaborative performance of agents in complex cooperative tasks. Experiments are performed in multiple test benchmark tasks to illustrate the effectiveness of this approach.

Photonic deep residual time-delay reservoir computing

Time-delay reservoir computing (TDRC) represents a simplified variant of recurrent neural networks, employing a nonlinear node with a feedback mechanism to construct virtual nodes. The capabilities of TDRC can be enhanced by transitioning to a deep architecture. In this work, we propose a novel photonic deep residual TDRC (DR-TDRC) with augmented capabilities. The additional time delay added to the residual structure enables DR-TDRC superior to traditional deep structures across various benchmark tasks, especially in memory capability and almost an order of magnitude improvement in nonlinear channel equalization. Additionally, a specifically designed clipping algorithm is utilized to counteract the damage of redundant layers in deep structures, enabling the extension of the deep TDRC to dozens rather than just a few layers, with higher performance. We experimentally demonstrate the proof-of-concept with a 4-layer DR-TDRC containing 960 interrelated neurons (240 neurons per layer), based on four injection-locked distributed feedback lasers. We confirm the potential for scalable deep RC with elevated performance. Our results provide a feasible approach for expanding deep photonic computing to satisfy the boosting demand for artificial intelligence.

Model-based controllers for CubeSat ORU installation: A comparative study

The increasing space debris population in critical orbits due to spacecraft failure dictates the need for action. On-Orbit Servicing (OOS) has been proposed as a method for mitigating this trend by repairing existing space assets. The development of large servicing spacecraft has been given significant attention. In this paper, we are approaching the problem with a new paradigm for a servicing spacecraft by proposing a CubeSat class servicer equipped with a one-degree-of-freedom (DoF) robotic arm. Considering the OOS context, we choose the Orbital Replacement Unit (ORU) installation on a target spacecraft as a challenging benchmark task for the proposed servicer configuration. To carry out this task, we formulate and compare performance of four controllers to achieve coordinated control of both the CubeSat and the robotic arm for the installation task. We start with a basic PD controller and progress to the more advanced impedance controller, Model Predictive Control (MPC) and Model Predictive Impedance Control (MPIC) designs. The controllers are evaluated and compared across several relevant metrics for the ORU installation and their merits for practical implementation are discussed.

A variational autoencoder trained with priors from canonical pathways increases the interpretability of transcriptome data.

Interpreting transcriptome data is an important yet challenging aspect of bioinformatic analysis. While gene set enrichment analysis is a standard tool for interpreting regulatory changes, we utilize deep learning techniques, specifically autoencoder architectures, to learn latent variables that drive transcriptome signals. We investigate whether simple, variational autoencoder (VAE), and beta-weighted VAE are capable of learning reduced representations of transcriptomes that retain critical biological information. We propose a novel VAE that utilizes priors from biological data to direct the network to learn a representation of the transcriptome that is based on understandable biological concepts. After benchmarking five different autoencoder architectures, we found that each succeeded in reducing the transcriptomes to 50 latent dimensions, which captured enough variation for accurate reconstruction. The simple, fully connected autoencoder, performs best across the benchmarks, but lacks the characteristic of having directly interpretable latent dimensions. The beta-weighted, prior-informed VAE implementation is able to solve the benchmarking tasks, and provide semantically accurate latent features equating to biological pathways. This study opens a new direction for differential pathway analysis in transcriptomics with increased transparency and interpretability.

A fine-grained self-adapting prompt learning approach for few-shot learning with pre-trained language models

Pre-trained language models have demonstrated remarkable performance in few-shot learning through the emergence of “prompt-based learning” methods, where the performance of these tasks highly rely on the quality of prompts. Existing prompt learning methods typically customize a single prompt to each few-shot learning task and all the examples in the task share the universal prompt. However, a fine-grained prompt design can enhance the performance of few-shot learning task by leveraging more diverse information hidden in the set of examples. In light of this motivation, this paper introduce an example-specific prompt learning method to embody fine-grained self-adapting prompts for few-shot learning with pre-trained models. Specifically, we introduce the concept of the “weak consistency assumption”, to trade-off the task-specific consistent and example-specific diversity. Based on this assumption, a novel method called Self-adapting Continuous Prompt Learning (SP-learning) to learn example-specific prompts is proposed. It employs a cross-attention prompt generator that considers the characteristics of input samples and utilizes a diversity calibration technique to adjust the prompt generator accordingly. By personalizing prompts for each example, SP-learning aims to improve few-shot learning performance. We perform a systematic evaluation on 10 public benchmark tasks and our method outperforms 8 of those tasks. Our research sheds light on the importance of personalized prompts and opens up new possibilities for improving few-shot learning tasks.

Fast physical reservoir computing, achieved with nonlinear interfered spin waves

Reservoir computing is a promising approach to implementing high-performance artificial intelligence that can process input data at lower computational costs than conventional artificial neural networks. Although reservoir computing enables real-time processing of input time-series data on artificial intelligence mounted on terminal devices, few physical devices are capable of high-speed operation for real-time processing. In this study, we introduce spin wave interference with a stepped input method to reduce the operating time of the physical reservoir, and second-order nonlinear equation task and second-order nonlinear autoregressive mean averaging, which are well-known benchmark tasks, were carried out to evaluate the operating speed and prediction accuracy of said physical reservoir. The demonstrated reservoir device operates at the shortest operating time of 13 ms/5000-time steps, compared to other compact reservoir devices, even though its performance is higher than or comparable to such physical reservoirs. This study is a stepping stone toward realizing an artificial intelligence device capable of real-time processing on terminal devices.

Improved synergistic swarm optimization algorithm to optimize task scheduling problems in cloud computing

Cloud computing has emerged as a cornerstone technology for modern computational paradigms due to its scalability and flexibility. One critical aspect of cloud computing is efficient task scheduling, which directly impacts system performance and resource utilization. In this paper, we propose an enhanced optimization algorithm tailored for task scheduling in cloud environments. Building upon the foundation of the Jaya algorithm and Synergistic Swarm Optimization (SSO), our approach integrates a Levy flight mechanism to enhance exploration-exploitation trade-offs and improve convergence speed. The Jaya algorithm's ability to exploit the current best solutions is complemented by the SSO's collaborative search strategy, resulting in a synergistic optimization framework. Moreover, the incorporation of Levy flights injects stochasticity into the search process, enabling the algorithm to escape local optima and navigate complex solution spaces more effectively. We evaluate the proposed algorithm against state-of-the-art approaches using benchmark task scheduling problems in cloud environments. Experimental results demonstrate the superiority of our method in terms of solution quality, convergence speed, and scalability. Overall, our proposed Improved Jaya Synergistic Swarm Optimization Algorithm offers a promising solution for optimizing TSCC (TSCC), contributing to enhanced resource utilization and system performance in cloud-based applications. The proposed method got 88 % accuracy overall and 10 % enhancement compared to the original method.

Exploring physical and digital architectures in magnetic nanoring array reservoir computers

Physical reservoir computing (RC) is a machine learning technique that is ideal for processing of time dependent data series. It is also uniquely well-aligned to in materio computing realisations that allow the inherent memory and non-linear responses of functional materials to be directly exploited for computation. We have previously shown that square arrays of interconnected magnetic nanorings are attractive candidates for in materio RC, and experimentally demonstrated their strong performance in a range of benchmark tasks (Dawidek et al 2021 Adv. Funct. Mater. 31 2008389, Vidamour et al 2022 Nanotechnology 33 485203, Vidamour et al 2023 Commun. Phys. 6 230). Here, we extend these studies to other lattice arrangements of rings, including trigonal and Kagome grids, to explore how these affect both the magnetic behaviours of the arrays, and their computational properties. We show that while lattice geometry substantially affects the microstate behaviour of the arrays, these differences manifest less profoundly when averaging magnetic behaviour across the arrays. Consequently the computational properties (as measured using task agnostic metrics) of devices with a single electrical readout are found to be only subtly different, with the approach used to time-multiplex data into and out of the arrays having a stronger effect on properties than the lattice geometry. However, we also find that hybrid reservoirs that combine the outputs from arrays with different lattice geometries show enhanced computational properties compared to any single array.

GateKeeper: An UltraLite malicious traffic identification method with dual-aspect optimization strategies on IoT gateways

The Internet of Things (IoT) landscape is booming, and a wide range of IoT endpoints has been integrated into various aspects of life, opening opportunities for malicious attacks that compromise IoT security. Consequently, effective malicious traffic identification methods play a crucial role in ensuring IoT security. However, the current lightweight methods face two main challenges. Firstly, they often generate feature sets that are time-consuming to construct and less adaptable to various scenarios. Secondly, classification models built upon inefficient neural networks incur a significant computational overhead. To address these drawbacks, in this work, we propose GateKeeper, which is an UltraLite method for malicious traffic identification on the IoT gateway. We first propose a base model as the cornerstone and target for subsequent optimization. Then we propose dual-aspect optimization strategies for reducing the input dimension and simplifying the model structure, i.e., Key Bytes Selection (KBS) and Attention Module Simplification (AMS) strategy. Finally, we obtain the optimized UltraLite model-GateKeeper, specifically tailored for integration into IoT gateway deployments, where high-speed real-time identification is paramount. Our extensive experiments demonstrate that GateKeeper performs remarkably in identifying malicious IoT traffic, achieving over 97% accuracy in all three IoT malicious traffic classification benchmark tasks, outperforming six state-of-the-art methods. Besides, GateKeeper’s parameters and FLOPs are 65% smaller than those of existing methods. Moreover, in the case of an IoT gateway platform, GateKeeper demonstrates remarkably low time overhead and hardware resource utilization, surpassing state-of-the-art methods.

A recurrent Gaussian quantum network for online processing of quantum time series

Over the last decade, researchers have studied the interplay between quantum computing and classical machine learning algorithms. However, measurements often disturb or destroy quantum states, requiring multiple repetitions of data processing to estimate observable values. In particular, this prevents online (real-time, single-shot) processing of temporal data as measurements are commonly performed during intermediate stages. Recently, it was proposed to sidestep this issue by focusing on tasks with quantum output, eliminating the need for detectors. Inspired by reservoir computers, a model was proposed where only a subset of the internal parameters are trained while keeping the others fixed at random values. Here, we also process quantum time series, but we do so using a Recurrent Gaussian Quantum Network (RGQN) of which all internal interactions can be trained. As expected, this increased flexibility yields higher performance in benchmark tasks. Building on this, we show that the RGQN can tackle two quantum communication tasks, while also removing some hardware restrictions of the currently available methods. First, our approach is more resource efficient to enhance the transmission rate of quantum channels that experience certain memory effects. Second, it can counteract similar memory effects if they are unwanted, a task that could previously only be solved when redundantly encoded input signals could be provided. Finally, we run a small-scale version of the last task on Xanadu’s photonic processor Borealis.

MDCNet: Long-term time series forecasting with mode decomposition and 2D convolution

Long-term time series forecasting is widely used in various real-world applications, such as weather, traffic, energy, healthcare, etc. Recently, time series decomposition techniques have been adopted in many mainstream forecasting models, such as the prevalent Transformer-based models, to help capture sophisticated temporal patterns and achieve success in several benchmark tasks. However, conventional decomposition algorithms are often based on simple operations or limited to specific fields, and therefore are not effective and applicable enough, especially for complex time series. In this paper, we propose Mode Decomposition and 2D Convolutional Network (MDCNet), a structure-simple yet effective forecasting architecture based on a more effective decomposition method and a multi-frequency time series feature extraction network with multi-scale 2D convolution. Specifically, we first introduce a Variational Mode Decomposition Block to discover intricate time patterns, which decompose time series into trend components and stationary modal components at different main frequencies. Then, we design a Trend Prediction Block and an Intrinsic Mode Functions Prediction Block to capture global correlation and hidden dependencies within different main frequencies, respectively. Furthermore, a Frequency Enhancement Module is designed to further reduce the impact of noise in long-term time series. Experiments on eight benchmark datasets show that MDCNet significantly reduces the error of the previous state-of-the-art method by 15.1% and 11.5% for multivariate and univariate time series, respectively.

Controlling estimation error in reinforcement learning via Reinforced Operation

In Value-based and Actor-Critic reinforcement learning (RL) methods, both inaccuracy and instability of value estimation will detrimentally affect their performance. Some typical RL methods, such as Maxmin Q-learning and QMD3, are plagued by the underestimation problem while failing to trade off the estimation bias and variance jointly. We propose the Reinforced Operation (RO) to address these shortcomings, which selects the closest median among multiple Q-function. RO is applicable to any model-free RL method. Theoretically, we introduce the Mean Square Error (MSE) to jointly analyze the estimation bias and variance of value estimation methods. We also demonstrate the superiority of RO in MSE reduction and give an upper bound for the estimation bias of value estimation methods with arbitrary distribution to guide the calculation of the estimation bias of value estimation methods. Based on RO, we propose the variants of Q-learning and TD3, Reinforced Q-learning (RQ) and Reinforced Delayed Deep Deterministic policy gradient (RD3), respectively, to tackle different tasks. We empirically demonstrate that our method can reduce estimation error and achieve superior performance on discrete and continuous benchmark tasks.

Experimental Demonstration of Reservoir Computing with Self-Assembled Percolating Networks of Nanoparticles.

The complex self-assembled network of neurons and synapses that comprises the biological brain enables natural information processing with remarkable efficiency. Percolating networks of nanoparticles (PNNs) are complex self-assembled nanoscale systems that have been shown to possess many promising brain-like attributes and which are therefore appealing systems for neuromorphic computation. Here experiments are performed that show that PNNs can be utilized as physical reservoirs within a nanoelectronic reservoir computing framework and demonstrate successful computation for several benchmark tasks (chaotic time series prediction, nonlinear transformation, and memory capacity). For each task, relevant literature results are compiled and it is shown that the performance of the PNNs compares favorably to that previously reported from nanoelectronic reservoirs. It is then demonstrated experimentally that PNNs can be used for spoken digit recognition with state-of-the-art accuracy. Finally, a parallel reservoir architecture is emulated, which increases the dimensionality and richness of the reservoir outputs and results in further improvements in performance across all tasks.

Time delay reservoir computing with a silicon microring resonator and a fiber-based optical feedback loop

Silicon microring resonators serve as critical components in integrated photonic neural network implementations, owing to their compact footprint, compatibility with CMOS technology, and passive nonlinear dynamics. Recent advancements have leveraged their filtering properties as weighting functions, and their nonlinear dynamics as activation functions with spiking capabilities. In this work, we investigate experimentally the linear and nonlinear dynamics of microring resonators for time delay reservoir computing, by introducing an external optical feedback loop. After effectively mitigating the impact of environmental noise on the fiber-based feedback phase dependencies, we evaluate the computational capacity of this system by assessing its performance across various benchmark tasks at a bit rate of few Mbps. We show that the additional memory provided by the optical feedback is necessary to achieve error-free operation in delayed-boolean tasks that require up to 3 bits of memory. In this case the microring was operated in the linear regime and the photodetection was the nonlinear activation function. We also show that the Santa Fe and Mackey Glass prediction tasks are solved when the microring nonlinearities are activated. Notably, our study reveals competitive outcomes even when employing only 7 virtual nodes within our photonic reservoir. Our findings illustrate the silicon microring’s versatile performance in the presence of optical feedback, highlighting its ability to be tailored for various computing applications.

Switching Dynamics in Anti‐Ferroelectric Transistor for Multimodal Reservoir Computing

AbstractSpatial‐temporal time series analysis and forecasting are crucial for understanding dynamic systems and making informed decisions. Recurrent neural networks (RNNs) have paved the way for reservoir computing (RC), a method enabling effective temporal information processing at low training costs. While software‐based RC performs well, physical RC systems face challenges like slow processing speed and limited state richness, leading to high hardware costs. This study introduces an innovative approach, i.e., the antiferroelectric field effect transistor‐based RC (AFeFET‐based RC) system for efficient temporal data processing. By exploiting the fading memory property inherent in hafnium oxide‐based antiferroelectric material, this system demonstrates promise for physical RC implementation. Moreover, it leverages the light sensitivity of 2D molybdenum disulfide (MoS2) channels for controllable temporal dynamics under electrical and optical stimuli. This dual‐mode modulation significantly enriches the reservoir state, boosting overall system performance. Experimental tests on standard benchmarking tasks using the AFeFET‐based RC system yielded impressive accuracy results (95.4%) in spoken‐digit recognition and a remarkable normalized root mean square error (NRMSE) of 0.015 in Mackey–Glass time series prediction.