State Transition Function Research Articles

Mapping state transition susceptibility in quantum annealing

Quantum annealing is a novel type of analog computation that aims to use quantum-mechanical fluctuations to search for optimal solutions for Ising problems. Quantum annealing in the transverse field Ising model, implemented on D-Wave devices, works by applying a time-dependent transverse field, which puts all qubits into a uniform state of superposition, and then applying a Hamiltonian over time, which describes a user-programed Ising problem. We present a method that utilizes two control features of D-Wave quantum annealers, namely reverse annealing and an $h$-gain schedule, to quantify the susceptibility, or the distance, between two classical states of an Ising problem. The starting state is encoded using reverse annealing, and the second state is encoded on the linear terms of a problem Hamiltonian. An $h$-gain schedule is specified, which incrementally increases the strength of the linear terms, thus allowing a quantification of the $h$-gain strength required to transition the anneal into a specific state at the final measurement. Because of the nature of quantum annealing, the state tends towards global minima, and therefore we restrict the second classical state to a minimum solution of the given Ising problem. This susceptibility mapping, when enumerated across all initial states, shows in detail the behavior of the quantum annealer during reverse annealing. The procedure is experimentally demonstrated on three small test Ising models which were embedded in parallel on the D-Wave Advantage_system4.1. Analysis of the state transition mapping shows detailed characteristics of the reverse annealing process, including intermediate state transition paths, which are visually represented as state transition networks.

Query-Efficient Black-Box Adversarial Attack With Customized Iteration and Sampling.

It is a challenging task to fool an image classifier based on deep neural networks under the black-box setting where the target model can only be queried. Among existing black-box attacks, transfer-based methods tend to overfit the substitute model on parameter settings. Decision-based methods have low query efficiency due to fixed sampling and greedy search strategy. To alleviate the above problems, we present a new framework for query-efficient black-box adversarial attack by bridging transfer-based and decision-based attacks. We reveal the relationship between current noise and variance of sampling, the monotonicity of noise compression, and the influence of transition function on the decision-based attack. Guided by the new framework, we propose a black-box adversarial attack named Customized Iteration and Sampling Attack (CISA). CISA estimates the distance from nearby decision boundary to set the stepsize, and uses a dual-direction iterative trajectory to find the intermediate adversarial example. Based on the intermediate adversarial example, CISA conducts customized sampling according to the noise sensitivity of each pixel to further compress noise, and relaxes the state transition function to achieve higher query efficiency. Extensive experiments demonstrate CISA's advantage in query efficiency of black-box adversarial attacks.

Improved ant colony algorithm for path planning of mobile robots based on compound prediction mechanism

For the issues of the ant colony algorithm (ACO) to solving the problems in mobile robot path planning, such as the slow optimization speed and the redundant paths in planning results, a high-precision improved ant colony algorithm (IPACO) with fast optimization and compound prediction mechanism is proposed. Firstly, aiming at maximizing the possibility of optimal node selection in the process of path planning, a composite optimal node prediction model is introduced to improve the state transition function. Secondly, a pheromone model with initialize the distribution and “reward or punishment” update mechanism is used to updates the global pheromone concentration directionally, which increases the pheromone concentration of excellent path nodes and the heuristic effect; Finally, a prediction-backward mechanism to deal with the “deadlock” problem in the ant colony search process is adopted in the IPACO algorithm, which enhance the success rate in the ACO algorithm path planning. Five groups of different environments are selected to compare and verify the performance of IPACO algorithm, ACO algorithm and three typical path planning algorithms. The experimental simulation results show that, compared with the ACO algorithm, the convergence speed and the planning path accuracy of the IPACO algorithm are improved by 57.69% and 12.86% respectively, and the convergence speed and the planning path accuracy are significantly improved; the optimal path length, optimization speed and stability of the IPACO algorithm are improved. Which verifies that the IPACO algorithm can effectively improve the environmental compatibility and stability of the ant colony algorithm path planning, and the effect is significantly improved.

Mapping Outputs and States Encoding Bits to Outputs Using Multiplexers in Finite State Machine Implementations

This paper proposes a new technique for implementing Finite State Machines (FSMs) in Field Programmable Gate Arrays (FPGAs). The proposed approach extends the called column compaction in two ways. First, it is applied to the state-encoding bits in addition to the outputs, allowing a reduction in the number of logic functions required both by the state transition function and by the output function. Second, the technique exploits the dedicated multiplexers usually included in FPGAs to increase the number of columns that can be compacted. Unlike conventional state-encoding techniques, the proposed approach reduces the number of logic functions instead of their complexity. An Integer Linear Programming (ILP) formulation that maximizes the number of compacted columns has been proposed. In order to evaluate the effectiveness of the proposed approach, experimental results using standard benchmarks are presented. In most cases, the proposed approach reduces the number of used Look-Up Tables (LUTs) with respect to the conventional FSM implementation.

Kernel Reinforcement Learning-assisted Adaptive Decoder Facilitates Stable and Continuous Brain Control Tasks.

Brain-Machine Interfaces (BMIs) assist paralyzed people to brain control (BC) the neuro-prosthesis continuously moving in space. During the BC process, the subject imagines the movement of the real limb and adapts the brain activity according to the sensory feedback. The neural adaptation in the closed-loop control results in complex and changing brain signals. Simultaneously, the decoder interprets the time-varying functional mapping between neural activity and continuous trajectory. It is crucial and challenging to accurately and adaptively track the mapping to help the subject accomplish the BC task with a stable performance. Existing Kalman Filter (KF) based decoders achieve continuous trajectory control by linearly interpreting neural firing observations into self-evolving prosthetic states. However, the linear neural-state mapping might not accurately reflect the movement intention of the subject. In this paper, we propose a novel method that allows subjects to achieve continuous brain control efficiently and stably. The proposed method incorporates a kernel reinforcement learning method into a state-observation model to decode the nonlinearly neural observation into a continuous trajectory state. The state transition function ensures the continuity of the prosthetic state. The kernel reinforcement learning allows the quick adaptation of the nonlinear neural-movement mapping during the BC process. The proposed method is tested in an online brain control reaching task for rats. Compared with KF, our method achieved more successful trials, faster response time, shorter inter-trial time, and remained stable over days. These results demonstrate that the proposed method is an efficient tool to assist subjects in brain control tasks.

The use of machine learning tools for modeling the countermeasure environment of robotic means and mobile objects of monitoring

In this article, an analysis of research related to the problems and areas of application of machine learning was carried out and it becomes clear that reinforcement learning is an important tool for the development of highly effective and adaptive control systems. This approach opens perspectives for the development of intelligent systems that can effectively adapt to changes in the environment and provide a reliable and optimal level of control under conditions of uncertainty and variability. In the course of the research, a complex environment modelling technique was developed for the task of searching and chasing moving objects by robotic means as part of reinforcement learning. The methodology includes defining goals and objectives, modelling the environment, defining states, actions, and rewards, formulating the state transition function, defining the reward function, modelling completion criteria, using convolutional neural networks to analyse the environment, and integrating the model into reinforcement learning algorithms. A feature of the developed methodology is its ability to adapt to various environmental conditions and dynamic changes occurring in it. The use of graphical representation of the environment allows you to reproduce in detail the real conditions in which the robotic means operate, including obstacles, areas of visibility and different types of surfaces. The use of convolutional neural networks for environmental analysis is an innovative approach that allows you to automate the process of determining the impact of the environment on the parameters of robotic vehicles and moving objects. CNN training on the collected data about the environment provides accurate prediction of influence coefficients in different areas, which allows optimizing strategies of robot behaviour depending on changes in conditions. In general, the developed environment modelling technique allows not only to accurately reproduce the real conditions in which robotic means operate, but also provides flexibility and adaptability when environmental conditions change. This opens up vast opportunities for training and improving reinforcement learning algorithms, increasing the effectiveness of robots in performing their tasks in various real-world scenarios.

Event-triggered adaptive tracking control of uncertain nonlinear systems with prescribed performance and actuator failures

This article studies adaptive prescribed performance tracking control problem for a class of strict-feedback nonlinear systems with parametric uncertainties and actuator failures. Firstly, in order to compensate the multiple uncertainties and eliminate the influence of actuator failure, a new adaptive tracking controller based on first-order filter technology will be proposed, which simplifies the algorithm design process. Then, by introducing an asymmetric state transition function, the transient and steady performances of the output tracking error are both constrained such that the predetermined performance control goal is achieved. Moreover, to reduce the communication burden from the controller to the actuator, the event-triggered mechanism is designed, and there will be no Zeno phenomenon. Based on Lyapunov stability theory, it is strictly proved that output signal can track the reference signal and all the signals of the closed-loop system are bounded. Finally, a simulation example is performed and the results demonstrate effectiveness of the proposed strategy.

A novel system identification algorithm for nonlinear Markov jump system

System identification of nonlinear Markov jump systems (NMJSs) is crucial in modeling complex systems that contain unknown continuous and discrete states. In this system, the dynamics of continuous states is governed by the discrete states, which means the system can operate among different modes. The existing studies primarily identify the mode transition probability and noise model, whereas the parameters concerning the nonlinear continuous state transition function and observation function are commonly not considered. To identify all the parameters, a novel NMJSs identification algorithm based on a new particle system is proposed. The hidden state inference is implemented via an extended smoother that embodies the new particle system and Rao-Blackwellized particle Gibbs ancestor sampling kernel. The smoother not only estimates hidden state but also enables parameter update of the nonlinear functions. After inferencing the hidden states’ posterior density, a log-likelihood-based loss function is utilized to reduce the uncertainty of identifying the parameters related to the continuous state transition function. Then expectation–maximization and gradient descent are adopted to update the parameters. Finally, different experiments are conducted to verify the proposed algorithm. The results show that the proposed algorithm can approximate the parameter that well describes the data, and outperform other related approaches.

Deep reinforcement learning control for non-stationary building energy management

Developing an optimal supervisory control policy for building energy management is a complex problem because the system exhibits non-stationary behaviors, and the target policy needs to evolve with changes in the state transition and reward functions. Non-stationary real-world problems often present a set of challenges: non-stationary changes are difficult to detect; and thermodynamic systems with larger time-constants can create sample-inefficiency problems for learning algorithms. In addition, the system may have to satisfy safety–critical constraints, and, therefore, the policy must be learned offline unless actuation rules are correctly designed.To address these challenges, we propose a data-driven deep reinforcement learning framework. A reinforcement learning supervisory controller is firstly developed and deployed on the building for the heating, ventilation and air conditioning (HVAC) system and monitored for performance degradation by tracking an aggregate metric. When degradation is detected, a relearning loop is triggered. Then, a set of data-driven models of the building behavior is updated with the latest real data. Subsequently, the deployed controller is re-tuned by letting it interact with the model and is then redeployed on the system.Our proposed approach is demonstrated extensively on the standard ASHRAE 5-zone testbed and a real building. It is benchmarked against state-of-the-art algorithms in building supervisory control: Guideline-36, Proximal Policy Optimization, Deep Deterministic Policy Gradient, and Model Predictive Path Integral control. Our approach performs significantly better than the previously mentioned supervisory control strategies and highlights the need for a condition-based offline relearning framework in dynamic systems.

Robotic Manipulation of Sperm as a Deformable Linear Object

The robotic manipulation of deformable linear objects is a classic and challenging topic. Apart from synthetic objects, such as wires and cables, linear objects are also commonly found in biological cells and organisms. Biomanipulation of such objects is hampered by difficulties, such as limited degrees of freedom of micromanipulators and varied mechanical properties of the biological entities to manipulate. This article presents a robotic manipulation of human sperm, which are deformable cells with a linear shape. The shape and movement of the cell are recapitulated by our developed geometric and kinematic models. Under unfixed constraints between the end-effector and the cell, path planning is designed to update the manipulation point to control cell deformation. A state transition function is formulated in path planning to handle the stiffness variations of sperm without force sensing. A model-predictive controller is designed to minimize the orientation error and manipulation path length. To detect sperm tail for visual feedback, an accuracy of 98% was achieved via deep neural networks. The robotic manipulation of human sperm was performed using a standard clinical setup of a glass micropipette to rotate a sperm to the target orientation. Experimental results showed that robotic sperm rotation achieved an orientation error of 0.8 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^{\circ }$</tex-math></inline-formula> , a tail curvedness of 0.14 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\mu$</tex-math></inline-formula> m <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^{-1}$</tex-math></inline-formula> , and an operation time of 5.6 s, all significantly less than those of the manual approach. The less orientation error and tail curvedness after robotic rotation led to a significantly lower speed of sperm entering the micropipette during sperm aspiration, resulting in a higher success rate of 97% (versus 76% after manual rotation) for aspiration control.

Comprehensive Ocean Information-Enabled AUV Path Planning Via Reinforcement Learning

The path planning of the autonomous underwater vehicle (AUV) has shown great potential in various Internet of Underwater Things (IoUT) applications. Although considerable efforts had been made, prior studies are confronted with some limitations. For one thing, existing work only uses the ocean current simulation model without introducing real ocean information, having not been supported by real data. For another, traditional path planning algorithms have strong environment dependence and lack flexibility: once the environment changes, they need to be remodeled and replanned. To overcome these challenges, this article proposes comprehensive ocean information D3QN (COID), an AUV path planning scheme exploiting comprehensive ocean information and reinforcement learning (RL), which consists of three steps. First, we introduce the comprehensive real ocean data, including weather, temperature, thermohaline, current, etc., and apply them into the regional ocean modeling system to generated reliable ocean current. Next, through well-designed state transition function and reward function, we build a 3-D grid model of ocean environment for RL. Furthermore, based on the framework of the double dueling deep <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula> network (D3QN), COID integrates local ocean current and position features to provide state input and uses priority sampling to accelerate network convergence. The performance of COID has been evaluated and proved by numerical results, which demonstrate efficient path planning and high flexibility for expansion into different ocean environments.

Factors associated with renal function state transitions: A population-based community survey in Taiwan.

BackgroundUnderstanding renal function state transition risk and associated factors in community residences is vital for appropriate preventive and care actions. We aim to investigate factors affecting renal function state transitions through 10-year longitudinal community screening surveys.MethodsThe prospective cohort study included participants who attended the screening program ≥2 times from 2001 to 2009 and were divided into two cohorts: those with baseline estimated glomerular filtration rate (eGFR) ≥60 (n = 46,278) and those with eGFR 59–30 mL/min/1.73 m2 (n = 4,656). We applied the illness-death model to identify associated factors with eGFR <60 and death for the cohort with baseline eGFR ≥60 and eGFR <30 and death for that with baseline eGFR ≥59–30.ResultsAmong the followed-up participants, 3,018 (6.5%) in the cohort of baseline eGFR ≥60 mL/min/1.73 m2 and 322 (6.9%) in the cohort of eGFR 59–30 mL/min/1.73 m2 experienced renal function state transition during a median over 7-year follow-up. Besides eGFR and grade of proteinuria, diabetes mellitus (adding nearly 50% hazard rate) is the main factor associated with both state transitions. Other early-phase eGFR state transition risk factors were metabolic syndrome score, triglyceride, uric acid, fasting blood sugar, and high-density lipoprotein cholesterol. Males, poor hemoglobin, high triglyceride, and high low-density lipoprotein cholesterol were all linked with the late-phase eGFR state transition hazard rate.ConclusionThe study developed the state transition functions for community participants with varying renal function levels. Further actions to develop precision screening plans and services that incorporate personal risk factors and state transition risks are necessary.

Dynamic evaluation and analysis of the uncertainty of roundness error measurement by Markov Chain Monte Carlo method

Measurement uncertainty is the necessary information to accurately describe the roundness error. This paper presents a dynamic evaluation model of Markov chain Monte Carlo(MCMC) roundness error measurement uncertainty based on stochastic process. The model regards the process of obtaining evaluation parameters as a stochastic process. The model is sampled using the MCMC method. By calculating the state transition function of the MCMC model, the autocorrelation characteristics of the parameters under the random process are reflected. In order to verify the accuracy and stability of the stochastic process-based MCMC model, a comparison between high-precision and low-precision measurements was carried out. The results show that compared with the traditional GUM method and Monte Carlo method, the MCMC method is consistent with the results of GUM method and Monte Carlo method. The MCMC method based on the stochastic process realizes the dynamic evaluation of the roundness error measurement uncertainty, which can obtain accurate results, run stably and improve the accuracy of the evaluation.

Remaining Useful Life Prediction of Lithium-Ion Batteries Using Neural Networks with Adaptive Bayesian Learning.

With smart electronic devices delving deeper into our everyday lives, predictive maintenance solutions are gaining more traction in the electronic manufacturing industry. It is imperative for the manufacturers to identify potential failures and predict the system/device’s remaining useful life (RUL). Although data-driven models are commonly used for prognostic applications, they are limited by the necessity of large training datasets and also the optimization algorithms used in such methods run into local minima problems. In order to overcome these drawbacks, we train a Neural Network with Bayesian inference. In this work, we use Neural Networks (NN) as the prediction model and an adaptive Bayesian learning approach to estimate the RUL of electronic devices. The proposed prognostic approach functions in two stages—weight regularization using adaptive Bayesian learning and prognosis using NN. A Bayesian framework (particle filter algorithm) is adopted in the first stage to estimate the network parameters (weights and bias) using the NN prediction model as the state transition function. However, using a higher number of hidden neurons in the NN prediction model leads to particle weight decay in the Bayesian framework. To overcome the weight decay issues, we propose particle roughening as a weight regularization method in the Bayesian framework wherein a small Gaussian jitter is added to the decaying particles. Additionally, weight regularization was also performed by adopting conventional resampling strategies to evaluate the efficiency and robustness of the proposed approach and to reduce optimization problems commonly encountered in NN models. In the second stage, the estimated distributions of network parameters were fed into the NN prediction model to predict the RUL of the device. The lithium-ion battery capacity degradation data (CALCE/NASA) were used to test the proposed method, and RMSE values and execution time were used as metrics to evaluate the performance.

Locally connected interrelated network: A forward propagation primitive

End-to-end learning for planning is a promising approach for finding good robot strategies in situations where the state transition, observation, and reward functions are initially unknown. Many neural network architectures for this approach have shown positive results. Across these networks, seemingly small components have been used repeatedly in different architectures, which means improving the efficiency of these components has great potential to improve the overall performance of the network. This paper aims to improve one such component: The forward propagation module. In particular, we propose Locally Connected Interrelated Network (LCI-Net) – a novel type of locally connected layer with unshared but interrelated weights – to improve the efficiency of learning stochastic transition models for planning and propagating information via the learned transition models. LCI-Net is a small differentiable neural network module that can be plugged into various existing architectures. For evaluation purposes, we apply LCI-Net to VIN and QMDP-Net. VIN is an end-to-end neural network for solving Markov Decision Processes (MDPs) whose transition and reward functions are initially unknown, while QMDP-Net is its counterpart for the Partially Observable Markov Decision Process (POMDP) whose transition, observation, and reward functions are initially unknown. Simulation tests on benchmark problems involving 2D and 3D navigation and grasping indicate promising results: Changing only the forward propagation module alone with LCI-Net improves VIN’s and QMDP-Net generalisation capability by more than 3× and 10×, respectively.

Replication study challenges and new number formats for chaotic pseudo random number generators

AbstractChaotic Pseudo Random Number Generators have been seen as a promising candidate for secure random number generation. Using the logistic map as state transition function, we perform number generation experiments that illustrate the challenges when trying to do a replication study. Those challenges range from uncertainties about the rounding mode in arithmetic hardware over chosen number representations for variables to compiler or programmer decisions on evaluation order for arithmetic expressions. We find that different decisions lead to different streams with different security properties, where we focus on period length. However, descriptions in articles often are not detailed enough to deduce all decisions unambiguously. To address similar problems in other replication studies for security applications, we propose recommendations for descriptions of numerical experiments on security applications to avoid the above challenges. Moreover, we use the results to propose the use of higher-radix and mixed-radix representations to trade storage size for period length, and investigate if exploiting the symmetry of the logistic map function for number representation is advantageous.

A new modeling framework for networked discrete-event systems

This paper proposes a new framework for modeling networked discrete-event systems (DES) with channel delays and losses. We construct a more elaborated model for the plant G and supervisor S by endowing them with an embedded module to preprocess the received messages and defining state transition mapping and output mapping to model the dynamics of G and S, which could provide users with flexibility to realize different mechanisms, and thus could potentially generate more permissive controllers. Two sets of new formulations are established to capture the temporal relationships between channel inputs and outputs without assuming the first-in-first-out (FIFO) property, which naturally contains the model without delays and losses in the Ramadge–Wonham framework. Based on the defined mappings of each component, the closed-loop behavior is defined to describe the concurrent system dynamics. We hope the new framework could serve as an analysis tool, and possibly the building block for a synthesis tool, for future theoretical explorations and practical implementations of networked DES control.

Visual Foresight Trees for Object Retrieval From Clutter With Nonprehensile Rearrangement

This paper considers the problem of retrieving an object from many tightly packed objects using a combination of robotic pushing and grasping actions. Object retrieval in dense clutter is an important skill for robots to operate in households and everyday environments effectively. The proposed solution, Visual Foresight Trees (VFT), intelligently rearranges the clutter surrounding a target object so that it can be grasped easily. Rearrangement with nested nonprehensile actions is challenging as it requires predicting complex object interactions in a combinatorially large configuration space of multiple objects. We first show that a deep neural network can be trained to accurately predict the poses of the packed objects when the robot pushes one of them. The predictive network provides visual foresight and is used in a tree search as a state transition function in the space of scene images. The tree search returns a sequence of consecutive push actions yielding the best arrangement of the clutter for grasping the target object. Experiments in simulation and using a real robot and objects show that the proposed approach outperforms model-free techniques as well as model-based myopic methods both in terms of success rates and the number of executed actions, on several challenging tasks. A video introducing VFT, with robot experiments, is accessible at https://youtu.be/7cL-hmgvyec. The full source code is available at https://github.com/arc-l/vft.

Resolvable Group Target Tracking via Multi-Bernoulli Filter and Its Application to Sensor Control Scenario

In this paper, we address the problem of resolvable group target tracking (RGTT) and the application of RGTT in the sensor control scenarios. The aim of our approach is to further improve the tracking performance of multi-Bernoulli (MB) filter by exploiting the group structure information. Firstly, a single-target state transition function (SSTF) is derived from the direct integral solution of the single-target state stochastic differential equation (SDE), which is used to revise the prediction process of the standard MB filter. Then, combing the Gaussian mixture (GM) implementation, the prediction process of the proposed SSTF-based MB (SSTF-MB) filter is detailed. Next, considering the benefits and importance of group structure in the sensor control scenarios, the proposed SSTF-MB filter is applied into a mobile sensor platform whose control actions (e.g., motion direction) are uncertain. Specifically, the proposed SSTF-MB filter is merged into a sensor control strategy where the Cauchy-Schwarz (CS) divergence is selected as the objective function to compute the discrepancy between two first-order moment approximations of MBs. Finally, the efficiency and performance of the proposed SSTF-MB filter and its application to sensor control scenario is well demonstrated in the simulation experiments.

On the effectiveness of Gated Echo State Networks for data exhibiting long-term dependencies

In the context of recurrent neural networks, gated architectures such as the GRU have contributed to the development of highly accurate machine learning models that can tackle long-term dependencies in the data. However, the training of such networks is performed by the expensive algorithm of gradient descent with backpropagation through time. On the other hand, reservoir computing approaches such as Echo State Networks (ESNs) can produce models that can be trained efficiently thanks to the use of fixed random parameters, but are not ideal for dealing with data presenting long-term dependencies. We explore the problem of employing gated architectures in ESNs from both theoretical and empirical perspectives. We do so by deriving and evaluating a necessary condition for the non-contractivity of the state transition function, which is important to overcome the fading-memory characterization of conventional ESNs. We find that using pure reservoir computing methodologies is not sufficient for effective gating mechanisms, while instead training even only the gates is highly effective in terms of predictive accuracy.