A Proposal for a State Space Secretariat: Why Does It Make Sense for the State of São Paulo?

An increasing number of system development projects are concerned with distributed and concurrent systems. There are numerous examples, ranging from large scale systems, in the areas of telecommunication and applications based on WWW technology, to medium or small scale systems, in the area of embedded systems. A typical distributed or concurrent system consists of a number of independent but communicating processes. This means that the execution of such systems may proceed in many different ways, e.g., depending on whether messages are lost, the speed of the processes involved, and the time at which input is received from the environment. As a result, distributed and concurrent systems are, by nature, complex and difficult to design and test. This has motivated the development of methods which support computer-aided analysis, validation, and verification of the behaviour of concurrent and distributed systems. <br /> State space methods are some of the most prominent approaches in this field. The basic idea underlying state spaces is (in its simplest form) to compute all reachable states and state changes of the system, and represent these as a directed graph. The virtue of a constructed state space is that it makes it possible to algorithmically reason about the behaviour of a system, e.g., verify that the system possesses certain desired properties or locate errors in the system. The main disadvantage of using state spaces is the state explosion problem: even relatively small descriptions/systems may have an astronomically or even infinite number of reachable states, and it is a serious limitation on the use of state space methods in the analysis of real-life systems. The development of reduction methods to alleviate this inherent complexity problem is, therefore, a central topic in the development of state space methods. Reduction methods avoid representing the entire state space of the system or represent the state space in a compact form. The reduction is done in such a way that properties of the system can still be derived from the reduced state space. <br /> In this thesis we study state space methods in the framework of Coloured Petri Nets which is a graphical language for modelling and analysis of concurrent and distributed systems. The thesis consists of two parts. Part I is the mandatory overview paper which summarises the work which has been done. Part II is composed of five individual papers and constitutes the core of this thesis. Four of the these papers have been published elsewhere as conference papers, journal papers, or book chapters. <br /> The overview paper introduces the research field of state space methods for Coloured Petri Nets and summarises the contents and contributions of the five individual papers. A substantial part of the overview paper has also been devoted to putting the results presented in the five individual papers in a broader perspective in the form of a discussion of related work. <br /> The first paper considers state space analysis of Coloured Petri Nets. It is well-known that almost all dynamic properties of the considered system can be verified when the state space is finite. However, state space analysis is more than just formulating a set of formal requirements and invoking a corresponding set of queries. State space analysis is also applicable during the design and debugging of a system. An approach towards this is to allow the user to analyse the behaviour of systems by drawing and generating selected parts of the state space. The paper presents a tool in which formal verification, partial state spaces, and analysis by means of graphical feedback and simulation are integrated entities. The focus of the paper is twofold: the support for graphical feedback and the way it has been integrated with simulation, and the underlying algorithms and datastructures which support computation and storage of state spaces and which exploit the hierarchical structure of the models. <br /> The second paper presents a computer tool for verification of distributed systems. The tool implements the method of state spaces with symmetries. The basic idea in the approach is to exploit the symmetries inherent in many distributed systems in order to construct a condensed state space. As an example, the correctness of Lamport's Fast Mutual Exclusion Algorithm is established. We demonstrate a significant increase in the number of states which can be analysed. State spaces with symmetries is not our invention. Our contribution is the development of the tool and verification of the example, demonstrating how the method of state spaces with symmetries can be put into practical use. <br /> The third paper demonstrates the potential of verification based on state spaces reduced by equivalence relations. The basic observation is that quite often some states of a system are similar, i.e., they induce similar behaviours. Similarity can be formally expressed by defining an equivalence relation on the set of states and on the set of actions of a system under consideration. A state space can be constructed in which the nodes correspond to equivalence classes of states, and the arcs correspond to equivalence classes of actions. Such a state space is often much smaller than the ordinary, full state space, but it does allow derivation of many verification results. State spaces with equivalence classes is not our invention. The contribution of the paper is the specification of a concrete notion of equivalence, and a demonstration of its application for verification of a communication protocol. Aided by a developed computer tool significant reductions of state spaces are exhibited, representing some first results on the practical use of state spaces with equivalence classes for Coloured Petri Nets. Exploiting the symmetries in systems induce a certain kind of equivalence. The verification of the communication protocol demonstrates the potential provided by more general notions of equivalence. <br /> The fourth paper addresses the issue of using the stubborn set method for Coloured Petri Nets without relying on unfolding to the equivalent Place/Transition Net. The stubborn set method exploits the independence between actions to avoid representing all possible interleavings of the system execution. We give a lower bound result which states that there exist Coloured Petri Nets for which computing good stubborn sets requires time proportional to the size of the equivalent Place/Transition Net. We suggest an approximative method for computing stubborn sets of so-called process-partitioned Coloured Petri Nets which does not rely on unfolding. The underlying idea is to add some structure to the Coloured Petri Net, which can be exploited during the stubborn set construction to avoid the unfolding. The practical applicability of the method is demonstrated with both theoretical and experimental case studies, in which reduction of the state space, as well as savings in time, are obtained. <br /> The fifth paper presents two new question-guided stubborn set methods for state properties. The first method makes it possible to determine whether a state is reachable in which a given state property holds. It generalises earlier results on stubborn sets for state properties in the sense that the earlier methods can be seen as an implementation of our more general method. We also propose alternative, more powerful implementations that have the potential of leading to better reduction results. This potential is demonstrated on some practical case studies. As an extension of the first method, we present a novel method which can be used to determine if it is always possible to reach a state where a given state property holds. Compared to earlier methods the benefit is again in the potential for better reduction results. omputer-aided analysis, validation, and verification of the behaviour of concurrent and distributed systems.

This paper presents a new adaptive segmentation of continuous state space based on vector quantization algorithm such as Linde---Buzo---Gray for high-dimensional continuous state spaces. The objective of adaptive state space partitioning is to develop the efficiency of learning reward values with an accumulation of state transition vector in a single-agent environment. We constructed our single-agent model in continuous state and discrete actions spaces using Q-learning function. Moreover, the study of the resulting state space partition reveals a Voronoi tessellation. In addition, the experimental results show that this proposed method can partition the continuous state space appropriately into Voronoi regions according to not only the number of actions, but also achieve a good performance of reward-based learning tasks compared with other approaches such as square partition lattice on discrete state space.

In this chapter we study the finite-state approximation problem for computing near optimal policies for discrete-time MDPs with Borel state and action spaces, under discounted and average costs criteria. Even though existence and structural properties of optimal policies of MDPs have been studied extensively in the literature, computing such policies is generally a challenging problem for systems with uncountable state spaces. This situation also arises in the fully observed reduction of a partially observed Markov decision process even when the original system has finite state and action spaces. Here we show that one way to compute approximately optimal solutions for such MDPs is to construct a reduced model with a new transition probability and one-stage cost function by quantizing the state space, i.e., by discretizing it on a finite grid. It is reasonable to expect that when the one-stage cost function and the transition probability of the original model has certain continuity properties, the cost of the optimal policy for the approximating finite model converges to the optimal cost of the original model as the discretization becomes finer. Moreover, under additional continuity conditions on the transition probability and the one stage cost function we also obtain bounds on the accuracy of the approximation in terms of the number of points used to discretize the state space, thereby providing a tradeoff between the computation cost and the performance loss in the system. In particular, we study the following two problems.

Safe deployment of a reinforcement learning robot using self stabilization

In this paper state space and circuit realizations are presented for tridiagonal discrete state space 2D filters. The proposed state space realizations are based on the corresponding circuit implementations. For the state space realization the 2D Roesser and Cyclic models are used. The dimension of the state vectors are minimal for both models. A low-order example is presented to illustrate the proposed results.

Robot learning such as reinforcement learning generally needs a well-defined state space in order to converge. However, to build such a state space is one of the main issues of the robot learning because of the inter-dependence between state and action spaces, which resembles to the well known "chicken and egg" problem. This paper proposes a method of action-based state space construction for vision-based mobile robots. Basic ideas to cope with the inter-dependence are that we define a state as a cluster of input vectors from which the robot can reach the goal state or the state already obtained by a sequence of one kind action primitive regardless of its length, and that this sequence is defined as one action. To realize these ideas, we need many data (experiences) of the robot and cluster the input vectors as hyper ellipsoids so that the whole state space is segmented into a state transition map in terms of action from which the optimal action sequence is obtained. To show the validity of the method, we apply it to a soccer robot which tries to shoot a ball into a goal. The simulation and real experiments are shown.

Robot learning, such as reinforcement learning, generally needs a well-defined state space in order to converge. However, building such a state space is one of the main issues of robot learning because of the interdependence between state and action spaces, which resembles the well-known "chicken and egg" problem. This article proposes a method of action-based state space construction for vision-based mobile robots. Basic ideas to cope with the interdependence are that we define a state as a cluster of input vectors from which the robot can reach the goal state or the state already obtained by a sequence of one kind of action primitive regardless of its length, and that this sequence is defined as one action. To realize these ideas, we need many data (experiences) of the robot and we must cluster the input vectors as hyper ellipsoids so that the whole state space is segmented into a state transition map in terms of action from which the optimal action sequence is obtained. To show the validity of the method, we apply it to a soccer robot that tries to shoot a ball into a goal. The simulation and real experiments are shown.

Robot learning such as reinforcement learning generally needs a well-defined state space in order to converge. However, to build such a state space is one of the main issues of the robot learning because of the inter-dependence between state and action spaces, which resembles to the well known“chicken and egg”problem. This paper proposes a method of action-based state space construction for vision-based mobile robots. Basic ideas to cope with the interdependence are that we define a state as a cluster of input vectors from which the robot can reach the goal state or the state already obtained by a sequence of one kind action primitive regardless of its length, and that this sequence is defined as one action. To realize these ideas, we need many data (experiences) of the robot and cluster the input vectors as hyper ellipsoids so that the whole state space is segmented into a state transition map in terms of action from which the optimal action sequence is obtained. To show the validity of the method, we apply it to a soccer robot which tries to shoot a ball into a goal. The simulation and real experiments are shown.

One of the conditions for the convergence of Q-Learning is to visit each state-action pair infinitely (or at least sufficiently) often. This requirement raises problems for large or continuous state spaces. Particularly, in continuous state spaces a discretization sufficiently fine to cover all relevant information usually results in an extremely large state space. In order to speed up and improve learning it is highly beneficial to add generalization to Q-Learning and thus being able to exploit experiences gained earlier. To achieve this, we compute a state space abstraction with a combination of growing neural gas and Q-Learning. This abstraction respects similarity in the state and action space and is constructed based on information achieved from interaction with the environment during learning. We examine the proposed algorithm on a continuous-state reinforcement learning problem and show that the approximated state space and the generalization speed up learning.

The planning process is modelled as an information processing system. The inputs to the system comprise observations of the state of the environment, and the outputs are goals and plans to guide actions in the environment. The processor consists of two interacting subsystems: the one carrying descriptions about material states is called the state space and the other holding information about decisions and values is called the decision space. The memory of the planning system stores its history: its ‘experiences’ both of its external and of its internal environments. The ‘facts’ which inform the planning system are represented as a relation between the state space and the decision space. Such a relation induces a pair of functions which is known to form a Galois connection between two sets. Compositions of these two functions are closure operators which define closure systems on the decision space and the state space, respectively. Each of these functions when restricted to the closure systems is an isomorphism. Isomorphic elements are paired together and the collection of such pairs forms a Galois lattice. A Galois lattice presents to the planners an overall view of the current ‘situation’ of the environment. Each element of the Galois lattice relates a set of ‘systems’ represented in the state space to a set of ‘values’ used to evaluate ‘systems’ in the decision space. The planning process is then viewed as a transformation of the Galois lattice which represents the current ‘situation’ to another Galois lattice which represents the desirable ‘situation’.

The recent literature contains several papers which explore mathematical programming formulations of particular Markov sequential decision problems. Each of these papers deals with finite state and action spaces; thus, the corresponding programming formulations yield dual finite linear programs. In this paper these investigations are extended to include countable action and/or state spaces for finite horison problems. Of particular interest are the duality aspects of the mathematical programming formulations. In addition, employing conditions analogous to fundamental concepts of Haar semi-infinite dual programming, we provide sufficient conditions for the existence of optimal rules for countable action spaces. Guided by the semi-infinite duality theory we explore mathematical programming formulations for two cases: 1) Countable action space and finite state space—the result is a pair of dual semi-infinite programs; and 2) Finite action space and countable state space—we obtain a pair of infinite programs. In the latter case we show that no duality gap occurs and obtain duality results comparable to those of finite linear programming.

In examples of reinforcement learning where state space is continuous, it seems impossible to use reference tables to store value-action .In these problems a method is required for value estimation for each state-action pair .The inputs to this estimation system are (characteristics of) state variables which reflect the status of agent in the environment .The system can be either linear of nonlinear .For each member in set of actions of an agent, there exists an estimation system which determines state value for the action . On the other hand, in most real world problems, just as the state space is continuous, so is the action space for an agent .In these cases, fuzzy systems may provide a useful solution in selection of final action from action space .In this paper we intend to combine reinforcement learning algorithm with fuzzified actions and state space along with a linear estimation system into an intelligent systems for parking Trailers in cases where both state and action spaces are continuous .Finally, the successful performance of the proposed algorithm is shown through simulations on trailer parking problem .

The uncertain stability of deep reinforcement learning training on complex tasks impedes its development and deployment, especially in intelligent vehicles, such as intelligent surface vessels and self-driving cars. Complex and varied environmental states puzzle training of decision-making networks. Here we propose an elastic adaptive deep reinforcement learning algorithm to address these challenges and achieve autonomous navigation in intelligent vehicles. Our method trains the decision-making network over the function and optimization learning stages, in which the state space and action space of autonomous navigation tasks are pruned by choosing classic states and actions to reduce data similarity, facilitating more stable training. We introduce a task-adaptive observed behaviour classification technique in the function learning stage to divide state and action spaces into subspaces and identify classic states and actions. In which the classic states and actions are accumulated as the training dataset that enhances its training efficiency. In the subsequent optimization learning stage, the decision-making network is refined through meticulous exploration and accumulation of datasets. The proposed elastic adaptive deep reinforcement learning enables the decision-making network to effectively learn from complex state and action spaces, leading to more efficient training compared to traditional deep reinforcement learning approaches. Simulation results demonstrate the remarkable effectiveness of our method in training decision-making networks for intelligent vehicles. The findings validate that our method provides reliable and efficient training for decision-making networks in intelligent vehicles. Moreover, our method exhibits stability in training other tasks characterized by continuous state and action spaces.

Reinforcement learning (RL) attracts much attention as a technique for realizing computational intelligence such as adaptive and autonomous decentralized systems. In general, however, it is not easy to put RL to practical use. The difficulty includes the problem of designing suitable state and action spaces for an agent. Previously, we proposed an adaptive state space construction method which is called a "state space filter," and an adaptive action space construction method which is called "switching RL," after the other space has been fixed. In this article, we reconstitute these two construction methods as one method by treating the former and the latter as a combined method for mimicking an infant's perceptual development. In this method, perceptual differentiation progresses as an infant become older and more experienced, and the infant's motor development, in which gross motor skills develop before fine motor skills, also progresses. The proposed method is based on introducing and referring to "entropy." In addition, a computational experiment was conducted using a so-called "path planning problem" with continuous state and action spaces. As a result, the validity of the proposed method has been confirmed.

Classifier fusion remains an effective method to improve classification performance. In applications, the classifiers learnt using different attributes may work with various frames of discernment (FoD) of classification. There generally exist more or less complementary knowledge among these classifiers. However, how to efficiently combine such classifiers under different FoD is a challenging problem. In this article, we propose a new method for classifier fusion with different FoD based on the belief functions, which allow to well represent and deal with uncertain information. The credal transformation rules are developed to map the various FoD into a common one. It allows to transfer the probability (or mass of belief) of one class in the given FoD not only to several singleton classes but also to the metaclasses (i.e., disjunction of several classes) and the ignorance in other chosen FoD according to a transformation matrix, which is estimated based on the training (pairwise) data by minimizing a certain error criteria. Thus, we can well characterize the uncertainty and imprecision during the transformation of FoD. After that, the outputs of different classifiers represented by basic belief assignments (BBAs) can be transformed to a common FoD. Then, the well-known Dempster's rule is employed to combine these transformed BBA to obtain final classification result under the chosen FoD. Several real data sets are used in the experiment to evaluate the performance of the proposed method. Our experimental results show that this new method can efficiently improve the classification accuracy with respect to other related methods.