Maze Problem Research Articles

Inverse Reinforcement Learning with Agents’ Biased Exploration Based on Sub-Optimal Sequential Action Data

Inverse reinforcement learning (IRL) estimates a reward function for an agent to behave along with expert data, e.g., as human operation data. However, expert data usually have redundant parts, which decrease the agent’s performance. This study extends the IRL to sub-optimal action data, including lack and detour. The proposed method searches for new actions to determine optimal expert action data. This study adopted maze problems with sub-optimal expert action data to investigate the performance of the proposed method. The experimental results show that the proposed method finds optimal expert data better than the conventional method, and the proposed search mechanisms perform better than random search.

Deep Q-learning with hybrid quantum neural network on solving maze problems

Quantum computing holds great potential for advancing the limitations of machine learning algorithms to handle higher dimensions of data and reduce overall training parameters in deep learning (DL) models. This study uses a trainable variational quantum circuit (VQC) on a gate-based quantum computing model to investigate the potential for quantum benefit in a model-free reinforcement learning problem. Through a comprehensive investigation and evaluation of the current model and capabilities of quantum computers, we designed and trained a novel hybrid quantum neural network based on the latest Qiskit and PyTorch framework. We compared its performance with a full-classical CNN with and without an incorporated VQC. Our research provides insights into the potential of deep quantum learning to solve a maze problem and, potentially, other reinforcement learning problems. We conclude that reinforcement learning problems can be practical with reasonable training epochs. Moreover, a comparative study of full-classical and hybrid quantum neural networks is discussed to understand these two approaches’ performance, advantages, and disadvantages to deep Q-learning problems, especially on larger-scale maze problems larger than 4times 4.

Bayesian reinforcement learning for navigation planning in unknown environments.

This study focuses on a rescue mission problem, particularly enabling agents/robots to navigate efficiently in unknown environments. Technological advances, including manufacturing, sensing, and communication systems, have raised interest in using robots or drones for rescue operations. Effective rescue operations require quick identification of changes in the environment and/or locating the victims/injuries as soon as possible. Several techniques have been developed in recent years for autonomy in rescue missions, including motion planning, adaptive control, and more recently, reinforcement learning techniques. These techniques rely on full knowledge of the environment or the availability of simulators that can represent real environments during rescue operations. However, in practice, agents might have little or no information about the environment or the number or locations of injuries, preventing/limiting the application of most existing techniques. This study provides a probabilistic/Bayesian representation of the unknown environment, which jointly models the stochasticity in the agent's navigation and the environment uncertainty into a vector called the belief state. This belief state allows offline learning of the optimal Bayesian policy in an unknown environment without the need for any real data/interactions, which guarantees taking actions that are optimal given all available information. To address the large size of belief space, deep reinforcement learning is developed for computing an approximate Bayesian planning policy. The numerical experiments using different maze problems demonstrate the high performance of the proposed policy.

Rosenblatt’s First Theorem and Frugality of Deep Learning

The Rosenblatt's first theorem about the omnipotence of shallow networks states that elementary perceptrons can solve any classification problem if there are no discrepancies in the training set. Minsky and Papert considered elementary perceptrons with restrictions on the neural inputs: a bounded number of connections or a relatively small diameter of the receptive field for each neuron at the hidden layer. They proved that under these constraints, an elementary perceptron cannot solve some problems, such as the connectivity of input images or the parity of pixels in them. In this note, we demonstrated Rosenblatt's first theorem at work, showed how an elementary perceptron can solve a version of the travel maze problem, and analysed the complexity of that solution. We also constructed a deep network algorithm for the same problem. It is much more efficient. The shallow network uses an exponentially large number of neurons on the hidden layer (Rosenblatt's A-elements), whereas for the deep network, the second-order polynomial complexity is sufficient. We demonstrated that for the same complex problem, the deep network can be much smaller and reveal a heuristic behind this effect.

Simulation and Implementation of a Mobile Robot Trajectory Planning Solution by Using a Genetic Micro-Algorithm

Robots able to roll and jump are used to solve complex trajectories. These robots have a low level of autonomy, and currently, only teleoperation is available. When researching the literature about these robots, limitations were found, such as a high risk of damage by testing, lack of information, and nonexistent tools. Therefore, the present research is conducted to minimize the dangers in actual tests, increase the documentation through a platform repository, and solve the autonomous trajectory of a maze with obstacles. The methodology consisted of: replicating a scenario with the parrot robot in the gazebo simulator; then the computational resources, the mechanism, and the available commands of the robot were studied; subsequently, it was determined that the genetic micro-algorithm met the minimum requirements of the robot; in the last part, it was programmed in simulation and the solution was validated in the natural environment. The results were satisfactory and it was possible to create a parrot robot in a simulation environment analogous to the typical specifications. The genetic micro-algorithm required only 100 generations to converge; therefore, the demand for computational resources did not affect the execution of the essential tasks of the robot. Finally, the maze problem could be solved autonomously in a real environment from the simulations with an error of less than 10% and without damaging the robot.

Controller-Based Interactive Proof to Ensure Trust in Multiagent Systems

When multiple agents need to perform tasks that need cooperation, communication is necessary for coordination and also to assess completion. In this case, trust needs to be guaranteed, so agents know if others are following a common strategy. If communication is explicit, other agents, including agents that are not involved directly in the mission, can forge information and lead agents to believe that they are behaving according to the mission plan. Several methods have been proposed in the literature, and in this article, we propose a novel method based on interactive proofs in which trust between two agents is asserted by one agent demonstrating to the other that both follow the same mission plan. We formally represent the problem as a series of Markov decision processes and demonstrate that the formulation is complete and sound. Simulations of multiple agents in a maze problem demonstrate the behavior of the system.

Multi-Faceted Decision Making Using Multiple Reinforcement Learning to Reducing Wasteful Actions

Reinforcement learning can lead to autonomous behavior depending on the environment. However, in complex and high-dimensional environments, such as real environments, a large number of trials are required for learning. In this paper, we propose a solution for the learning problem using local learning to select an action based on the surrounding environmental information. Simulation experiments were conducted using maze problems, pitfall problems, and environments with random agents. The actions that did not contribute to task accomplishment were compared between the proposed method and ordinary reinforcement learning method.

Quantum reinforcement learning: the maze problem

Quantum machine learning (QML) is a young but rapidly growing field where quantum information meets machine learning. Here, we will introduce a new QML model generalising the classical concept of reinforcement learning to the quantum domain, i.e. quantum reinforcement learning (QRL). In particular, we apply this idea to the maze problem, where an agent has to learn the optimal set of actions in order to escape from a maze with the highest success probability. To perform the strategy optimisation, we consider a hybrid protocol where QRL is combined with classical deep neural networks. In particular, we find that the agent learns the optimal strategy in both the classical and quantum regimes, and we also investigate its behaviour in a noisy environment. It turns out that the quantum speedup does robustly allow the agent to exploit useful actions also at very short time scales, with key roles played by the quantum coherence and the external noise. This new framework has the high potential to be applied to perform different tasks (e.g. high transmission/processing rates and quantum error correction) in the new-generation noisy intermediate-scale quantum (NISQ) devices whose topology engineering is starting to become a new and crucial control knob for practical applications in real-world problems. This work is dedicated to the memory of Peter Wittek.

Path discovering in maze area using mobile robot

Robotic maze pathfinding problems deal with detecting the correct route from the start point to the end-point in a virtual maze environment consisting of walls. Automated robot mobility is a significant feature, which enables a mobile robot to traverse a maze independently, from one position to another, without human intervention. There is a myriad of autonomous industrial mobile robot applications, including the transportation of goods and parts, domestic cleaning, indoor security surveillance, airport baggage couriering, and a plethora of other applications to traverse dangerous locations. This paper proposes a pathfinding mobile robot in a virtual maze based on a combination of a simplified left-hand algorithm and a line-following control algorithm. The mobile robot works in any maze to determine a route from the initial starting point to the end-point. The approach outlined in this paper uses a left-hand algorithm to solve the maze problem and a line-follower control algorithm to enable the robot to move in a straight line through the virtual maze. The algorithm used is less complicated and prevents the robot from falling into infinity loops compared to the traditional wall-follower algorithm.

Recombination and Novelty in Neuroevolution: A Visual Analysis

Neuroevolution has re-emerged as an active topic in the last few years. However, there is a lack of accessible tools to analyse, contrast and visualise the behaviour of neuroevolution systems. A variety of search strategies have been proposed such as Novelty search and Quality-Diversity search, but their impact on the evolutionary dynamics is not well understood. We propose using a data-driven, graph-based model, search trajectory networks (STNs) to analyse, visualise and directly contrast the behaviour of different neuroevolution search methods. Our analysis uses NEAT for solving maze problems with two search strategies: novelty-based and fitness-based, and including and excluding the crossover operator. We model and visualise the trajectories, contrasting and illuminating the behaviour of the studied neuroevolution variants. Our results confirm the advantages of novelty search in this setting, but challenge the usefulness of recombination.

Exploration of Applying Lego NXT and Arduino in Situated Engineering Teaching: A Case Study of a Robotics Contest at King Saud University

Saudi students in engineering courses suffer from a lack of science, technology, engineering, and mathematics (STEM) knowledge due to the teaching philosophy and programs offered in secondary and intermediate schools. This weakness naturally impacts their motivation, grades, and their relationship with teachers. In this paper, we introduce a new experimental teaching experience in the Applied Mechanical Department of the Applied Engineering College of King Saud University, based on situated learning theory, which emphasizes that knowledge must be learned in constructed situational context). This experience involved introducing Lego NXT and Arduino to enhance the enthusiasm and interest of students through designing and building robots in agreement with the “Introduction to design” course syllabus. Two experimental challenges were associated with: the line-follower problem and the maze problem. These challenges took the form of an internal completion at the end of each semester. The experience was conducted for two consecutive terms (30 students, the academic year 2019-2020), and the results were compared to those in six previous terms (100 students, academic years from 2016 to 2019). The experimental group demonstrated grades improvement (course mean grade rose from 77.1 to 85.3), the progress of academic achievement, and interest that enables them to actively explore and construct knowledge.

A learning search algorithm with propagational reinforcement learning

When reinforcement learning with a deep neural network is applied to heuristic search, the search becomes a learning search. In a learning search system, there are two key components: (1) a deep neural network with sufficient expression ability as a heuristic function approximator that estimates the distance from any state to a goal; (2) a strategy to guide the interaction of an agent with its environment to obtain more efficient simulated experience to update the Q-value or V-value function of reinforcement learning. To date, neither component has been sufficiently discussed. This study theoretically discusses the size of a deep neural network for approximating a product function of p piecewise multivariate linear functions. The existence of such a deep neural network with O(n + p) layers and O(dn + dnp + dp) neurons has been proven, where d is the number of variables of the multivariate function being approximated, 𝜖 is the approximation error, and n = O(p + log2(pd/𝜖)). For the second component, this study proposes a general propagational reinforcement-learning-based learning search method that improves the estimate h(.) according to the newly observed distance information about the goals, propagates the improvement bidirectionally in the search tree, and consequently obtains a sequence of more accurate V-values for a sequence of states. Experiments on the maze problems show that our method increases the convergence rate of reinforcement learning by a factor of 2.06 and reduces the number of learning episodes to 1/4 that of other nonpropagating methods.

FAOD: Fast Automatic Option Discovery in Hierarchical Reinforcement Learning

The hierarchical reinforcement learning framework breaks down the reinforcement learning problem into subtasks or extended actions called options in order to facilitate its resolution. Different models have been proposed where options were manually predefined or semi-automatically discovered. However, the automatic discovery of options has become a real challenge for research in hierarchical reinforcement learning and the new proposed approaches are very greedy in learning time or space. Thus we opt for a faster and less consuming approach. In this paper we propose an automatic option discovery method for hierarchical reinforcement learning, that we call FAOD (Fast Automatic Option Discovery). We take inspiration from robot learning methods to categorize the sensorimotor flow during navigation. Here, the agent moves along the walls to discover the rooms’ contour, closed spaces, doors and bottleneck regions to define terminate states for options. The FAOD method is evaluated on different classical maze problems, demonstrating fast and promising results.

A Scale Balanced Loss for Bounding Box Regression

Object detectors typically use bounding box regressors to improve the accuracy of object localization. Currently, the two types of bounding box regression loss are $\ell _{n}$ -norm-based and intersection over union ( $IoU$ )-based. However, we found that these two types of losses have their drawbacks. First, for $\ell _{n}$ -norm-based loss, large-scale objects are more likely to obtain a larger penalty than the smaller ones when calculating localization errors, which will cause regression loss imbalance. Second, $\ell _{n}$ -norm-based loss has symmetry so that when the predicted bounding boxes are in some unique symmetrical relationships (i.e., Symmetric Trap), the regression loss remains unchanged. Third, for $IoU$ -based loss, the overlap area and the union area do not change as the shape or relative position of two bounding boxes changes in some cases(i.e., Area Maze). To address these problems, we propose the scale balanced loss( $\mathcal {L}_{SB}$ ), which is asymmetric, position-sensitive, and scale-invariant. First, in order to obtain the property of scale invariance, it is designed as a fraction to eliminate the scale information contained in the numerator and denominator. Second, by incorporating the Euclidean distance between different corner points instead of the area, $\mathcal {L}_{SB}$ is sensitive to the changes of coordinates of any corner point, so as to solve the area maze problem. Finally, by incorporating the diagonals of the overlap and the smallest enclosing rectangle, this fraction is no longer symmetric, thus solving the symmetry trap problem. To validate the proposed algorithm, we have replaced the $\ell _{n}$ -norm-based loss of YOLOv3 and SSD with $\mathcal {L}_{GIoU}$ and $\mathcal {L}_{SB}$ and evaluate their performance on Pascal Visual Object Classes and Microsoft Common Objects in Context benchmarks. The final results show that $\mathcal {L}_{SB}$ has improved their average precisions at different $IoU$ thresholds and scales. We envision that this regression loss can also improve the performance of other visual tasks.

Analyzing trajectories of learning processes through behaviour-based entropy

ABSTRACTOver recent decades, it has been asserted that the essence of developmental learning processes is change through learning. However, capturing the essence of change in learning processes remains an open question. To study learning processes, we take up a maze problem and conduct an experiment in which a participant draws a route for the maze problem over 10 sessions. To understand how a participant learns to draw a route, we draw learning curves by plotting, for each session, the number of mazes for which a participant succeeds in drawing correct routes. To analyze the learning process, we introduce a new metric called behaviour-based entropy, which quantifies the extent of how intensively a participant is devoted to drawing a route. A crucial finding is that substantial improvement in performance is preceded by a few sessions (plateau) during which the behaviour-based entropy is quite high. We run a program that simulates drawing of routes, and thereby obtain a learning curve based on the simulation. The resultant learning curves turn out to coincide roughly with the corresponding learning curves based on the experiment, which demonstrates the plausibility of the computational model for the simulation.

Online Model-Free n-Step HDP With Stability Analysis.

Because of a powerful temporal-difference (TD) with λ [TD( λ )] learning method, this paper presents a novel n -step adaptive dynamic programming (ADP) architecture that combines TD( λ ) with regular TD learning for solving optimal control problems with reduced iterations. In contrast with a backward view learning of TD( λ ) that is required an extra parameter named eligibility traces to update at the end of each episode (offline training), the new design in this paper has forward view learning, which is updated at each time step (online training) without needing the eligibility trace parameter in various applications without mathematical models. Therefore, the new design is called the online model-free n -step action-dependent (AD) heuristic dynamic programming [NSHDP( λ )]. NSHDP( λ ) has three neural networks: the critic network (CN) with regular one-step TD [TD(0)], the CN with n -step TD learning [or TD( λ )], and the actor network (AN). Because the forward view learning does not require any extra eligibility traces associated with each state, the NSHDP( λ ) architecture has low computational costs and is memory efficient. Furthermore, the stability is proven for NSHDP( λ ) under certain conditions by using Lyapunov analysis to obtain the uniformly ultimately bounded (UUB) property. We compare the results with the performance of HDP and traditional action-dependent HDP( λ ) [ADHDP( λ )] with different λ values. Moreover, a complex nonlinear system and 2-D maze problem are two simulation benchmarks in this paper, and the third one is an inverted pendulum simulation benchmark, which is presented in the supplemental material part of this paper. NSHDP( λ ) performance is examined and compared with other ADP methods.

Parallel Accelerated Virtual Physarum Lab Based on Cellular Automata Agents

Self-aware and self-expressive physical systems are inspiring new methodologies for engineering solutions of complex computing problems. Among many other examples, the slime mold Physarum Polycephalum exhibits self-awareness and self-expressiveness while adapting to changes in its dynamical environment and solving resource-consuming problems like shortest path, proximity graphs or optimization of transport networks. As such, the modeling of the slime mold’s behavior is essential when designing bio-inspired algorithms and hardware prototypes. The goal of this paper is to combine one of the powerful parallel computational tools, cellular automata (CA) with the adaptive potential of Physarum slime mold. Namely, we propose a CA model and multi-agent approach to imitate the behavior of the plasmodium. We then test the efficacy of the proposed model on graph problems such as the maze problem or the traveling salesman problem (TSP). Finally, the virtual Physarum model is evaluated on a data set for pattern recognition purposes and achieves to form very effectively the letters of the alphabet, especially when compared with real experiments performed to prove the efficacy of the proposed model. Furthermore, to exploit the CA’s inherent parallelism and make the model’s responses faster, both GPU and hardware implementations are proposed and compared. As a result, an accelerated virtual lab is developed which uses a multi-agent CA model to describe the behavior of plasmodium and can be used as an intelligent, autonomous, self-adaptive system in various heterogeneous and unknown environments spanning from different types of graph problems up to real life-time applications.

Adaptive reinforcement learning with active state-specific exploration for engagement maximization during simulated child-robot interaction

AbstractUsing assistive robots for educational applications requires robots to be able to adapt their behavior specifically for each child with whom they interact.Among relevant signals, non-verbal cues such as the child’s gaze can provide the robot with important information about the child’s current engagement in the task, and whether the robot should continue its current behavior or not. Here we propose a reinforcement learning algorithm extended with active state-specific exploration and show its applicability to child engagement maximization as well as more classical tasks such as maze navigation. We first demonstrate its adaptive nature on a continuous maze problem as an enhancement of the classic grid world. There, parameterized actions enable the agent to learn single moves until the end of a corridor, similarly to “options” but without explicit hierarchical representations.We then apply the algorithm to a series of simulated scenarios, such as an extended Tower of Hanoi where the robot should find the appropriate speed of movement for the interacting child, and to a pointing task where the robot should find the child-specific appropriate level of expressivity of action. We show that the algorithm enables to cope with both global and local non-stationarities in the state space while preserving a stable behavior in other stationary portions of the state space. Altogether, these results suggest a promising way to enable robot learning based on non-verbal cues and the high degree of non-stationarities that can occur during interaction with children.

Multi-Agent Cooperation Based on Reinforcement Learning with Internal Reward in Maze Problem

This paper introduces a reinforcement learning technique with an internal reward for a multi-agent cooperation task. The proposed methods is an extension of Q-learning which changes the ordinary (e...

Planning and navigation as active inference

This paper introduces an active inference formulation of planning and navigation. It illustrates how the exploitation–exploration dilemma is dissolved by acting to minimise uncertainty (i.e. expected surprise or free energy). We use simulations of a maze problem to illustrate how agents can solve quite complicated problems using context sensitive prior preferences to form subgoals. Our focus is on how epistemic behaviour—driven by novelty and the imperative to reduce uncertainty about the world—contextualises pragmatic or goal-directed behaviour. Using simulations, we illustrate the underlying process theory with synthetic behavioural and electrophysiological responses during exploration of a maze and subsequent navigation to a target location. An interesting phenomenon that emerged from the simulations was a putative distinction between ‘place cells’—that fire when a subgoal is reached—and ‘path cells’—that fire until a subgoal is reached.