Complex Simulation Environment Research Articles

Learn Zero-Constraint-Violation Safe Policy in Model-Free Constrained Reinforcement Learning.

We focus on learning the zero-constraint-violation safe policy in model-free reinforcement learning (RL). Existing model-free RL studies mostly use the posterior penalty to penalize dangerous actions, which means they must experience the danger to learn from the danger. Therefore, they cannot learn a zero-violation safe policy even after convergence. To handle this problem, we leverage the safety-oriented energy functions to learn zero-constraint-violation safe policies and propose the safe set actor-critic (SSAC) algorithm. The energy function is designed to increase rapidly for potentially dangerous actions, locating the safe set on the action space. Therefore, we can identify the dangerous actions prior to taking them and achieve zero-constraint violation. Our major contributions are twofold. First, we use the data-driven methods to learn the energy function, which releases the requirement of known dynamics. Second, we formulate a constrained RL problem to solve the zero-violation policies. We prove that our Lagrangian-based constrained RL solutions converge to the constrained optimal zero-violation policies theoretically. The proposed algorithm is evaluated on the complex simulation environments and a hardware-in-loop (HIL) experiment with a real autonomous vehicle controller. Experimental results suggest that the converged policies in all environments achieve zero-constraint violation and comparable performance with model-based baseline.

3D real-time dynamic path planning for UAV based on improved interfered fluid dynamical system and artificial neural network

In complex and volatile unknown flight environments, the limited environmental information obtained by sensors in the face of sudden dynamic and static obstacles makes it extremely challenging for unmanned aerial vehicles (UAVs) to obtain a safe and efficient path to avoid obstacles and reach a designated target point. Therefore, a real-time dynamic path planning method based on an improved interfered fluid dynamical system (IFDS) and artificial neural network (ANN) is proposed to enhance path quality and computational efficiency. Firstly, to address the issue of insufficient sample quality and quantity, IFDS is employed as the fundamental method for path planning to simulate and generate an adequate amount of sample data for the ANN training. Then, an enhanced sand cat swarm optimization algorithm (ESCSO) with an adaptive social neighborhood search mechanism and Lévy flight strategy is proposed to improve the sample quality. Secondly, the information between the UAV and the target points and obstacles is extracted from the sample data as the input for the network, the parameters of the IFDS are used as the feature extraction at the output of the network, and the ESCSO is applied to optimize the weights and biases of the ANN, enabling offline training of the neural network. Finally, the trained neural network is utilized to dynamically output IFDS parameters based on the real-time environmental information obtained from the sensors, enabling the generation of real-time obstacle avoidance paths. Experimental results in a series of complex simulated environments demonstrate that the proposed method outperforms other algorithms in terms of path quality and meets real-time requirements. It provides excellent obstacle avoidance characteristics for the UAV.

In situ monitoring of phthalate esters (PAEs) pollution and environmental risk assessment in Poyang Lake Basin by DGT Technology using cyclodextrin polymer as binding phase

Poyang Lake is the first freshwater lake in China, which is an important drinking water source. In recent years, industrial pollution has led to the increased phthalate acid esters (PAEs) in Poyang Lake. PAEs are a class of typical endocrine disruptors that can accumulate in organisms and interfere with their secretion systems. Thus, the accurate determination of PAEs in Poyang Lake is important for health risk prediction and the development of corresponding control means. Monitoring organic pollutants in water using the diffusive gradient in thin films technique (DGT) has attracted much attention due to more accuracy and convenience than the traditional methods. This study used an inexpensive amphiphilic cyclodextrin polymer (PBCD) as the sorbent for the binding gel. This new binding gel has an ultra-high specific surface area and excellent adsorption performance. Diffusion coefficients of the five PAEs were determined, and the performance of DGT such as adsorption capacity and deployment time (1–4 days) was tested using five PAEs as models. The assembled PBCD-DGT was used to examine the performance in a complex simulated water environment. The sampling capability of PBCD-DGT was verified in Yangshan Lake, and a large-scale field application was conducted in Poyang Lake basin. The results of 11 sampling points showed that the concentration ranges of dimethyl phthalate, diethyl phthalate, diallyl phthalate, dipropyl phthalate, and dibutyl phthalate were 434–2594 ng/L, 40–314 ng/L, 80–527 ng/L, 45–308 ng/L, and ND-182 ng/L, respectively. The health risk index (HI) and ecological risk quotient (RQ) values of PAEs in the Poyang Lake watershed were far below 1, indictating a lower health and ecological risk. Considering that PAEs are bioaccumulative and persistent, it is very necessary to continue to pay attention to its pollution status and health and ecological risk changes in Poyang Lake Basin in the future.

A Novel Behavioral Strategy for RoboCode Platform Based on Deep Q-Learning

This paper addresses a new machine learning-based behavioral strategy using the deep Q-learning algorithm for the RoboCode simulation platform. According to this strategy, a new model is proposed for the RoboCode platform, providing an environment for simulated robots that can be programmed to battle against other robots. Compared to Atari Games, RoboCode has a fairly wide set of actions and situations. Due to the challenges of training a CNN model for such a continuous action space problem, the inputs obtained from the simulation environment were generated dynamically, and the proposed model was trained by using these inputs. The trained model battled against the predefined rival robots of the environment (standard robots) by cumulatively benefiting from the experience of these robots. The comparison between the proposed model and standard robots of RoboCode Platform was statistically verified. Finally, the performance of the proposed model was compared with machine learning based-customized robots (community robots). Experimental results reveal that the proposed model is mostly superior to community robots. Therefore, the deep Q-learning-based model has proven to be successful in such a complex simulation environment. It should also be noted that this new model facilitates simulation performance in adaptive and partially cluttered environments.

Optimizing Mobility of Robotic Arms in Collision-free Motion Planning

A major task in motion planning is to find paths that have a high ability to react to external influences while ensuring a collision-free operation at any time. This flexibility is even more important in human-robot collaboration since unforeseen events can occur anytime. Such ability can be described as mobility, which is composed of two characteristics. First, the ability to manipulate, and second, the distance to joint limits. This mobility needs to be optimized while generating collision-free motions so that there is always the flexibility of the robot to evade dynamic obstacles in the future execution of generated paths. For this purpose, we present a Rapidly-exploring Random Tree (RRT), which applies additional costs and sampling methods to increase mobility. Additionally, we present two methods for the optimization of a generated path. Our first approach utilizes the built-in capabilities of the RRT*. The second method optimize the path with the stochastic trajectory optimization for motion planning (STOMP) approach with Gaussian Mixture Models. Moreover, we evaluate the algorithms in complex simulation and real environments and demonstrate an enhancement of mobility.

Identification and modelling of race driving styles

A good understanding and modelling of the human driver is essential for modern vehicle development, particularly in motorsports, where the race car should fit its driver perfectly. At the same time, an objective assessment and especially imitation of professional race drivers is difficult due to individual driving styles, complex and non-deterministic decision making processes, and small stability margins. In this paper, we present a holistic approach to identify and model individual race driving styles in a robust way. We develop the Driver Identification and Metric Ranking Algorithm (DIMRA) as a data-based method for an in-depth objective analysis and assessment of professional race drivers. Supported by this knowledge, we extend and adapt the imitation learning framework Probabilistic Modeling of Driver Behavior (ProMoD) in order to model race drivers in a complex simulation environment. An evaluation with data from professional race drivers shows the capability of DIMRA to derive metrics which describe human race driving styles, as well as ProMoD to robustly generate competitive laps with human-like controls in a professional motorsport driving simulator. The ability to identify and imitate individual driving styles does not only support the performance optimisation of race cars but could also aid the development of road cars and driver assistance systems in future work.

Circular Fields and Predictive Multi-Agents for Online Global Trajectory Planning

Safe and efficient trajectory planning for autonomous robots is becoming increasingly important in both industrial applications and everyday life. The demands on a robot which has to react quickly and precisely to changes in cluttered, unknown and dynamic environments are particularly high. Towards this end, based on the initial idea proposed in [27] we propose the Circular Field Predictions approach, which unifies reactive collision avoidance and global trajectory planning while providing smooth, fast and collision free trajectories for robotic motion planningreactive collision avoidance and global trajectory planning while providing smooth, fast and collision free trajectories for robotic motion planning. The proposed approach is inspired by electromagnetic fields, free of local minima and extended with artificial multi-agents to efficiently explore the environment. The algorithm is extensively analysed in complex simulation environments where it is shown to be able to generate smooth trajectories around arbitrarily shaped obstacles. Moreover, we experimentally verified the approach with a 7 Degree-of-Freedom (DoF) Franka Emika robot.

A Deep Reinforcement Learning Approach for Active SLAM

In this paper, we formulate the active SLAM paradigm in terms of model-free Deep Reinforcement Learning, embedding the traditional utility functions based on the Theory of Optimal Experimental Design in rewards, and therefore relaxing the intensive computations of classical approaches. We validate such formulation in a complex simulation environment, using a state-of-the-art deep Q-learning architecture with laser measurements as network inputs. Trained agents become capable not only to learn a policy to navigate and explore in the absence of an environment model but also to transfer their knowledge to previously unseen maps, which is a key requirement in robotic exploration.

Virtual and Augmented Reality Applications to Support Data Analysis and Assessment of Science and Engineering

For decades, virtual reality (VR) and augmented reality (AR) technologies have remained absent from science and engineering workflows due to the custom-built nature of the hardware and software and oftentimes highly task-specific design. In addition to high cost, these characteristics create a critical entry barrier for most users to invest in VR and AR applications. Recent developments in both VR and AR technologies in the commercial sector have resulted in the availability of affordable consumer-grade and reliable VR and AR hardware and software tools. However, significant research is needed to adapt VR and AR technology use for data-intensive scientific and engineering applications such as network modeling research, human sciences research, and visualization of complex simulated environments. Our VR and AR applications provide examples of how scientists and engineers can use advanced visualization technologies to perform data analysis and assessment, thus transforming scientific discovery.

3D Design Using Generative Adversarial Networks and Physics-Based Validation

Abstract The authors present a generative adversarial network (GAN) model that demonstrates how to generate 3D models in their native format so that they can be either evaluated using complex simulation environments or realized using methods such as additive manufacturing. Once initially trained, the GAN can create additional training data itself by generating new designs, evaluating them in a physics-based virtual environment, and adding the high performing ones to the training set. A case study involving a GAN model that is initially trained on 4045 3D aircraft models is used for demonstration, where a training data set that has been updated with GAN-generated and evaluated designs results in enhanced model generation, in both the geometric feasibility and performance of the designs. Z-tests on the performance scores of the generated aircraft models indicate a statistically significant improvement in the functionality of the generated models after three iterations of the training-evaluation process. In the case study, a number of techniques are explored to structure the generate-evaluate process in order to balance the need to generate feasible designs with the need for innovative designs.

A generic testbed for simulation and physical-based testing of maritime cyber-physical system of systems

e-Navigation Systems are CPSoS, which are regarded as systems based on systems with physical and information technology-based components. SoS are usually open systems which can have emerging behaviour and are not fully defined during design time. Future maritime bridge systems, for instance, will be embedded into the vessel infrastructure and its navigation and safety subsystems receive continuous updates and additional functionalities for the ship’s command and navigation. Suitable testbeds are required to support the technological and regulatory development and certification processes. To support the development and integration of CPS, this paper introduces the testbed eMIR as an open and modular research, development and test environment. The test environment is suitable for early testing, validation and verification or demonstration of new maritime technologies in a complex simulation environment (traffic, environment and sensors) and supports a seamless transfer of these technologies from a virtual into a physical testbed. eMIR comprised various interconnected virtual and physical components via a standardised co-simulation environment based on HLA, combined with a message passing middleware as a backbone for the physical testbed elements. The testbed was effectively applied in various maritime projects for the evaluation of maritime technologies in navigation, management and surveillance or maritime (transportation) systems.

Optimized data exploration applied to the simulation of a chemical process

In complex simulation environments, certain parameter space regions may result in non-convergent or unphysical outcomes. All parameters can therefore be labeled with a binary class describing whether or not they lead to valid results. In general, it can be very difficult to determine feasible parameter regions, especially without previous knowledge. We propose a novel algorithm to explore such an unknown parameter space and improve its feasibility classification in an iterative way. Moreover, we include an additional optimization target in the algorithm to guide the exploration towards regions of interest and to improve the classification therein. In our method we make use of well-established concepts from the field of machine learning like kernel support vector machines and kernel ridge regression. From a comparison with a Kriging-based exploration approach based on recently published results we can show the advantages of our algorithm in a binary feasibility classification scenario with a discrete feasibility constraint violation. In this context, we also propose an improvement of the Kriging-based exploration approach. We apply our novel method to a fully realistic, industrially relevant chemical process simulation to demonstrate its practical usability and find a comparably good approximation of the data space topology from relatively few data points.

Motion Planning with Energy Reduction for a Floating Robotic Platform Under Disturbances and Measurement Noise Using Reinforcement Learning

This paper investigates the use of reinforcement learning for the navigation of an over-actuated, i.e. more control inputs than degrees of freedom, marine platform in unknown environment. The proposed approach uses an online least-squared policy iteration scheme for value function approximation in order to estimate optimal policy, in conjunction with a low-level control system that controls the magnitude of the linear velocity, and the orientation of the platform. Primary goal of the proposed scheme is the reduction of the consumed energy. To that end, we propose a variable reward function that depends on the energy consumption of the platform. We evaluate our approach in a complex and realistic simulation environment and report results concerning its performance on estimating optimal navigation policies under different environmental disturbances, and position GPS measurement noise. The proposed framework is compared, in terms of energy consumption, to a baseline approach based on virtual potential fields. The results show that the marine platform successfully discovers the target point following a sub-optimal path, maintaining reduced energy consumption.

Electric energy converter development and diagnostics in mixed-signal simulation environment

<p class="Abstract"><span lang="EN-US">In this work the input and output capacitors of a digitally controlled Buck converter have been tested in 4 different cases with unused and worn out capacitors. The information extracted from interactions between the subsystems of an electric energy conversion system requires the implementation of the whole system from network to load. A complex simulation environment has been implemented where digital and analogue domains can be connected. The input and output capacitor aging have a significant influence on time domain characteristics, but the evaluation of frequency-domain data is supposed to be required to extract information from the noisy and distorted time-domain signal. The presented diagnostic approach aims to analyze signals that are used by the control loop in order to avoid expensive additions to existing circuits.</span></p>

On the Compatibility of Electric Equivalent Circuit Models for Enhanced Flooded Lead Acid Batteries Based on Electrochemical Impedance Spectroscopy

Electric equivalent circuit (EEC) models have been widely used to interpret the inner dynamics of all type of batteries. Added to this, they also have been used to estimate state of charge (SOC) and state of health (SOH) values in combination with different methods. Four EEC models are considered for enhanced flooded lead acid batteries (EFB) which are widely used in micro hybrid vehicles. In this study, impedance and phase prediction capabilities of models throughout a frequency spectrum from 1 mHz to 10 kHz are compared with those of experimental results to investigate their consistency with the data. The battery is charged, discharged, and aged according to appropriate standards which imitates a lifetime of a micro hybrid vehicle battery under high current partial cycling. Impedance tests are repeated between different charge and health states until the end of the battery’s lifetime. It is seen that adding transmission line elements to mimic the high porous electrode electrolyte interface to a double parallel constant phase element resistance model (ZARC) can increase the model data representing capability by 100%. The mean average percentage error (MAPE) of the conventional model with respect to data is 3.2% while the same value of the transmission line added model found as 1.6%. The results can be helpful to represent an EFB in complex simulation environments, which are used in automobile industry.

Customized dynamic pricing of airline fare products

Firms practice dynamic pricing when they charge different customers different prices for the same products, as a function of an observable state of nature. In the airline industry, dynamic pricing has historically been limited by the capabilities of distribution systems, which require filing a finite set of fare products with fixed prices. However, advancements in airline distribution technology will soon allow the generation of “customized offers,” which could include a dynamically generated price. In this paper, we propose a heuristic for customized dynamic pricing of airfares when the observable state of nature includes an observation of passenger characteristics. We first introduce a general model for decoupling the customized offer generation problem from the well-studied airline revenue management problem. After generating a baseline assortment of fare products and observing the characteristics of a passenger’s request, an airline can choose to customize that passenger’s offer by dynamically incrementing or discounting the prices that would ordinarily be offered. The methodology could be applied with existing airline revenue management methods, and would be compatible with IATA’s New Distribution Capability (NDC). For implementation, we propose a straightforward heuristic approach based on simple estimates of passenger willingness-to-pay distributions. The heuristics are simulated in the Passenger Origin–Destination Simulator, a complex airline revenue management simulation environment that takes into account passenger choice and competition. The results show that the heuristics can lead to revenue gains of up to 3–4% when practiced by one airline. Furthermore, the heuristics remain revenue positive in competitive environments.

An Efficient Approach to Model-Based Hierarchical Reinforcement Learning

We propose a model-based approach to hierarchical reinforcement learning that exploits shared knowledge and selective execution at different levels of abstraction, to efficiently solve large, complex problems. Our framework adopts a new transition dynamics learning algorithm that identifies the common action-feature combinations of the subtasks, and evaluates the subtask execution choices through simulation. The framework is sample efficient, and tolerates uncertain and incomplete problem characterization of the subtasks. We test the framework on common benchmark problems and complex simulated robotic environments. It compares favorably against the state-of-the-art algorithms, and scales well in very large problems.

Social Skill Acquisition Model through Face-to-Face Interaction: Local Contingency for Open-Ended Development

Behavior associated with joint attention is among the most important human functionalities for communicating with others. Previous studies indicate that even a robot can learn these behavioral patterns as social skills through interaction with a modeled/real caregiver by contingency evaluation. However, existing mechanisms are too time-consuming, especially for implementation on a real-world interactive robot. Also, they are poor in the acquisition of complex skills. In this paper, we propose a fast mechanism that enables the acquisition of many complex social skills within a short interaction time. The mechanism is realized by the utilization of two significant ideas: evaluating contingency locally, and acquiring social skills by finding synergistic contributions of values in contingencies. A comparison of our proposed mechanism in a simple environment of computer simulation with other mechanisms in terms of speed, accuracy, complexity, and noise resistance confirms the superior performance of our mechanism. Further, experimental results obtained with the proposed mechanism in a more complex computer simulation environment, which more closely resembles a real-world environment, indicate that the mechanism can be applied in real-world interaction between a robot and a human.

Sensorimotor Model of Obstacle Avoidance in Echolocating Bats.

Bat echolocation is an ability consisting of many subtasks such as navigation, prey detection and object recognition. Understanding the echolocation capabilities of bats comes down to isolating the minimal set of acoustic cues needed to complete each task. For some tasks, the minimal cues have already been identified. However, while a number of possible cues have been suggested, little is known about the minimal cues supporting obstacle avoidance in echolocating bats. In this paper, we propose that the Interaural Intensity Difference (IID) and travel time of the first millisecond of the echo train are sufficient cues for obstacle avoidance. We describe a simple control algorithm based on the use of these cues in combination with alternating ear positions modeled after the constant frequency bat Rhinolophus rouxii. Using spatial simulations (2D and 3D), we show that simple phonotaxis can steer a bat clear from obstacles without performing a reconstruction of the 3D layout of the scene. As such, this paper presents the first computationally explicit explanation for obstacle avoidance validated in complex simulated environments. Based on additional simulations modelling the FM bat Phyllostomus discolor, we conjecture that the proposed cues can be exploited by constant frequency (CF) bats and frequency modulated (FM) bats alike. We hypothesize that using a low level yet robust cue for obstacle avoidance allows bats to comply with the hard real-time constraints of this basic behaviour.

Unknown environment exploration of multi-robot system with the FORDPSO

Effective environment exploration in unknown environment is precondition of constructing the environment map and carrying out other tasks for multi-robot system. Due to its excellent performance, particle swarm optimization (PSO) has been widely used in multi-robot exploration field. To deal with its drawback – easily trapped in local optima, Darwinian PSO (DPSO) optimization is proposed by Tillett et al. [1] with the natural selection function and first used in real world robot exploration by Couceiro et al. [2], forming the robotic DPSO (RDPSO). To increase the algorithm performance and control its convergence rate, fractional calculus is used to replace inertia component in RDPSO for its “memory” ability and forming the fractional order RDPSO (FORDPSO). This paper presents a formal analysis of RDPSO and studies the influence of the coefficients on FORDPSO algorithm. To satisfy the requirement of dynamically changing robots׳ behaviors during the exploration, fuzzy inferring system is designed to achieve better control coefficients. Experiment results obtained in two complex simulated environments illustrate that biological and sociological inspiration is effective to meet the challenges of multi-robot system application in unknown environment exploration, and the exploration effect of the fuzzy adaptive FORDPSO is better than that of the fixed coefficient FORDPSO. Furthermore, the performance of FORDPSO with different neighborhood topologies are studied and compared with other six PSO variations. All the results demonstrate the effect of the FORDPSO on the multi-robot environment exploration.