Moving Target Search Research Articles

Reinforcement-Learning-Based Multi-UAV Cooperative Search for Moving Targets in 3D Scenarios

Most existing multi-UAV collaborative search methods only consider scenarios of two-dimensional path planning or static target search. To be close to the practical scenario, this paper proposes a path planning method based on an action-mask-based multi-agent proximal policy optimization (AM-MAPPO) algorithm for multiple UAVs searching for moving targets in three-dimensional (3D) environments. In particular, a multi-UAV high–low altitude collaborative search architecture is introduced that not only takes into account the extensive detection range of high-altitude UAVs but also leverages the benefit of the superior detection quality of low-altitude UAVs. The optimization objective of the search task is to minimize the uncertainty of the search area while maximizing the number of captured moving targets. The path planning problem for moving target search in a 3D environment is formulated and addressed using the AM-MAPPO algorithm. The proposed method incorporates a state representation mechanism based on field-of-view encoding to handle dynamic changes in neural network input dimensions and develops a rule-based target capture mechanism and an action-mask-based collision avoidance mechanism to enhance the AM-MAPPO algorithm’s convergence speed. Experimental results demonstrate that the proposed algorithm significantly reduces regional uncertainty and increases the number of captured moving targets compared to other deep reinforcement learning methods. Ablation studies further indicate that the proposed action mask mechanism, target capture mechanism, and collision avoidance mechanism of the AM-MAPPO algorithm can improve the algorithm’s effectiveness, target capture capability, and UAVs’ safety, respectively.

DRL-Searcher: A Unified Approach to Multirobot Efficient Search for a Moving Target.

This article studies the multirobot efficient search (MuRES) for a nonadversarial moving target problem, whose objective is usually defined as either minimizing the target's expected capture time or maximizing the target's capture probability within a given time budget. Different from canonical MuRES algorithms, which target only one specific objective, our proposed algorithm, named distributional reinforcement learning-based searcher (DRL-Searcher), serves as a unified solution to both MuRES objectives. DRL-Searcher employs distributional reinforcement learning (DRL) to evaluate the full distribution of a given search policy's return, that is, the target's capture time, and thereafter makes improvements with respect to the particularly specified objective. We further adapt DRL-Searcher to the use case without the target's real-time location information, where only the probabilistic target belief (PTB) information is provided. Lastly, the recency reward is designed for implicit coordination among multiple robots. Comparative simulation results in a range of MuRES test environments show the superior performance of DRL-Searcher to state of the arts. Additionally, we deploy DRL-Searcher to a real multirobot system for moving target search in a self-constructed indoor environment with satisfying results.

Research on Dynamic Target Search for Multi-UAV Based on Cooperative Coevolution Motion-Encoded Particle Swarm Optimization

Effectively strategizing the trajectories of multiple Unmanned Aerial Vehicles (UAVs) within a dynamic environment to optimize the search for and tracking of mobile targets presents a formidable challenge. In this study, a cooperative coevolution motion-encoded particle swarm optimization algorithm called the CC-MPSO search algorithm is designed to tackle the moving target search issue effectively. Firstly, a Markov process-based target motion model considering the uncertainty of target motion is investigated. Secondly, Bayesian theory is used to formulate the moving target search as an optimization problem where the objective function is defined as maximizing the cumulative probability of detection of the target in finite time. Finally, the problem is solved based on the CC-MPSO algorithm to obtain the optimal search path nodes. The motion encoding mechanism converts the search path nodes into a set of motion paths, which enables more flexible handling of UAV trajectories and improves the efficiency of dynamic path planning. Meanwhile, the cooperative coevolution optimization framework enables collaboration between different UAVs to improve global search performance through multiple swarm information sharing, which helps avoid falling into local optimal solutions. The simulation results show that the CC-MPSO algorithm demonstrates efficacy, reliability, and superior overall performance when compared to the five commonly used swarm intelligence algorithms.

An improved sand cat swarm optimization for moving target search by UAV

Moving target search is a challenging dynamic path planning problem. In this scenario, unmanned aerial vehicles endeavor to locate a moving entity based on sensor information, utilizing the optimal path generated by a search algorithm. Based on the Bayesian principle, the task can be transformed into an optimization issue of the fitness function with the maximum probability of capturing the objective. In this study, an improved version of the sand cat swarm optimization algorithm, called the ISCSO search algorithm, is designed to tackle the moving target search issue effectively. Firstly, the presented ISCSO algorithm enhances the planning efficiency of the algorithm by encoding the unmanned aerial vehicle search path information into a set of motion paths through the motion-encoded mechanism. Secondly, the elite pooling strategy and the adaptive T-distribution are constructed to effectively improve the algorithm’s ability to escape local optima and enhance its variability. Finally, ISCSO proposes a main architecture that seamlessly merges the search and attack methods of the sand cat swarm optimization algorithm, striking a balance between global and local search capabilities. To evaluate the superiority of the proposed algorithm, nine diverse search scenarios are constructed to verify its performance. The simulation results demonstrate that ISCSO achieves higher detection accuracy and offers more effective search paths for locating dynamic targets in comparison to other well-established metaheuristic algorithms. Code has been available at https://github.com/yb-niu1/ISCSO.

Cross-Entropy Regularized Policy Gradient for Multirobot Nonadversarial Moving Target Search

This article investigates the multirobot efficient search (MuRES) for a nonadversarial moving target problem from the multiagent reinforcement learning (MARL) perspective. MARL is deemed as a promising research field for cooperative multiagent applications. However, one of the main bottlenecks of applying MARL to the MuRES problem is the nonstationarity introduced by multiple learning agents. With learning agents simultaneously updating their policies, the environment cannot be modeled as a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">stationary</i> Markov decision process, which results in the inapplicability of fundamental reinforcement learning techniques such as deep <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$Q$</tex-math></inline-formula> -network and policy gradient (PG). In view of that, we adopt the centralized training and decentralized execution scheme and thereby propose a cross-entropy regularized policy gradient (CE-PG) method to train the learning agents/robots. We let the robots <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">commit</i> to a predetermined policy during execution, collect the trajectories, and then perform centralized training for the corresponding policy improvement. In this way, the nonstationarity problem is overcome, in that the robots do not update their policies during execution. During the centralized training stage, we improve the canonical PG method to consider the interactions among robots by adding a cross-entropy regularization term, which essentially functions to “disperse” the robots in the environment. Extensive simulation results and comparisons with state of the art show CE-PG's superior performance, and we also validate the algorithm with a real multirobot system in an indoor moving target search scenario.

An Efficient Evolution-Based Technique for Moving Target Search with Unmanned Aircraft Vehicle: Analysis and Validation

Recent advances in technology have led to a surge in interest in unmanned aerial vehicles (UAVs), which are remote-controlled aircraft that rely on cameras or sensors to gather information about their surroundings during flight. A UAV requires a path-planning technique that can swiftly recalculate a viable and quasi-optimal path in flight if a new obstacle or hazard is recognized or if the target is moved during the mission. In brief, the planning of UAV routes might optimize a specific problem determined by the application, such as the moving target problem (MTP), flight time and threats, or multiobjective navigation. The complexity of MTP ranges from NP-hard to NEXP-complete because there are so many probabilistic variables involved. Therefore, it is hard to detect a high-quality solution for this problem using traditional techniques such as differential calculus. Therefore, this paper hybridizes differential evolution (DE) with two newly proposed updating schemes to present a new evolution-based technique named hybrid differential evolution (HDE) for accurately tackling the MTP in a reasonable amount of time. Using Bayesian theory, the MTP can be transformed into an optimization problem by employing the target detection probability as the fitness function. The proposed HDE encodes the search trajectory as a sequence of UAV motion pathways that evolve with increasing the current iteration for finding the near-optimal solution, which could maximize this fitness function. The HDE is extensively compared to the classical DE and several rival optimizers in terms of several performance metrics across four different scenarios with varying degrees of difficulty. This comparison demonstrates the proposal’s superiority in terms of the majority of used performance metrics.

Convex combination multiple populations competitive swarm optimization for moving target search using UAVs

Unmanned aerial vehicles (UAVs) searching for a moving target is a complex search optimization problem that aims to find a path with maximum probability of observing the target. However, as the area where the target may be located increases in size, existing methods tend to cause easy entrapment into local optima, resulting in poor performance. Hence, we propose a Convex Combination Multiple Populations Competitive Swarm Optimization algorithm (CDCSO) in this paper. First, we suggest applying the multiple population strategy to enhance the search ability of the algorithm. The population of interest is divided into two subgroups that each conduct the CSO. Second, we propose a novel convex combination update strategy to coordinate the two subgroups to search jointly for the global optimal solution. At each iteration, this strategy allows the two subgroups to compete and learn from each other, which prevents the population from falling into the local optimum. We then designed new scenarios containing 3 to 6 probability areas to investigate the performance of our proposed algorithm in multiple and complex scenarios.Results: To demonstrate the effectiveness and stability of the proposed algorithm, we compared it with several swarm optimization algorithms from different perspectives. Experimental results showed that the proposed algorithm has superior performance in following and intercepting a high-probability area. To further demonstrate the generality of the convex combination update strategy, we evaluated the strategy in several developed particle swarm optimization algorithms. The results demonstrated that the convex combination update strategy can be integrated into other algorithms to better address the moving target search problem.

PD-FAC: Probability Density Factorized Multi-Agent Distributional Reinforcement Learning for Multi-Robot Reliable Search

This letter presents a new range of multi-robot search for a non-adversarial moving target problems, namely multi-robot reliable search (MuRRS). The term ‘reliability’ in MuRRS is defined as the expectation of a predefined utility function over the probability density function (PDF) of the target’s capture time. We argue that MuRRS subsumes the canonical multi-robot efficient search (MuRES) problem, which minimizes the target’s expected capture time, as its special case, and offers the end user with a wide range of objective selection options. Since state-of-the-art algorithms are usually targeting the MuRES problem, and cannot offer up-to-standard performance to the various MuRRS objectives, we, thereby, propose a probability density factorized multi-agent distributional reinforcement learning method, namely PD-FAC, as a unified solution to the MuRRS problem. PD-FAC decomposes the PDF of the multi-robot system’s overall value distribution into a set of individual value distributions and guarantees that any reliability objective defined as a function of the overall system’s value distribution can be linearly approximated by the same reliability metric defined over the agent’s individual value distribution. In this way, the individual global maximum (IGM) principle is satisfied for all the pre-defined reliability metrics. It means that when each reinforcement learning agent is executing the individual policy, which maximizes its own reliability metric, the system’s overall reliability performance is also maximized. We evaluate and compare the performance of PD-FAC with state of the arts in a range of canonical multi-robot search environments with satisfying results, and also deploy PD-FAC to a real multi-robot system for non-adversarial moving target search.

Motion-Encoded Electric Charged Particles Optimization for Moving Target Search Using Unmanned Aerial Vehicles.

In this paper, a new optimization algorithm called motion-encoded electric charged particles optimization (ECPO-ME) is developed to find moving targets using unmanned aerial vehicles (UAV). The algorithm is based on the combination of the ECPO (i.e., the base algorithm) with the ME mechanism. This study is directly applicable to a real-world scenario, for instance the movement of a misplaced animal can be detected and subsequently its location can be transmitted to its caretaker. Using Bayesian theory, finding the location of a moving target is formulated as an optimization problem wherein the objective function is to maximize the probability of detecting the target. In the proposed ECPO-ME algorithm, the search trajectory is encoded as a series of UAV motion paths. These paths evolve in each iteration of the ECPO-ME algorithm. The performance of the algorithm is tested for six different scenarios with different characteristics. A statistical analysis is carried out to compare the results obtained from ECPO-ME with other well-known metaheuristics, widely used for benchmarking studies. The results found show that the ECPO-ME has great potential in finding moving targets, since it outperforms the base algorithm (i.e., ECPO) by as much as 2.16%, 5.26%, 7.17%, 14.72%, 0.79% and 3.38% for the investigated scenarios, respectively.

Cooperative Target Search of UAV Swarm with Communication Distance Constraint

This paper proposes a cooperative search algorithm to enable swarms of unmanned aerial vehicles (UAVs) to capture moving targets. It is based on prior information and target probability constrained by inter-UAV distance for safety and communication. First, a rasterized environmental cognitive map is created to characterize the task area. Second, based on Bayesian theory, the posterior probability of a target’s existence is updated using UAV detection information. Third, the predicted probability distribution of the dynamic time-sensitive target is obtained by calculating the target transition probability. Fourth, a customized information interaction mechanism switches the interaction strategy and content according to the communication distance to produce cooperative decision-making in the UAV swarm. Finally, rolling-time domain optimization generates interactive information, so interactive behavior and autonomous decision-making among the swarm members are realized. Simulation results showed that the proposed algorithm can effectively complete a cooperative moving-target search when constrained by communication distance yet still cooperate effectively in unexpected situations such as a fire.

Moving Target Search with Subgoal Graphs

Moving Target Search (MTS) is a dynamic path planning problem, where an agent is trying to reach a moving entity with a minimum path cost. Problems of this nature can be found in video games and dynamic robotics, which require fast processing time (real time). In this work, we introduce a new algorithm for this problem - the Moving Target Search with Subgoal Graphs (MTSub). MTSub is based on environment abstraction and uses Subgoal Graphs to speed up the searches for a minimal cost route to the target. The algorithm is optimal with respect to the knowledge that the agent has during the search. Experimental results show that MTSub can be used in real-time applications (e.g., applications requiring 5 microseconds response time per step). The experiments compared MTSub to G-FRA*, which is the best known dynamic algorithm so far, showing that MTSub is up to 29 times faster in average time per step, and up to 186 times faster in maximum time per step.

Context-Aware Decision Support for Anti-Submarine Warfare Mission Planning Within a Dynamic Environment

Anti-submarine warfare (ASW) missions are the linchpin of maritime operations involving effective allocation and path planning of scarce assets to search for, detect, classify, track, and prosecute hostile submarines within a dynamic and uncertain mission environment. Motivated by the need to assist ASW commanders to make better decisions within an evolving mission context, we investigate a moving target search problem with multiple searchers and develop a context-driven decision support tool for the ASW mission planning problem. Given the spatial probability distribution of a target submarine, sensor detection probability surfaces from meteorological and oceanographic products, and the risk to the fleet as a function of distance of the target from the fleet, we model and formulate the ASW asset allocation and search path planning problem using a hidden Markov modeling framework. We propose a two phase approach to solve this NP-hard problem. In phase I, we partition the geographic area, satisfying contiguity constraints, into search regions using an evolutionary algorithm (EA) coupled with a Voronoi tessellation approach, and allocate the assets to partitioned search areas using the auction algorithm. In phase II, we construct a dynamic search plan for each asset over the search interval using EA. We evaluate our approach via a hypothetical ASW scenario to monitor an enemy submarine in a geographic region via multiple assets. We compare our results to various search path planning strategies that, using the context-driven decision support tool developed here, revise the search regions at periodic intervals given a fixed total search time.

Energy Aware - Bio-Inspired Hybrid WSN for Area Surveillance (E-BHAS)

Objectives: Randomly moving target search in an unknown location is a complex problem which does not have a linear solution. This paper proposes Energy aware - Bio-inspired Hybrid technique for target finding Area surveillance application. Methods/Statistical Analysis: In recent years, many researchers tried to find a generic solution which requires minimum energy for target searching problem. For the proposed work four algorithms are proposed which are MSN’s Movement Prediction Algorithm (MMPA), Leader Selection Algorithm (LSA), and Leader’s Movement Predication Algorithm (LMPA), and Follower’s Algorithm. Particle Swarm Optimization (PSO) technique is used to move the Mobile Sensor Nodes (MSN) in searching area. Findings: There is a need to study the hybrid or heterogeneous Sensor network (combination of the Static sensor network and Mobile sensor network) as it is less studied in research literature. The major hurdle is communication between heterogeneous sensor nodes. Due to the mobility of MSN, topology changes frequently and it affects the communication in the network. In area surveillance domain, finding the randomly moving target using sensor node is a complex and non-linear problem. There is no straight forward solution. Nature based techniques may be more effective for such problem. Bio-inspired Hybrid WSN technique is to be studied for its suitability for target finding and target tracking application in area surveillance domain. To perform the work, two scenarios are considered and calculated their energy to prove that ifmore number of SSNs is planted, the target can be found in less time and ultimately overall energy of network reduces. Through the simulation, it is shown that E-BHAS reduces the energy when more number of SSNs is involved in the network. Application/ Improvements: Borderline Surveillance, Disaster Management, under water Surveillance, Traffic Monitoring etc. Keywords: Area Surveillance, Energy, Hybrid Wireless Sensor Network, Particle Swarm Optimization

Trajectory optimization under kinematical constraints for moving target search

Various recent events in the Mediterranean sea have shown the enormous importance of maritime search-and-rescue missions. By reducing the time to find floating victims, the number of casualties can be reduced. A major improvement can be achieved by employing unmanned aerial systems for autonomous search missions. In this context, the need for efficient search trajectory planning methods arises. Existing approaches either consider K-step-lookahead optimization without accounting for kinematics of fixed-wing platforms or propose a suboptimal myopic method. A few approaches consider both aspects, however only applicable to stationary target search. The contribution of this article consists of a novel method for Markovian target search-trajectory optimization. This is a unified method for fixed-wing and rotary-wing platforms, taking kinematical constraints into account. It can be classified as K-step-lookahead planning method, which allows for anticipation to the estimated future position and motion of the target. The method consists of a mixed integer linear program that optimizes the cumulative probability of detection. We show the applicability and effectiveness in computational experiments for three types of moving targets: diffusing, conditionally deterministic, and Markovian. This approach is the first K-step-lookahead method for Markovian target search under kinematical constraints.

Aerial Vehicle Search-Path Optimization: A Novel Method for Emergency Operations

This paper presents a novel search-path optimization method for moving target search by an aerial vehicle, applicable to realistically sized search areas. For such missions, long endurance vehicles are needed, which are usually fixed-winged. The proposed method accounts for flight kinematics of fixed-wing and rotary-wing aerial vehicles. It additionally accounts for movements of the target, considerably increasing complexity of search-path optimization, compared to a static target. The objective is to maximize the probability to detect a conditionally deterministic moving target within a given time period. We propose a first K-step-lookahead planning method that takes flight kinematic constraints into account and in which the target and platform state space are heterogeneous. It consists of a binary integer linear program that yields a physically feasible search-path, while maximizing the probability of detection. It is based on the Max-K-Coverage problem, as it selects K waypoints while maximizing the probability that a target is within the field of view of a platform at one of these waypoints. This K-step-lookahead planning method is embedded in an iterative framework, where the probability of overlooking a target is fed back to the controller after observations are made. Simulations show the applicability and effectiveness of this method.

An Empirical Comparison of Search Approaches for Moving Agents

This study empirically compares existing search approaches used for path planning of moving agents, namely, incremental and real‐time search approaches. The comparisons are performed in both stationary and moving target search problems separately. In each problem domain, well‐known representatives of both approaches are evaluated in partially observable environments where the agent senses a limited area based on its sensor range. In addition to the available algorithms, we propose two algorithms to be used in each problem. The simulations conducted on random grid and maze structures show that the algorithms behave differently and have advantages over each other especially as the sensor range varies. Therefore, the proposed study enables the agent to determine the most appropriate algorithm depending on its priorities and sensor range.

Fast Algorithm for Catching a Prey Quickly in Known and Partially Known Game Maps

In moving target search, the objective is to guide a hunter agent to catch a moving prey. Even though in game applications maps are always available at developing time, current approaches to moving target search do not exploit preprocessing to improve search performance. In this paper, we propose MtsCopa, an algorithm that exploits precomputed information in the form of compressed path databases (CPDs), and that is able to guide a hunter agent in both known and partially known terrain. CPDs have previously been used in standard, fixed-target pathfinding but had not been used in the context of moving target search. We evaluated MtsCopa over standard game maps. Our speed results are orders of magnitude better than current state of the art. The time per individual move is improved, which is important in real-time search scenarios, where the time available to make a move is limited. Compared to state of the art, the number of hunter moves is often better and otherwise comparable, since CPDs provide optimal moves along shortest paths. Compared to previous successful methods, such as I-ARA*, our method is simple to understand and implement. In addition, we prove MtsCopa always guides the agent to catch the prey when possible.

Moving Target Search with Subgoal Graphs

Moving Target Search (MTS) is a dynamic path planning problem, where an agent is trying to reach a moving entity with a minimum path cost. Problems of this nature can be found in video games and dynamic robotics, which require fast processing time (real time). In this work, we introduce a new algorithm for this problem - the Moving Target Search with SubgoalGraphs (MTSub). MTSub is based on environment abstraction and uses Subgoal Graphs to speed up searches without giving up cost minimal paths. The algorithm is optimal with respect to the knowledge that the agent has during the search. Experimental results show that MTSub meets the requirement of real time performance (e.g., 5 microseconds per step). Compared to G-FRA*, which is the best known dynamic algorithm so far, MT-Sub is up to 29 times faster in average time per step and 186 times faster in maximum time per step. MTSub also compares fairly well against MtsCopa. Although in this case MTSub is up to 3.89 times slower in average response time and up to 6.81 times slower in maximum response time, it performed much better than MtsCopain the processing phase - up to 220,000 times faster and requiring up to 44 times less space.

Two phase solution for an intelligent moving target search problem based on a 0–1 linear model

This paper presents a generic discrete model for the moving, intelligent target problem. Our objective is to maximise the probability of detection of the moving target with respect to target and searcher’s constraints. The solution method proposed here is composed on two stages. The first one aims at providing a large-scale strategy by solving an Integer Linear Program approach. As a direct solution of this problem is not practically possible, we use a decomposition of the problem into a searcher’s strategy on one side, and the target’s strategy on the other side. A good strategy for the searcher is determined using a sliding window procedure. Concerning the target, our approach consists in simulating some of the target’s possible strategies and considering each of these possibilities as an independent and deterministic entity. The second stage is dedicated to adjusting the large-scale strategy provided by stage 1. Finally, numerical results are presented so as to assess the impact of our approach.

Moving Target Search with Compressed Path Databases

Moving target search, where the goal state changes during a search, has recently seen a revived interest. Incremental Anytime Repairing A* (I-ARA*) is a very recent, state-ofthe-art algorithm for moving target search in a known terrain. In this work, we address the problem using compressed path databases (CPDs) in moving target search. CPDs have previously been used in standard, fixed-target pathfinding. They encode all-pairs shortest paths in a compressed form and require preprocessing and memory to store the database. In moving-target search, our speed results are orders of magnitude better than state of the art. The time per individual move is improved, which is important in real-time search scenarios, where the time available to make a move is limited. The number of hunter moves is very good, since CPDs provide optimal moves along shortest paths. Compared to previous successful methods, such as I-ARA*, our method is simple to understand and implement.