Forward-looking Sonars Research Articles

Autonomous Underwater Environment Perceiving and Modeling: An Experimental Campaign With FeelHippo AUV for Forward Looking Sonar-Based Automatic Target Recognition and Data Association

Seabed inspection is one of the most sought-after applications for autonomous underwater vehicles (AUVs). Acoustical sensors, such as side-scan sonars and forward-looking sonars (FLSs), are commonly favored over optical cameras to carry out such a task. Indeed, sonars are not influenced by illumination conditions and can provide high-range data. However, due to the lack of features and low resolution, acoustical images are often hard to interpret with conventional automatic techniques, forcing human operators to analyze thousands of collected images to identify the so-called objects of potential interest (OPIs). In this article, we report the development of an automatic target recognition (ATR) methodology to identify and localize OPIs in FLS imagery. Such detections have been then exploited to realize a virtual world model with the probabilistic multiple hypothesis anchoring data association and model tracking algorithm. Distinct models of convolutional neural networks have been trained with a data set acquired in May 2019 at the Naval Support and Experimentation Centre (Centro di Supporto e Sperimentazione Navale—CSSN) basin in La Spezia, Italy. The ATR strategy has been successfully validated offline with the data gathered in October 2019 in the same site where the seabed targets were replaced and relocated. As regards the world modeling technique, it has been preliminarily tested on a simulated scenario built upon unmanned underwater vehicle Simulator. Finally, both the ATR and world modeling systems were on-field tested in October 2020 at the CSSN basin in a multivehicle architecture by employing an acoustical channel between FeelHippo AUV and an autonomous moving buoy.

Evaluation of UKF-Based Fusion Strategies for Autonomous Underwater Vehicles Multisensor Navigation

In the underwater domain, guaranteeing accurate navigation for an autonomous underwater vehicle (AUV) is a complex but fundamental task to be achieved. As a matter of fact, only by ensuring a correct AUV localization, it is possible to accomplish surveillance, monitoring, and inspection missions. Most of the navigation filters for AUVs are based on Bayesian estimators, such as the Kalman filter, the extended Kalman filter (EKF), or the unscented Kalman filter (UKF), and employ different instruments, including the Doppler velocity log to perform the localization task. Recently, the use of payload sensors, such as cameras or forward-looking SONARs, in navigation-aiding has arisen as an interesting research field in the attempt to reduce the localization error drift. Such sensors, if used simultaneously, can provide multiple observations, which can be combined in a Kalman filtering framework to increase navigation robustness against noise sources. Navigation techniques that employ multiple devices can provide a high improvement of the estimation quality, but they can also cause an increase in terms of computational load. Consequently, strategies that can represent a tradeoff between these two conflicting goals have to be investigated. In this contribution, two different frameworks have been implemented and compared: on the one hand, a centralized iterative UKF-based navigation approach and, on the other hand, a sensor fusion framework with parallel local UKFs. The sequential (or iterated) UKF, where the correction step is iteratively performed for each available measurement, belongs to the first class of filters. The federated and the consensus-based decentralized UKFs can be categorized as the second class and they differ in the employed fusion strategy. Experimental navigation data obtained during sea trials performed at Vulcano Island, Messina, Italy has been used for offline validation. The results analysis focuses on both the navigation quality and the filter robustness against the reduction of the available measurements.

Performance Comparison of Feature Detectors on Various Layers of Underwater Acoustic Imagery

Image feature matching is essential in many computer vision applications, and the foundation of matching is feature detection, which is a crucial feature quantification process. This manuscript focused on detecting more features from underwater acoustic imageries for further ocean engineering applications of autonomous underwater vehicles (AUVs). Currently, the mainstream feature detection operators are developed for optical images, and there is not yet a feature detector oriented to underwater acoustic imagery. To better analyze the suitability of existing feature detectors for acoustic imagery and develop an operator that can robustly detect feature points in underwater imageries in the future, this manuscript compared the performance of well-established handcrafted feature detectors and that of the increasingly popular deep-learning-based detectors to fill the gap in the literature. The datasets tested are from the most commonly used side-scan sonars (SSSs) and forward-looking sonars (FLSs). Additionally, the detection idea of these detectors on the acoustic imagery phase congruency (PC) layer was innovatively proposed with the aim of finding a solution that balances detection accuracy and speed. The experimental results show that the ORB (Oriented FAST and Rotated BRIEF) and BRISK (Binary Robust Invariant Scalable Keypoints) detectors achieve the best overall performance, the FAST detector is the fastest, and the PC and Sobel layers are the most favorable for implementing feature detection.

Optimization Method for Wide Beam Sonar Transmit Beamforming.

Imaging and mapping sonars such as forward-looking sonars (FLS) and side-scan sonars (SSS) are sensors frequently used onboard autonomous underwater vehicles. To acquire information from around the vehicle, it is desirable for these sonar systems to insonify a large area; thus, the sonar transmit beampattern should have a wide field of view. In this work, we study the problem of the optimization of wide transmission beampatterns. We consider the conventional phased-array beampattern design problem where all array elements transmit an identical waveform. The complex weight vector is adjusted to create the desired beampattern shape. In our experiments, we consider wide transmission beampatterns (≥20) with uniform output power. In this paper, we introduce a new iterative-convex optimization method for narrowband linear phased arrays and compare it to existing approaches for convex and concave–convex optimization. In the iterative-convex method, the phase of the weight parameters is allowed to be complex as in disciplined convex–concave programming (DCCP). Comparing the iterative-convex optimization method and DCCP to the standard convex optimization, we see that the former methods archive optimized beampatterns closer to the desired beampatterns. Furthermore, for the same number of iterations, the proposed iterative-convex method achieves optimized beampatterns, which are closer to the desired beampattern than the beampatterns achieved by optimization with DCCP.

Automatic Target Recognition for Mine Countermeasure Missions Using Forward-Looking Sonar Data

The detection of objects on the seafloor is a complex task. The domain of the detection and classification of naval mines is additionally complicated by the high risk nature of the task. Autonomous underwater vehicles (AUVs) have been used in naval mine countermeasures (MCM) operations to search large areas using sensors such as sidescan or synthetic aperture sonars. These sensors generally have a high coverage rate, while sacrificing spatial resolution. Conversely, sensors with higher resolution but lower coverage (such as forward-looking sonars and electro–optical cameras) are employed for the later classification and identification stages of the MCM mission. However, to autonomously execute a target reacquisition mission, it is important to be able to collect and process data automatically and, in near real time, onboard an AUV. For this purpose, an automatic target recognition (ATR) system is required. This article proposes an ATR, which can be used onboard an autonomous vehicle, capable of detecting mine-like objects in forward-looking sonar data. The ATR combines a detector and a classifier, based on convolutional neural network models, with a probabilistic grid map that filters out false positives and combines reported detections at nearby locations. A strategy, combining a survey pattern with target-mapping maneuvers automatically activated by the ATR, has been designed to maximize the performance of this ATR. The whole system has been tested in simulation as well as using data from previous MCM exercises, the results of which are presented here.

CycleGAN-based realistic image dataset generation for forward-looking sonar

ABSTRACT In this paper, we propose a novel method to generate realistic acoustic datasets for forward-looking sonars. A forward-looking sonar, which is also known as an acoustic camera, outperforms other imaging sensor when applied in underwater tasks as it can provide more accurate and detailed information about the environment, even in dark or turbid water. However, the difficulty and high cost of acquiring acoustic images in real experiments encourage researchers to consider the generation of simulated acoustic image datasets. In particular, deep learning-based methods demonstrated high performance in computer vision tasks, such as in object detection. However, a large dataset is necessary in most cases. In the proposed method, we first build a novel user-friendly acoustic image simulator based on 3D modeling software. Then, the CycleGAN is applied to generate realistic acoustic images based on the generated dataset from the simulator. The experimental results demonstrate that our method can generate effective and realistic acoustic datasets with relatively simple operations.

Underwater navigation with 2D forward looking SONAR: An adaptive unscented Kalman filter‐based strategy for AUVs

AbstractOne of the most significant challenges in the underwater domain is to retrieve the autonomous underwater vehicle (AUV) position within the surrounding environment. Indeed, reliable navigation systems are fundamental to perform complex tasks and missions. Most of the navigation filters for AUVs are based on Bayesian estimators such as the linear Kalman Filter (KF), the extended KF, the unscented KF, or the particle filter where, usually, different instruments including a Doppler velocity log (DVL) contribute to the localization task. The usage of forward‐looking SONARs (FLS) in navigation‐aiding is, most of the time, devoted to limiting the navigation drift of the AUV by using simultaneous localization and mapping methods. Therefore, these devices are commonly employed with a standard navigation sensors set comprising an attitude heading reference system and a DVL. In this contribution, the authors propose a novel navigation strategy specifically tailored to AUVs based on an adaptive unscented KF, where linear speed estimations are obtained with a 2D FLS instead of with a DVL and therefore promoting the employment of FLSs as an aid for underwater navigation. The marine robotics community could gain significant benefits from reliable navigation achieved with an FLS‐based navigation architecture. Most importantly, a single FLS can be used for imaging‐related applications (i.e., sonograms acquisition) and navigation, where, instead, different dedicated devices are currently employed for the two tasks. Smaller AUVs usually possess reduced payload carrying capabilities; thus, multitasking use of onboard sensors, which leads to compactness, is a desirable feature. Navigation data obtained during sea trials performed in La Spezia (Italy) at the NATO STO Centre for Maritime Research and Experimentation has been used for offline validation. Afterward, the online results of real autonomous underwater missions undertaken in La Spezia (Italy) and at Vulcano Island, Messina (Italy), are reported.

ACMarker: Acoustic Camera-Based Fiducial Marker System in Underwater Environment

ACMarker is an acoustic camera-based fiducial marker system designed for underwater environments. Optical camera-based fiducial marker systems have been widely used in computer vision and robotics applications such as augmented reality (AR), camera calibration, and robot navigation. However, in underwater environments, the performance of optical cameras is limited owing to water turbidity and illumination conditions. Acoustic cameras, which are forward-looking sonars, have been gradually applied in underwater situations. They can acquire high-resolution images even in turbid water with poor illumination. We propose methods to recognize a simply designed marker and to estimate the relative pose between the acoustic camera and the marker. The proposed system can be applied to various underwater tasks such as object tracking and localization of unmanned underwater vehicles. Simulation and real experiments were conducted to test the recognition of such markers and pose estimation based on the markers.

Motion Estimation From Doppler and Spatial Data in SONAR Images

Motion estimation is critical for the localization of autonomous underwater vehicles. Current SONAR-based techniques exclusively utilize either Doppler or spatial measurements. However, these measurement domains are complementary to each other; Doppler measurements directly measure radial motion, whereas spatial measurements uniquely observe angular motion. Therefore, this article presents SONARODO (SONAR Odometry), a novel real-time motion estimation algorithm for 2-D forward-looking SONARs. It depends on a largely decoupled motion estimation process that better utilizes each measurement domain for their respective strengths. Specifically, it estimates translational motion from Doppler-azimuth images and rotational motion from range-azimuth images. While this method does require a SONAR that can provide both image types, it was designed to ensure robustness to featureless seafloor environments and low-resolution images. This article's validation with high-fidelity simulation data demonstrated that SONARODO offers accuracy and computational cost advantages over related motion estimation techniques.

A novel GPU-based sonar simulator for real-time applications

Mainly when applied in the underwater environment, sonar simulation requires great computational effort due to the complexity of acoustic physics. Simulation of sonar operation allows evaluating algorithms and control systems without going to the real underwater environment; that reduces the costs and risks of in-field experiments. This paper tackles with the problem of real-time underwater imaging sonar simulation by using the OpenGL shading language chain on GPU. Our proposed system is able to simulate two main types of acoustic devices: mechanical scanning imaging sonars and forward-looking sonars. The underwater scenario simulation is performed based on three frameworks: (i) OpenSceneGraph reproduces the ocean visual effects, (ii) Gazebo deals with physical forces, and (iii) the Robot Construction Kit controls the sonar in underwater environments. Our system exploits the rasterization pipeline in order to simulate the sonar devices, which are simulated by means of three parameters: the pulse distance, the echo intensity and the sonar field-of-view, being all calculated over observable objects shapes in the 3D rendered scene. Sonar-intrinsic operational parameters, speckle noise and object material properties are also considered as part of the acoustic image. Our evaluation demonstrated that the proposed system is able to operate close to or faster than the real-world devices. Also, our method generates visually realistic sonar images when compared with real-world sonar images of the same scenes.

Description and Matching of Acoustic Images Using a Forward Looking Sonar: A Topological Approach.

Autonomous underwater vehicles (AUVs) are an outstanding tool for underwater exploration. Due to the characteristics of the marine environment that limit the sensors capability, autonomous navigation is still a challenge. This paper proposes a loop closure detector addressed to the Simultaneous Localization and Mapping (SLAM) problem using acoustic images of forward-looking sonars. It is presented an acoustic image descriptor that uses a graph to represent the topologic relationship between the observed objects, a graph comparison approach to determine the similarity between acoustic images and also a segmentation method that uses an adaptative threshold strategy based on the acoustic beams analysis. The method is evaluated using a real dataset of a marina.

Obstacle Avoidance Approaches for Autonomous Underwater Vehicle: Simulation and Experimental Results

Autonomous underwater vehicles (AUVs) operate in unknown underwater environments and must be able to avoid submerged obstacles such as cliffs, wrecks, and seabed changes. This paper proposes a methodology for obstacle avoidance by AUVs that are equipped with forward-looking sonars (FLSs). The data collected from two FLSs placed in horizontal and vertical orientation are processed in real time to provide obstacle detection information in the $xz$ - and $xy$ -planes, respectively. Due to the necessity of maintaining constant height when employing sidescan sonar (SSS) and lower energy consumption, horizontal avoidance maneuvers are preferred over vertical ones. For the horizontal obstacle avoidance, a two-layer obstacle avoidance algorithm (OAA) contains two different processes, an OAA based on Bandler and Kohout (BK) product of fuzzy relation used as a preplanning method and a reactive approach based on potential field and edge detection methods. The preplanning technique has a clear advantage on a line segment of the path, and the reactive method is more efficient on a curved segment. In case a horizontal approach cannot find a path to safely avoid the obstacle, the reactive vertical approach is activated. A seabed gradient detection technique that allows prediction of seabed and altitude changes up to 40 m ahead of the AUV is presented. Simulation and experimental results clearly demonstrate that the proposed methodology enables AUVs to navigate safely through obstacles and provide crucial information about the seafloor terrain changes.

Forward looking sonar mosaicing for Mine Countermeasures

Forward looking sonars (FLS) are nowadays popular for many different applications. In particular, they can be used for Automatic Target Recognition (ATR) in the context of Mine Countermeasures. Currently, ATR techniques are applied to raw data which generates many false positives and the need for human supervision. Mosaicing FLS data increases target contrast and thus reduces false positive rate. Moreover, it implies a considerable data size reduction which is important if one thinks of exchange of data in real time through an acoustic channel with very limited bandwidth. Results of applying a real-time mosaicing algorithm to FLS data generated during Mine Countermeasures missions are shown and discussed thoroughly in this article.

ブースティングによる機械学習に基づく海底物体の検出

Side-scan and forward-looking sonars are some of the most widely used imaging systems for obtaining large scale images of the seafloor, and their use continues to expand rapidly with their increased deployment on autonomous underwater vehicles. However, it is difficult to extract quantitative information from the images generated from these processes, particularly for the detection and extraction of information on the objects within these images. We propose in this paper an algorithm for automatic detection of underwater objects in side-scan images based on machine learning employing adaptive boosting. Experimental results show that the method produces consistent maps of the seafloor.

A voting-based approach for fast object recognition in underwater acoustic images

This paper describes a voting-based approach for the fast automatic recognition of man-made objects and related attitude estimation in underwater acoustic images generated by forward-looking sonars or acoustic cameras. In general, the continuous analysis of sequences of images is a very heavy task for human operators and this is due to the poor quality of acoustic images. Hence, algorithms able to recognize an object on the basis of a priori knowledge of the model and to estimate its attitude with reference to a global coordinate system are very useful to facilitate underwater operations like object manipulation or vehicle navigation. The proposed method is capable of recognizing objects and estimating their two-dimensional attitude by using information coming from boundary segments and their angular relations. It is based on a simple voting approach directly applied to the edge discontinuities of underwater acoustic images, whose quality is usually affected by some undesired effects such as object blurring, speckle noise, and geometrical distortions degrading the edge detection. The voting approach is robust, with respect to these effects, so that good results are obtained even with images of very poor quality. The sequences of simulated and real acoustic images are presented in order to test the validity of the proposed method in terms of average estimation error and computational load.