Multi-beam Forward Looking Sonar Research Articles

Advanced deep learning framework for underwater object detection with multibeam forward-looking sonar

Underwater object detection (UOD) is an essential activity in maintaining and monitoring underwater infrastructure, playing an important role in their efficient and low-risk asset management. In underwater environments, sonar, recognized for overcoming the limitations of optical imaging in low-light and turbid conditions, has increasingly gained popularity for UOD. However, due to the low resolution and limited foreground-background contrast in sonar images, existing sonar-based object detection algorithms still face challenges regarding precision and transferability. To solve these challenges, this article proposes an advanced deep learning framework for UOD that uses the data from multibeam forward-looking sonar. The framework is adapted from the network architecture of YOLOv7, one of the state-of-the-art vision-based object detection algorithms, by incorporating unique optimizations in three key aspects: data preprocessing, feature fusion, and loss functions. These improvements are extensively tested on a dedicated public dataset, showing superior object classification performance compared to the selected existing sonar-based methods. Through experiments conducted on an underwater remotely operated vehicle, the proposed framework validates significant enhancements in target classification, localization, and transfer learning capabilities. Since the engineering structures have similar geometric shapes to the objects tested in this study, the proposed framework presents potential applicability to underwater structural inspection and monitoring, and autonomous asset management.

Multibeam Forward-Looking Sonar Video Object Tracking Using Truncated - Sparsity and Aberrances Repression Regularization

Multibeam Forward-Looking Sonar Video Object Tracking Using Truncated - Sparsity and Aberrances Repression Regularization

Acoustic communication and imaging sonar guided AUV docking: system infrastructure, docking methodology and lake trials

Autonomous underwater vehicle (AUV) docking technique attracts attention which leverages the long-term, on-station vehicle launch and recovery for the underwater missions. In this paper, an acoustic communication and imaging sonar guided docking method is proposed to improve the docking accuracy for AUV. In docking system, multibeam forward looking sonar (MFLS) is deployed on the submerged docking station to reduce the noise in acoustic images rather than being fixed on AUV, and the ultra short base line (USBL) transceiver with acoustic communication modules is used to achieve positioning while AUV is out of the view of MFLS. For the docking methodology, a dedicated three-stage docking strategy is presented to guide AUV to the docking station. In the homing stage, AUV navigates to a preset homing place not very far from the docking station via the way-point guidance and self-positioning of the on-board inertial navigation system (INS). In the next stage, USBL provides the relative position for AUV controller via acoustic channel. Moreover, an exponential decay model based filtering algorithm is proposed to eliminate the positioning outliers of USBL. It is expected that AUV will be guided into the perceptual field of MFLS in this stage. In the final stage, the docking accuracy in short range should be improved while USBL may generate outliers due to the louder noise. In this case, the relative positioning is implemented by using kernelized correlation filter (KCF), which is a fast objective tracking algorithm to keep track of the AUV feature in acoustic image until AUV navigates into the docking station or out of the view of sensor. To validate feasibility and performance of the proposed system infrastructure and docking methodology, the docking results are analyzed by contrast with the experiments only use USBL during lake trials.

Three-Dimensional Underwater Path Planning Based on Modified Potential Field Algorithm in Time-Varying Current

Abstract The article addresses the three-dimensional (3D) underwater path planning problem of an autonomous underwater vehicle (AUV) in a time-varying current. A modified artificial potential field algorithm combining the velocity vector synthesis method is proposed to search for the optimal path. The modified potential field (MPF) algorithm is designed to dynamically plan the non-collision path. Meanwhile, this modified method is also proved to be an effective solution to the goals not reachable with obstacles nearby (GNRON), U-shaped trap, and rotation unreachable problems. To offset the influence of time-varying current, the velocity synthesis approach is designed to adjust the AUV movement direction. Besides, considering path planning in the complex underwater environment, the multi-beam forward-looking sonar (FLS) model is used. Finally, simulation studies substantiate that the designed algorithm can implement the AUV path planning effectively and successfully in a 3D underwater environment.

A Dataset with Multibeam Forward-Looking Sonar for Underwater Object Detection

Multibeam forward-looking sonar (MFLS) plays an important role in underwater detection. There are several challenges to the research on underwater object detection with MFLS. Firstly, the research is lack of available dataset. Secondly, the sonar image, generally processed at pixel level and transformed to sector representation for the visual habits of human beings, is disadvantageous to the research in artificial intelligence (AI) areas. Towards these challenges, we present a novel dataset, the underwater acoustic target detection (UATD) dataset, consisting of over 9000 MFLS images captured using Tritech Gemini 1200ik sonar. Our dataset provides raw data of sonar images with annotation of 10 categories of target objects (cube, cylinder, tyres, etc). The data was collected from lake and shallow water. To verify the practicality of UATD, we apply the dataset to the state-of-the-art detectors and provide corresponding benchmarks for its accuracy and efficiency.

Underwater Localization and Mapping Based on Multi-Beam Forward Looking Sonar.

SLAM (Simultaneous Localization And Mapping) plays a vital role in navigation tasks of AUV (Autonomous Underwater Vehicle). However, due to a vast amount of image sonar data and some acoustic equipment's inherent high latency, it is a considerable challenge to implement real-time underwater SLAM on a small AUV. This paper presents a filter based methodology for SLAM algorithms in underwater environments. First, a multi-beam forward looking sonar (MFLS) is utilized to extract environmental features. The acquired sonar image is then converted to sparse point cloud format through threshold segmentation and distance-constrained filtering to solve the calculation explosion issue caused by a large amount of original data. Second, based on the proposed method, the DVL, IMU, and sonar data are fused, the Rao-Blackwellized particle filter (RBPF)-based SLAM method is used to estimate AUV pose and generate an occupancy grid map. To verify the proposed algorithm, the underwater vehicle is equipped as an experimental platform to conduct field tasks in both the experimental pool and wild lake, respectively. Experiments illustrate that the proposed approach achieves better performance in both state estimation and suppressing divergence.

Multiple Receptive Field Network (MRF-Net) for Autonomous Underwater Vehicle Fishing Net Detection Using Forward-Looking Sonar Images.

Underwater fishing nets represent a danger faced by autonomous underwater vehicles (AUVs). To avoid irreparable damage to the AUV caused by fishing nets, the AUV needs to be able to identify and locate them autonomously and avoid them in advance. Whether the AUV can avoid fishing nets successfully depends on the accuracy and efficiency of detection. In this paper, we propose an object detection multiple receptive field network (MRF-Net), which is used to recognize and locate fishing nets using forward-looking sonar (FLS) images. The proposed architecture is a center-point-based detector, which uses a novel encoder-decoder structure to extract features and predict the center points and bounding box size. In addition, to reduce the interference of reverberation and speckle noises in the FLS image, we used a series of preprocessing operations to reduce the noises. We trained and tested the network with data collected in the sea using a Gemini 720i multi-beam forward-looking sonar and compared it with state-of-the-art networks for object detection. In order to further prove that our detector can be applied to the actual detection task, we also carried out the experiment of detecting and avoiding fishing nets in real-time in the sea with the embedded single board computer (SBC) module and the NVIDIA Jetson AGX Xavier embedded system of the AUV platform in our lab. The experimental results show that in terms of computational complexity, inference time, and prediction accuracy, MRF-Net is better than state-of-the-art networks. In addition, our fishing net avoidance experiment results indicate that the detection results of MRF-Net can support the accurate operation of the later obstacle avoidance algorithm.

An improved recurrent neural network for unmanned underwater vehicle online obstacle avoidance

This paper focuses on online obstacle avoidance planning for unmanned underwater vehicles. To improve the autonomous ability and intelligence of obstacle avoidance planning, a recurrent neural network with convolution is proposed. In the proposed method, convolution replaces full connection in the standard recurrent neural network, thus reducing the number of parameters and improving the feature extraction capability. Training and test datasets are generated for the deep learning process and, combined with multibeam forward-looking sonar, this learning system can automatically realize online obstacle avoidance for unmanned underwater vehicles. Experiments are designed to compare the performance of the proposed structure with that of a recurrent neural network, gated recurrent units, and conventional ant colony optimization. The results fully verify the effectiveness and feasibility of the proposed method, and show that the powerful learning and memory capabilities of the proposed structure can be used in the unmanned underwater vehicle autonomous learning environment. This study demonstrates that recurrent neural networks with convolution greatly enhance the ability of unmanned systems to sense and adapt to unknown environments.

Attitude-Trajectory Estimation for Forward-Looking Multibeam Sonar Based on Acoustic Image Registration

This work considers the processing of acoustic data from a multibeam forward-looking sonar (FLS) on a moving underwater platform to estimate the platform's attitude and trajectory. We propose an algorithm to produce an estimate of the attitude trajectory for an FLS based on the optical flow between consecutive sonar frames. The attitude trajectory can be used to locate an underwater platform, such as an autonomous underwater vehicle, to a degree of accuracy suitable for navigation. It can also be used to build a mosaic of the underwater scene. The estimation is performed in three steps. First, a selection of techniques based on the optical flow model is used to estimate a pixel displacement map (DM) between consecutive sonar frames represented in the native polar (range/bearing) format. The second step finds the best match between the estimated DM and DMs for a set of modeled sonar sensor motions. To reduce complexity, it is proposed to describe the DM with a small parameter vector derived from the displacement distribution. Thus, an estimate of the incremental sensor motion between frames is made. Finally, using a weighted regularized spline technique, the incremental interframe motions are integrated into an attitude trajectory for the sonar sensor. To assess the accuracy of the attitude-trajectory estimate, it is used to register FLS frames from a field experiment data set and build a high-quality mosaic of the underwater scene.

Beam Slice-Based Recognition Method for Acoustic Landmark With Multi-Beam Forward Looking Sonar

Autonomous underwater vehicles (AUVs) have been widely used for many risky underwater tasks. For these tasks, AUVs require navigational aids, such as active beacon-type underwater acoustic landmarks, which are the most widely used. It requires heavy installation loads and regular maintenance with high price sensors. In this paper, we propose a novel localization method in shallow water with a multi-beam forward looking sonar (MFLS), where positioning is based on passive-type acoustic landmarks. The proposed landmark comprises a combination of concrete pillars with a highly reliable and efficient recognition method for the real-time use of AUVs. The proposed landmark system has many advantages in the practical use, such as no maintenance, being strong on the marine bio-fouling, and light installation load with low price concrete pillars. The proposed recognition method is divided into two processes to verify the landmark. First, a pillarlike object is separated from the background, and the height of the pillar is calculated by the length of the shadow, which has a very high recognition rate and accuracy. Through modeling, the distance from the landmark to the MFLS can be calculated. The proposed method stochastically updates the navigation data based on the positional relationship with the landmark. The performance of the proposed landmark and its recognition method is verified through a water tank and field experiments using the AUV (Cyclops). The recognition rate and accuracy are also discussed.

A Real-Time Image Processing Method for Multi-Beam Forward-Looking Sonar of AUV

The key of real-time obstacle avoidance for AUV is to accurately and fast identify underwater obstacle. In this paper, according to the characteristics of the multi-beam forward-looking sonar of AUV, we extracted the obstacle information from sonar image with the image filtering, fuzzy clustering and morphological methods. Finally, it can be proved that the real-time image processing method is feasible and effective by the lake trial.

Application of A* algorithm for real-time path re-planning of an unmanned surface vehicle avoiding underwater obstacles

This paper describes path re-planning techniques and underwater obstacle avoidance for unmanned surface vehicle (USV) based on multi-beam forward looking sonar (FLS). Near-optimal paths in static and dynamic environments with underwater obstacles are computed using a numerical solution procedure based on an A* algorithm. The USV is modeled with a circular shape in 2 degrees of freedom (surge and yaw). In this paper, two-dimensional (2-D) underwater obstacle avoidance and the robust real-time path re-planning technique for actual USV using multi-beam FLS are developed. Our real-time path re-planning algorithm has been tested to regenerate the optimal path for several updated frames in the field of view of the sonar with a proper update frequency of the FLS. The performance of the proposed method was verified through simulations, and sea experiments. For simulations, the USV model can avoid both a single stationary obstacle, multiple stationary obstacles and moving obstacles with the near-optimal trajectory that are performed both in the vehicle and the world reference frame. For sea experiments, the proposed method for an underwater obstacle avoidance system is implemented with a USV test platform. The actual USV is automatically controlled and succeeded in its real-time avoidance against the stationary undersea obstacle in the field of view of the FLS together with the Global Positioning System (GPS) of the USV.