Forward-looking Sonar Images Research Articles

SimNFND: A Forward-Looking Sonar Denoising Model Trained on Simulated Noise-Free and Noisy Data

Given the propagation characteristics of sound waves and the complexity of the underwater environment, denoising forward-looking sonar image data presents a formidable challenge. Existing studies often add noise to sonar images and then explore methods for its removal. This approach neglects the inherent complex noise in sonar images, resulting in inaccurate evaluations of traditional denoising methods and poor learning of noise characteristics by deep learning models. To address the lack of high-quality data for FLS denoising model training, we propose a simulation algorithm for forward-looking sonar data based on RGBD data. By utilizing rendering techniques and noise simulation algorithms, high-quality noise-free and noisy sonar data can be rapidly generated from existing RGBD data. Based on these data, we optimize the loss function and training process of the FLS denoising model, achieving significant improvements in noise removal and feature preservation compared to other methods. Finally, this paper performs both qualitative and quantitative analyses of the algorithm’s performance using real and simulated sonar data. Compared to the latest FLS denoising models based on traditional methods and deep learning techniques, our method demonstrates significant advantages in denoising capability. All inference results for the Marine Debris Dataset (MDD) have been made open source, facilitating subsequent research and comparison.

WTCRNet: a wavelet transform and contrastive regularization network for sonar denoising by self-supervision

In the field of underwater acoustics, forward-looking sonar represents a pivotal tool for acquiring subaqueous imagery. However, this technique is susceptible to the inherent ambient noise prevalent in underwater environments, resulting in degraded image quality. A notable challenge in this domain is the scarcity of pristine image exemplars, making it difficult to apply many advanced deep denoising networks for the purification of sonar images. To address this issue, the study introduces a novel self-supervised methodology specifically designed for denoising forward-looking sonar images. The proposed model employs a blind-spot network architecture to reconstruct unblemished images. Additionally, it integrates wavelet transform technology within a convolutional neural network (CNN) framework, combining frequency and structural information. Furthermore, the model incorporates contrastive regularization to augment denoising efficiency. This innovative denoising network, which leverages wavelet transform and contrastive regularization (CR), is henceforth referred to as WTCRNet. To evaluate the performance of WTCRNet, this study constructs a dual dataset comprising both simulated and authentic forward-looking sonar images, thereby furnishing a comprehensive dataset for network training and evaluation. Empirical assessments conducted on these datasets demonstrate that WTCRNet substantially outperforms existing denoising methodologies by effectively mitigating noise. The code is available at https://gitee.com/sichengling/wtcrnet.git.

Target positioning of dual forward looking sonars based on orthogonal detection

Accurate positioning of underwater targets is an important basis for many underwater tasks. As an imagery equipment, forward-looking sonar (FLS) can overcome the problem of poor brightness in underwater environment, and has a longer detection distance than the camera, which plays an important role in oceanic target positioning. However, there remain two challenging problems: (1) due to the complex underwater environment and the propagation characteristics of sound waves in the water, there are a lot of noises in the FLS images, which impair the accuracy of positioning; (2) FLS can only obtain two-dimensional information, which is insufficient for target positioning. Aiming at these two problems, a three-dimensional underwater target positioning algorithm based on dual FLSs is proposed in this paper. Firstly, a Threshold-Mask-Guided Filter (TMGF) image denoising algorithm is designed to deal with the obvious background noises and same-pitch noises in FLS images, which greatly improves the denoising effect compared with the original guided filter algorithm. Secondly, a target positioning algorithm based on orthogonal vertical placement of dual sonars is proposed to calculate the three-dimensional position of the target. Finally, pool experiments and sea trial tests are carried out respectively to verify the effectiveness of TMGF algorithm in denoising and the accuracy and stability of positioning algorithm.

A Lightweight Underwater Target Detection Network for Forward-Looking Sonar Images

A Lightweight Underwater Target Detection Network for Forward-Looking Sonar Images

Forward-Looking Sonar Image Stitching Based on Midline Template Matching in Polar Image

Forward-Looking Sonar Image Stitching Based on Midline Template Matching in Polar Image

SonarNet: Hybrid CNN-Transformer-HOG Framework and Multifeature Fusion Mechanism for Forward-Looking Sonar Image Segmentation

SonarNet: Hybrid CNN-Transformer-HOG Framework and Multifeature Fusion Mechanism for Forward-Looking Sonar Image Segmentation

Estimating forward-looking sonar (FLS) image quality using water column data features

Forward-looking sonar (FLS) underwater imaging systems are primarily used for measuring seafloor backscatter. These returns, however, may be occluded due to scattering from other objects in the water such as fish, bubbles, or other suspended objects, and in-band acoustic interference from sources like engines or other acoustic transmissions. The presence of non-seafloor scatterers or interference is most easily observed and estimated in the water column data, the portion of the time-series after the sonar has transmitted a ping, but before that ping has scattered off the seafloor and returned to the FLS. Analysis of the water column time-series returns can be leveraged to estimate the quality of seafloor imagery. In this study, we characterize bright scatterers that appear in the water column data using statistical tools and generate corresponding quality metrics to evaluate imagery.

Time-domain Kirchhoff approximation extensions modeling of acoustic camera imagery with multiple scattering

High-frequency acoustic cameras have provided high-resolution sonar images to enable the development of signal processing algorithms, such as 3-D reconstruction, target recognition, and precision navigation. However, the lack of large datasets has hampered the development of such algorithms. In this context, this paper proposes a time-domain Kirchhoff approximation extensions (TDKAX)-based model to generate realistic acoustic camera images, capable of accounting for multiple scattering from arbitrary-shapetargets. The images are formed by a coherent summation of different orders of transient echoes, along with a pipeline of the range conversion, time-varying gain compensation, noise addition, and image rendering. Validating the proposed model using tank experiments of isolated cylindrical concave reflectors, we then present the ray paths based on the coordinates of intersection points. They can help interpret the mechanism of image features in the absence of amplitude information. Results demonstrate that, with the temporal and spatial coherence of different orders of scattering, the proposed model successfully reproduces forward-looking sonar (FLS) images with multiple scattering arcs and coherent speckles.

An underwater small target boundary segmentation method in forward-looking sonar images

Underwater target boundary segmentation is integral to forward-looking sonar image processing. However, small target boundary segmentation is always challenging due to the low resolution, low signal-to-noise ratio, and inhomogeneity of intensity of forward-looking sonar images. This paper proposes an improved level set segmentation method to accurately obtain the contour of small targets in the forward-looking sonar images: morphological reconstruction combined with the level set method (MRLSM). Compared with the classical level set and the level set method combined with the morphological method, MRLSM improves the accuracy and stability for forward-looking sonar image segmentation. Furthermore, the fuzzy C-means, Markov random field, and MRLSM are applied to the numerical simulations and experimental data for comparison. The deep learning methods are also used to compare the performances. The results demonstrated that the proposed method is more accurate, robust, and considerable for underwater small target boundary segmentation.

Beam-Domain Image Mosaic of Forward-Looking Sonar Using Expression Domain Mapping Model

Since the existing forward-looking sonar (FLS) image mosaic algorithm is completed in the image-domain with reference to the optical image, and the characteristics of FLS imaging are not fully considered, obvious edge-seams and pseudo-target values are prone to generate after mosaic, resulting in poor visual effect and low operation efficiency. To avoid the aforementioned problems, the mosaic operation of beam-domain FLS images is proposed. Combined with the imaging principle of FLS, this study takes advantage of the variation of the sampling beams and sampling points of the same target point during sonar movement to analyze the deformation law of the target in beam-domain which is scanned in image-domain, and then proposes the concept of expression domain mapping model (EDMM). Considering the need of aligning acoustic origin, the target is deformed according to this model in the process of beam-domain mosaic, and the target alignment is completed. Subsequently, the Laplace pyramid (LP) layered algorithm is used for fusion and mosaic, and finally mapped to image-domain for display. Experimental results illustrate that this kind of algorithm can remove the mosaic edge-seams generated during mosaicking in the image-domain because it uses the sonar raw image data for mosaic in the beam-domain and avoids the appearance of sector edges in the image-domain; the actual pixel points sampled by sonar are used for mosaic, which effectively eliminates the pseudo-target value generated by the mosaic of the inter-beam interpolation in the image-domain; moreover, the number of image pixels in the beam-domain is small, and the mosaic time consuming of adjacent frames and continuous frames is more than half shorter than that of the image domain, which significantly improves the mosaic efficiency and has good mosaic performance.

Dynamic Target Tracking Control of Autonomous Underwater Vehicle Based on Trajectory Prediction.

Underwater dynamic target tracking technology has a wide application prospect in marine resource exploration, underwater engineering operations, naval battlefield monitoring, and underwater precision guidance. Aiming at the underwater dynamic target tracking problem, an autonomous underwater vehicle tracking control method based on trajectory prediction is studied. First, a deep learning-based target detection algorithm is developed. For the image collected by the multibeam forward-looking sonar image, this algorithm uses the YOLO v3 network to determine the target in a sonar image and obtain the position of the target. Then, a time profit Elman neural network (TPENN) is constructed to predict the trajectory information of the dynamic target. Compared with an ordinary Elman neural network, its accuracy of dynamic target prediction is increased. Finally, underwater tracking of the dynamic target is realized using the model predictive controller (MPC), and the tracking result is stable and reliable. Through simulations and experiment, the proposed underwater dynamic target tracking control method is demonstrated to be effective and feasible.

A Texture Feature Removal Network for Sonar Image Classification and Detection

Deep neural network (DNN) was applied in sonar image target recognition tasks, but it is very difficult to obtain enough sonar images that contain a target; as a result, the direct use of a small amount of data to train a DNN will cause overfitting and other problems. Transfer learning is the most effective way to address such scenarios. However, there is a large domain gap between optical images and sonar images, and common transfer learning methods may not be able to effectively handle it. In this paper, we propose a transfer learning method for sonar image classification and object detection called the texture feature removal network. We regard the texture features of an image as domain-specific features, and we narrow the domain gap by discarding the domain-specific features, and hence, make it easier to complete knowledge transfer. Our method can be easily embedded into other transfer learning methods, which makes it easier to apply to different application scenarios. Experimental results show that our method is effective in side-scan sonar image classification tasks and forward-looking sonar image detection tasks. For side-scan sonar image classification tasks, the classification accuracy of our method is enhanced by 4.5% in a supervised learning experiment, and for forward-looking sonar detection tasks, the average precision (AP) is also significantly improved.

Underwater Forward-Looking Sonar Images Target Detection via Speckle Reduction and Scene Prior

Forward-looking sonar (FLS) imagery system plays a significant role in oceanic object recognition and detection since it can overcome the limitation of lighting conditions and reflect the real situation of the underwater environment. However, object detection algorithms for FLS images remain challenging for two main reasons: 1) The noise caused by the coherent characteristic of the scattering phenomenon impairs the detector capture of target information; 2) The scene prior based on the uneven target scale distribution is generally neglected, which leads to the detector generating redundant anchors and slows down detection efficiency. Confronting such challenges, this paper characterizes the noise and the uneven target scale distribution in FLS images as multiplicative speckle noise and scene prior, respectively. Therefore, we propose a novel underwater FLS image detection network, namely UFIDNet, to further improve detection performance by considering speckle noise reduction and scene prior in FLS images. More specifically, a speckle reduction auxiliary branch (SRAB) is designed to introduce additional despeckled supervision information to encourage the feature extractor to produce clean features and share them with the detection pipeline during the training phase. In particular, the noise distribution of FLS images is excavated for synthetic dataset construction and Despeckle Network (DSN) design to obtain despeckled supervision images. In addition, a feature selection strategy (FSS) embedded in detection branch is designed to screen out feature levels that do not match the target size, thus significantly reducing the generation of redundant anchors and improving detection speed. Experimental results show that our UFIDNet achieves 70.5% and 47.3% average precision (AP), 81.3% and 54.6% average recall (AR <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">max=10</sub> ), 27.0 and 26.1 FPS on two real FLS datasets respectively, outperforming many state-of-the-art general detectors and sonar image detectors.

LPMsDE: Multi-Scale Denoising and Enhancement Method Based on Laplacian Pyramid Framework for Forward-Looking Sonar Image

LPMsDE: Multi-Scale Denoising and Enhancement Method Based on Laplacian Pyramid Framework for Forward-Looking Sonar Image

Feature Pyramid U-Net with Attention for Semantic Segmentation of Forward-Looking Sonar Images.

Forward-looking sonar is a technique widely used for underwater detection. However, most sonar images have underwater noise and low resolution due to their acoustic properties. In recent years, the semantic segmentation model U-Net has shown excellent segmentation performance, and it has great potential in forward-looking sonar image segmentation. However, forward-looking sonar images are affected by noise, which prevents the existing U-Net model from segmenting small objects effectively. Therefore, this study presents a forward-looking sonar semantic segmentation model called Feature Pyramid U-Net with Attention (FPUA). This model uses residual blocks to improve the training depth of the network. To improve the segmentation accuracy of the network for small objects, a feature pyramid module combined with an attention structure is introduced. This improves the model's ability to learn deep semantic and shallow detail information. First, the proposed model is compared against other deep learning models and on two datasets, of which one was collected in a tank environment and the other was collected in a real marine environment. To further test the validity of the model, a real forward-looking sonar system was devised and employed in the lake trials. The results show that the proposed model performs better than the other models for small-object and few-sample classes and that it is competitive in semantic segmentation of forward-looking sonar images.

Forward looking sonar image segmentation based on empirical mode decomposition

Due to the low imaging quality and serious noise pollution of forward-looking sonar images, traditional image segmentation methods based on edge information or statistical information are difficult to obtain high precision and robust segmentation results. Therefore, it is very complicated to segment forward-looking sonar images. Based on the analysis of the characteristics of forward-looking sonar, a new image segmentation method for forward-looking sonar is proposed.Firstly, 2d maximum entropy segmentation principle combined with chicken flock optimization algorithm is used to remove the background of the forward-looking sonar image.Then, based on 2d empirical mode decomposition, appropriate eigenmode functions are selected to denoise and enhance the forward-looking sonar image.Finally, the reconstructed forward-looking sonar image is segmented by enhanced fuzzy C-means clustering, and the segmented forward-looking sonar image result is obtained. This method can not only suppress noise interference effectively, but also protect the details of image edge for segmentation.

Detection of Weak and Small Targets in Forward-Looking Sonar Image Using Multi-Branch Shuttle Neural Network

Autonomous Underwater Vehicle (AUV) is an automated navigation device, which can independently complete the underwater investigation and detection without supervision. However, the marine environment is getting increasingly severe, and the garbage in the ocean and fishnets cast by fishermen have a great negative influence on the work of AUV. To solve the problems of low precision and slow speed in the state-of-the-art network detection of weak and small targets in underwater sonar images, Yolo5 was improved by building the multi-branch shuttle neural network The dataset is collected by multi-beam forward-looking sonar Gemini720i, and includes “fishnet” and two representative types of knitted and plastic garbage, i.e., “cloth” and “plasticbag”. The original dataset is enhanced and balanced, and the effect of dataset distribution on the model performance is studied. Utilizing pre-training, the effects of Yolo5 family, i.e., Yolo5s, Yolo5m, Yolo5l and Yolo5x are compared using the balanced dataset, to explore the impacts on detecting weak and small targets of the deeper and wider networks of this family. In addition, the combinations of Yolo5s with lightweight networks, i.e., MobileNet3 and ShuffleNet2 are experimented, which further illustrates the effectiveness of the proposed network, and can satisfy the high accuracy and real-time requirements of AUV.

Target Detection of Forward-Looking Sonar Image Based on Improved YOLOv5

Forward-looking sonar is a commonly used underwater detection device at present, but the detection accuracy is poor due to the complex underwater environment, small target highlight area and fuzzy feature details. Therefore, this paper proposes a forward sonar image target detection model based on You Only Look Once Version 5 (YOLOv5) network using transfer learning method. First, the YOLOv5 network is pretrained with COCO data set. Then the pre-training model is fine-tuned according to the training set of forward-looking sonar images. Before fine-tuning, the traditional k-means clustering is improved. The intersection over union ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$IoU$ </tex-math></inline-formula> ) value is used as the distance function to cluster the labeling information of the training set of the forward-looking sonar image. The results of clustering serve as the initial anchor frame of the training network. This operation greatly improves the detection speed. Second, due to the characteristics of weak echo intensity and small target area of forward-looking sonar image, an improved feature extraction method of CoordConv was proposed to give corresponding coordinate information to high-level features which improves the accuracy of network detection regression. Finally, the fine-tuned network is used to detect the target in the forward-looking sonar image. The experimental results show that the improved model based on YOLOv5 network is superior to the original YOLOv5 network and other popular deep neural networks for target detection in the forward-looking sonar image, which has a reference significance for underwater target detection. The CoordConv-YOLOv5 network based on transfer learning proposed in this paper shows the best performance in both detection accuracy and detection speed. Detection accuracy mAP@0.5:0.95 can reach 56.95%, and detection speed can reach 9ms.

The Acoustic System of the Fendouzhe HOV.

Due to the strong absorption and attenuation of electromagnetic waves by water, radio communications and global positioning systems are lacking in the deep-sea environment. Therefore, underwater long-distance communications, positioning, detection and other functions depend on acoustic technology. In order to realize the above functions, the acoustic system of the Fendouzhe human occupied vehicle (HOV) is composed of eight kinds of sonars and sensors, which is one of the core systems of manned submersible. Based on the Jiaolong/Shenhai Yongshi HOVs, the acoustic system of the Fendouzhe HOV has been developed. Compared with the previous technology, there are many technical improvements and innovations: 10,000-m underwater acoustic communication, 10,000-m underwater acoustic positioning, multi-beam forward-looking imaging sonar, an integrated navigation system, etc. This study introduces the structure of the acoustic system of the Fendouzhe HOV and the technical improvements compared with the Jiaolong/Shenhai Yongshi HOVs. The results of the acoustic system are illustrated by the 10,000-m sea trails in the Mariana Trench from October to December 2020.

A submarine pipeline segmentation method for noisy forward-looking sonar images using global information and coarse segmentation

The submarine pipeline is an important way to transport marine resources such as oil and gas. For maintaining submarine pipelines, autonomous underwater vehicles (AUVs) equipped with sonars are leveraged. Whether the complete submarine pipelines can be segmented from the complex and noisy background is the key to path planning and navigation, which determines the efficiency of AUVs. Existing methods for segmenting sonar images such as Markov random field (MRF) are insufficient to achieve this goal. Inspired by the similarities between submarine pipelines in sonar images and salient objects in natural images, we propose a novel method to extract pipelines which can be summarized as three steps. First, an effective denoising method via curvelet thresholding is proposed to suppress speckle noise. Second, we attempt to compute the probability whether a pixel belongs to submarine pipelines by global information containing four similarities. Two of them are shape similarity and angle similarity, which are proposed according to the characteristics of pipelines in sonar images. Third, we propose a coarse segmentation method based on bootstrapped support vector machines (SVMs) and a fusion strategy to generate binary segmentation results precisely and completely. We captured 15 sonar sequences via the AUV equipped with a forward-looking sonar at Qingdao National Deep-Sea Base and selected 262 sonar images from the sequences for evaluating the segmentation performance. The proposed method is compared with other advanced methods. The experimental results show that our method achieves a better performance than other methods, and demonstrate the feasibility of segmenting submarine pipelines inspired by saliency segmentation.