Side-scan Sonar Images Research Articles

Side-Scan Sonar Analysis Using ROI Analysis and Deep Neural Networks

Side-scan sonar generates an image based on the reflection of sound waves. The reflection takes place with the first encountered object, which, in practice, may be a passing fish or even a wave from the boat turbine. Therefore, such data are sensitive to all noises. In this article, we propose a real-time automatic system of side-scan sonar analysis that can detect and classify different objects. Our proposition is based on the processing of a given image to quickly verify whether there is anything other than the bottom of the river/sea. If so, this image is analyzed in terms of regions of interest (ROIs) by the histogram module. This action allows the extraction of only objects of interest, which are then classified by convolutional neural networks (CNNs). A proposed model also contains an automatic mechanism of adding the sample to the database in order to later guarantee the accuracy of the classifiers. The model is a hybridization of image processing techniques with a machine learning approach to analyze difficult images. The presented system has been tested on the bottom of a river in Szczecin, Poland. We reached 90% of accurate classification in case scenario and 88% in simulation on used datasets. The obtained results were presented and discussed in terms of the advantages of practical application in analyzing side-scan sonar images.

Comprehensive Sample Augmentation by Fully Considering SSS Imaging Mechanism and Environment for Shipwreck Detection Under Zero Real Samples

To solve the shortage of training samples when using deep learning to detect shipwrecks, a comprehensive sample augmentation method is proposed. The method fully considers the imaging mechanism and environment of side-scan sonar (SSS), such as acoustic emission and reception, waterbody, target reflection, and seafloor background, and generates diverse and representative shipwreck samples from five aspects of target diversity, target texture, imaging resolution, equipment and environmental noises, and background through a series of novel sample augmentation methods. Under the condition of zero real SSS samples, a detection model of YOLOv5s was established with these amplified samples and achieved a mean average precision (MAP) better than 96% for real SSS data detection.

Side-scan sonar image processing: Seabed classification based on acoustic backscattering

The smoothness of vessel traffic flow is the most important thing in the shipping industry of port. Traffic problems are commonly solved by development and maintenance programs. Seabed conditions in the port-channel should be known to be considered in port development and maintenance programs related to port efficiency, safety navigation, and berthing. The objective of this paper is to characterize seabed into several classes of geological features. The Seabed condition and characteristics are classified based on image processing of side scan sonar data. The image processing will extract pixel value parameters; intensity, entropy, and standard deviation. Classification use combination of these pixel view parameter to define each class. Seabed classification has been successfully carried out in Teluk Bayur Port and classified into five classes, sandy silt, silty sand, fine sand, coarse sand, and rocks or reefs. Indication of crack or shallow structure was also identified. These results of classification are necessary to verify by sediment sampling and visual inspection, and then it should be reclassified to become a valid classification.

A Robust and Fast Method for Sidescan Sonar Image Segmentation Based on Region Growing.

For high-resolution side scan sonar images, accurate and fast segmentation of sonar images is crucial for underwater target detection and recognition. However, due to the characteristics of low signal-to-noise ratio (SNR) and complex environmental noise of sonar, the existing methods with high accuracy and good robustness are mostly iterative methods with high complexity and poor real-time performance. For this purpose, a region growing based segmentation using the likelihood ratio testing method (RGLT) is proposed. This method obtains the seed points in the highlight and the shadow regions by likelihood ratio testing based on the statistical probability distribution and then grows them according to the similarity criterion. The growth avoids the processing of the seabed reverberation regions, which account for the largest proportion of sonar images, thus greatly reducing segmentation time and improving segmentation accuracy. In addition, a pre-processing filtering method called standard deviation filtering (STDF) is proposed to improve the SNR and remove the speckle noise. Experiments were conducted on three sonar databases, which showed that RGLT has significantly improved quantitative metrics such as accuracy, speed, and segmentation visual effects. The average accuracy and running times of the proposed segmentation method for 100 × 400 images are separately 95.90% and 0.44 s.

A Mosaic Method for Side-Scan Sonar Strip Images Based on Curvelet Transform and Resolution Constraints.

Due to the complex marine environment, side-scan sonar signals are unstable, resulting in random non-rigid distortion in side-scan sonar strip images. To reduce the influence of resolution difference of common areas on strip image mosaicking, we proposed a mosaic method for side-scan sonar strip images based on curvelet transform and resolution constraints. First, image registration was carried out to eliminate dislocation and distortion of the strip images. Then, the resolution vector of the common area in two strip images were calculated, and a resolution model was created. Curvelet transform was then performed for the images, the resolution fusion rules were used for Coarse layer coefficients, and the maximum coefficient integration was applied to the Detail layer and Fine layer to calculate the fusion coefficients. Last, inverse Curvelet transform was carried out on the fusion coefficients to obtain images in the fusion area. The fusion images in multiple areas were then combined in the registered images to obtain the final image. The experiment results showed that the proposed method had better mosaicking performance than some conventional fusion algorithms.

Real-Time Underwater Maritime Object Detection in Side-Scan Sonar Images Based on Transformer-YOLOv5

To overcome the shortcomings of the traditional manual detection of underwater targets in side-scan sonar (SSS) images, a real-time automatic target recognition (ATR) method is proposed in this paper. This method consists of image preprocessing, sampling, ATR by integration of the transformer module and YOLOv5s (that is, TR–YOLOv5s), and target localization. By considering the target-sparse and feature-barren characteristics of SSS images, a novel TR–YOLOv5s network and a down-sampling principle are put forward, and the attention mechanism is introduced in the method to meet the requirements of accuracy and efficiency for underwater target recognition. Experiments verified the proposed method achieved 85.6% mean average precision (mAP) and 87.8% macro-F2 score, and brought 12.5% and 10.6% gains compared with the YOLOv5s network trained from scratch, and had the real-time recognition speed of about 0.068 s per image.

Contour Extraction Based on Adaptive Thresholding in Sonar Images

A common problem in underwater side-scan sonar images is the acoustic shadow generated by the beam. Apart from that, there are a number of reasons impairing image quality. In this paper, an innovative algorithm with two alternative histogram approximation methods is presented. Histogram approximation is based on automatically estimating the optimal threshold for converting the original gray scale images into binary images. The proposed algorithm clears the shadows and masks most of the impairments in side-scan sonar images. The idea is to select a proper threshold towards the rightmost local minimum of the histogram, i.e., closest to the white values. For this purpose, the histogram envelope is approximated by two alternative contour extraction methods: polynomial curve fitting and data smoothing. Experimental results indicate that the proposed algorithm produces superior results than popular thresholding methods and common edge detection filters, even after corrosion expansion. The algorithm is simple, robust and adaptive and can be used in automatic target recognition, classification and storage in large-scale multimedia databases.

Seabed features associated with cold seep activity at the Formosa Ridge, South China Sea: Integrated application of high-resolution acoustic data and photomosaic images

High resolution acoustic data, together with high-definition photomosaic images and video observations of the Formosa Ridge Cold Seep Site were obtained using a remotely operated vehicle (ROV). Seabed features of the cold seep site were described in detail and the scales and distribution patterns of these features displayed on the seabed were quantitatively analyzed. The cold seep site is characterized by irregular and hummocky topography on the seabed relief map and patches with high to medium backscatter intensities on side-scan sonar (SSS) images. Based on visual data, six kinds of seabed features related to cold seep activity were identified: gas flares, dense benthic chemosynthetic communities, mussel beds with low population densities, shells and/or shell debris areas, authigenic carbonate concretions, and reduced sediments. Spatially, these seabed features are distributed in an approximately zoned pattern with gas plumes and benthic communities in the innermost part of Site F and the reduced sediments in the outermost part. Based on the integrated acoustic and visual images and in-situ geochemical data, a conceptual model describing the cold seep fluid conduits within the shallow subsurface at Site F at the present is established. Briefly, favorable conduits have developed within the authigenic carbonate mound. The conduits channelize the cold seep fluids and result in relatively focused discharge of the seep fluids at the intersections of the conduits and the seabed, to sustain the dense communities there. Additionally, the authigenic carbonate concretions redirect some of the cold seep fluids to seeps at the periphery of the site, resulting in the reduced sediments. Integrated acoustic and visual investigations combined with in situ measurements are an effective way to quantitatively study the spatial extent of a cold seep site and can provide detailed information for further study of the cold seep system.

Side scan sonar image augmentation for sediment classification using deep learning based transfer learning approach

Object detection in underwater acoustics especially sea floor object has been overwhelming mission chiefly owing to strident environment of sonar images as well as because of visibly existing sonar images. Side Scan Sonar is the primary sensor for Autonomous Underwater Vehicles to perform survey on sea water. Hence, we are using this SSS images for categorizing several objects like sand, mud, clay, graves, ridges and sediments in underwater sea through any size subsequent to training. We applied two-layer CNN architecture to train the model as well as we utilized three pre-trained network models such as VGG-19, ResNet50 and EfficientNet model for evaluating the performance of the model based on training and validation accuracy measures. Moreover, we utilized deep learning based transfer learning approach in which the parameters are tuned for classifying the images into sediments, clay, mud, stones etc. Our experimental outcomes shows that pre-trained EfficientNet model generates better accuracy of 100% after fine tuning the parameter in object recognition along with classification using SSS images.

Side-Scan Sonar Image Classification Based on Style Transfer and Pre-Trained Convolutional Neural Networks

Side-scan sonar is widely used in underwater rescue and the detection of undersea targets, such as shipwrecks, aircraft crashes, etc. Automatic object classification plays an important role in the rescue process to reduce the workload of staff and subjective errors caused by visual fatigue. However, the application of automatic object classification in side-scan sonar images is still lacking, which is due to a lack of datasets and the small number of image samples containing specific target objects. Secondly, the real data of side-scan sonar images are unbalanced. Therefore, a side-scan sonar image classification method based on synthetic data and transfer learning is proposed in this paper. In this method, optical images are used as inputs and the style transfer network is employed to simulate the side-scan sonar image to generate “simulated side-scan sonar images”; meanwhile, a convolutional neural network pre-trained on ImageNet is introduced for classification. In this paper, we experimentally demonstrate that the maximum accuracy of target classification is up to 97.32% by fine-tuning the pre-trained convolutional neural network using a training set incorporating “simulated side-scan sonar images”. The results show that the classification accuracy can be effectively improved by combining a pre-trained convolutional neural network and “similar side-scan sonar images”.

Object detection/tracking toward underwater photographs by remotely operated vehicles (ROVs)

Owing to the shortcoming of image blur, scale diversity, and complex backgrounds, underwater target detection and tracking is a tough challenge in real-world ROVs applications. In this paper, an underwater target detection method based on the class-weighted YOLO network is proposed. The key technique is to construct the class-weighted loss function based on the deep network YOLO to balance the sample difficulty and achieve better results. In addition, an adaptive dimensional clustering algorithm of target box is introduced to further improve the detection performance. Firstly, according to the imaging characteristics of the small underwater target in the side-scan sonar image, the sonar image is filtered and segmented by clustering to significantly reduce noises in each image. Afterward, the speckle detection is deployed to extract the suspected target area in the side-scan sonar image. Finally, the sonar image is segmented according to the automatic threshold segmentation and receive the binary image of the target region. The target scale is estimated by the second moment and the false target is eliminated. Correspondingly, the accurate detection of small underwater target can be achieved. The moving object pixel is detected by background difference and three-frame difference respectively. On this basis, we combine the background difference and the three-frame difference. The background difference characterizes the objects capturing the incomplete message detected by the three-frame difference. Finally, morphological operation is leveraged to eliminate the noise caused by non-static background objects. Experiments on multiple ROVs have validated that, the proposed algorithm can improve the average accuracy and the recall rate remarkably in the dense target detection task that contains 20 targets averagely.

Side-scan sonar images segmentation for AUV with recurrent residual convolutional neural network module and self-guidance module

This paper designed an end-to-end image segmentation method for side-scan sonar mounted on Autonomous Underwater Vehicle (AUV) to obtain accurate results. Accurate segmentation results depend on precise feature extraction, and an effective segmentation network can guarantee the autonomous recognition ability of Autonomous Underwater Vehicle (AUV) when performing tasks. In this paper, the R2CNN module is used to obtain accurate sonar image features, which reduces errors and improves accuracy. Besides, the Self-guidance module was introduced to ensure the network’s stability and optimize the segmentation results. The experimental results show that the proposed method can achieve better segmentation results than UNet, SegNet and their derivative networks and has better generalization ability.

Side scan sonar image super-resolution using an improved initialization structure

Side scan sonar image super-resolution using an improved initialization structure

Image Recognition Technology in Texture Identification of Marine Sediment Sonar Image

Through the recognition of ocean sediment sonar images, the texture in the image can be classified, which provides an important basis for the classification of ocean sediment. Aiming at the problems of low efficiency, waste of human resources, and low accuracy in the traditional manual side-scan sonar image discrimination, this paper studies the application of image recognition technology in sonar image substrate texture discrimination, which is popular in many fields. At the same time, considering the scale complexity, diversity, multisources, and small sample characteristics of the marine sediment sonar image texture, the transfer learning is introduced into the image recognition, and the K-means clustering algorithm is used to reset the prior frame parameters to improve the speed and accuracy of image recognition. Through the experimental comparison between the original model and the new model based on transfer learning, the AP (average precision) value of the yolov3 model based on transfer learning can reach 84.39%, which is 0.97% higher than that of the original model, with considerable accuracy and room for improvement; it takes less than 0.2 seconds. This shows the applicability and development of image recognition technology in texture discrimination of bottom sonar images.

Generating Synthetic Sidescan Sonar Snippets Using Transfer-Learning in Generative Adversarial Networks

The training of a deep learning model requires a large amount of data. In case of sidescan sonar images, the number of snippets from objects of interest is limited. Generative adversarial networks (GAN) have shown to be able to generate photo-realistic images. Hence, we use a GAN to augment a baseline sidescan image dataset with synthetic snippets. Although the training of a GAN with few data samples is likely to cause mode collapse, a combination of pre-training using simple simulated images and fine-tuning with real data reduces this problem. However, for sonar data, we show that this approach of transfer-learning a GAN is sensitive to the pre-training step, meaning that the vanishing of the gradients of the GAN’s discriminator becomes a critical problem. Here, we demonstrate how to overcome this problem, and thus how to apply transfer-learning to GANs for generating synthetic sidescan snippets in a more robust way. Additionally, in order to further investigate the GAN’s ability to augment a sidescan image dataset, the generated images are analyzed in the image and the frequency domain. The work helps other researchers in the field of sonar image processing to augment their dataset with additional synthetic samples.

Automatic Overlapping Area Determination and Segmentation for Multiple Side Scan Sonar Images Mosaic

Seafloor image is important for marine scientific research and ocean engineering. With the wide use of side scan sonar (SSS), multiple measured SSS images need to be mosaicked to form a large-scale and high-resolution seafloor image. This article proposes an automatic mosaic method. First, the overlapping areas between adjacent strips are determined automatically by the track lines and swath width. In the overlapping areas, the image matching is conducted using the speeded-up robust features algorithm with the constraint of geographic coordinates of detected feature points. Then, the overlapping area is segmented using the k-means cluster method. Within each segmented region, the coordinate transformation model is established to make the positions of common features in the overlapping areas unique. With regards to multiple SSS images mosaic, the method to mosaic in sequence is also proposed according to the permutation parameter of each strip. Finally, the SSS images in the overlapping areas are fused and a large-scale seafloor image is formed. This proposed method was applied in one water area of Shenzhen, demonstrating good performance in terms of coordinate consistency of the same features in the mosaicked image. The mean coordinate deviations of the same feature points in overlapping areas were nearly zero and the standard deviation of them also decreased. This proposed method is easily transferable to other study areas and provides an objective, repeatable means for multiple SSS images mosaic.

Optimizing the Sediment Classification of Small Side-Scan Sonar Images Based on Deep Learning

Acoustic seabed classification (ASC) is a fast and large-scale seabed sediment survey method. In particular, combining it with an automated classifier can theoretically achieve fast automatic seabed sediment classification. However, owing to the cost of sampling, a lack of labeled data for sediment classification based on seabed acoustic images impedes the training and deployment of classifiers. Herein, we use shallow-water, side-scan sonar images collected from the Pearl River Estuary combined with deep learning to study sediment classification and optimization methods for a small dataset of seabed acoustic images. In this paper, we applied different and deeper convolutional neural networks (CNNs) and used grayscale CIFAR-10 for pretraining to achieve large-span parameter migration and improve model performance. The best result in the experiment is a 3.459% error rate achieved by ResNet after fine tuning, verifying the improvement brought by our fine tuning strategy and the deeper models used in such tasks. The results of data enhancement based on generative adversarial networks (GANs) indicated that this method can improve the accuracy of sediment classification; however, the effects of GANs are limited and they are computationally expensive. Overall, our findings resolve, to an extent, the dilemma of using small datasets of seabed acoustic images for sediment classification and provide a framework for future studies on sediment classification, which has a certain significance in helping people better understand the seabed.

Mixed Noise Removal by Processing of Patches

Sonar images are degraded by mixed noise which has an adverse impact on detection and classification of underwater objects. Existing denoising methods of sonar images remove either additive noise or multiplicative noise. In this study, the mixed noise in sonar images, the additive Gaussian noise and the multiplicative speckle effect are handled by the data adaptive methods. A patch based denoising is applied in two phases to remove the additive Gaussian and multiplicative speckle noises. In the first phase, the adaptive processing of local patches is used to remove the additive Gaussian noise by exploiting the sonar image local sparsity. The PCA and SVD methods are used for denoising the noisy image patches and blocks of patches. In the second phase, the weighted maximum likelihood denoising of the nonlocal patches reduces the speckle effect by exploiting the non-local similarity in a probability distribution. Experiments on side scan sonar images are conducted and the results show the importance of removing both the additive and multiplicative components from the sonar images.

Noise reduction for sonar images by statistical analysis and fields of experts

Sonar images are usually suffering from speckle noise which results in poor visual quality. In order to improve the sonar imaging quality, removing or reducing these speckle noises is a very important and arduous task. In this paper, the imaging principle and noise characteristics of the side-scan sonar (SSS) are analyzed, and five typical probability distribution functions are used to fit the seabed reverberation. Through experiment comparison, the Gamma distribution is selected to simulate the noise of the SSS image caused by the reverberation. Simultaneously, the fields of experts denoising algorithm based on the Gamma distribution (Gamma FoE) is proposed for SSS image denoising. In order to perceive and measure the denoising effect better, evaluation indexes of Fast Noise Variance Estimation (FNVE, an image noise estimation method) and Blind Referenceless Image Spatial Quality Evaluator (BRISQUE, an image quality evaluation method) are selected for image quality perception. The final results of the SSS image denoise experiment show that the Gamma FoE denoise algorithm has a better effect on SSS image denoise application than other denoise algorithms.

Study on Unmanned Hybrid Unmanned Surface Vehicle and Unmanned Underwater Vehicle System

Underwater operating platforms face difficulties regarding power supply and communications. To overcome these difficulties, this study proposes a hybrid surface and underwater vehicle (HSUV) and presents the development of the platform, control algorithms, and results of field tests. The HSUV is capable of supplying reliable power to the unmanned underwater vehicle (UUV) and obtaining data in real time by using a tether cable between the UUV and the unmanned surface vehicle (USV). The HSUV uses global positioning system (GPS) and ultra-short base line sensors to determine the relative location of the UUV. Way point (WP) and dynamic positioning (DP) algorithms were developed to enable the HSUV to perform unmanned exploration. After reaching the target point using the WP algorithm, the DP algorithm enables USV to maintain position while withstanding environmental disturbances. To ensure the navigation performance at sea, performance tests of GPS, attitude/heading reference system, and side scan sonar were conducted. Based on these results, manual operation, WP, and DP tests were conducted at sea. WP and DP test results and side scan sonar images during the sea trials are presented.