Maritime Images Research Articles

Deep Learning-Based Maritime Environment Segmentation for Unmanned Surface Vehicles Using Superpixel Algorithms

Unmanned surface vehicles (USVs) are receiving increasing attention in recent years from both academia and industry. To make a high-level autonomy for USVs, the environment situational awareness is a key capability. However, due to the richness of the features in marine environments, as well as the complexity of the environment influenced by sun glare and sea fog, the development of a reliable situational awareness system remains a challenging problem that requires further studies. This paper, therefore, proposes a new deep semantic segmentation model together with a Simple Linear Iterative Clustering (SLIC) algorithm, for an accurate perception for various maritime environments. More specifically, powered by the SLIC algorithm, the new segmentation model can achieve refined results around obstacle edges and improved accuracy for water surface obstacle segmentation. The overall structure of the new model employs an encoder–decoder layout, and a superpixel refinement is embedded before final outputs. Three publicly available maritime image datasets are used in this paper to train and validate the segmentation model. The final output demonstrates that the proposed model can provide accurate results for obstacle segmentation.

Infrared maritime small target detection based on edge and local intensity features

In infrared maritime images, small targets are always dim, and there are interferences such as iands and wave clutter. Thus it is easy to miss the real target or generate a large number of false alarms. This paper proposes a detection method based on edge and local intensity features to improve detection accuracy. According to the difference in the similarity of directional edge components between targets and interference, constructing and multiplying directional subbands reduces false alarms. According to the local internal-external intensity features, we designed the candidate target descriptor. The proposed method has higher detection accuracy than several state-of-the-art methods and has stronger parameter robustness through experiments on images with different backgrounds and targets.

Territorial Distribution of Maritime Symbols in Public Spaces of Klaipėda City

The geographical location of Klaipeda city determined that a suitable space for the formation of maritime culture was created here. There are a number of exceptional objects of small and large architecture in the public spaces of the city, some of which are related to the sea. However, it must be stated that the city still lacks objects that glorify famous personalities who have contributed to the fostering of maritime culture and the promotion of Lithuania as a maritime country. Methods used in the research: observation, object photography, cartographic, expert surveys. The study analyzes the diversity of maritime symbols and their territorial distribution in the public spaces of Klaipeda city. Trends in the renewal and further development of maritime elements are also presented in order to form the image of the city and foster maritime culture. This research should expand the boundaries of cognition of maritime culture and contribute to the strengthening of the maritime image of Klaipeda city. Keywords: Klaipeda city, maritime symbols, maritime themes, public spaces.

Maritime filtering for images and videos

Maritime signal processing technologies have emerged as an important area of study because of the increasing popularity of autonomous ships and automatic maritime surveillance systems. However, the various techniques developed for detecting or tracking objects remain unable to address various maritime noise challenges that cause several types of false positives in maritime visual surveillance. Maritime signal processing is challenging because of the prevalence of noise sources such as severe dynamic backgrounds, wakes, and reflections, owing to the complex, unconstrained, and diverse nature of such scenes caused by the surface properties of water. Moreover, few studies have investigated specific maritime noise filtering as a general integrated processing approach with image and video technologies in the context of maritime visual surveillance. In this study, we propose a novel maritime noise prior (MNP) based on a dark channel prior and observations of the characteristics of the sea. A general maritime filtering technique is developed to suppress noise originating from the properties of water in maritime images and videos. The proposed method employs a noniterative, nonlinear, and simple maritime filtering approach based on MNP that does not require specialized knowledge of application scene conditions or structure. We conducted image and video experiments by applying our approach to three publicly available databases. In experiments with color images, our method successfully filtered related background noise and water, i.e., severe boat wakes and reflections, while preserving objects other than water in color images. In the experiments with video sequences, the results demonstrated that the proposed filter improved the overall performance of state-of-the-art background subtraction (BS) algorithms from 36.60%–50.63%. By combining BS algorithms and filtering to enhance foreground detection in video sequences, the proposed method ensures the universal applicability and flexibility required to eliminate noise from images and videos obtained in challenging maritime environments. The results indicate that the proposed method is appropriate for maritime surveillance applications implementing image segmentation and foreground detection, and it can potentially increase the accuracy of maritime visual surveillance.

Ship Target Detection Algorithm Based on Improved YOLOv3 for Maritime Image

Accurate identification of ships is the key technology of intelligent transportation in water. At the same time, it also provides a judgment basis for water traffic safety control. This paper proposed a detection method of ships in water based on improved You Only Look Once version 3 (YOLOv3), which is called Feature Attention, Feature Enhancement YOLOv3 (AE-YOLOv3). The feature attention module was constructed by introducing the attention mechanism, which was embedded in Darknet-53 for feature recalibration, which improved the feature extraction ability of the model in the complex navigable background. For the problem of insufficient semantic information of low-level features in the feature fusion process, a feature enhancement module was constructed and applied to the feature fusion part to enhance the receptive field size of the corresponding feature layer and the correlation degree of feature extraction network. Experiments were carried out on the public SeaShips dataset. Experiments show that the detection accuracy is as high as 98.72%, which is better than that of other mainstream ship identification models, fully verifying the superiority of this method in the detection of waterborne traffic ships.

Ship detection from coastal surveillance videos via an ensemble Canny-Gaussian-morphology framework

AbstractCoastal surveillance video helps officials to obtain on-site visual information on maritime traffic situations, which benefits building up the maritime transportation detection infrastructure. The previous ship detection methods focused on detecting distant small ships in maritime videos, with less attention paid to the task of ship detection from coastal surveillance video. To address this challenge, a novel framework is proposed to detect ships from coastal maritime images in three typical traffic situations in three consecutive steps. First the Canny detector is introduced to determine the potential ship edges in each maritime frame. Then, the self-adaptive Gaussian descriptor is employed to accurately rule out noisy edges. Finally, the morphology operator is developed to link the detected separated edges to connected ship contours. The model's performance is tested under three typical maritime traffic situations. The experimental results show that the proposed ship detector achieved satisfactory performance (in terms of precision, accuracy and time cost) compared with other state-of-the-art algorithms. The findings of the study offer the potential of providing real-time visual traffic information to maritime regulators, which is crucial for the development of intelligent maritime transportation.

CNN-Enabled Visibility Enhancement Framework for Vessel Detection under Haze Environment

Maritime images captured under haze environment often have a terrible visual effect, making it easy to overlook important information. To avoid the failure of vessel detection caused by fog, it is necessary to preprocess the collected hazy images for recovering vital information. In this paper, a novel CNN-enabled visibility dehazing framework is proposed, consisting of two subnetworks, that is, Coarse Feature Extraction Module (C-FEM) and Fine Feature Fusion Module (F-FFM). Specifically, C-FEM is a multiscale haze feature extraction network, which can learn information from three scales. Correspondingly, F-FFM is an improved encoder-decoder network to fuse multiscale information obtained by C-FEM and enhance the visual effect of the final output. Meanwhile, a hybrid loss function is designed for monitoring the multiscale output of C-FEM and the final result of F-FFM simultaneously. It is worth mentioning that massive maritime images are considered the training dataset to further adapt the vessel detection task under haze environment. Comprehensive experiments on synthetic and realistic images have verified the superior effectiveness and robustness of our CNN-enabled visibility dehazing framework compared to several state-of-the-art methods. Our method preprocesses images before vessel detection to demonstrate our framework has the capacity of promoting maritime video surveillance.

Infrared maritime dim small target detection based on spatiotemporal cues and directional morphological filtering

The existing small target detection algorithms will suffer severe performance deterioration when the weak target is submerged in heavy cluttered infrared (IR) maritime image. To settle this problem, a small target detection scheme is developed based on spatiotemporal cues and multidirectional morphological filtering. Initially, considering the small target will be isotropic because of the point spread function of the long-distance thermal imaging, while background edge clutters are generally local directional, the multidirectional structuring elements (MSEs) are constructed. By incorporating constructed MSEs and morphological operations, the directional morphological filtering (DMF) is established to measure the multidirectional differences between target region and local surroundings. Then, according to the brightness and isotropic Gaussian-like shape of small target in local spatial domain, the local intensity difference measure (LIDM) and the local Gaussian-like shape measure (LGSM) are presented to differentiate the small target from background clutters. After that, in order to integrate the two spatial properties, the multidirectional improved top-hat filter (MITHF) is proposed by fusing LIDM and LGSM to highlight potential small targets and suppress strong structural edge clutters. Finally, considering the consistency of targets and the fluctuation of sea clutters in temporal domain, a new appearance descriptor namely histogram of oriented morphological filtering (HOMF) is designed to characterize the small target by fully exploiting the advantages of DMF, and the HOMF-based multi-frame confirmation strategy (HMCS) is developed to further identify real small targets and eliminate false alarms. Extensive experiments prove that proposed algorithm is superior to the compared methods, especially for dim small target buried in intricate maritime background clutter.

Single Maritime Image Defogging Based on Illumination Decomposition Using Texture and Structure Priors

Removing fog components from maritime images is more challenging than removing them from land images because of the complicated scattering effects. In this paper, we propose a single maritime image defogging algorithm based on illumination decomposition. First, we decompose a foggy input image into glow and glow-free layers by formulating an optimization problem and then solving it effectively, where the optimization variables are updated by closed-form solutions. Then, the fog components in the glow-free layer are removed using a defogging algorithm. Finally, the natural illumination in the glow layer is recovered using a compensation scheme to obtain the final fog-free image. Experimental results on real test images demonstrate that the proposed algorithm outperforms state-of-the-art algorithms in terms of both subjective and objective evaluation criteria.

Infrared Small Maritime Target Detection Based on Integrated Target Saliency Measure

Robust and effective detection of a small target in an infrared maritime image is a key technology of maritime target search and tracking applications. Infrared small target detection is a challenging task due to the factors such as dim small targets and various complex backgrounds caused by sun glitters and strong waves. In this article, the integrated target saliency measure (ITSM) based on local and nonlocal spatial information is proposed to improve target detection performance. We combine local heterogeneity property of targets and nonlocal self-correlation property of background with targets' sparsity to separate real targets from background clutters. First, local heterogeneity calculation based on cross-window standard deviation (CSD) is proposed to extract candidate targets preliminarily, which enhances the local intensity difference between small targets and neighboring background. Meanwhile, low-rank representation (LRR) is applied to background prediction and removal, which is followed by adaptive threshold segmentation to enhance target saliency. Finally, we integrate the results obtained from the two steps mentioned above to further enhance targets and suppress background clutters. Then, real targets are extracted by an iterative threshold on the integrated map so as to generate the seed map, in addition, the target expansion strategy is exploited to keep full target areas. Experimental results on six datasets show that the proposed method outperforms the state-of-the-art methods in terms of robustness and target detection accuracy.

Visual ship tracking via a hybrid kernelized correlation filter and anomaly cleansing framework

Ship tracking from maritime visual sensing data (namely maritime surveillance videos) provides various kinematic maritime traffic information, which significantly benefits remote maritime traffic controlling and management, off-site law enforcement, etc. But, it is difficult to extract distinct ship visual features when the target ship is sheltered by the neighboring ship in the maritime images (or the images are shot in low visibility condition). To address the difficulty, we proposed a hybrid ship tracking framework via the help of kernelized correlation filter (KCF) and anomaly cleaning models (including curve fitting method and Kalman filter). First, we employed the KCF model to obtain raw ship trajectories in consecutive maritime images. Second, our ship tracker is accurately initialized (to be specific, ground truth ship position in the first frame is employed to initialize the ship tracker). Third, the Kalman filter is introduced to suppress the trivial ship position oscillations in the raw ship trajectories. We verified the proposed framework performance on the four typical maritime scenarios. The experimental results indicate that the proposed ship tracker showed more accurate ship tracking results compared to other four popular ship trackers in terms of average root mean square error (RMSE), mean absolute deviation (MAD) and mean square error (MSE). The research findings can help maritime traffic participants obtain visual on-spot maritime kinematic information, and thus further enhance maritime traffic safety.

Кодек для передачи морских сюжетов в условиях ограничений канала

В работе представлен видеокодек с адаптивным способом сканирования спектральных трансформант, основанный на применении трехмерных (3D) дискретных косинусных преобразований. Кодек имеет низкую вычислительную сложность и высокую устойчивость к ошибкам передачи по каналу связи и предназначен для мобильных устройств. Кодек представляет собой устройство, которое последовательно выполняет дискретное косинусное преобразование для устранения пространственной избыточности в пределах кадра и временной межкадровой избыточности в последовательности кадров с учетом скорости движения объектов на изображениях морских сюжетов. Приведены результаты моделирования алгоритмов кодирования и декодирования видеоинформации для различных видеопотоков, полученных из камер наблюдения. Получены результаты тестирования алгоритмов кодирования и декодирования изображений в виде графиков зависимости точности восстановления от скорости передачи сжатых видеоданных и зависимости точности от сложности устройств сжатия изображений. This paper presents a video codec with an adaptive method for scanning spectral transformants based on the use of three-dimensional (3D) discrete cosine transformations. The codec has a low computational complexity and high resistance to transmission errors over the communication channel and is designed for mobile devices. A codec is a device that sequentially performs a discrete cosine transformation to eliminate spatial redundancy within a frame and temporal inter-frame redundancy in a sequence of frames, taking into account the speed of movement of objects in images of marine subjects. The results of modeling algorithms for encoding and decoding video information for various video streams obtained from surveillance cameras are presented. The results of testing algorithms for encoding and decoding images in the form of graphs of the dependence of the recovery accuracy on the speed of transmission of compressed video data and the dependence of accuracy on the complexity of image compression devices are obtained.

Robust Detection of Infrared Maritime Targets for Autonomous Navigation

This paper addresses a problem on infrared maritime target detection robustly in various situations. Its main contribution is to improve the infrared maritime target detection accuracy in different backgrounds, for various targets, using multiple infrared wave bands. The accuracy and the computational time of traditional infrared maritime searching systems are improved by our proposed Local Peak Singularity Measurement (LPSM)-Based Image Enhancement and Grayscale Distribution Curve Shift Binarization (GDCSB)-Based Target Segmentation. The first part uses LPSM to quantize the local singularity of each peak. Additionally, an enhancement map (EM) is generated based on the quantitative local singularity. After multiplying the original image by the EM, targets can be enhanced and the background will be suppressed. The second part of GDCSB-Based Target Segmentation calculates the desired threshold by cyclic shift of the grayscale distribution curve (GDC) of the enhanced image. After binarizing the enhanced image, real targets can be segmented from the image background. To verify the proposed algorithm, experiments based on 13,625 infrared maritime images and five comparison algorithms were conducted. Results show that the proposed algorithm has solid performance in strong and weak background clutters, different wave bands, different maritime targets, etc.

EEG-Based Maritime Object Detection for IoT-Driven Surveillance Systems in Smart Ocean

Automated maritime object detection is a significant research challenge in intelligent marine surveillance systems for the Internet of Things (IoT) and smart ocean applications. In particular, ship detection is recognized as one of the core research issues of these IoT-driven intelligent marine surveillance systems. Traditional methods based on machine learning have made some achievements in detection tasks for specific objects. However, the ship objects are relatively small, and they are usually not accurately detected. In this article, we propose an electroencephalography (EEG)-based maritime object detection algorithm for IoT-driven surveillance systems in the smart ocean. For this purpose, we conduct experiments to record the EEG signals of subjects when they are watching the maritime image scenes. With the feature analysis of EEG signals, the event-related potential (ERP) components associated with detecting objects are induced, such as the $P3$ and $N2$ components. Employing classification based on linear discriminant analysis (LDA), the area under curve (AUC) of the receiver operating characteristic (ROC) is used to evaluate the detection accuracy. We use this novel method to determine and identify essential objects and areas from IoT devices, such as digital camera imaging sensors. Our proposed method can not only help to detect small objects accurately using fewer samples but can also be used to reduce the data volume needed to be stored and transmitted in IoT-driven marine surveillance systems.

Improved denoising autoencoder for maritime image denoising and semantic segmentation of USV

Unmanned surface vehicle (USV) is currently a hot research topic in maritime communication network (MCN), where de-noising and semantic segmentation of maritime images taken by USV have been rarely studied. The former has recently researched on autoencoder model used for image denoising, but the existed models are too complicated to be suitable for real-time detection of USV. In this paper, we proposed a lightweight autoencoder combined with inception module for maritime image denoising in different noisy environments and explore the effect of different inception modules on the denoising performance. Furthermore, we completed the semantic segmentation task for maritime images taken by USV utilizing the pretrained U-Net model with tuning, and compared them with original U-Net model based on different backbone. Subsequently, we compared the semantic segmentation of noised and denoised maritime images respectively to explore the effect of image noise on semantic segmentation performance. Case studies are provided to prove the feasibility of our proposed denoising and segmentation method. Finally, a simple integrated communication system combining image denoising and segmentation for USV is shown.

A Robust Detection Algorithm for Infrared Maritime Small and Dim Targets.

Infrared maritime target detection is the key technology of maritime target search systems. However, infrared images generally have the defects of low signal-to-noise ratio and low resolution. At the same time, the maritime environment is complicated and changeable. Under the interference of islands, waves and other disturbances, the brightness of small dim targets is easily obscured, which makes them difficult to distinguish. This is difficult for traditional target detection algorithms to deal with. In order to solve these problems, through the analysis of infrared maritime images under a variety of sea conditions including small dim targets, this paper concludes that in infrared maritime images, small targets occupy very few pixels, often do not have any edge contour information, and the gray value and contrast values are very low. The background such as island and strong sea wave occupies a large number of pixels, with obvious texture features, and often has a high gray value. By deeply analyzing the difference between the target and the background, this paper proposes a detection algorithm (SRGM) for infrared small dim targets under different maritime background. Firstly, this algorithm proposes an efficient maritime background filter for the common background in the infrared maritime image. Firstly, the median filter based on the sensitive region selection is used to extract the image background accurately, and then the background is eliminated by image difference with the original image. In addition, this article analyzes the differences in gradient features between strong interference caused by the background and targets, proposes a small dim target extraction operator with two analysis factors that fit the target features perfectly and combines the adaptive threshold segmentation to realize the accurate extraction of the small dim target. The experimental results show that compared with the current popular small dim target detection algorithms, this paper has better performance for target detection in various maritime environments.

Robust Visual Ship Tracking with an Ensemble Framework via Multi-View Learning and Wavelet Filter.

Maritime surveillance videos provide crucial on-spot kinematic traffic information (traffic volume, ship speeds, headings, etc.) for varied traffic participants (maritime regulation departments, ship crew, ship owners, etc.) which greatly benefits automated maritime situational awareness and maritime safety improvement. Conventional models heavily rely on visual ship features for the purpose of tracking ships from maritime image sequences which may contain arbitrary tracking oscillations. To address this issue, we propose an ensemble ship tracking framework with a multi-view learning algorithm and wavelet filter model. First, the proposed model samples ship candidates with a particle filter following the sequential importance sampling rule. Second, we propose a multi-view learning algorithm to obtain raw ship tracking results in two steps: extracting a group of distinct ship contour relevant features (i.e., Laplacian of Gaussian, local binary pattern, Gabor filter, histogram of oriented gradient, and canny descriptors) and learning high-level intrinsic ship features by jointly exploiting underlying relationships shared by each type of ship contour features. Third, with the help of the wavelet filter, we performed a data quality control procedure to identify abnormal oscillations in the ship positions which were further corrected to generate the final ship tracking results. We demonstrate the proposed ship tracker’s performance on typical maritime traffic scenarios through four maritime surveillance videos.

Video-Based Detection Infrastructure Enhancement for Automated Ship Recognition and Behavior Analysis

Video-based detection infrastructure is crucial for promoting connected and autonomous shipping (CAS) development, which provides critical on-site traffic data for maritime participants. Ship behavior analysis, one of the fundamental tasks for fulfilling smart video-based detection infrastructure, has become an active topic in the CAS community. Previous studies focused on ship behavior analysis by exploring spatial-temporal information from automatic identification system (AIS) data, and less attention was paid to maritime surveillance videos. To bridge the gap, we proposed an ensemble you only look once (YOLO) framework for ship behavior analysis. First, we employed the convolutional neural network in the YOLO model to extract multi-scaled ship features from the input ship images. Second, the proposed framework generated many bounding boxes (i.e., potential ship positions) based on the object confidence level. Third, we suppressed the background bounding box interferences, and determined ship detection results with intersection over union (IOU) criterion, and thus obtained ship positions in each ship image. Fourth, we analyzed spatial-temporal ship behavior in consecutive maritime images based on kinematic ship information. The experimental results have shown that ships are accurately detected (i.e., both of the average recall and precision rate were higher than 90%) and the historical ship behaviors are successfully recognized. The proposed framework can be adaptively deployed in the connected and autonomous vehicle detection system in the automated terminal for the purpose of exploring the coupled interactions between traffic flow variation and heterogeneous detection infrastructures, and thus enhance terminal traffic network capacity and safety.

Three Adaptive Sub-Histograms Equalization Algorithm for Maritime Image Enhancement

According to maritime image histograms' statistic and analysis, the histogram of pure maritime image obeys Gaussian distribution approximately, thus Three Adaptive Sub-histograms Equalization (TASHE) algorithm for maritime image enhancement is proposed in this paper. First, the characteristics of pure maritime image's histogram are studied, then the adaptive threshold's optimal selection strategy for the histogram's division is discussed, finally the implement of three sub-histograms is described. This paper employs visible gray maritime image, visible color maritime image and infrared maritime image to verify the enhancement algorithm's effectiveness and robustness, the experimental results show that TASHE algorithm can not only keep the maritime image's mean brightness and naturalness, but also improve the maritime image's contrast without the noise and artifacts. The objective image quality assessment also indicates that TASHE algorithm can improve the original maritime image's Enhancement Measure by Entropy (EME) value, furthermore, when a maritime image is pre-processed by TASHE algorithm, the maritime target's Detection Rate (DR) can be improved.

Low-Light Image Enhancement With Regularized Illumination Optimization and Deep Noise Suppression

Maritime images captured under low-light imaging condition easily suffer from low visibility and unexpected noise, leading to negative effects on maritime traffic supervision and management. To promote imaging performance, it is necessary to restore the important visual information from degraded low-light images. In this article, we propose to enhance the low-light images through regularized illumination optimization and deep noise suppression. In particular, a hybrid regularized variational model, which combines L0-norm gradient sparsity prior with structure-aware regularization, is presented to refine the coarse illumination map originally estimated using Max-RGB. The adaptive gamma correction method is then introduced to adjust the refined illumination map. Based on the assumption of Retinex theory, a guided filter-based detail boosting method is introduced to optimize the reflection map. The adjusted illumination and optimized reflection maps are finally combined to generate the enhanced maritime images. To suppress the effect of unwanted noise on imaging performance, a deep learning-based blind denoising framework is further introduced to promote the visual quality of enhanced image. In particular, this framework is composed of two sub-networks, i.e., E-Net and D-Net adopted for noise level estimation and non-blind noise reduction, respectively. The main benefit of our image enhancement method is that it takes full advantage of the regularized illumination optimization and deep blind denoising. Comprehensive experiments have been conducted on both synthetic and realistic maritime images to compare our proposed method with several state-of-the-art imaging methods. Experimental results have illustrated its superior performance in terms of both quantitative and qualitative evaluations.