Maritime Situational Awareness Research Articles

Vessel trajectory prediction with recurrent neural networks: An evaluation of datasets, features, and architectures

Maritime situational awareness tasks such as port management, collision avoidance, and search-and-rescue missions rely on accurate knowledge of vessel locations. The availability of historical vessel trajectory data through the Automatic Identification System (AIS) has enabled the development of prediction methods, with a recent focus on trajectory prediction via recurrent neural networks (RNNs) and other deep learning architectures. While these methods have shown promising performance benefits over kinematic and clustering-based models, comparing among RNN-based models remains difficult due to variations in evaluation datasets, region sizes, vessel types, and numerous other design choices. As a result, it is not clear whether recent methods based on highly-sophisticated network architectures are necessary to achieve strong prediction performance. In this work, we present a simple fusion-based RNN approach to vessel trajectory prediction that allows for easy incorporation of exogenous variables. We perform an extensive ablation study to measure the impact of various modeling choices, including preprocessing, loss functions, and the choice of features, as well as the first usage of surface current information in vessel trajectory prediction. We demonstrate that our approach achieves state-of-the-art performance on three large regions off the United States coast, obtaining an improvement of up to 0.88 km over competing methods when predicting three hours into the future. We conclude that our simple architecture can outperform more complicated architectures while incurring a lower memory cost. Further, we show that the choice of loss function and the inclusion of surface current information both have significant impact on prediction performance.

Unsupervised maritime anomaly detection for intelligent situational awareness using AIS data

With the mandatory implementation of the automatic identification system and the rapid advancement of relevant satellite communication technologies, a vast amount of vessel trajectory data has been amassed. It has catalyzed advancements in the field of maritime anomaly detection, significantly contributing to the enhancement of intelligent maritime situational awareness. However, detecting abnormal vessel trajectories from massive data is a highly challenging task that requires extensive manual effort. To address the challenge, this paper develops an unsupervised deep learning method for detecting abnormal vessel trajectories. Specifically, we first generate the informative trajectory images by remapping the raw vessel trajectories into two-dimensional matrices while maintaining the spatio-temporal properties. Then, a Wasserstein generative adversarial network with gradient penalty (WGAN-GP) is trained on normal vessel trajectories. Next, an encoder is trained to map these trajectory images to a latent space. Finally, the trained encoder and WGAN-GP are used to design an anomaly score for detecting abnormal vessel trajectories and anomalies in vessel trajectories. Experimental evaluations using multiple evaluation metrics on diverse simulated and real-world maritime environments validate the efficacy of the proposed method in unsupervised maritime anomaly detection. The results indicate that the proposed model can provide accurate anomaly detection on vessel trajectories without manual labeling.

Graph-based ship traffic partitioning for intelligent maritime surveillance in complex port waters

Maritime Situational Awareness (MSA) is a critical component of intelligent maritime traffic surveillance. However, it becomes increasingly challenging to gain MSA accurately given the growing complexity of ship traffic patterns due to multi-ship interactions possibly involving classical manned ships and emerging autonomous ships. This study proposes a new traffic partitioning methodology to realise the optimal maritime traffic partition in complex waters. The methodology combines conflict criticality and spatial distance to generate conflict-connected and spatially compact traffic clusters, thereby improving the interpretability of traffic patterns and supporting ship anti-collision risk management. First, a composite similarity measure is designed using a probabilistic conflict detection approach and a newly formulated maritime traffic route network learned through maritime knowledge mining. Then, an extended graph-based clustering framework is used to produce balanced traffic clusters with high intra-connections but low inter-connections. The proposed methodology is thoroughly demonstrated and tested using Automatic Identification System (AIS) trajectory data in the Ningbo-Zhoushan Port. The experimental results show that the proposed methodology 1) has effective performance in decomposing the traffic complexity, 2) can assist in identifying high-risk/density traffic clusters, and 3) is sufficiently generic to handle various traffic scenarios in complex geographical waters. Therefore, this study makes significant contributions to intelligent maritime surveillance and provides a theoretical foundation for promoting maritime anti-collision risk management for the future mixed traffic of both manned and autonomous ships.

AIS-based maritime anomaly traffic detection: A review

Maritime transportation plays an essential role in global trade. Due to the huge number of vessels worldwide, there is also a non-negligible volume of Maritime incidents such as collisions/sinking and illegal events (e.g., piracy, smuggling, and unauthorized fishing). Electronic equipment/systems, such as radars and Automatic Identification Systems (AIS), have contributed to improving maritime situational awareness. AIS provides one of the fundamental sources of vessel kinematics and static data. Today, many approaches are focused on automatically detecting the vessels’ traffic behavior and discovering useful patterns and deviations from those data. These studies contribute to detecting suspicious activities and anomalous trajectories, whose developed techniques could be applied in the surveillance systems, helping the authorities to anticipate proper actions. Several concerns and difficulties are involved in the analyses of vessel kinematics data: how to deal with big data generated, inconsistencies, irregular updates, dynamic data, unlabeled data, and evaluation. This article presents the approaches, constraints, and challenges in maritime traffic anomaly detection research, presenting a review, a taxonomy, and a discussion of the proposed approaches.

Interval prediction of vessel trajectory based on lower and upper bound estimation and attention-modified LSTM with bayesian optimization

Uncertainty prediction of vessel trajectory is essential to enhance maritime situational awareness and traffic safety. Traditional approaches for trajectory prediction face challenges in accurately quantifying uncertainties, thereby limiting effectiveness in decision-making. To address this challenge, we propose a hybrid interval prediction frame of vessel trajectory using the lower and upper bound estimation (LUBE) and attention-modified long short-term memory (LSTM) network with bayesian optimization (BO). Firstly, trajectory data is preprocessed to solve the scale irregularity. Then, a novel trajectory interval prediction model for perceiving the prediction uncertainties is designed based on an advanced attention-modified LSTM with interval prediction capability. Meanwhile, a supervised training strategy with differentiating the interval widths of latitude and longitude is put forward to devise sample labels for training the model. Additionally, a new prediction interval-based objective function is proposed considering the target of maximizing coverage and minimizing width of trajectory interval. The BO algorithm optimizes the weighted parameters in the LSTM network by minimizing the objective function value. Finally, cases from two water areas are implemented to test and verify the proposed method. The results illustrate the superiority of the proposed approach in (1) outperforming other methods used for comparison in both coverage probability and width criteria of prediction interval. (2) quantifying the prediction uncertainty and improving the reliability of trajectory predictor. (3) performing anomaly detection tasks using visualized trajectory prediction intervals. The research can help maritime traffic participants obtain more reliable trajectory data, which supports making more reasonable traffic supervision decisions.

Comparing Multiple Extended Object Tracking with Point Based Multi Object Tracking for LiDAR in a Maritime Context

In this paper, we compare a recently developed multiple extended object tracking method with a point-based tracker for maritime applications by evaluating the methods on both real and simulated LiDAR data. Being able to track other vessels is a key part of maritime situational awareness, and multi-object tracking is well suited to this task. Traditionally, multi-object tracking uses the point approximation to simplify the target tracking problem, meaning that an object is assumed to only generate a single measurement. With recent advances in sensor technology, this approximation is no longer valid. Multiple extended object tracking instead makes use of methods that estimates the extent of a target using all available measurements. However, the target models used in extended object tracking are by necessity more complex. We compare these two approaches on real and simulated LiDAR data, which was gathered by tracking smaller vessels. We find that the extended object method has greater performance on simulated data and experimental data with comparatively little wake clutter. However, on experimental data with more wake clutter, the point-based method outperforms the extended object method since it is less sensitive to the disturbance caused by wake clutter. We can conclude that multiple extended object tracking methods have the potential to be more accurate for LiDAR data, but since the models are more complex, measurement sources need to be modeled correctly.

National and international maritime situational awareness model examples and the effects of North Stream Pipelines sabotage

While many countries and international organizations with maritime security interests and rights at sea have developed new security strategies or policies in the wake of the 9/11 terrorist attacks in the United States of America, they have accordingly changed or created new maritime security strategies or doctrines with appropriate Maritime Situational Awareness (MSA) models as well. Maritime deterioration, climate change, cyberattacks, serious and organized crime, epidemics, and state-made threats are just some of the new and growing concerns affecting maritime security. The sabotage of the Nord Stream gas pipelines in the Baltic Sea has given maritime security doctrines and frameworks a new dimension. In this article, the current maritime security approaches and maritime domain or situational awareness (MDA/MSA) model examples of some countries and international organizations from different geographic regions and also the ones that are located in the maritime choke point regions where global maritime trade routes are located and also the effects of the Nord Stream Pipelines sabotages on these are examined in light of the new threats and risks. The principle result reached in this study is that countries and international structures should have a cross governmental maritime security strategy, or at least a doctrine, in order to guide their own maritime situational awareness models and identify information sharing architectures. The most important result of the sabotages on Nord Stream Pipelines for MSA models in this study is that the fastest and most cost-effective method for protecting critical infrastructure under the seas is the concept of systems such as Mothership controlled autonomous and unmanned underwater vehicles, extra large unmanned undersea vehicles and Synthetic-aperture radar (SAR) satellites.

Identification of Spoofing Ships from Automatic Identification System Data via Trajectory Segmentation and Isolation Forest

Outliers of ship trajectory from the Automatic Identification System (AIS) onboard a ship will affect the accuracy of maritime situation awareness, especially for a regular ship trajectory mixed with a spoofing ship, which has an unauthorized Maritime Mobile Service Identification code (MMSI) owned by a regular ship. As has been referred to in the literature, the trajectory of these spoofing ships would simply be removed, and more AIS data would be lost. The pre-processing of AIS data should aim to retain more information, which is more helpful in maritime situation awareness for the Maritime Safety Administration (MSA). Through trajectory feature mining, it has been found that there are obvious differences between the trajectory of a regular ship and that of a regular ship mixed with a spoofing ship, such as in terms of speed and distance between adjacent trajectory points. However, there can be a long update time interval in the results of severe missing trajectories of a ship, bringing challenges in terms of the identification of spoofing ships. In order to accurately divide the regular ship trajectory and spoofing ship trajectory, combined with trajectory segmentation by the update time interval threshold, the isolation forest was adopted in this work to train the labeled trajectory point of a regular ship mixed with a spoofing ship. The experimental results show that the average accuracy of the identification of spoofing ships using isolation forest is 88.4%, 91%, 93.1%, and 93.3%, corresponding to different trajectory segmentation by update time intervals (5 h, 10 h, 15 h, and 20 h). The research conducted in this study can almost eliminate the outliers of ship trajectory, and it also provides help for maritime situation awareness for the MSA.

Online event recognition over noisy data streams

Composite event recognition (CER) systems process streams of sensor data and infer composite events of interest by means of pattern matching. Data uncertainty is frequent in CER applications and results in erroneous detection. To support streaming applications, we present oPIECbd, an extension of oPIEC with a bounded memory, leveraging interval duration statistics to resolve memory conflicts. oPIECbd may achieve comparable predictive accuracy to batch reasoning, avoiding the prohibitive cost of such reasoning. Furthermore, the use of interval duration statistics allows oPIECbd to outperform significantly earlier versions of bounded oPIEC. The empirical evaluation demonstrates the efficacy of oPIECbd on a benchmark activity recognition dataset, as well as real data streams from the field of maritime situational awareness.

LPDNet: A Lightweight Network for SAR Ship Detection Based on Multi-Level Laplacian Denoising.

Intelligent ship detection based on synthetic aperture radar (SAR) is vital in maritime situational awareness. Deep learning methods have great advantages in SAR ship detection. However, the methods do not strike a balance between lightweight and accuracy. In this article, we propose an end-to-end lightweight SAR target detection algorithm, multi-level Laplacian pyramid denoising network (LPDNet). Firstly, an intelligent denoising method based on the multi-level Laplacian transform is proposed. Through Convolutional Neural Network (CNN)-based threshold suppression, the denoising becomes adaptive to every SAR image via back-propagation and makes the denoising processing supervised. Secondly, channel modeling is proposed to combine the spatial domain and frequency domain information. Multi-dimensional information enhances the detection effect. Thirdly, the Convolutional Block Attention Module (CBAM) is introduced into the feature fusion module of the basic framework (Yolox-tiny) so that different weights are given to each pixel of the feature map to highlight the effective features. Experiments on SSDD and AIR SARShip-1.0 demonstrate that the proposed method achieves 97.14% AP with a speed of 24.68FPS and 92.19% AP with a speed of 23.42FPS, respectively, with only 5.1 M parameters, which verifies the accuracy, efficiency, and lightweight of the proposed method.

Embedded 3D reconstruction of dynamic objects in real time for maritime situational awareness pictures

Assessing the security status of maritime infrastructures is a key factor for maritime safety and security. Facilities such as ports and harbors are highly active traffic zones with many different agents and infrastructures present, like containers, trucks or vessels. Conveying security-related information in a concise and easily understandable format can support the decision-making process of stakeholders, such as port authorities, law enforcement agencies and emergency services. In this work, we propose a novel real-time 3D reconstruction framework for enhancing maritime situational awareness pictures by joining temporal 2D video data into a single consistent display. We introduce and verify a pipeline prototype for dynamic 3D reconstruction of maritime objects using a static observer and stereoscopic cameras on an GPU-accelerated embedded device. A simulated dataset of a harbor basin was created and used for real-time processing. Usage of a simulated setup allowed verification against synthetic ground-truth data. The presented pipeline runs entirely on a remote, low-power embedded system with ∼\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\sim $$\\end{document}6 Hz. A Nvidia Jetson Xavier AGX module was used, featuring 512 CUDA-cores, 16 GB memory and an ARMv8 64-bit octa-core CPU.

Maritime information sharing environment deployment using the advanced multilayered Data Lake capabilities

Establishing an efficient information sharing network among national agencies in maritime domain is of essential importance in enhancing the operational performance, increasing the situational awareness and enabling interoperability among all involved maritime surveillance assets. Based on various data-driven technologies and sources, the EU initiative of Common Information Sharing Environment (CISE), enables the networked participants to timely exchange information concerning vessel traffic, joint SAR & operational missions, emergency situations and other events at sea. In order to host and process vast amounts of vessels and related maritime data consumed from heterogeneous sources (e.g. SAT-AIS, UAV, radar, METOC), the deployment of big data repositories in the form of Data Lakes is of great added value. The different layers in the Data Lakes with capabilities for aggregating, fusing, routing and harmonizing data are assisted by decision support tools with combined reasoning modules with semantics aiming at providing a more accurate Common Operational Picture (COP) among maritime agencies. Based on these technologies, the aim of this paper is to present an end-to-end interoperability framework for maritime situational awareness in strategic and tactical operations at sea, developed in EFFECTOR EU-funded project, focusing on the multilayered Data Lake capabilities. Specifically, a case study presents the important sources and processing blocks, such as the SAT-AIS, CMEMS, UAV components, enabling maritime information exchange in CISE format and communication patterns. Finally, the technical solution is validated in the project’s recently implemented maritime operational trials and the respective results are documented.

Grid-Based Vessel Deviation from Route Identification with Unsupervised Learning

The application of anomaly-monitoring and surveillance systems is crucial for improving maritime situational awareness. These systems must work on the fly in order to provide the operator with information on potentially dangerous or illegal situations as they are occurring. We present a system for identifying vessels deviating from their normal course of travel, from unlabelled AIS data. Our approach attempts to solve problems with scalability and on-line learning of other grid-based systems available in the literature, by applying a dynamic grid size, adjustable per vessel characteristics, combined with a binary-search tree method for data discretization and vessel grid search. The results of this study have been validated during the Portuguese Maritime Trial in April 2022, conducted by the Portuguese navy along the southern coast of Portugal.

A Method for Vessel’s Trajectory Prediction Based on Encoder Decoder Architecture

Data-driven technologies and automated identification systems (AISs) provide unprecedented opportunities for maritime surveillance. As part of enhancing maritime situational awareness and safety, in this paper, we address the issue of predicting a ship’s future trajectory using historical AIS observations. The objective is to use past data in the training phase to learn the predictive distribution of marine traffic patterns and then use that information to forecast future trajectories. To achieve this, we investigate an encoder–decoder architecture-based sequence-to-sequence prediction model and CNN model. This architecture includes a long short-term memory (LSTM) RNN that encodes sequential AIS data from the past and generates future trajectory samples. The effectiveness of sequence-to-sequence neural networks (RNNs) for forecasting future vessel trajectories is demonstrated through an experimental assessment using an AIS dataset.

Incorporation of Deep Kernel Convolution into Density Clustering for Shipping AIS Data Denoising and Reconstruction

Automatic Identification System (AIS) equipment can aid in identifying ships, reducing ship collision risks and ensuring maritime safety. However, the explosion of massive AIS data has caused increasing data processing challenges affecting their practical applications. Specifically, mistakes, noise, and missing data are presented during AIS data transmission and encoding, resulting in poor data quality and inaccurate data sources that negatively impact maritime safety research. To address this issue, a robust AIS data denoising and reconstruction methodology was proposed to realise the data preprocessing for different applications in maritime transportation. It includes two parts: Density-Based Spatial Clustering of Applications with Noise based on Deep Kernel Convolution (DBSCANDKC) and the reconstruction method, which can extract high-quality AIS data to guarantee the accuracy of the related maritime research. Firstly, the kinematics feature was employed to remove apparent noise from the AIS data. The square deep kernel convolution was then incorporated into density clustering to find and remove possibly anomalous data. Finally, a piecewise cubic spline interpolation approach was applied to construct the missing denoised trajectory data. The experiments were implemented in the Arctic Ocean and Strait of Dover to demonstrate the effectiveness and performance of the proposed methodology in different shipping environments. This methodology makes significant contributions to future maritime situational awareness, collision avoidance, and robust trajectory development for safety at sea.

Optimizing vessel trajectory compression for maritime situational awareness

Optimizing vessel trajectory compression for maritime situational awareness

Multiparametric sea state fields from synthetic aperture radar for maritime situational awareness

This paper introduces a method for estimating a series of sea state parameters from satellite-borne synthetic aperture radar (SAR). The method was realized in a near real time (NRT) application which allows for the processing of data from different satellites and modes. The algorithm estimates the total significant wave height Hs, dominant and secondary swell and windsea wave heights, first, and second moment wave periods, the mean wave period and the period of wind sea. The algorithm was applied to the Sentinel-1 (S1) C-band Interferometric Wide Swath Mode (IW), Extra Wide (EW) and Wave Mode (WV) Level-1 (L1) products and also extended to X-band TerraSAR-X (TS-X) StripMap (SM) products. The scenes are processed in raster and result in continuous sea state fields, with the exception of S1 WV, where averaged values for sea state parameters for along-orbit imagettes of 20 km × 20 km are presented.The developed empirical algorithm consists of two parts: a first CWAVE_EX (extended CWAVE) part, based on a linear regression approach, and a subsequent machine learning part using the support vector machine (SVM) technique. A series of new data preparation steps (i.e. filtering, denoising) and new features estimated from SAR images are also introduced. The algorithm was tuned and validated using two independent global wave model hindcasts, WaveWatch-3 and MFWAM as well as National Data Buoy Center (NDBC) measurements. The achieved root mean squared errors (RMSE) for CWAVE_EX for the total Hs are 0.60 m for low-resolution modes S1 IW (10 m pixel) and EW (40 m pixel) and 0.35 m for S1 WV and TS-X SM (pixel spacing ca.1–4 m) in comparison to model predictions. The RMSEs of the retrieved wave periods are in the range of 0.45–0.90 s for all of the satellites and models considered. Similarly, the dominant and secondary swell, and wind sea wave height RMSEs are in the range of 0.35–0.80 m. The SVM postprocessing improves the accuracy of the initial results of CWAVE_EX for Hs and reaches an RMSE of 0.25 m for S1 WV. Comparisons to 64 NDBC buoys, collocated at distances shorter than 50 km to S1 WV imagettes worldwide, result in an RMSE of 0.41 m. All results and the methods presented are novel in terms of the accuracy achieved, combining the classical approach with machine learning techniques, and performing an automatic NRT processing of multiparametric sea state fields from L1 data with automatic switching for different satellites and modes.The complete archive of S1 WV L1 Single Look Complex products from December 2014 until February 2021 was processed to create a sea state parameter database and validated using model hindcast and buoy measurements. The derived parameters are available to the public within the scope of the European Space Agency's Climate Change Initiative.

Ship Target Identification via Bayesian-Transformer Neural Network

Ship target identification is of great significance in both military and civilian fields. Many methods have been proposed to identify the targets using tracks information. However, most of existing studies can only identify two or three types of targets, and the accuracy of identification needs to be further improved. Meanwhile, they do not provide a reliable probability of the identification result under a high-noise environment. To address these issues, a Bayesian-Transformer Neural Network (BTNN) is proposed to complete the ship target identification task using tracks information. The aim of the research is improving the ability of ship target identification to enhance the maritime situation awareness and strengthen the protection of maritime traffic safety. Firstly, a Bayesian-Transformer Encoder (BTE) module that contains four different Bayesian-Transformer Encoders is used to extract discriminate features of tracks. Then, a Bayesian fully connected layer and a SoftMax layer complete the classification. Benefiting from the superiority of the Bayesian neural network, BTNN can provide a reliable probability of the result, which captures both aleatoric uncertainty and epistemic uncertainty. The experiments show that the proposed method can successfully identify nine types of ship targets. Compared with traditional methods, the identification accuracy of BTNN increases by 3.8% from 90.16%. In addition, compared with non-Bayesian Transformer Neural Network, the BTNN can provide a more reliable probability of the identification result under a high-noise environment.

Ship Segmentation and Georeferencing from Static Oblique View Images.

Camera systems support the rapid assessment of ship traffic at ports, allowing for a better perspective of the maritime situation. However, optimal ship monitoring requires a level of automation that allows personnel to keep track of relevant variables in the maritime situation in an understandable and visualisable format. It therefore becomes important to have real-time recognition of ships present at the infrastructure, with their class and geographic position presented to the maritime situational awareness operator. This work presents a novel dataset, ShipSG, for the segmentation and georeferencing of ships in maritime monitoring scenes with a static oblique view. Moreover, an exploration of four instance segmentation methods, with a focus on robust (Mask-RCNN, DetectoRS) and real-time performances (YOLACT, Centermask-Lite) and their generalisation to other existing maritime datasets, is shown. Lastly, a method for georeferencing ship masks is proposed. This includes an automatic calculation of the pixel of the segmented ship to be georeferenced and the use of a homography to transform this pixel to geographic coordinates. DetectoRS provided the highest ship segmentation mAP of 0.747. The fastest segmentation method was Centermask-Lite, with 40.96 FPS. The accuracy of our georeferencing method was (22 ± 10) m for ships detected within a 400 m range, and (53 ± 24) m for ships over 400 m away from the camera.

Real-Time Ship Tracking under Challenges of Scale Variation and Different Visibility Weather Conditions

Visual ship tracking provides crucial kinematic traffic information to maritime traffic participants, which helps to accurately predict ship traveling behaviors in the near future. Traditional ship tracking models obtain a satisfactory performance by exploiting distinct features from maritime images, which may fail when the ship scale varies in image sequences. Moreover, previous frameworks have not paid much attention to weather condition interferences (e.g., visibility). To address this challenge, we propose a scale-adaptive ship tracking framework with the help of a kernelized correlation filter (KCF) and a log-polar transformation operation. First, the proposed ship tracker employs a conventional KCF model to obtain the raw ship position in the current maritime image. Second, both the previous step output and ship training sample are transformed into a log-polar coordinate system, which are further processed with the correlation filter to determine ship scale factor and to suppress the negative influence of the weather conditions. We verify the proposed ship tracker performance on three typical maritime scenarios under typical navigational weather conditions (i.e., sunny, fog). The findings of the study can help traffic participants efficiently obtain maritime situation awareness information from maritime videos, in real time, under different visibility weather conditions.