Optical Image Sequences Research Articles

Optical flow-based vertical angular rate fault detection on UAVs

This paper discusses optical flow-based vertical angular rate fault detection based-on real fight IMU data and camera image sequences. The fight test platform utilizes a gimbal stabilized downward looking camera that is why only the angular rate around the inertial vertical axis can be estimated. Improvements relative to the previous work of the authors are the application of forward-backward optical flow to filter outliers, the direct detection of angular rate faults without the need for Euler angles or GPS velocities and tuning and verification on real images and fight data. After presenting the optical flow equations and the method for image-based vertical angular rate estimation the fight test scenarios and video processing steps are introduced. Four different Artificial errors are added to the IMU measurements and the fault detection applies residual thresholding and up-down counters. Tuning is based on false alarm and missed detection rates and the application of receiver operating characteristics. Finally, the estimation results and possible further improvements are presented.

Contrastive-Learning-Based Time-Series Feature Representation for Parcel-Based Crop Mapping Using Incomplete Sentinel-2 Image Sequences

Parcel-based crop classification using multi-temporal satellite optical images plays a vital role in precision agriculture. However, optical image sequences may be incomplete due to the occlusion of clouds and shadows. Thus, exploring inherent time-series features to identify crop types from incomplete optical image sequences is a significant challenge. This study developed a contrastive-learning-based framework for time-series feature representation to improve crop classification using incomplete Sentinel-2 image sequences. Central to this method was the combined use of inherent time-series feature representation and machine-learning-based classifications. First, preprocessed multi-temporal Sentinel-2 satellite images were overlaid onto precise farmland parcel maps to generate raw time-series spectral features (with missing values) for each parcel. Second, an enhanced contrastive learning model was established to map the raw time-series spectral features to their inherent feature representation (without missing values). Thirdly, eXtreme Gradient-Boosting-based and Long Short-Term Memory-based classifiers were applied to feature representation to produce crop classification maps. The proposed method is further discussed and validated through parcel-based time-series crop classifications in two study areas (one in Dijon of France and the other in Zhaosu of China) with multi-temporal Sentinel-2 images in comparison to the existing methods. The classification results, demonstrating significant improvements greater than 3% in overall accuracy and 0.04 in F1 scores over comparison methods, indicate the effectiveness of the proposed contrastive-learning-based time-series feature representation for parcel-based crop classification utilizing incomplete Sentinel-2 image sequences.

Implementation of Real-Time Space Target Detection and Tracking Algorithm for Space-Based Surveillance

Space-based target surveillance is important for aerospace safety. However, with the increasing complexity of the space environment, the stellar target and strong noise interference pose difficulties for space target detection. Simultaneously, it is hard to balance real-time processing with computational performance for the onboard processing platform owing to resource limitations. The heterogeneous multi-core architecture has corresponding processing capabilities, providing a hardware implementation platform with real-time and computational performance for space-based applications. This paper first developed a multi-stage joint detection and tracking model (MJDTM) for space targets in optical image sequences. This model combined an improved local contrast method and the Kalman filter to detect and track the potential targets and use differences in movement status to suppress the stellar targets. Then, a heterogeneous multi-core processing system based on a field-programmable gate array (FPGA) and digital signal processor (DSP) was established as the space-based image processing system. Finally, MJDTM was optimized and implemented on the above image processing system. The experiments conducted with simulated and actual image sequences examine the accuracy and efficiency of the MJDTM, which has a 95% detection probability while the false alarm rate is 10−4. According to the experimental results, the algorithm hardware implementation can detect targets in an image with 1024 × 1024 pixels in just 22.064 ms, which satisfies the real-time requirements of space-based surveillance.

Analysis-ready optical underwater images of Manganese-nodule covered seafloor of the Clarion-Clipperton Zone

We provide a sequence of analysis-ready optical underwater images from the Clarion-Clipperton Zone (CCZ) of the Pacific Ocean. The images were originally recorded using a towed camera sledge that photographed a seabed covered with polymetallic manganese-nodules, at an average water depth of 4,250 meters. The original degradation in visual quality and inconsistent scale among individual raw images due to different altitude implies that they are not scientifically comparable in their original form. Here, we present analysis-ready images that have already been pre-processed to account for this degradation. We also provide accompanying metadata for each image, which includes their geographic coordinates, depth of the seafloor, absolute scale (cm/pixel), and seafloor habitat class obtained from a previous study. The provided images are thus directly usable by the marine scientific community e.g., to train machine learning models for seafloor substrate classification and megafauna detection.

An assessment of microvascular hemodynamics in human macula

An adequate blood supply to meet the energy demands is essential for any tissue, particularly for high energy demand tissues such as the retina. A critical question is: How is the dynamic match between neuronal demands and blood supply achieved? We present a quantitative assessment of temporal and spatial variations in perfusion in the macular capillary network in 10 healthy human subjects using a non-invasive and label-free imaging technique. The assessment is based on the calculation of the coefficient of variation (CoV) of the perfusion signal from arterioles, venules and capillaries from a sequence of optical coherence tomography angiography images centred on the fovea. Significant heterogeneity of the spatial and temporal variation was found within arterioles, venules and capillary networks. The CoV values of the capillaries and smallest vessels were significantly higher than that in the larger vessels. Our results demonstrate the presence of significant heterogeneity of spatial and temporal variation within each element of the macular microvasculature, particularly in the capillaries and finer vessels. Our findings suggest that the dynamic match between neuronal demands and blood supply is achieved by frequent alteration of local blood flow evidenced by capillary perfusion variations both spatially and temporally in the macular region.

Visual Detection and Association Tracking of Dim Small Ship Targets from Optical Image Sequences of Geostationary Satellite Using Multispectral Radiation Characteristics

By virtue of the merits of wide swath, persistent observation, and rapid operational response, geostationary remote sensing satellites (e.g., GF-4) show tremendous potential for sea target system surveillance and situational awareness. However, ships in such images appear as dim small targets and may be affected by clutter, reef islands, clouds, and other interferences, which makes the task of ship detection and tracking intractable. Considering the differences in visual saliency characteristics across multispectral bands between ships and jamming targets, a novel approach to visual detecting and association tracking of dense ships based on the GF-4 image sequences is proposed in this paper. First, candidate ship blobs are segmented in each single-spectral image of each frame through a multi-vision salient features fusion strategy, to obtain the centroid position, size, and corresponding spectral grayscale information of suspected ships. Due to the displacement of moving ships across multispectral images of each frame, multispectral association with regard to the positions of ship blobs is then performed to determine the final ship detections. Afterwards, precise position correction of detected ships is implemented for each frame in image sequences via multimodal data association between GF-4 detections and automatic identification system data. Last, an improved multiple hypotheses tracking algorithm with multispectral radiation and size characteristics is put forward to track ships across multi-frame corrected detections and estimate ships’ motion states. Experiment results demonstrate that our method can effectively detect and track ships in GF-4 remote sensing image sequences with high precision and recall rate, yielding state-of-the-art performance.

A Driver Drowsiness Detection Scheme Based on 3D Convolutional Neural Networks

It is an obvious fact that drivers’ drowsiness is more likely to cause traffic accidents. Recently, driver drowsiness detection has drawn considerable attention. In this paper, a novel drowsiness detection scheme is proposed, which can recognize drivers’ drowsiness actions through their facial expressions. First, a drowsiness action recognition model based on 3D-CNN is proposed, which can effectively distinguish drivers’ drowsiness actions and nondrowsiness actions. Second, a fusion algorithm of the two input streams is proposed, which can fuse gray image sequence and optical image sequence containing target motion information. Finally, the proposed model is evaluated on National Tsinghua University Driver Drowsiness Detection (NTHU-DDD) dataset. The experimental results show that the algorithm performs better than other algorithms, and its accuracy reaches 86.64%.

COCO-Net: A Dual-Supervised Network With Unified ROI-Loss for Low-Resolution Ship Detection From Optical Satellite Image Sequences

Low-resolution ship detection from optical satellite image sequences is critical in high-orbit remote sensing satellite applications. However, it is still a difficult problem due to the following challenges: 1) the size of the ship is tiny in the low-resolution image; 2) the ship target is dim and the contrast with the background is low; 3) the interference of cloud and fog covering is complex and changeable. For these reasons, the targets are easily lost during the detection. In fact, the Clearer the Objects against to the background, the more Confident the Observers can detect it. In light of these considerations, we propose a COCO-Net to detect the small dynamic objects on low-resolution images in this paper. First, the multi-frame images are associated by introducing motion information as an effective compensation for small object features. Second, an integrated dual-supervised network that processes single-level tasks hierarchically is presented to adaptively enhance the input data quality of object detection without being limited by diverse scene disturbances. Third, a unified ROI-loss scheme that modulates the loss function of the first component by introducing ROI-masks from the second component is utilized to make the first component also work for object detection. In addition, we construct a new dataset for the small dynamic object detection based on the GaoFen-4 satellite imagery. Comprehensive experiments on a self-assembled dataset from the GaoFen-4 satellite show the superior performance of the proposed method compared to state-of-the-art object detectors.

SEN12MS-CR-TS: A Remote-Sensing Data Set for Multimodal Multitemporal Cloud Removal

About half of all optical observations collected via spaceborne satellites are affected by haze or clouds. Consequently, cloud coverage affects the remote-sensing practitioner’s capabilities of a continuous and seamless monitoring of our planet. This work addresses the challenge of optical satellite image reconstruction and cloud removal by proposing a novel multimodal and multitemporal data set called SEN12MS-CR-TS. We propose two models highlighting the benefits and use cases of SEN12MS-CR-TS: First, a multimodal multitemporal 3-D convolution neural network that predicts a cloud-free image from a sequence of cloudy optical and radar images. Second, a sequence-to-sequence translation model that predicts a cloud-free time series from a cloud-covered time series. Both approaches are evaluated experimentally, with their respective models trained and tested on SEN12MS-CR-TS. The conducted experiments highlight the contribution of our data set to the remote-sensing community as well as the benefits of multimodal and multitemporal information to reconstruct noisy information. Our data set is available at <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><uri>https://patrickTUM.github.io/cloud_removal</uri></i> .

100 kHz CH2O imaging realized by lower speed planar laser-induced fluorescence and deep learning.

This paper reports an approach to interpolate planar laser-induced fluorescence (PLIF) images of CH2O between consecutive experimental data by means of computational imaging realized with convolutional neural network (CNN). Such a deep learning based method can achieve higher temporal resolution for 2D visualization of intermediate species in combustion based on high-speed experimental images. The capability of the model was tested for generating 100 kHz PLIF images by interpolating single and multiple PLIF frames into the sequences of experimental images of lower frequencies (50, 33, 25 and 20 kHz). Results show that the prediction indices, including intersection over union (IoU), peak signal to noise ratio (PSNR), structural similarity index (SSIM), and time averaged correlation coefficient at various axial positions could achieve acceptable accuracy. This work sheds light on the utilization of CNN-based models to achieve optical flow computation and image sequence interpolation, also providing an efficient off-line model as an alternative pathway to overcome the experimental challenges of the state-of-the-art ultra-high speed PLIF techniques, e.g., to further increase repetition rate and save data transfer time.

Real-time crowd behavior recognition in surveillance videos based on deep learning methods

Automatic video surveillance in public crowded places has been an active research area for security purposes. Traditional approaches try to solve the crowd behavior recognition task using a sequential two-stage pipeline as low-level feature extraction and classification. Lately, deep learning has shown promising results in comparison to traditional methods by extracting high-level representation and solving the problem in an end-to-end pipeline. In this paper, we investigate a deep architecture for crowd event recognition to detect seven behavior categories in PETS2009 event recognition dataset. More especially, we apply an integrated handcrafted and Conv-LSTM-AE method with optical flow images as input to extract a high-level representation of data and conduct classification. After achieving a latent representation of input optical flow image sequences in the bottleneck of autoencoder(AE), the architecture is split into two separate branches, one as AE decoder and the other as the classifier. The proposed architecture is jointly trained for representation and classification by defining two different losses. The experimental results in comparison to the state-of-the-art methods demonstrate that our algorithm can be promising for real-time event recognition and achieves a better performance in calculated metrics.

Space target detection in optical image sequences for wide-field surveillance

ABSTRACT A wide-field surveillance system with a long exposure time has a stronger detection capability for faint space targets. However, some of the complexities it generates also pose difficulties for space target detection; a large amount of image data, numberless object points, some stars manifesting as streak-like sources, and possible discontinuous or nonlinear target trajectories. This paper presents a high precision and low computational-cost space target detection method to overcome these obstacles. Firstly, the minimum external rectangle method is implemented to effectively remove stars and noise. Secondly, the motion velocity of the targets is calculated as the basis for predicting the allowed state transition region in each image of the frame set. Finally, a dynamic programming sliding window method is proposed to detect space targets with continuous, discontinuous, linear or nonlinear trajectories. The experimental results show that this method can effectively detect faint space targets in wide-field surveillance under a long exposure time. This method also has the advantage of a low computational cost.

Sea–sky line detection using gray variation differences in the time domain for unmanned surface vehicles

Sea–sky lines (SKLs) can provide valuable reference information for the obstacle avoidance systems of unmanned surface vehicles (USVs) because obstacles that can threaten the safety of USVs, such as ships and rocks, are generally located below SKLs. Existing methods detect SKLs only using gray, texture or line features in an optical image. However, the background features in images obtained by onboard cameras are complex and change continuously over time, which leads to poor robustness of these methods. Conversely, the method that we proposed in this paper detects SKLs according to the gray variation differences in the time domain between the sky, the SKL and the sea surface when the USVs actually move on the sea. In this way, we can decrease the gray complexity of the image data of different backgrounds, which reduces interference while enhancing the edge features of the SKLs to obtain superior robustness. The proposed method is tested on optical image sequences collected by the ‘Jiu Hang 490’ USV. The experimental results demonstrate that the average accuracy of SKL detection is higher than that of existing methods with approximately 10% improvement while maintaining computational efficiency.

Inferring Surface Flow Velocities in Sediment-Laden Alaskan Rivers from Optical Image Sequences Acquired from a Helicopter

The remote, inaccessible location of many rivers in Alaska creates a compelling need for remote sensing approaches to streamflow monitoring. Motivated by this objective, we evaluated the potential to infer flow velocities from optical image sequences acquired from a helicopter deployed above two large, sediment-laden rivers. Rather than artificial seeding, we used an ensemble correlation particle image velocimetry (PIV) algorithm to track the movement of boil vortices that upwell suspended sediment and produce a visible contrast at the water surface. This study introduced a general, modular workflow for image preparation (stabilization and geo-referencing), preprocessing (filtering and contrast enhancement), analysis (PIV), and postprocessing (scaling PIV output and assessing accuracy via comparison to field measurements). Applying this method to images acquired with a digital mapping camera and an inexpensive video camera highlighted the importance of image enhancement and the need to resample the data to an appropriate, coarser pixel size and a lower frame rate. We also developed a Parameter Optimization for PIV (POP) framework to guide selection of the interrogation area (IA) and frame rate for a particular application. POP results indicated that the performance of the PIV algorithm was highly robust and that relatively large IAs (64–320 pixels) and modest frame rates (0.5–2 Hz) yielded strong agreement ( R 2 &gt; 0.9 ) between remotely sensed velocities and field measurements. Similarly, analysis of the sensitivity of PIV accuracy to image sequence duration showed that dwell times as short as 16 s would be sufficient at a frame rate of 1 Hz and could be cut in half if the frame rate were doubled. The results of this investigation indicate that helicopter-based remote sensing of velocities in sediment-laden rivers could contribute to noncontact streamgaging programs and enable reach-scale mapping of flow fields.

Space Debris Detection Using Feature Learning of Candidate Regions in Optical Image Sequences

Space debris detection is important in space situation awareness and space asset protection. In this article, we propose a method to detect space debris using feature learning of candidate regions. The acquired optical image sequences are first processed to remove hot pixels and flicker noise, and the nonuniform background information is removed by the proposed one dimensional mean iteration method. Then, the feature learning of candidate regions (FLCR) method is proposed to extract the candidate regions and to detect space debris. The candidate regions of space debris are precisely extracted, and then classified by a trained deep learning network. The feature learning model is trained using a large number of simulated space debris with different signal to noise ratios (SNRs) and motion parameters, instead of using real space debris, which make it difficult to extract a sufficient number of real space debris with diverse parameters in optical image sequences. Finally, the candidate regions are precisely placed in the optical image sequences. The experiment is performed using the simulated data and acquired image sequences. The results show that the proposed method has good performance when estimating and removing background, and it can detect low SNR space debris with high detection probability.

Topological Sweep for Multi-Target Detection of Geostationary Space Objects

Conducting surveillance of the geocenric orbits is a key task towards achieving space situational awareness (SSA). Our work focuses on the optical detection of man-made objects (e.g., satellites, space debris) in Geostationary orbit (GEO), which is home to major space assets such as telecommunications and Earth observing satellites. GEO object detection is challenging due to the distance of the targets, which appear as small dim point-like objects among a background of streak-like objects. In this paper, we propose a novel multi-target detection technique based on topological sweep, to find GEO objects from a short sequence of optical images. Our topological sweep technique exploits the geometric duality that underpins the approximately linear trajectory of target objects across the sequence, to extract the targets from significant clutter and noise. Unlike standard multi-target methods, our algorithm deterministically solves a combinatorial problem to ensure high-recall rates without requiring accurate initializations. The usage of geometric duality also yields an algorithm that is computationally efficient and suitable for online processing.

Vision-Based Hazard Detection with End-to-end Spatiotemporal Networks for Planetary Landing

Hazard detection is the key technique for autonomous landing during a planetary exploration mission. This paper proposed an end-to-end spatiotemporal network to detect all hazards in the image sequences captured by optical camera. The spatial stream processes the colour image sequences and the temporal one learns features from optical flow image sequences. The spatial features and temporal features are fused by the proposed metric fusion method, and then the spatiotemporal features become more discriminative through triplet loss. A hazard map can ultimately be obtained in the testing phase after processing on the full-size image sequences. The evaluation results prove the effectiveness of the proposed network, and the testing results show the feasibility of practical application.

Multiview-based group behavior analysis in optical image sequence

Group behavior analysis is a hot topic in intelligent video surveillance, and has attracted a surge of interest in the field of artificial intelligence. Groups are the basic components of a crowd system, and provide a high-level representation of the crowd phenomenon. By investigating the motion dynamics within each image patch, this paper proposes a multiview-based group behavior analysis method that is able to divide the paths into different groups. The main contributions are threefold: (1) the correlation between image paths is captured from four views (interaction, distance, motion direction, and motion transition), (2) a multiview clustering method with diversity regularization is proposed to perceive the complementary information within the multiview data and alleviate the influence of redundant features, and (3) a cluster merging strategy is designed to combine the highly correlated clusters and determine the final groups automatically. Experimental results on several benchmark datasets validate the good performance of the proposed method.

ЭВОЛЮЦИЯ МЕЗОМАСШТАБНОГО АНТИЦИКЛОНИЧЕСКОГО ВИХРЯ И ВИХРЕВЫХ ДИПОЛЕЙ/МУЛЬТИПОЛЕЙ НА ЕГО ОСНОВЕ В ЮГО-ВОСТОЧНОЙ БАЛТИКЕ (СПУТНИКОВАЯ ИНФОРМАЦИЯ: МАЙ–ИЮЛЬ 2015 г.)

Based on the analysis of a sequence of optical (MODIS-Aqua / Terra, AVHRR NOAA-18, VIIRS-SNPP, TIRS and OLI Landsat-8, ETM + Landsat-7) and radar (Sentinel-1A, Radarsat-2) satellite images, the evolution of mesoscale anticyclonic vortex with a diameter of about 35 km and associated cyclones (two to four) at its periphery was traced for more than a month and a half (from May 15 to July 7, 2015) in the South-Eastern Baltic. Within about a month the position of the anticyclone varied little (center – at about 54° 55’ N, 19°15’ E), but in the interval from June 22 to July 1 the vortex dipole formed by this anticyclone and the cyclone at its northern periphery with a diameter comparable with the diameter of the anticyclone moved eastward with an average speed of about 4 km/day. It is assumed that such movement of this quasisymmetric vortex dipole was due to its location to the north of the Gdansk Bay and the property of a dipole (mushroom-like current) to move in the direction of its jet part under wind forcing (in this case, with the strengthening of the westerly wind).

Development of a multichannel microfluidic system with Schlieren imaging microscopy for online chip-based moving boundary electrophoresis

The concentration gradient detection method based on the Schlieren optics employed for electrophoresis analyses by extending the technology to a multi-channel system using a prototyped microfluidic chip (thinXXS Micro-technology, Germany). The results prove that coupling a chip-based microfluidic device with Schlieren detection is an appropriate approach to improve the electrophoretic separations. The effects of channel's geometry and dimension were investigated by conducting the experiments in channels with different cross sectional areas. Fast kinetic data acquisition of the charge-coupled device (CCD) camera facilitated recording of a time sequence of optical images, demonstrating the potential of the CCD camera as a powerful tool for studying dynamic processes such as diffusion. Diffusion coefficients of sample proteins were measured under static and dynamic conditions, where the static mode demonstrated more accurate results. Furthermore, the Fourier transformation was employed to improve the Schlieren images for quantitative analysis of the diffusion coefficient measurement.