Color Image Sequences Research Articles

Ocean Color Image Sequences Reveal Diurnal Changes in Water Column Stability Driven by Air–Sea Interactions

The southward propagation of cold-air frontal boundaries into the Gulf of Mexico region initiates a cascade of coupled air–sea processes that manifests along the coastlines as an apparent brightness anomaly in the ocean color signals. Our hypothesis is that the color anomaly is largely due to the turbulent resuspension of sedimentary particles. Initially, there is significant wind-driven ocean turbulence as the frontal boundary passes, followed by the potential for sustained convective instability due to significant heat losses from the ocean surface. These cold front events occur during boreal autumn, winter, and into early spring, and the latter episodes occur in the context of the seasonally recurring thermal stratification of shelf waters. Here, we show that as stratification reasserts thermal stability in the waning days of a cold front episode, daily to hourly ocean color patterns are temporally coherent with the air–sea heat flux changes and the resulting impact on water column stability. Concomitant results from a nested, data-assimilative, and two-way coupled ocean-atmosphere numerical modeling system provides both corroboration and insight into how surface air–sea fluxes and in-water turbulent mixing manifest as hourly changes in apparent surface water turbidity due to the potential excitation and settling of reflective particles. A simple model of particle mixing and settling driven by the simulated turbulence mimics patterns seen in the satellite image sequences. This study offers a preview of potential application areas that may emerge following the launch of a dedicated ocean color geostationary sensor.

Phenomics for Komatsuna plant growth tracking using deep learning approach

Plant growth tracking has potential applications in plant breeding research, crop management, understanding gene–environment interactions, stress analysis, and yield prediction. In this work, we determine plant phenotypes to track the plant’s growth over time. We use a publicly available Komatsuna plant dataset that includes time series color image sequences of Komatsuna plants captured over time to track plant growth. First, we use a deep neural network-based multi-class semantic segmentation algorithm to track each plant leaf in the image sequence. We achieved average multi-class semantic segmentation accuracy of 97.41%, MIoU of 0.9032 and dice coefficient of 0.9147 using DeepLabV3+ model. The model could track partially occluded and tiny emerging leaves throughout image sequences. In the second phase, we determine plant phenotypes like leaf count, projected plant area, emergence time of leaves using segmented images obtained in the first stage. We investigate the algorithm’s performance by evaluating and comparing plant phenotype values for segmented and ground-truth images. Growth curves indicate similar patterns for all the image sequences and demonstrate the method’s applicability for plant growth analysis.

Unsupervised Deep Learning Registration of Uterine Cervix Sequence Images.

Simple SummaryA sequence of images can be taken after the application of acetic acid during a colposcopic examination of the uterine cervix to capture the dynamic visual variations due to the aceto-whitening reaction on the cervical epithelium. Automated analyis of these changes require spatial alignment of the apparent change in the cervix location in the image sequence due to patient movement or imaging device repositioning. We developed a new self-supervised RGB-colored deep learning-based image registration method to automatically align the images that does not require a manually-provided reference standard. We also fine-tuned a transformer-based segmentation network to evaluate the result of our registration method which achieved 12.62% higher in Dice/IoU scores in cervix boundary detection compared to the unregistered images.During a colposcopic examination of the uterine cervix for cervical cancer prevention, one or more digital images are typically acquired after the application of diluted acetic acid. An alternative approach is to acquire a sequence of images at fixed intervals during an examination before and after applying acetic acid. This approach is asserted to be more informative as it can capture dynamic pixel intensity variations on the cervical epithelium during the aceto-whitening reaction. However, the resulting time sequence images may not be spatially aligned due to the movement of the cervix with respect to the imaging device. Disease prediction using automated visual evaluation (AVE) techniques using multiple images could be adversely impacted without correction for this misalignment. The challenge is that there is no registration ground truth to help train a supervised-learning-based image registration algorithm. We present a novel unsupervised registration approach to align a sequence of digital cervix color images. The proposed deep-learning-based registration network consists of three branches and processes the red, green, and blue (RGB, respectively) channels of each input color image separately using an unsupervised strategy. Each network branch consists of a convolutional neural network (CNN) unit and a spatial transform unit. To evaluate the registration performance on a dataset that has no ground truth, we propose an evaluation strategy that is based on comparing automatic cervix segmentation masks in the registered sequence and the original sequence. The compared segmentation masks are generated by a fine-tuned transformer-based object detection model (DeTr). The segmentation model achieved Dice/IoU scores of 0.917/0.870 and 0.938/0.885, which are comparable to the performance of our previous model in two datasets. By comparing our segmentation on both original and registered time sequence images, we observed an average improvement in Dice scores of 12.62% following registration. Further, our approach achieved higher Dice and IoU scores and maintained full image integrity compared to a non-deep learning registration method on the same dataset.

Applying a Deep Learning Neural Network to Gait-Based Pedestrian Automatic Detection and Recognition

Gait recognition is a noncontact biometric procedure that determines the identity or health status of a person by analyzing his or her walking posture and habits, including skeletal and joint movements. The most remarkable feature of this method is the possibility of conducting recognition without demanding much cooperation from participants. Therefore, this recognition technique has attracted much attention from scholars. Additionally, because of the rapid development of graphics processing unit technology, related hardware and computation performance, the applications of deep-learning technology are considerably enhanced. The objective of this study was to apply a deep neural network (DNN), which employs deep-learning technology, to achieve gait-based automatic pedestrian detection and recognition. In contrast to using wearable devices to precisely capture skeletal and joint movements, pedestrian color-image sequences were used as input in this study. Subsequently, a pretraining convolutional neural network (CNN) was employed to capture pedestrian location and extract pedestrian dense optical flow to serve as concrete low-level feature inputs. Then, a finely-tuned DNN based on the wide residual network was employed to extract high-level abstract features. In addition, to overcome the difficulty of obtaining local temporal features by using a 2D CNN, part of the 3D convolutional structure was introduced into the CNN. This design enabled use of limited memory to acquire more effective features and enhance the DNN performance. The experimental results show that the proposed method has exceptional performance for pedestrian detection and recognition.

Architecture for Automatic Recognition of Group Activities Using Local Motions and Context

Currently, the ability to automatically detect human behavior in image sequences is one of the most important challenges in the area of computer vision. Within this broad field of knowledge, the recognition of activities of people groups in public areas is receiving special attention due to its importance in many aspects including safety and security. This paper proposes a generic computer vision architecture with the ability to learn and recognize different group activities using mainly the local group’s movements. Specifically, a multi-stream deep learning architecture is proposed whose two main streams correspond to a representation based on a descriptor capable of representing the trajectory information of a sequence of images as a collection of local movements that occur in specific regions of the scene. Additional information (e.g. location, time, etc.) to strengthen the classification of activities by including it as additional streams. The proposed architecture is capable of classifying in a robust way different activities of a group as well to deal with the one-class problems. Moreover, the use of a simple descriptor that transforms a sequence of color images into a sequence of two-image streams can reduce the curse of dimensionality using a deep learning approach. The generic deep learning architecture has been evaluated with different datasets outperforming the state-of-the-art approaches providing an efficient architecture for single and multi-class classification problems.

Robust object tracking using sparse color appearance model

Distinguishing the target from the background, judging target occlusion, and real-time processing are the problems that the visual tracking algorithm still needs to solve. Color information and position information of the target block are fused as new features to track the target under the framework of particle filtering. First, the hues, saturation, value space, and color integral graph of the image are constructed. The vector representation of the target is obtained on the color integral image by sparse matrix. Then, candidate particles are produced by a particle filter and the sampling mode of particles is adjusted by a uniform acceleration model. The difference of particles reflects the position and scale change of the target. Finally, the candidate with the smallest eigenvector projection error is taken as the tracking target and the feature template is updated based on the tracking results. The presented algorithm can be used to track a single target in the color image sequence and has some robustness to the scale change, occlusion, and morphological change of the target. Experiment results on public datasets show that the proposed algorithm performs favorably in both speed and tracking effect when compared with other conventional trackers.

Impulse noise removal in color image sequences using fuzzy logic

This paper presents a two-step fuzzy filter to remove impulse noise from a color image sequence in RGB color space. The primary filter recognizes the pixels corrupted by impulse noise. It also computes the extent of noise and afterward rectifies them. The output of the first filter acts as an input to the secondary filter and further refines the outcome to give the final output. Excellent alignment is seen between noise removal and structure conservation of an image due to the classifying nature of the algorithm. To minimize blurring, noisy pixels are exclusively rectified and noise-free pixels remain intact. The proposed filter is a 3-D Spatio-temporal filter that considers spatial, temporal as well as color information. A pixel of one color component is compared to its neighboring pixels within the same frame and with the corresponding pixels in neighboring frames. It is likewise compared with the pixels of the other two color components. Peak Signal to Noise Ratio (PSNR) and structural similarity index (SSIM), Miss Detection (MD), and False Alarm (FA) rates are utilized as a performance metric. The experimental result of several color image sequences demonstrates the efficacy of the proposed fuzzy filter both qualitatively and quantitatively.

Occlusion-aware Visual Tracker using Spatial Structural Information and Dominant Features

To overcome the problem of occlusion in visual tracking, this paper proposes an occlusion-aware tracking algorithm. The proposed algorithm divides the object into discrete image patches according to the pixel distribution of the object by means of clustering. To avoid the drifting of the tracker to false targets, the proposed algorithm extracts the dominant features, such as color histogram or histogram of oriented gradient orientation, from these image patches, and uses them as cues for tracking. To enhance the robustness of the tracker, the proposed algorithm employs an implicit spatial structure between these patches as another cue for tracking; Afterwards, the proposed algorithm incorporates these components into the particle filter framework, which results in a robust and precise tracker. Experimental results on color image sequences with different resolutions show that the proposed tracker outperforms the comparison algorithms on handling occlusion in visual tracking

Observing the Ocean Submesoscale with Enhanced-Color GOES-ABI Visible Band Data.

Ocean color remote sensing has long been utilized as a fundamental research tool in the oceanographic investigations of coupled biological-physical processes. Despite numerous technical advances in the application of space borne ocean-viewing radiometers, host satellite platforms in a polar-orbiting configuration often render the temporal frequency of sensor data acquisition insufficient for studies of ocean processes that occur within increasingly smaller space-time scales. Whereas geostationary ocean color missions are presently the exception (GOCI) rather than the rule, this paper presents a method to convolve ocean reflectance data obtained from contemporary ocean-viewing multispectral radiometers (VIIRS, OLCI) with spectrally-limited Advanced Baseline Imager (ABI) data obtained from the GOES-R meteorological satellites. The method, Chromatic Domain Mapping (CDM), employs a colorimetry approach to visible range ocean reflectance data. The true color space is used as a frame-of-reference that is mapped by the dedicated yet temporally sparse ocean color sensors; coincident and spectrally coarse information from ABI is then used to estimate the evolution of the true color scene. The procedure results in very high resolution (~5 min) true color image sequences. Herein, example CDM applications of rapid frontal boundary evolution and feature displacement in the Gulf of Mexico are presented and future applications of this technique are discussed.

Molecular Orientation Change Nearby Topological Defects Observed by Photo-Induced Polarization/Phase Microscopy.

Topological defects in liquid crystals (LCs) have been intensively studied and intentionally generated in an organized way recently because they could control the alignment and motion of LCs. We studied how the topological defects could change the molecular orientation/alignment from the observation of photo-induced orientation change of a photo-responsive LC. The photo-induced dynamics was observed by an LED-induced time-resolved polarization/phase microscopy with white light illumination. From the color image sequence, we found that the molecular orientation change started from the topological defects and the orientation change propagated as a pair of defects and was connected, and further disordering was induced as a next step after the initial orientation change finished.

Vision-Based Hazard Detection with End-to-end Spatiotemporal Networks for Planetary Landing

Hazard detection is the key technique for autonomous landing during a planetary exploration mission. This paper proposed an end-to-end spatiotemporal network to detect all hazards in the image sequences captured by optical camera. The spatial stream processes the colour image sequences and the temporal one learns features from optical flow image sequences. The spatial features and temporal features are fused by the proposed metric fusion method, and then the spatiotemporal features become more discriminative through triplet loss. A hazard map can ultimately be obtained in the testing phase after processing on the full-size image sequences. The evaluation results prove the effectiveness of the proposed network, and the testing results show the feasibility of practical application.

Dynamic multiple-image encryption based on chirp z-transform

A dynamic encryption scheme is demonstrated based on chirp z-transform (CZT). The novel design of this setup allows us to protect multiple images in a multiplexed format and fully recover them without the influence of crosstalk and sample noise. The encoding stage is implemented using the 2D CZT and a Fourier lens. The CZT works with proper frequency parameters to control the position and size of the input frequency spectrum. The accurate positioning leads to an efficient packing of the multiplexed images and a remarkable improvement of the signal-to-noise ratio of the decrypted pictures. The introduction of the CZT in the field of encryption is significant, since it can expand the functionality to a dynamic regime of optical and digital architectures for static encoding. We validated the feasibility of our proposal with the encryption and decryption of a color image sequence and a set of binary images.

Motion Detection and Clustering Using PCA and NN in Color Image Sequence

This paper presents a motion detection method with the use of the Principal Component Analysis. This method is able to detect and track moving objects in a sequence of images. The tested sequence is segmented within the meaning of movement. In this paper, the concept of extracting significant information from a large number of data is adopted to provide an effective method for tracking moving objects on the video image. The principal components are different in term of getting significant information, the nature of motion (the nature of information) is responsible of this difference, the algorithm in this paper distinguish the motion nature and choose the appropriate components to give a best segmentation.

Dance motion capture and composition using multiple RGB and depth sensors

Dynamic human movements such as dance are difficult to capture without using external markers due to the high complexity of a dancer’s body. This article introduces a marker-free motion capture and composition system for dance motion that uses multiple RGB and depth sensors. Our motion capture system utilizes a set of high-speed RGB and depth sensors to generate skeletal motion data from an expert dancer. During the motion acquisition process, a skeleton tracking method based on a particle filter is provided to estimate the motion parameters for each frame from a sequence of color images and depth features retrieved from the sensors. The expert motion data become archived in a database. The authoring methods in our composition system automate most of the motion editing processes for general users by providing an online motion search with an input posture and then performing motion synthesis on an arbitrary motion path. Using the proposed system, we demonstrate that various dance performances can be composed in an intuitive and efficient way on client devices such as tablets and kiosk PCs.

A spatiotemporal restoration of partial color artifacts in archival films

With the rapid growth in video technology and marketability of all forms of visual media, we encounter the need to exploit archive material more effectively and guarantee picture quality. Old films and video often get corrupted by many artifacts due to extensive usage or improper storage. These defects not only lead to reduction in perceptual quality, but also require large transmission bandwidth. In this paper, we present a novel method to restore color image sequences corrupted by partial color artifact. This degradation visible in old color film/video appear as random color patches in the frames. It reduces the perceptual quality of the video. The proposed approach minimizes a convex function, formulated as rank minimization problem, seeking the temporal correlation among the frames in a video sequence. The main feature of the proposed algorithm is its accurate detection and fast restoration of the artifacts present in a video shot.

Multifocus color image sequence fusion based on mean shift segmentation.

This paper presents a region-based technique for fusion of a multifocus color image sequence in the LUV color space. First, mean shift segmentation was applied on the weighted average image of the image sequence to obtain the fusion reference areas. Second, for each segmented area, the well-known modified Laplacian (LAP2) was used as a focus measure to select the clearest parts within the image sequence and then a final image focused with all parts can be generated. Mutual information, QAB/F metric, entropy, standard deviation, image sharpness metric, image contrast metric, average gradient, and spatial frequency were adopted to assess the quality of the fused image. Experiments carried out using standard image sequences from HeliconSoft demonstrated that the results obtained through our technique offer good performance. The proposed technique can be used to extend the depth of field (DOF) of a camera system effectively.

Enhancing Color Images of Extremely Low Light Scenes Based on RGB/NIR Images Acquisition With Different Exposure Times.

We propose a novel method to synthesize a noise- and blur-free color image sequence using near-infrared (NIR) images captured in extremely low light conditions. In extremely low light scenes, heavy noise and motion blur are simultaneously produced in the captured images. Our goal is to enhance the color image sequence of an extremely low light scene. In this paper, we augment the imaging system as well as enhancing the image synthesis scheme. We propose a novel imaging system that can simultaneously capture the red, green, blue (RGB) and the NIR images with different exposure times. An RGB image is taken with a long exposure time to acquire sufficient color information and mitigates the effects of heavy noise. By contrast, the NIR images are captured with a short exposure time to measure the structure of the scenes. Our imaging system using different exposure times allows us to ensure sufficient information to reconstruct a clear color image sequence. Using the captured image pairs, we reconstruct a latent color image sequence using an adaptive smoothness condition based on gradient and color correlations. Our experiments using both synthetic images and real image sequences show that our method outperforms other state-of-the-art methods.

Satellite Image Denoising Using Local Spayed and Optimized Center Pixel Weights

Now a day’s digital image processing applications are widely used in various fields such as medical, military, satellite, remote sensing and even web applications also. In any application image denoising is a challenging task because noise removal will increase the digital quality of an image and will improve the perceptual visual quality. In this paper we proposed a new method “local spayed and optimized center pixel weights (LSOCPW) with non local means” to improve the denoising performance of digital color image sequences. Simulation results show that the proposed method has given the better performance when compared to the existing algorithms in terms of peak signal to noise ratio (PSNR) and mean square error (MSE). DOI: http://dx.doi.org/10.11591/ijece.v4i5.6624

A tensor-based deep learning framework

This paper presents an unsupervised deep learning framework that derives spatio-temporal features for human–robot interaction. The respective models extract high-level features from low-level ones through a hierarchical network, viz. the Hierarchical Temporal Memory (HTM), providing at the same time a solution to the curse of dimensionality in shallow techniques. The presented work incorporates the tensor-based framework within the operation of the nodes and, thus, enhances the feature derivation procedure. This is due to the fact that tensors allow the preservation of the initial data format and their respective correlation and, moreover, attain more compact representations. The computational nodes form spatial and temporal groups by exploiting the multilinear algebra and subsequently express the samples according to those groups in terms of proximity. This generic framework may be applied in a diverse of visual data, while it has been examined on sequences of color and depth images, exhibiting remarkable performance.

Study on Moving Object Detection Based on RGB Color Model

The traditional background difference method is based on gray image. Some information is lost when color image is transformed into gray image. So it is difficult to discriminate different colors with similar gray values and easily disturbed by noise and shadows. In this paper, the background difference is based on RGB color model. It is proposed to use the average value of each pixel of the color image sequences to extract the background, and then use the three-dimensional color values of the current frame and background image to compute the difference to detect the moving objects. The proposed approach is simple and easy to implement. The experimental results show that it is more sensitive to colors and has higher accuracy and robustness than the traditional background difference method. Besides, it is more resistant to shadows.