Input Image Sequence Research Articles

Multimodal Social Media Fake News Detection Based on 1D-CCNet Attention Mechanism

Due to the explosive rise of multimodal content in online social communities, cross-modal learning is crucial for accurate fake news detection. However, current multimodal fake news detection techniques face challenges in extracting features from multiple modalities and fusing cross-modal information, failing to fully exploit the correlations and complementarities between different modalities. To address these issues, this paper proposes a fake news detection model based on a one-dimensional CCNet (1D-CCNet) attention mechanism, named BTCM. This method first utilizes BERT and BLIP-2 encoders to extract text and image features. Then, it employs the proposed 1D-CCNet attention mechanism module to process the input text and image sequences, enhancing the important aspects of the bimodal features. Meanwhile, this paper uses the pre-trained BLIP-2 model for object detection in images, generating image descriptions and augmenting text data to enhance the dataset. This operation aims to further strengthen the correlations between different modalities. Finally, this paper proposes a heterogeneous cross-feature fusion method (HCFFM) to integrate image and text features. Comparative experiments were conducted on three public datasets: Twitter, Weibo, and Gossipcop. The results show that the proposed model achieved excellent performance.

A CNN-LSTM model using elliptical constraints for temporally consistent sun position estimation

More accurate sun position estimation could transform the design and operation of solar power systems, weather forecasting services, and outdoor augmented reality systems. Although several image-based approaches to sun position estimation have been proposed, their performance is significantly affected by momentary disruptions in cloud cover because they use only a single image as input. This study proposes a deep learning-based sun position estimation system that leverages spatial, temporal, and geometric features to accurately regress sun positions even when the sun is partially or entirely occluded. In the proposed approach, spatial features are extracted from an input image sequence by applying a separate Resnet-based convolution network to each frame. To ensure that the temporal changes in the brightness distribution across frames are considered, the spatial features are concatenated and passed on to a stack of LSTM layers prior to regressing the final sun position. The proposed network is also trained with elliptical (geometric) constraints to ensure that predicted sun positions are consistent with the natural elliptical path of the sun in the sky. The proposed approach's performance was evaluated on the Sirta and Laval datasets along with a custom dataset, and an R2 Score of 0.98 was achieved, which is at least 0.1 higher than that of previous approaches. The proposed approach is capable of identifying the position of the sun even when occluded and was employed in a novel sky imaging system consisting of only a camera and fisheye lens in place of a complex array of sensors.

Large-Scale Road Network Traffic Congestion Prediction Based on Recurrent High-Resolution Network

Traffic congestion is a significant problem that adversely affects the economy, environment, and public health in urban areas worldwide. One promising solution is to forecast road-level congestion levels in the short-term and long-term, enabling commuters to avoid congested areas and allowing traffic agencies to take appropriate action. In this study, we propose a hybrid deep neural network algorithm based on High-Resolution Network (HRNet) and ConvLSTM decoder for 10, 30, and 60-min traffic congestion prediction. Our model utilizes the HRNet’s multi-scale feature extraction capability to capture rich spatial features from a sequence of past traffic input images. The ConvLSTM module learns temporal information from each HRNet multi-scale output and aggregates all feature maps to generate accurate traffic forecasts. Our experiments demonstrate that the proposed model can efficiently and effectively learn both spatial and temporal relationships for traffic congestion and outperforms four other state-of-the-art architectures (PredNet, UNet, ConvLSTM, and Autoencoder) in terms of accuracy, precision, and recall. A case study was conducted on the dataset from Seoul, South Korea.

Drought stress prediction and propagation using time series modeling on multimodal plant image sequences.

The paper introduces two novel algorithms for predicting and propagating drought stress in plants using image sequences captured by cameras in two modalities, i.e., visible light and hyperspectral. The first algorithm, VisStressPredict, computes a time series of holistic phenotypes, e.g., height, biomass, and size, by analyzing image sequences captured by a visible light camera at discrete time intervals and then adapts dynamic time warping (DTW), a technique for measuring similarity between temporal sequences for dynamic phenotypic analysis, to predict the onset of drought stress. The second algorithm, HyperStressPropagateNet, leverages a deep neural network for temporal stress propagation using hyperspectral imagery. It uses a convolutional neural network to classify the reflectance spectra at individual pixels as either stressed or unstressed to determine the temporal propagation of stress in the plant. A very high correlation between the soil water content, and the percentage of the plant under stress as computed by HyperStressPropagateNet on a given day demonstrates its efficacy. Although VisStressPredict and HyperStressPropagateNet fundamentally differ in their goals and hence in the input image sequences and underlying approaches, the onset of stress as predicted by stress factor curves computed by VisStressPredict correlates extremely well with the day of appearance of stress pixels in the plants as computed by HyperStressPropagateNet. The two algorithms are evaluated on a dataset of image sequences of cotton plants captured in a high throughput plant phenotyping platform. The algorithms may be generalized to any plant species to study the effect of abiotic stresses on sustainable agriculture practices.

Multi-exposure electric power monitoring image fusion method without ghosting based on exposure fusion framework and color dissimilarity feature.

To solve the ghosting artifacts problem in dynamic scene multi-scale exposure fusion, an improved multi-exposure fusion method has been proposed without ghosting based on the exposure fusion framework and the color dissimilarity feature of this study. This fusion method can be further applied to power system monitoring and unmanned aerial vehicle monitoring. In this study, first, an improved exposure fusion framework based on the camera response model was applied to preprocess the input image sequence. Second, the initial weight map was estimated by multiplying four weight items. In removing the ghosting weight term, an improved color dissimilarity feature was used to detect the object motion features in dynamic scenes. Finally, the improved pyramid model as adopted to retain detailed information about the poor exposure areas. Experimental results indicated that the proposed method improves the performance of images in terms of sharpness, detail processing, and ghosting artifacts removal and is superior to the five existing multi-exposure image fusion (MEF) methods in quality evaluation.

3D Convolutional Neural Network for Low-Light Image Sequence Enhancement in SLAM

Typical visual simultaneous localization and mapping (SLAM) systems rely on front-end odometry for feature extraction and matching to establish the relations between adjacent images. In a low-light environment, the image obtained by a camera is dim and shows scarce information, hindering the extraction of sufficient stable feature points, consequently undermining visual SLAM. Most existing methods focus on low-light enhancement of a single image, neglecting the strong temporal correlation across images in visual SLAM. We propose a method that leverages the temporal information of an input image sequence to enhance the low-light image and employed the enhanced result to improve the feature extraction and matching quality of visual SLAM. Our method trains a three-dimensional convolutional neural network to estimate pixelwise grayscale transformation curves to obtain a low-light enhancement image. Then, the grayscale transformation curves are iteratively applied to obtain the final enhanced result. The training process of the network does not require any paired reference images. We also introduced a spatial consistency loss for the enhanced image to retain the content and texture of the original image. We further integrated our method into VINS-Mono and compared with similar low-light image enhancement methods on the TUM-VI public dataset. The proposed method provides a lower positioning error. The positioning root-mean-squared error of our method is 19.83% lower than that of Zero-DCE++ in low-light environments. Moreover, the proposed network achieves real-time operation, being suitable for integration into a SLAM system.

Multi-Exposure Image Fusion Based on Weighted Average Adaptive Factor and Local Detail Enhancement

In order to adapt to the local brightness and contrast of input image sequences, we propose a new weighted average adaptive factor well-exposure weight evaluation scheme. The exposure weights of brighter and darker pixels are determined according to the local average brightness and expected brightness. We find that in the traditional multi-exposure image fusion scheme, the brighter and darker regions of the scene lose many details. To solve this problem, we first propose a standard to determine the brighter and darker regions and then use a fast local Laplacian filter to enhance the image in the region. This paper selects 16 multi-exposure images of different scenes for subjective and objective analysis and compares them with eight existing multi-exposure fusion schemes. The experimental results show that the proposed method can enhance the details appropriately while preserving the details in static scenes and adapting to the input image brightness.

Dynamic mode decomposition via convolutional autoencoders for dynamics modeling in videos

Extracting the underlying dynamics of objects in image sequences is one of the challenging problems in computer vision. Besides, dynamic mode decomposition (DMD) has recently attracted attention as a method for obtaining modal representations of nonlinear dynamics from general multivariate time-series data without explicit prior information about the dynamics. In this paper, we propose a convolutional autoencoder (CAE)-based DMD (CAE-DMD) to perform accurate modeling of underlying dynamics in videos. We develop a modified CAE model that encodes images to latent vectors and incorporated DMD on the latent vectors to extract DMD modes. These modes are split into background and foreground modes for foreground modeling in videos, or used for video classification tasks. And the latent vectors are mapped so as to recover the input image sequences through a decoder. We perform the network training in an end-to-end manner, i.e., by minimizing the mean square error between the original and reconstructed images. As a result, we obtain accurate extraction of underlying dynamic information in the videos. We empirically investigate the performance of CAE-DMD in two applications background foreground extraction and video classification on synthetic and publicly available datasets.

Tracking Axonal Transports in Time-Lapse Images Obtained from a Microfluidic Culture Platform

In this paper, a procedure is described for tracking moving object trajectories from image sequences acquired from a microfluidic culture platform. Since particles move along the axons, curve structures need to be detected first from the input image sequence. A kymograph analysis technique is applied to detect axon structures from the consolidated image of the input sequence. Horizontally and vertically oriented axons are then detected by applying the process twice to the original and the 90-degree rotated image. Multiple kymographs are generated along the detected axons by projecting image intensity variation through the time-axis. The trajectory detection process is then applied to each kymograph image. To obtain the particle motion information from the entire image sequence, an integration process is applied to each horizontal and vertical kymograph data set. The proposed technique has been applied to image sequences in the present application area. It is demonstrated that practical results can be obtained using time-lapse image sequence data.

Learning to Discriminate Information for Online Action Detection: Analysis and Application.

Online action detection, which aims to identify an ongoing action from a streaming video, is an important subject in real-world applications. For this task, previous methods use recurrent neural networks for modeling temporal relations in an input sequence. However, these methods overlook the fact that the input image sequence includes not only the action of interest but background and irrelevant actions. This would induce recurrent units to accumulate unnecessary information for encoding features on the action of interest. To overcome this problem, we propose a novel recurrent unit, named Information Discrimination Unit (IDU), which explicitly discriminates the information relevancy between an ongoing action and others to decide whether to accumulate the input information. This enables learning more discriminative representations for identifying an ongoing action. In this paper, we further present a new recurrent unit, called Information Integration Unit (IIU), for action anticipation. Our IIU exploits the outputs from IDN as pseudo action labels as well as RGB frames to learn enriched features of observed actions effectively. In experiments on TVSeries and THUMOS-14, the proposed methods outperform state-of-the-art methods by a significant margin in online action detection and action anticipation. Moreover, we demonstrate the effectiveness of the proposed units by conducting comprehensive ablation studies.

Automatic segmentation of uterine endometrial cancer on multi-sequence MRI using a convolutional neural network

Endometrial cancer (EC) is the most common gynecological tumor in developed countries, and preoperative risk stratification is essential for personalized medicine. There have been several radiomics studies for noninvasive risk stratification of EC using MRI. Although tumor segmentation is usually necessary for these studies, manual segmentation is not only labor-intensive but may also be subjective. Therefore, our study aimed to perform the automatic segmentation of EC on MRI with a convolutional neural network. The effect of the input image sequence and batch size on the segmentation performance was also investigated. Of 200 patients with EC, 180 patients were used for training the modified U-net model; 20 patients for testing the segmentation performance and the robustness of automatically extracted radiomics features. Using multi-sequence images and larger batch size was effective for improving segmentation accuracy. The mean Dice similarity coefficient, sensitivity, and positive predictive value of our model for the test set were 0.806, 0.816, and 0.834, respectively. The robustness of automatically extracted first-order and shape-based features was high (median ICC = 0.86 and 0.96, respectively). Other high-order features presented moderate-high robustness (median ICC = 0.57–0.93). Our model could automatically segment EC on MRI and extract radiomics features with high reliability.

Robust Focus Volume Regularization in Shape From Focus.

Shape from focus (SFF) reconstructs 3D shape of the scene from a sequence of multi-focus images, and the quality of reconstructed shape mainly depends on the accuracy of image focus volume (FV). Traditional SFF techniques exhibit poor performance in preserving structural edges and fine details while removing noisy artifacts, and mostly they do not incorporate any additional shape prior. Therefore, in this paper, we propose to refine FV by formulating an energy minimization framework that employs a nonconvex regularizer and incorporates two types of shape priors. The proposed regularizer is robust against noisy focus values. The first proposed shape prior is input image sequence and it is a single and static shape prior. While, the second shape prior corresponds to a series of shape priors. These shape priors are FVs which are iteratively obtained on-the-fly. Both of these shape priors constrain the solution space for output FV. We optimize nonconvex energy function through majorize-minimization algorithm which iteratively guarantees a local minimum and converges quickly. Experiments have been conducted to evaluate accuracy and convergence properties of the proposed method. Experimental results of synthetic and real image sequences demonstrate that our method achieves superior results in terms of ability to reconstruct accurate 3D shapes as compared to existing approaches.

L, r-Stitch Unit: Encoder-Decoder-CNN Based Image-Mosaicing Mechanism for Stitching Non-Homogeneous Image Sequences

Image-stitching (or) mosaicing is considered an active research-topic with numerous use-cases in computer-vision, AR/VR, computer-graphics domains, but maintaining homogeneity among the input image sequences during the stitching/mosaicing process is considered as a primary-limitation & major-disadvantage. To tackle these limitations, this article has introduced a robust and reliable image stitching methodology (l,r-Stitch Unit), which considers multiple non-homogeneous image sequences as input to generate a reliable panoramically stitched wide view as the final output. The l,r-Stitch Unit further consists of a pre-processing, post-processing sub-modules & a l,r-PanoED-network, where each sub-module is a robust ensemble of several deep-learning, computer-vision & image-handling techniques. This article has also introduced a novel convolutional-encoder-decoder deep-neural-network (l,r-PanoED-network) with a unique split-encoding-network methodology, to stitch non-coherent input left, right stereo image pairs. The encoder-network of the proposed l,r-PanoED extracts semantically rich deep-feature-maps from the input to stitch/map them into a wide-panoramic domain, the feature-extraction & feature-mapping operations are performed simultaneously in the l,r-PanoED's encoder-network based on the split-encoding-network methodology. The decoder-network of l,r-PanoED adaptively reconstructs the output panoramic-view from the encoder networks' bottle-neck feature-maps. The proposed l,r-Stitch Unit has been rigorously benchmarked with alternative image-stitching methodologies on our custom-built traffic dataset and several other public-datasets. Multiple evaluation metrics (SSIM, PSNR, MSE, L <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">α,β,γ</sub> , FM-rate, Average-latency-time) & wild-Conditions (rotational/color/intensity variances, noise, etc) were considered during the benchmarking analysis, and based on the results, our proposed method has outperformed among other image-stitching methodologies and has proved to be effective even in wild non-homogeneous inputs.

Estimating Local Cellular Density in Glioma Using MR Imaging Data.

Increased cellular density is a hallmark of gliomas, both in the bulk of the tumor and in areas of tumor infiltration into surrounding brain. Altered cellular density causes altered imaging findings, but the degree to which cellular density can be quantitatively estimated from imaging is unknown. The purpose of this study was to discover the best MR imaging and processing techniques to make quantitative and spatially specific estimates of cellular density. We collected stereotactic biopsies in a prospective imaging clinical trial targeting untreated patients with gliomas at our institution undergoing their first resection. The data included preoperative MR imaging with conventional anatomic, diffusion, perfusion, and permeability sequences and quantitative histopathology on biopsy samples. We then used multiple machine learning methodologies to estimate cellular density using local intensity information from the MR images and quantitative cellular density measurements at the biopsy coordinates as the criterion standard. The random forest methodology estimated cellular density with R 2 = 0.59 between predicted and observed values using 4 input imaging sequences chosen from our full set of imaging data (T2, fractional anisotropy, CBF, and area under the curve from permeability imaging). Limiting input to conventional MR images (T1 pre- and postcontrast, T2, and FLAIR) yielded slightly degraded performance (R2 = 0.52). Outputs were also reported as graphic maps. Cellular density can be estimated with moderate-to-strong correlations using MR imaging inputs. The random forest machine learning model provided the best estimates. These spatially specific estimates of cellular density will likely be useful in guiding both diagnosis and treatment.

Simple Shading Correction Method for Brightfield Whole Slide Imaging.

Whole slide imaging (WSI) refers to the process of creating a high-resolution digital image of a whole slide. Since digital images are typically produced by stitching image sequences acquired from different fields of view, the visual quality of the images can be degraded owing to shading distortion, which produces black plaid patterns on the images. A shading correction method for brightfield WSI is presented, which is simple but robust not only against typical image artifacts caused by specks of dust and bubbles, but also against fixed-pattern noise, or spatial variations in pixel values under uniform illumination. The proposed method comprises primarily of two steps. The first step constructs candidates of a shading distortion model from a stack of input image sequences. The second step selects the optimal model from the candidates. The proposed method was compared experimentally with two previous state-of-the-art methods, regularized energy minimization (CIDRE) and background and shading correction (BaSiC) and showed better correction scores, as smooth operations and constraints were not imposed when estimating the shading distortion. The correction scores, averaged over 40 image collections, were as follows: proposed method, 0.39 ± 0.099; CIDRE method, 0.67 ± 0.047; BaSiC method, 0.55 ± 0.038. Based on the quantitative evaluations, we can confirm that the proposed method can correct not only shading distortion, but also fixed-pattern noise, compared with the two previous state-of-the-art methods.

Facial landmark detection and tracking with dynamically adaptive matched filters

A reliable method for the detection and tracking of facial landmarks in image sequences is presented. Given a set of prespecified facial landmarks in a reference face image, a bank of composite matched filters is constructed for reliable detection and accurate location of the landmarks in an input image sequence. The filter bank is dynamically adapted to each captured frame by learning from current and past landmark detections and considering geometrical modifications of the landmarks. The detected landmarks are accurately tracked using a kinematic motion model that predicts their coordinates in future frames. The performance of facial landmark detection and tracking obtained with the proposed method is tested by processing real-life image sequences. The obtained results are analyzed and discussed in terms of objective measures.

Adaptive embedding gate for attention-based scene text recognition

Scene text recognition has attracted particular research interest because it is a very challenging problem and has various applications. The most cutting-edge methods are attentional encoder-decoder frameworks that learn the alignment between the input image and output sequences. In particular, the decoder recurrently outputs predictions, using the prediction of the previous step as a guidance for every time step. In this study, we point out that the inappropriate use of previous predictions in existing attentional decoders restricts the recognition performance and brings instability. To handle this problem, we propose a novel module, namely adaptive embedding gate (AEG). The proposed AEG focuses on introducing high-order character language models to attentional decoders by controlling the information transmission between adjacent characters. AEG is a flexible module and can be easily integrated into the state-of-the-art attentional decoders for scene text recognition. We evaluate its effectiveness as well as robustness on a number of standard benchmarks, including the IIIT5K, SVT, SVT-P, CUTE80, and ICDAR datasets. Experimental results demonstrate that AEG can significantly boost recognition performance and bring better robustness.

ScalarFlow

In this paper, we present ScalarFlow , a first large-scale data set of reconstructions of real-world smoke plumes. In addition, we propose a framework for accurate physics-based reconstructions from a small number of video streams. Central components of our framework are a novel estimation of unseen inflow regions and an efficient optimization scheme constrained by a simulation to capture real-world fluids. Our data set includes a large number of complex natural buoyancy-driven flows. The flows transition to turbulence and contain observable scalar transport processes. As such, the ScalarFlow data set is tailored towards computer graphics, vision, and learning applications. The published data set contains volumetric reconstructions of velocity and density as well as the corresponding input image sequences with calibration data, code, and instructions how to reproduce the commodity hardware capture setup. We further demonstrate one of the many potential applications: a first perceptual evaluation study, which reveals that the complexity of the reconstructed flows would require large simulation resolutions for regular solvers in order to recreate at least parts of the natural complexity contained in the captured data.

Reward-driven U-Net training for obstacle avoidance drone

Along with the fast progress in deep learning, an autonomous drone with obstacle avoidance capability has been studied mainly by two machine learning paradigms, i.e. supervised learning, and reinforcement learning. The former has some advantages since the trained network is light and fast, but it needs a large amount of data that requires laborious manual labeling. With the latter, such a drawback can be overcome as an agent learns by itself in a simulated environment, although the gap between the real and simulated one has to be minimized in the end. This study proposes a new framework where a supervised segmentation network is trained with labels made by an actor-critic network in a reward-driven manner, wherein this U-Net based network infers the next moving direction from the sequence of input images. For the actor-critic part, several recent policy gradient algorithms have been tested for controlling the drone with the continuous action space. After training in the Airsim simulation environment, the model is transferred to a Bebop drone flying in the real environment, built as a reconfigurable maze using panels and a hoop. The result suggests that our network enables the drone to navigate through the obstacles using only monocular RGB input in the trained environment as well as in the reconfigured ones without retraining.

Shape from focus based on 3D structure tensor using optical microscopy.

Shape from focus (SFF) is a widely used passive optical method for 3D shape reconstruction. In SFF, a focus measure, which is used to estimate the relative focus level, plays a critical role in depth estimation. In this article, we present a new focus measure for accurate 3D shape estimation in optical microscopy based on the analysis of 3D structure tensor. First, the 3D tensors are computed from the input image sequence for each pixel. Then, each tensor is decomposed into point, curve, and surface tensors by decomposing tensors into eigenvalues and eigenvectors. Finally, the surfaceness is used to measure the quality of sharpness. The proposed focus measure provides accurate focus values and better resistance against noise. The proposed measure is evaluated by conducting experiments using image sequences of simulated and microscopic real objects. The comparative analysis demonstrates the effectiveness of the proposed focus measure in recovering 3D shape.