Scene Change Detection Research Articles

Damage Scene Change Detection Based on Infrared Polarization Imaging and Fast-PCANet

Change detection based on optical image processing plays a crucial role in the field of damage assessment. Although existing damage scene change detection methods have achieved some good results, they are faced with challenges, such as low accuracy and slow speed in optical image change detection. To solve these problems, an image change detection approach that combines infrared polarization imaging with a fast principal component analysis network (Fast-PCANet) is proposed. Firstly, the acquired infrared polarization images are analyzed, and pixel image blocks are extracted and filtered to obtain the candidate change points. Then, the Fast-PCANet network framework is established, and the candidate pixel image blocks are sent to the network to detect the change pixel points. Finally, the false-detection points predicted by the Fast-PCANet are further corrected by region filling and filtering to obtain the final binary change map of the damage scene. Comparisons with typical PCANet-based change detection algorithms are made on a dataset of infrared-polarized images. The experimental results show that the proposed Fast-PCANet method improves the PCC and the Kappa coefficient of infrared polarization images over infrared intensity images by 6.77% and 13.67%, respectively. Meanwhile, the inference speed can be more than seven times faster. The results verify that the proposed approach is effective and efficient for the change detection task with infrared polarization imaging. The study can be applied to the damage assessment field and has great potential for object recognition, material classification, and polarization remote sensing.

SpectralGPT: Spectral Remote Sensing Foundation Model.

The foundation model has recently garnered significant attention due to its potential to revolutionize the field of visual representation learning in a self-supervised manner. While most foundation models are tailored to effectively process RGB images for various visual tasks, there is a noticeable gap in research focused on spectral data, which offers valuable information for scene understanding, especially in remote sensing (RS) applications. To fill this gap, we created for the first time a universal RS foundation model, named SpectralGPT, which is purpose-built to handle spectral RS images using a novel 3D generative pretrained transformer (GPT). Compared to existing foundation models, SpectralGPT 1) accommodates input images with varying sizes, resolutions, time series, and regions in a progressive training fashion, enabling full utilization of extensive RS Big Data; 2) leverages 3D token generation for spatial-spectral coupling; 3) captures spectrally sequential patterns via multi-target reconstruction; and 4) trains on one million spectral RS images, yielding models with over 600 million parameters. Our evaluation highlights significant performance improvements with pretrained SpectralGPT models, signifying substantial potential in advancing spectral RS Big Data applications within the field of geoscience across four downstream tasks: single/multi-label scene classification, semantic segmentation, and change detection.

Co-scale cross-attentional transformer for rearrangement target detection

Rearranging objects (e.g. vase, door) back in their original positions is one of the most fundamental skills for domestic service robots (DSRs). In rearrangement tasks, it is crucial to detect the objects that need to be rearranged according to the goal and current states. In this study, we focus on Rearrangement Target Detection (RTD), where the model generates a change mask for objects that should be rearranged. Although many studies have been conducted in the field of Scene Change Detection (SCD), most SCD methods often fail to segment objects with complex shapes and fail to detect the change in the angle of objects that can be opened or closed. In this study, we propose a Co-Scale Cross-Attentional Transformer for RTD. We introduce the Serial Encoder which consists of a sequence of serial blocks and the Cross-Attentional Encoder which models the relationship between the goal and current states. We built a new dataset consisting of RGB images and change masks regarding the goal and current states. We validated our method on the dataset and the results demonstrated that our method outperformed baseline methods on F 1 -score and mean IoU.

RICE: A Dataset and Baseline for Cloud Removal in Remote Sensing Images

Removing clouds is an essential preprocessing step in analyzing remote sensing images, as cloud-based overlays commonly occur in optical remote sensing images and can significantly limit the usability of the acquired data. Deep learning has exhibited remarkable progress in remote sensing, encompassing scene classification and change detection tasks. Nevertheless, the appli-cation of deep learning techniques to cloud removal in remote sensing images is currently con-strained by the limited availability of training datasets explicitly tailored for neural networks. This paper presents the Remote sensing Image Cloud rEmoving dataset (RICE) to address this challenge and proposes baseline models incorporating a convolutional attention mechanism, which has demonstrated superior performance in identifying and restoring cloud-affected regions, with quantitative results indicating a 3.08% improvement in accuracy over traditional methods. This mechanism empowers the network to comprehend better the spatial structure, local details, and inter-channel correlations within remote sensing images, thus effectively addressing the diverse distributions of clouds. Moreover, by integrating this attention mechanism, our models achieve a crucial comparison advantage, outperforming existing state-of-the-art techniques in terms of both visual quality and quantitative metrics. We propose adopting the Learned Per-ceptual Image Patch Similarity metric, which emphasizes perceptual similarity, to evaluate the quality of cloud-free images generated by the models. Our work not only contributes to advancing cloud removal techniques in remote sensing but also provides a comprehensive evaluation framework for assessing the fidelity of the generated images.

AstrobeeCD: Change detection in microgravity with free-flying robots

This work presents AstrobeeCD, a system for 3D scene change detection toward near-real-time environmental awareness of space outposts using the Astrobee free-flying robot in microgravity. Assistive free-flyer robots autonomously caring for future crewed space habitats must be able to detect day-to-day interior changes to track inventory, detect and diagnose faults, and monitor the outpost status. A set of image and depth data from one time step is used to reconstruct a 3D model of the environment. The 3D model is used as the basis for comparison for free-flyer environment surveys at future time steps, where an image-based change detection algorithm identifies inconsistencies against the 3D model. Change detection is demonstrated using real image and pose data collected by an Astrobee robot in a test environment on Earth at NASA Ames Research Center and from microgravity aboard the International Space Station. Change detection computation time and performance are quantitatively evaluated on the test data captured on Earth, and it identifies scene changes more quickly than a point cloud clustering-based algorithm applied to data from the same surveys.

Edge-Guided Hierarchical Network for Building Change Detection in Remote Sensing Images

Building change detection monitors building changes by comparing and analyzing multi-temporal images acquired from the same area and plays an important role in land resource planning, smart city construction and natural disaster assessment. Different from change detection in conventional scenes, buildings in the building change detection task usually appear in a densely distributed state, which is easy to be occluded; at the same time, building change detection is easily interfered with by shadows generated by light and similar-colored features around the buildings, which makes the edges of the changed region challenging to be distinguished. Aiming at the above problems, this paper utilizes edge information to guide the neural network to learn edge features related to changes and suppress edge features unrelated to changes, so as to accurately extract building change information. First, an edge-extracted module is designed, which combines deep and shallow features to supplement the lack of feature information at different resolutions and to extract the edge structure of the changed features; second, an edge-guided module is designed to fuse the edge features with different levels of features and to guide the neural network to focus on the confusing building edge regions by increasing the edge weights to improve the network’s ability to detect the edges that have changed. The proposed building change detection algorithm has been validated on two publicly available data (WHU and LEVIR-CD building change detection datasets). The experimental results show that the proposed model achieves 91.14% and 89.76% in F1 scores, respectively, demonstrating superior performance compared to some recent learning change detection methods.

A fast hyperspectral change detection algorithm for agricultural crops based on low-rank matrix and morphological feature extraction

Crop change detection study is the foundation of agricultural sustainability. The inherent high spectral resolution of hyperspectral images, combined with multi-temporal datasets, facilitates the detection of subtle changes. To enhance the accuracy and applicability of hyperspectral change detection in agricultural scenes, this paper introduces a fast hyperspectral change detection approach for agricultural crops based on low-rank matrix and morphological feature extraction (FLRaMF). The goal is to improve detection precision and computational efficiency of the change detection process. The method initially employs rapid low-rank matrix extraction to separate changing and non-changing pixels in the spectral domain. Subsequently, spatial information is introduced using attribute profiles, restricting spectral anomalies through hyperspectral morphology, which ultimately improves the detection results. This study utilized four hyperspectral change detection datasets in agricultural crop scenarios, optimizing and analyzing parameters. Experimental results and analysis indicate that the FLRaMF method can achieve higher detection accuracy with reduced computation cost in unsupervised, default parameter scenarios when performing agricultural crop change detection tasks.

Analyzing the Impact of Scene Transitions on Indoor Camera Localization through Scene Change Detection in Real-Time

Analyzing the Impact of Scene Transitions on Indoor Camera Localization through Scene Change Detection in Real-Time

A Weakly Supervised Bitemporal Scene Change Detection Approach for Pixel-Level Building Damage Assessment Using Pre- and Post-Disaster High-Resolution Remote Sensing Images

A Weakly Supervised Bitemporal Scene Change Detection Approach for Pixel-Level Building Damage Assessment Using Pre- and Post-Disaster High-Resolution Remote Sensing Images

Contrastive Scene Change Representation Learning for High-Resolution Remote Sensing Scene Change Detection

Contrastive Scene Change Representation Learning for High-Resolution Remote Sensing Scene Change Detection

MultiFusedNet: A Multi-Feature Fused Network of Pretrained Vision Models via Keyframes for Student Behavior Classification

This research proposes a deep learning method for classifying student behavior in classrooms that follow the professional learning community teaching approach. We collected data on five student activities: hand-raising, interacting, sitting, turning around, and writing. We used the sum of absolute differences (SAD) in the LUV color space to detect scene changes. The K-means algorithm was then applied to select keyframes using the computed SAD. Next, we extracted features using multiple pretrained deep learning models from the convolutional neural network family. The pretrained models considered were InceptionV3, ResNet50V2, VGG16, and EfficientNetB7. We leveraged feature fusion, incorporating optical flow features and data augmentation techniques, to increase the necessary spatial features of selected keyframes. Finally, we classified the students’ behavior using a deep sequence model based on the bidirectional long short-term memory network with an attention mechanism (BiLSTM-AT). The proposed method with the BiLSTM-AT model can recognize behaviors from our dataset with high accuracy, precision, recall, and F1-scores of 0.97, 0.97, and 0.97, respectively. The overall accuracy was 96.67%. This high efficiency demonstrates the potential of the proposed method for classifying student behavior in classrooms.

A large scale training sample database system for intelligent interpretation of remote sensing imagery

ABSTRACT Artificial Intelligence (AI) Machine Learning (ML) technologies, particularly Deep Learning (DL), have demonstrated significant potential in the interpretation of Remote Sensing (RS) imagery, covering tasks such as scene classification, object detection, land-cover/land-use classification, change detection, and multi-view stereo reconstruction. Large-scale training samples are essential for ML/DL models to achieve optimal performance. However, the current organization of training samples is ad-hoc and vendor-specific, lacking an integrated approach that can effectively manage training samples from different vendors to meet the demands of various RS AI tasks. This article proposes a solution to address these challenges by designing and implementing LuoJiaSET, a large-scale training sample database system for intelligent interpretation of RS imagery. LuoJiaSET accommodates over five million training samples, providing support for cross-dataset queries and serving as a comprehensive training data store for RS AI model training and calibration. It overcomes challenges related to label semantic categories, structural heterogeneity in label representation, and interoperable data access.

An Experimental Study of the Accuracy and Change Detection Potential of Blending Time Series Remote Sensing Images with Spatiotemporal Fusion

Over one hundred spatiotemporal fusion algorithms have been proposed, but convolutional neural networks trained with large amounts of data for spatiotemporal fusion have not shown significant advantages. In addition, no attention has been paid to whether fused images can be used for change detection. These two issues are addressed in this work. A new dataset consisting of nine pairs of images is designed to benchmark the accuracy of neural networks using one-pair spatiotemporal fusion with neural-network-based models. Notably, the size of each image is significantly larger compared to other datasets used to train neural networks. A comprehensive comparison of the radiometric, spectral, and structural losses is made using fourteen fusion algorithms and five datasets to illustrate the differences in the performance of spatiotemporal fusion algorithms with regard to various sensors and image sizes. A change detection experiment is conducted to test if it is feasible to detect changes in specific land covers using the fusion results. The experiment shows that convolutional neural networks can be used for one-pair spatiotemporal fusion if the sizes of individual images are adequately large. It also confirms that the spatiotemporally fused images can be used for change detection in certain scenes.

A cooperative three-player game theory approach for designing an ideal video steganography framework

ABSTRACT This paper presents a cooperative game theory approach to improve the video steganography framework. Wherein, the video steganography framework comprises the following steps: (1) Cover video devising, (2) Secret image pre-processing, and (3) Embedding process. In the first step, the cover video is segmented using scene change detection method. Once the scenes are segmented the motion vectors are identified by Block Matching Motion Estimation Algorithm (BMMEA). Based on these motion vectors, the Region of Interest (ROI) is selected. The selected ROI is grouped using their momentum. In order to add an additional layer of security, the secret image is scrambled using pixel-wise Arnold Transform in the next step. Finally, based on the ROI groups, the scrambled secret data is embedded into the ROI's locations. An ideal and perfect video steganography system should be able to maintain the quality of the video after embedding the secret data. The quality factors of the video steganography approach include perceptible invisibility, payload capacity, and robustness. However, these quality factors are conflicted with each other. In order to address this issue, having these three quality factors as players a cooperative 3-players game theory approach is proposed. This technique provides an optimal solution for the video steganography framework using strategy adaption. The optimal solution is acquired by applying the Iterative Elimination of Strictly Dominant Strategies (IESDS) method. The achieved optimal solution comes up with the best trade-off between the quality factors. Experimental results have deduced the best optimal solution from the proposed three-player game theoretic approach that helps the video steganography approach to resolve the quality and security issues. It is also observed that the optimized proposed method outperforms the contemporary methods by attaining significant outcomes.

SARAS-Net: Scale and Relation Aware Siamese Network for Change Detection

Change detection (CD) aims to find the difference between two images at different times and output a change map to represent whether the region has changed or not. To achieve a better result in generating the change map, many State-of-The-Art (SoTA) methods design a deep learning model that has a powerful discriminative ability. However, these methods still get lower performance because they ignore spatial information and scaling changes between objects, giving rise to blurry boundaries. In addition to these, they also neglect the interactive information of two different images. To alleviate these problems, we propose our network, the Scale and Relation-Aware Siamese Network (SARAS-Net) to deal with this issue. In this paper, three modules are proposed that include relation-aware, scale-aware, and cross-transformer to tackle the problem of scene change detection more effectively. To verify our model, we tested three public datasets, including LEVIR-CD, WHU-CD, and DSFIN, and obtained SoTA accuracy. Our code is available at https://github.com/f64051041/SARAS-Net.

Cost-efficient information extraction from massive remote sensing data: When weakly supervised deep learning meets remote sensing big data

With many platforms and sensors continuously observing the earth surface, the large amount of remote sensing data presents a big data challenge. While remote sensing data acquisition capability can fully meet the requirements of many application domains, there is still a need to further explore how to efficiently mine the useful information from remote sensing big data (RSBD). Many researchers in the remote sensing community have introduced deep learning in the process of RSBD, and deep learning-based methods have achieved better performance compared with traditional methods. However, there are still substantial obstacles to the application of deep learning in remote sensing. One of the major challenges is the generation of pixel-level labels with high quality for training samples, which is essential to deep learning models. Weakly supervised deep learning (WSDL) is a promising solution to address this problem as WSDL can utilize greedily labeled datasets that are easy to collect but not ideal to train the deep networks. In this review, we summarize the achievements of WSDL-driven cost-efficient information extraction from RSBD. We first analyze the opportunities and challenges of information extraction from RSBD. Based on the analysis of the theoretical foundations of WSDL in the computer vision (CV) domain, we conduct a survey on the WSDL-based information extraction methods under the data characteristic and task demand of RSBD in four different tasks: (i) scene classification, (ii) object detection, (iii) semantic segmentation and (iv) change detection. Finally, potential research directions are outlined to guide researchers to further exploit WSDL-based information extraction from RSBD.

Scene Change aware Inter-Frame Forgeries Detection Technique for Surveillance Videos Based on Similarities Analysis

Surveillance videos provide security and increases work efficiency in places of work and homes. as the most acceptable form of evidence, surveillance videos are now tampered to hide actions or convey wrong information. Researchers have proposed ways to mitigate the effect of activities of the attackers through checking the authenticity of the video. The proposed schemes suffer performance degradation in the presence of scene changes. Recently a scheme that addresses the effects of scene change on inter-frame forgery detection was developed where it detects scene changes and divides multiple scenes in to shots. The scheme improves the overall performance of the inter-frame forgery detection at the expense of high average computational time. In this research, a video scene change aware forgery detection scheme is proposed to mitigate the effect of scene change on inter-frame forgery detection with low average computational time. The proposed scheme utilizes the luminance level within frame region which is a more efficient feature to detect scene change. The experimental results show that the scheme has 57% decreases in computational average time and increased in accuracy to 99.03%.

LuoJiaAI: A cloud-based artificial intelligence platform for remote sensing image interpretation

ABSTRACT The rapid processing, analysis, and mining of remote-sensing big data based on intelligent interpretation technology using remote-sensing cloud computing platforms (RS-CCPs) have recently become a new trend. The existing RS-CCPs mainly focus on developing and optimizing high-performance data storage and intelligent computing for common visual representation, which ignores remote sensing data characteristics such as large image size, large-scale change, multiple data channels, and geographic knowledge embedding, thus impairing computational efficiency and accuracy. We construct a LuoJiaAI platform composed of a standard large-scale sample database (LuoJiaSET) and a dedicated deep learning framework (LuoJiaNET) to achieve state-of-the-art performance on five typical remote sensing interpretation tasks, including scene classification, object detection, land-use classification, change detection, and multi-view 3D reconstruction. The details of the LuoJiaAI application experiment can be found at the white paper for LuoJiaAI industrial application. In addition, LuoJiaAI is an open-source RS-CCP that supports the latest Open Geospatial Consortium (OGC) standards for better developing and sharing Earth Artificial Intelligence (AI) algorithms and products on benchmark datasets. LuoJiaAI narrows the gap between the sample database and deep learning frameworks through a user-friendly data-framework collaboration mechanism, showing great potential in high-precision remote sensing mapping applications.

Wipe Scene Change Detection in Object-Camera Motion Based on Linear Regression and an Inflated Spatial-Motion Neural Network

To facilitate content-based video analysis, automatic scene change detection (SCD) with large-scale motion activity is an essential fundamental step for locating a transition from one video scene to another. With the exponential increase in digital media usage, SCD has become more challenging in processing large motion content with minimal information loss and maximum perseverance. Wipe SCD in object-camera motion is noticeable evidence of this issue. Wipe transitions, which are a type of gradual transition, have diverse motion pattern changes when influenced by object-camera motion (camera pan, large-object, and zoom-in/out), creating a velocity imbalance in the same frame. Furthermore, this motion imbalance leads to false detection. Due to the loss of motion information and longer processing time of existing frameworks, we propose a novel method of wipe scene change detection (WSCD) based on deep spatial-motion feature analysis. First, large input videos are segmented into shots using dimensionality reduction and adaptive threshold. Secondly, linear regression is used to compute slope angle changes in shots for candidate selection and wipe localization. Finally, only selected candidates are processed to extract features using a two-stream inflated 3D-convolutional neural network for RGB stream and optical flow velocity for motion stream network (I3DCNN) and then classified into wipe in-motion and no-motion clips. The experimental results are obtained by classifying wipe patterns using a detection reviewing and merging strategy on corresponding wipe frames. The average improvement in wipe scene change detection accuracy evaluated on the benchmark TRECVID dataset is 11.9%, demonstrating the efficacy of our proposed method.

Scene Change Detection by Differential Aggregation Network and Class Probability-Based Fusion Strategy

Scene Change Detection by Differential Aggregation Network and Class Probability-Based Fusion Strategy