Multi-temporal Information Research Articles

Small object change detection in UAV imagery via a Siamese network enhanced with temporal mutual attention and contextual features: A case study concerning solar water heaters

Small object change detection (SOCD) based on high-spatial resolution (HSR) images is of significant practical value in applications such as the investigation of illegal urban construction, but little research is currently available. This study proposes an SOCD model called TMACNet based on a multitask network architecture. The model modifies the YOLOv8 network into a Siamese network and adds structures, including a feature difference branch (FDB), temporal mutual attention layer (TMAL) and contextual attention module (CAM), to merge differential and contextual features from different phases for the accurate extraction and analysis of small objects and their changes. To verify the proposed method, an SOCD dataset called YZDS is created based on unmanned aerial vehicle (UAV) images of small-scale solar water heaters on rooftops. The experimental results show that TMACNet exhibits strong resistance to image registration errors and building height displacement and prevents error propagation from object detection to change detection originating from overlay-based change detection. TMACNet also provides an enhanced approach to small object detection from the perspective of multitemporal information fusion. In the change detection task, TMACNet exhibits notable F1 improvements exceeding 5.96% in comparison with alternative change detection methods. In the object detection task, TMACNet outperforms the single-temporal object detection models, increasing accuracy with an approximately 1–3% improvement in the AP metric while simplifying the technical process.

A Lightweight Machine-Learning Method for Cloud Removal in Remote Sensing Images Constrained by Conditional Information

Reconstructing cloud-covered regions in remote sensing (RS) images holds great promise for continuous ground object monitoring. A novel lightweight machine-learning method for cloud removal constrained by conditional information (SMLP-CR) is proposed. SMLP-CR constructs a multilayer perceptron with a presingle-connection layer (SMLP) based on multisource conditional information. The method employs multi-scale mean filtering and local neighborhood sampling to gain spatial information while also taking into account multi-spectral and multi-temporal information as well as pixel similarity. Meanwhile, the feature importance from the SMLP provides a selection order for conditional information—homologous images are prioritized over images from the same season as the restoration image, and images with close temporal distances rank last. The results of comparative experiments indicate that SMLP-CR shows apparent advantages in terms of visual naturalness, texture continuity, and quantitative metrics. Moreover, compared with popular deep-learning methods, SMLP-CR samples locally around cloud pixels instead of requiring a large cloud-free training area, so the samples show stronger correlations with the missing data, which demonstrates universality and superiority.

Deep Attention Networks With Multi-Temporal Information Fusion for Sleep Apnea Detection.

Sleep Apnea (SA) is a prevalent sleep disorder with multifaceted etiologies that can have severe consequences for patients. Diagnosing SA traditionally relies on the in-laboratory polysomnogram (PSG), which records various human physiological activities overnight. SA diagnosis involves manual scoring by qualified physicians. Traditional machine learning methods for SA detection depend on hand-crafted features, making feature selection pivotal for downstream classification tasks. In recent years, deep learning has gained popularity in SA detection due to its capability for automatic feature extraction and superior classification accuracy. This study introduces a Deep Attention Network with Multi-Temporal Information Fusion (DAN-MTIF) for SA detection using single-lead electrocardiogram (ECG) signals. This framework utilizes three 1D convolutional neural network (CNN) blocks to extract features from R-R intervals and R-peak amplitudes using segments of varying lengths. Recognizing that features derived from different temporal scales vary in their contribution to classification, we integrate a multi-head attention module with a self-attention mechanism to learn the weights for each feature vector. Comprehensive experiments and comparisons between two paradigms of classical machine learning approaches and deep learning approaches are conducted. Our experiment results demonstrate that (1) compared with benchmark methods, the proposed DAN-MTIF exhibits excellent performance with 0.9106 accuracy, 0.9396 precision, 0.8470 sensitivity, 0.9588 specificity, and 0.8909 [Formula: see text] score at per-segment level; (2) DAN-MTIF can effectively extract features with a higher degree of discrimination from ECG segments of multiple timescales than those with a single time scale, ensuring a better SA detection performance; (3) the overall performance of deep learning methods is better than the classical machine learning algorithms, highlighting the superior performance of deep learning approaches for SA detection.

Incorporating landscape ecological approach in machine learning classification for agricultural land-use mapping based on a single date imagery

Land-use maps containing crop rotational information are very important in land management and physical planning. Such maps are usually generated using multitemporal data. Although recent technology allows analysts to process multitemporal information effectively, the use of single date imagery for such purpose is more efficient. This study aimed to map detailed agricultural land-use with crop rotational information based on a single date Landsat 8 imagery and SRTM-derived terrain attributes. A landscape ecological approach assuming the influence of terrain characteristics on the existence of crop and land-use types was implemented in multisource classification using random decision forest (RDF) machine learning algorithm. The use of seven optical bands and five terrain attributes could provide a land-use map at 88.03% accuracy, compared to seven optical bands only that generate 82.45% accuracy. These results are also better than those of maximum likelihood. The most influential variables in the achieved accuracy are elevation and thermal band.

Enhancing RABASAR for Multi-Temporal SAR Image Despeckling through Directional Filtering and Wavelet Transform.

The presence of speckle noise severely hampers the interpretability of synthetic aperture radar (SAR) images. While research on despeckling single-temporal SAR images is well-established, there remains a significant gap in the study of despeckling multi-temporal SAR images. Addressing the limitations in the acquisition of the "superimage" and the generation of ratio images within the RABASAR despeckling framework, this paper proposes an enhanced framework. This enhanced framework proposes a direction-based segmentation approach for multi-temporal SAR non-local means filtering (DSMT-NLM) to obtain the "superimage". The DSMT-NLM incorporates the concept of directional segmentation and extends the application of the non-local means (NLM) algorithm to multi-temporal images. Simultaneously, the enhanced framework employs a weighted averaging method based on wavelet transform (WAMWT) to generate superimposed images, thereby enhancing the generation process of ratio images. Experimental results demonstrate that compared to RABASAR, Frost, and NLM, the proposed method exhibits outstanding performance. It not only effectively removes speckle noise from multi-temporal SAR images and reduces the generation of false details, but also successfully achieves the fusion of multi-temporal information, aligning with experimental expectations.

GLTF-Net: Deep-Learning Network for Thick Cloud Removal of Remote Sensing Images via Global–Local Temporality and Features

Remote sensing images are very vulnerable to cloud interference during the imaging process. Cloud occlusion, especially thick cloud occlusion, significantly reduces the imaging quality of remote sensing images, which in turn affects a variety of subsequent tasks using the remote sensing images. The remote sensing images miss ground information due to thick cloud occlusion. The thick cloud removal method based on a temporality global–local structure is initially suggested as a solution to this problem. This method includes two stages: the global multi-temporal feature fusion (GMFF) stage and the local single-temporal information restoration (LSIR) stage. It adopts the fusion feature of global multi-temporal to restore the thick cloud occlusion information of local single temporal images. Then, the featured global–local structure is created in both two stages, fusing the global feature capture ability of Transformer with the local feature extraction ability of CNN, with the goal of effectively retaining the detailed information of the remote sensing images. Finally, the local feature extraction (LFE) module and global–local feature extraction (GLFE) module is designed according to the global–local characteristics, and the different module details are designed in this two stages. Experimental results indicate that the proposed method performs significantly better than the compared methods in the established data set for the task of multi-temporal thick cloud removal. In the four scenes, when compared to the best method CMSN, the peak signal-to-noise ratio (PSNR) index improved by 2.675, 5.2255, and 4.9823 dB in the first, second, and third temporal images, respectively. The average improvement of these three temporal images is 9.65%. In the first, second, and third temporal images, the correlation coefficient (CC) index improved by 0.016, 0.0658, and 0.0145, respectively, and the average improvement for the three temporal images is 3.35%. Structural similarity (SSIM) and root mean square (RMSE) are improved 0.33% and 34.29%, respectively. Consequently, in the field of multi-temporal cloud removal, the proposed method enhances the utilization of multi-temporal information and achieves better effectiveness of thick cloud restoration.

MULTI-SCALE AND MULTI-SENSOR APPROACHES FOR THE PROTECTION OF CULTURAL NATURAL HERITAGE: THE ISLAND OF SANTO SPIRITO IN VENICE

Abstract. The study of Cultural Natural Heritage (CNH) requires the development of multi-disciplinary and multi-scale methodologies for data recording, representation, and correlation from various platforms such as terrestrial, aerial and satellite sensors. The heterogeneity of geo-databases currently available demands on-site validation and time monitoring to control the phenomena related to climate change that inevitably affect the Cultural Heritage (CH). The pressures stressing the territorial dimension due to climatic changes lead to the decrease of essential resources and burden on the CH. To overcome the lack of information needed at various territorial scales, it becomes necessary to construct detailed and dynamic cognitive frameworks. This paper establishes a multitemporal information framework regarding the case study area, the Island of Santo Spirito in Venice, using several geomatic techniques to investigate the island's ecological significance and constructed heritage. The suggested methodology uses the integration of multitemporal data resulting from the processing of satellite images provided by the Copernicus satellites (Sentinel-2) and data from geomatic documentation techniques. Two separate methods were used in the survey operations: a Terrestrial Laser Scanning (TLS) and aerial photogrammetry from Uncrewed Aerial Vehicles (UAV) survey. The integration of satellite, aerial, and terrestrial data has allowed a complete knowledge of the necessary parameters for the monitoring of the CH of the area. In order to manage conservative policy from a preventive perspective and to recreate and digitally visualize missing historical phases, programmed monitoring is a crucial instrument.

Multi-Sensor and Multi-Scale Remote Sensing Approach for Assessing Slope Instability along Transportation Corridors Using Satellites and Uncrewed Aircraft Systems

Rapid slope instabilities (i.e., rockfalls) involving highway networks in mountainous areas pose a threat to facilities, settlements and life, thus representing a challenge for asset management plans. To identify different morphological expressions of degradation processes that lead to rock mass destabilization, we combined satellite and uncrewed aircraft system (UAS)-based products over two study sites along the State Highway 133 sector near Paonia Reservoir, Colorado (USA). Along with a PS-InSAR analysis covering the 2017–2021 interval, a high-resolution dataset composed of optical, thermal and multi-spectral imagery was systematically acquired during two UAS surveys in September 2021 and June 2022. After a pre-processing step including georeferencing and orthorectification, the final products were processed through object-based multispectral classification and change detection analysis for highlighting moisture or lithological variations and for identifying areas more susceptible to deterioration and detachments at the small and micro-scale. The PS-InSAR analysis, on the other hand, provided multi-temporal information at the catchment scale and assisted in understanding the large-scale morpho-evolution of the displacements. This synergic combination offered a multiscale perspective of the superimposed imprints of denudation and mass-wasting processes occurring on the study site, leading to the detection of evidence and/or early precursors of rock collapses, and effectively supporting asset management maintenance practices.

Dilated Multi-Temporal Modeling for Action Recognition

Action recognition involves capturing temporal information from video clips where the duration varies with videos for the same action. Due to the diverse scale of temporal context, uniform size kernels utilized in convolutional neural networks (CNNs) limit the capability of multiple-scale temporal modeling. In this paper, we propose a novel dilated multi-temporal (DMT) module that provides a solution for modeling multi-temporal information in action recognition. By using dilated convolutions with different dilation rates in different feature map channels, the DMT module captures information at multiple scales without the need for costly multi-branch networks, input-level frame pyramids, or feature map stacking that previous works have usually incurred. Therefore, this approach enables the integration of temporal information from multiple scales. In addition, the DMT module can be integrated into existing 2D CNNs, making it a straightforward and intuitive solution for addressing the challenge of multi-temporal modeling. Our proposed method has demonstrated promising results in performance and has achieved about 2% and 1% accuracy improvement on FineGym99 and SthV1. We conducted an empirical analysis that demonstrates how DMT improves the classification accuracy for action classes with varying durations.

Land use classification over smallholding areas in the European Common Agricultural Policy framework

Land use (LU) monitoring and mapping from remote sensing (RS) data are relevant topics in Earth observation (EO). LU monitoring requires powerful/robust classification approaches able to provide reliable identifications in decision-making applications, even over challenging areas. The European Common Agricultural Policy (CAP) promotes the use of remote sensing for LU monitoring in Europe useful in the compliance control of the subsidy payments. In this context, the present study proposes a novel classification chain to identify ten land uses, namely: forest (FOR), pasture with trees (TRE), pasture with shrubs (SHR), pastureland (PAS), vineyard (VIN), arable land (ARL), fruit trees (FRU), nut trees (NUT), citrus (CIT), and olive grove (OLI), over three areas in the Valencian Autonomous Region (Spain) where smallholding farming predominates. The approach exploits the multitemporal information provided by Sentinel-2 data using a novel spatial strategy specifically designed to deal with heterogeneous agricultural areas. More precisely, we implemented a deep learning algorithm based on bidirectional recurrent neural networks to account for complex temporal dynamics. The classification results over the three areas showed accuracies higher than 95.5% over validation sets composed of in-situ checks never used during the training process. The proposed methodology outperformed standard approaches that not consider the spatial variability of the training samples, and revealed very good agreement concerning the Land Parcel Identification System (LPIS) of Spain. In addition, the developed chain proposes the novelty of using the kernel normalized difference vegetation index (kNDVI) as a predictor in a LU classification processing chain. Including the kNDVI instead of the traditional NDVI outperformed the classification accuracy in all metrics and classes. Ultimately, the obtained classifications were used for assessing the 2020 farmers’ declarations in the three study areas. The declarations showed a level of agreement concerning the proposed approach near 99%.

Essential tensor learning for multimodal information-driven stock movement prediction

In the literature, an increasing amount of information from various sources related to the stock market is being considered for stock movement prediction. However, previous studies usually modeled market information as a vector, failing to effectively utilize the inner structure in terms of multimodal and multitemporal characteristics. Moreover, the release, dissemination, and absorption of information causing spillover effects from stocks related to the target stock should not be neglected in today’s information society. Thus, this study proposes a general tensor representation and fusion framework to capture the intrinsic interactions of multimodal and multitemporal stock market information based on the invariant correlations among stocks within a short period. Specifically, we construct a general correlation matrix to represent the correlation between the stocks with respect to a given mode of information for a single day. Then, for a short period, with multimodal information, the matrices are concatenated into a tensor, which is highly inner correlated. A tensor robust principal component analysis (TRPCA) model is then employed to fuse the multimodal and multitemporal information, adaptively infer essential interactions, and faithfully enhance the inner correlation of the constructed tensor. Experiments on real datasets show that the proposed tensor representation and fusion framework can efficiently improve the performance of stock movement prediction. The performance of the investment simulation further illustrates the superiority of the proposed method in terms of the return rate (26.73%) for a full year.

SPATIO-TEMPORAL ANALYSIS OF LAND COVER CHANGES OF DHAKA CITY IN BANGLADESH

The objective of this study was to identify the Spatio-temporal variation in landcover of Dhaka city, the capital of Bangladesh. The study was conducted to highlight the issues of changing land-cover of mega cities of Bangladesh and its future environmental consequences. Land cover changes and temporal maps for the selected study areas were developed by using multi-temporal information data sets by Multispectral Scanner (MSS), Thematic Map (TM), Enhanced Thematic Mapper (ETM), Enhanced Thematic Mapper Plus (ETM+),or Operational Land Imager (OLI) and Thermal Infrared Sensor (TIRS) for the years 1993, 2000, 2010 and 2015. The results of this study highlighted the rapid urbanization in Dhaka which caused significant decline in vegetation cover and increase of built-up or paved surfaces. The land-cover classified images highlighted, that in 1993 only 22% per cent area of the study area was covered with built-up surface. However, in 2015, it increased up to 48%. This substantial decline in vegetation cover and upsurge in built-up area of Dhaka city may put petrifying effect on local environments.

High-Fidelity Reconstruction of Structured Illumination Microscopy by an Amplitude-Phase Channel Attention Network With Multitemporal Information

Structured illumination microscopy (SIM) has been a popular method for live-cell super-resolution (SR) imaging due to its excellent photon efficiency. However, SIM is always marred by artifacts in the reconstructed SR images. To address this problem, we develop an end-to-end amplitude-phase channel attention network (APCAN) for SIM reconstruction based on a-priori frequency and temporal knowledge. The APCAN reinforces both amplitude and phase information of the raw images to guide network reconstruction and attains SR images with fewer artifacts. Moreover, inspired by the continuity knowledge in the traditional method, we design a temporal processing module in APCAN to utilize multiple time points data. Trained with the video dataset imaged from our setup, APCAN can reconstruct an artifact-minimized SR image, achieving a reduction of 38% in reconstruction errors. Finally, we demonstrate that the APCAN reconstructed images have great resistance to noise and photobleaching, achieving the best fidelity among all methods tested.

Despeckling Multitemporal Polarimetric SAR Data Based on Tensor Decomposition

Despeckling is an essential task in polarimetric synthetic aperture radar (PolSAR) image processing. Most existing filters developed for multi-temporal SAR images make use of either real or complex information. Real information refers to amplitude or intensity values, whereas complex information refers to complex covariance matrix (CCM) derived from either interferometric SAR (InSAR) or PolSAR data. The InSAR CCM is formed using images of the same polarimetric channel but acquired at different dates, and the PolSAR CCM contains information acquired simultaneously in different polarimetric channels. Therefore, these despeckling methods may present a good performance in some applications and scenes but fail in other cases, due to differences in input sources. In order to achieve a more robust result in all cases, we develop a method for multi-temporal polarimetric SAR data filtering based on tensor decomposition (TD-MPF), which aims at improving the identification of homogeneous pixels for spatially adaptive filtering. The key element of this approach consists of exploiting tensor theory to construct a new CCM that contains both polarimetric and interferometric information, as well as multi-temporal information for each pixel. The effectiveness of the proposed method and its performance are evaluated with simulated and real SAR data in comparison with several established methods.

Spatio-temporal habitat assessment of the Gangetic floodplain in the Hastinapur wildlife sanctuary

Remote sensing provides multi-dimensional and multi-temporal information about habitat, insights into the significant drivers of change, and the key factors affecting landscape dynamics. Such information is crucial to provide perspective and a more profound understanding of ecological surveys. This study utilizes Google Earth Engine's capability to assess a riverine wetland grassland floodplain in Hastinapur Wildlife Sanctuary (HWS) along the Ganga River, which is a critical habitat for wintering migratory birds, the critically endangered gharial, turtles, aquatic mammals such as otters and dolphins, and cervid species such as swamp deer. We have developed a framework for regular monitoring through rapid habitat assessment, which visualizes the spatio-temporal change in land cover, the seasonal dynamics in water and vegetation, changes due to anthropogenic influences on the landscape, and finally, how these factors affect the habitat availability of species of concern in the HWS. The results show a dynamic river system with high seasonal variations. The vegetation trend shows increasing greenness, indicative of the conversion of grassland, scrubland, and herbaceous cover to more permanent vegetation, which will adversely affect the riparian habitat structure. A habitat suitability index generated through geospatial analysis using the weighted overlay method suggests that 40.07% of the HWS, nearly 767.12 km2, comprising mainly the riverscape, wetland, and riparian grasslands, is suitable habitat for aquatic and semi-aquatic species. However, only 9.93% of the sanctuary comprising 197.27 km2, is available as a core habitat. Further, the area is threatened by encroachment evident from the rapidly increasing land-use intensity and night light pollution, which puts the grasslands sheltered to swamp deer and other wildlife at higher risk leaving almost 27.9% of the area comprising 533.63 km2 unsuitable for wildlife. Urban and agricultural land use has taken over 67% of the sanctuary. An increase in minimum radiance value representing ALAN, from 0.41 to 0.73 W/m2-sr in just six years from 2015 to 2021, shows a reduction in nocturnal darkness, reducing safe niches for wildlife. The study results provide critical baseline information for ecological surveys and a rapid assessment platform for future sanctuary management. Constant monitoring of anthropogenic activities such as farming, settlements, and transport routes threatening the habitat is essential for informed conservation management decisions.

Study of image fusion optimization techniques for medical applications

Image fusion is becoming increasingly important in computer vision activities because of the larger number of capture methods. The integration of different views like multi view, multi temporal view and information which is larger together termed as image fusion (IF). In image fusion the information of multiple-sensors are converted into a unified image while maintaining the fidelity of critical characteristics.Healthcare image fusion procedures are used to improve picture quality by achieving the conspicuous characteristics in the fusion findings. As a result, they increase the practical usefulness of medical pictures for issue assessment and identification. Fusion of medical images are generally evaluated using the modalities like MRI-Magnetic Resonance Imaging, MRA-Magnetic Resonance Angiogram, PET-Positron Emission Tomography, SPET-Structural Positron Emission Tomography,CT-Computed Tomography, and SPECT-Single-Photon Emission Computed Tomography. Neural network and optimization techniques help in improving the quality of fused image. In this paper various fusion techniques are studied. Along with different fuson approaches, the outcomes of diverse research projects are contrasted in terms of how the pulse coupled neural network is employed and different optimization techniques. The Pulse Couple Neural Networks (PCNN) using various optimization techniques are compared. Among which the swarming mechanism of salps improves the performance of the system. The PCNN is combined with SSO algorithm and evaluated the results.From the results it is shown that PCNN- Salp Swarm Optimization (SSO)ishaving good value of Peak Signal to Noise Ratio (PSNR) with 45.93 and Structural Similarity Index(SSIM) is 0.996.

Remote Sensing Approaches for Meteorological Disaster Monitoring: Recent Achievements and New Challenges.

Meteorological disaster monitoring is an important research direction in remote sensing technology in the field of meteorology, which can serve many meteorological disaster management tasks. The key issues in the remote sensing monitoring of meteorological disasters are monitoring task arrangement and organization, meteorological disaster information extraction, and multi-temporal disaster information change detection. To accurately represent the monitoring tasks, it is necessary to determine the timescale, perform sensor planning, and construct a representation model to monitor information. On this basis, the meteorological disaster information is extracted by remote sensing data-processing approaches. Furthermore, the multi-temporal meteorological disaster information is compared to detect the evolution of meteorological disasters. Due to the highly dynamic nature of meteorological disasters, the process characteristics of meteorological disasters monitoring have attracted more attention. Although many remote sensing approaches were successfully used for meteorological disaster monitoring, there are still gaps in process monitoring. In future, research on sensor planning, information representation models, multi-source data fusion, etc., will provide an important basis and direction to promote meteorological disaster process monitoring. The process monitoring strategy will further promote the discovery of correlations and impact mechanisms in the evolution of meteorological disasters.

A review of hyperspectral remote sensing of crops

With the development of space science and technology, various resource monitoring environmental satellites provide multi-platform, multi-spectral, multi-temporal and wide-range real-time information for the study of surface dynamic changes, and remote sensing technology has become a powerful technical means for human to study the earth’s resources and environment, while high-resolution and hyperspectral remote sensing has become the main source for fruit tree growth monitoring and fruit quality detection acquisition. This paper has the following aspects to introduce the current situation of application of high-resolution and hyperspectral remote sensing data.

Robust Thick Cloud Removal for Multitemporal Remote Sensing Images Using Coupled Tensor Factorization

The existing nonblind cloud and cloud shadow (cloud/shadow) removal methods for remote sensing (RS) images are based on the assumption that cloud/shadow masks are accurately given. Since the masks are usually manually labeled or detected by cloud detection methods, whose accuracy cannot be well guaranteed, the cloud/shadow removal effect may be affected. In this article, we suggest a robust thick cloud/shadow removal (RTCR) method that meets the problem with an inaccurate mask. To faithfully reconstruct the multitemporal information, a coupled tensor factorization is used to explore the relationship between the abundances of the multitemporal images in the same scene. Moreover, an efficient algorithm is developed to solve the proposed model based on the augmented Lagrange multiplier method. The experimental results under accurate masks and inaccurate masks demonstrate its robustness and superiority for thick cloud/shadow removal.

A Learnable Model With Calibrated Uncertainty Quantification for Estimating Canopy Height From Spaceborne Sequential Imagery

Global-scale canopy height mapping is an important tool for ecosystem monitoring and sustainable forest management. Various studies have demonstrated the ability to estimate canopy height from a single spaceborne multi-spectral image using end-to-end learning techniques. In addition to texture information of a single-shot image, our study exploits multi-temporal information of image sequences to improve estimation accuracy. We adopt a convolutional variant of a long short-term memory (LSTM) model for canopy height estimation from multi-temporal instances of Sentinel-2 products. Furthermore, we utilize deep ensembles technique for meaningful uncertainty estimation on the predictions and post-processing isotonic regression model for calibrating them. Our lightweight model (~ 320k trainable parameters) achieves mean absolute error (MAE) of 1.29m in a European test area of 79km2. It outperforms state-of-the-art methods based on single-shot spaceborne images as well as costly airborne images, while providing additional confidence maps that are shown to be well calibrated. Moreover, the trained model is shown to be transferable in a different country of Europe using a fine-tuning area of as low as ~ 2km2 with MAE = 1.94m.