Fine-grained Changes Research Articles

Prediction of changes in war-induced population and CO2 emissions in Ukraine using social media

Monitoring the changes in population and anthropogenic CO2 emissions caused by geopolitical conflicts is significant for humanitarian assistance and also for revealing the CO2 emission patterns of human activities. However, the changes in population and anthropogenic CO2 emissions are highly dynamic, and representative survey data are generally unavailable. We monitor the near-real-time and fine-grained changes in war-induced population and anthropogenic CO2 emissions in Ukraine through social media data. One year after the invasion, over 11 million Ukrainians are displaced from the baseline 0.1∘*0.1∘\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$${0.1}^{\\circ}\\,*\\,{0.1}^{\\circ}$$\\end{document} gridded regions. There is a significant correlation between the estimated changes and the reference changes for each month in all CO2 emission sectors, with R2 respectively ranging from 0.57 to 0.93, 0.41 to 0.9, 0.74 to 0.99 for residential consumption, ground transport, and industry sectors. Overall, the proposed method provides a new perspective to monitor Ukrainian refugee crisis and measure the spatio-temporal response of anthropogenic CO2 emissions.

SMART: Syntax-Calibrated Multi-Aspect Relation Transformer for Change Captioning.

Change captioning aims to describe the semantic change between two similar images. In this process, as the most typical distractor, viewpoint change leads to the pseudo changes about appearance and position of objects, thereby overwhelming the real change. Besides, since the visual signal of change appears in a local region with weak feature, it is difficult for the model to directly translate the learned change features into the sentence. In this paper, we propose a syntax-calibrated multi-aspect relation transformer to learn effective change features under different scenes, and build reliable cross-modal alignment between the change features and linguistic words during caption generation. Specifically, a multi-aspect relation learning network is designed to 1) explore the fine-grained changes under irrelevant distractors (e.g., viewpoint change) by embedding the relations of semantics and relative position into the features of each image; 2) learn two view-invariant image representations by strengthening their global contrastive alignment relation, so as to help capture a stable difference representation; 3) provide the model with the prior knowledge about whether and where the semantic change happened by measuring the relation between the representations of captured difference and the image pair. Through the above manner, the model can learn effective change features for caption generation. Further, we introduce the syntax knowledge of Part-of-Speech (POS) and devise a POS-based visual switch to calibrate the transformer decoder. The POS-based visual switch dynamically utilizes visual information during different word generation based on the POS of words. This enables the decoder to build reliable cross-modal alignment, so as to generate a high-level linguistic sentence about change. Extensive experiments show that the proposed method achieves the state-of-the-art performance on the three public datasets.

Integrating bi-temporal VHR optical and long-term SAR images for built-up area change detection

ABSTRACT With the rapid expansion of urbanization, it is imperative to monitor built-up areas changes to promote the sustainable development of cities, aligning with the goals of Sustainable Development Goal 11(SDG 11). Remote sensing big data is valuable for automatically mapping these changes. Bi-temporal very high-resolution (Bi-VHR) optical images have been widely utilized for fine-grained change detection (CD). However, the significant spatiotemporal inconsistency due to imaging conditions and seasonal variations poses challenges for VHR optical CD. Unlike optical images, synthetic aperture radar (SAR) images are unaffected by atmospheric interference and provide robust spatiotemporal features as a supplement. Previous CD algorithms with SAR overlooked the exploration of long-term features of time series. In this study, we propose a novel CD framework combining long-term SAR with Bi-VHR images. It incorporates a spatial-frequency learning module to enhance SAR temporal features and a multisource feature fusion module to adaptively fuse the heterogeneous features. The experiments are conducted on OS-BCD dataset, which is the first dataset specifically designed for this task. The results demonstrate that our proposal outperforms advanced CD methods with F1 score, IoU and OA of 64.99%, 48.13%, and 95.11%, validating the efficacy of our proposal in accurately detecting changes in built-up areas.

MSLR: A Self-supervised Representation Learning Method for Tabular Data Based on Multi-scale Ladder Reconstruction

Tabular data are widely used for prediction tasks, but they often suffer from the curse of dimensionality and noise, leading to degradation in the performance and robustness of prediction models. Self-supervised representation learning has emerged as a promising technique to overcome these challenges, but most existing methods are applicable to images, text, and others rather than tabular data. In this study, we propose a novel self-supervised representation learning method for tabular data based on multi-scale ladder reconstruction (MSLR). The method effectively learns low-dimensional and noise-resistant representations, thereby improving the prediction performance across various tabular datasets. The idea of MSLR is to employ a binning method to generate a sequence of fuzzy data with different noise scales, followed by training a neural network to recover the raw data from the most corrupted data in a circular manner. This process allows MSLR to learn fine-grained changes caused by noise while maintaining consistency and similarity at a coarse granularity. The proposed method is evaluated on five real-world datasets, namely, MIMIC-IV, Thyroid, Heart, Pima, and Adult, and compared with several baselines. The experimental results of downstream prediction tasks show that MSLR is robust to noisy data and performs better than other existing baseline methods.

Feature alignment and refinement for Remote Sensing images change Detection

ABSTRACT Change detection (CD) plays a critical role in extracting ground changes from bi-temporal remote sensing (RS) images and is instrumental in understanding surface dynamics. In recent years, deep learning has made significant breakthroughs in CD. However, typical CD methods that employ the Siamese network for temporal feature extraction lack feature alignment ability for bi-temporal heterogeneous RS images, resulting in inadequate temporal discriminative capability. Moreover, deep learning-based CD methods are still susceptible to the problem of minor changes missing due to scale variation with deeper network layers. In this article, we propose a bi-temporal feature alignment and refinement network (FARNet). To improve the discriminative capability of the Siamese network, an adversarial learning-based temporal discriminatory loss function is designed to align temporal-level features and eliminate bi-temporal domain shift, and a cosine similarity-based loss function is employed to measure feature distance at the Pixel-level. To address the problem of minor changes missing, we adopt a dilated convolution-based Siamese network to prevent feature map size reduction, and a multi-level feature detail supplement (MFDS) module is designed to supplement the deep layer features with shallow layer features. Additionally, we construct a change map refinement (CMR) module that refines the coarse change map to the fine-grained change map. Furthermore, we design a cross-temporal feature interaction (CFI) module to learn more fine-grained change features by combining features across temporal. Comprehensive experimental results on two popular CD datasets demonstrate the effectiveness and efficiency of FARNet compared with state-of-the-art (SOTA) methods.

The effect of musical training and language background on vocal imitation of pitch in speech and song.

Vocal imitation plays a critical function in the development and use of both language and music. Previous studies have reported more accurate imitation for sung pitch than spoken pitch, which might be attributed to the structural differences in acoustic signals and/or the distinct mental representations of pitch patterns across speech and music. The current study investigates the interaction between bottom-up (i.e., acoustic structure) and top-down (i.e., participants' language and musical background) factors on pitch imitation by comparing speech and song imitation accuracy across four groups: English and Mandarin speakers with or without musical training. Participants imitated pitch sequences that were characteristic of either song or speech, derived from pitch patterns in English and Mandarin spoken sentences. Overall, song imitation was more accurate than speech imitation, and this advantage was larger for English than Mandarin pitch sequences, regardless of participants' musical and language experiences. This effect likely reflects the perceptual salience of linguistic tones in Mandarin relative to English speech. Music and language knowledge were associated with optimal imitation of different acoustic features. Musicians were more accurate in matching absolute pitch across syllables and musical notes compared to nonmusicians. By contrast, Mandarin speakers were more accurate at imitating fine-grained changes within and across pitch events compared to English speakers. These results suggest that different top-down factors (i.e., language and musical background) influence pitch imitation ability for different dimensions of bottom-up features (i.e., absolute pitch and relative pitch patterns). (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Phase-based fine-grained change detection

Phase-based fine-grained change detection

Deep Insights of Learning-Based Micro Expression Recognition: A Perspective on Promises, Challenges, and Research Needs

Micro expression recognition (MER) is a very challenging area of research due to its intrinsic nature and fine-grained changes. In the literature, the problem of MER has been solved through handcrafted/descriptor-based techniques. However, in recent times, deep learning (DL) based techniques have been adopted to gain higher performance for MER. Also, rich survey articles on MER are available by summarizing the datasets, experimental settings, conventional and deep learning methods. In contrast, these studies lack the ability to convey the impact of network design paradigms and experimental setting strategies for DL based MER. Therefore, this paper aims to provide a deep insight into the DL-based MER frameworks with a perspective on promises in network model designing, experimental strategies, challenges, and research needs. Also, the detailed categorization of available MER frameworks is presented in various aspects of model design and technical characteristics. Moreover, an empirical analysis of the experimental and validation protocols adopted by MER methods is presented. The challenges mentioned earlier and network design strategies may assist the affective computing research community in forge ahead in MER research. Finally, we point out the future directions, research needs and draw our conclusions.

An Overview of Facial Micro-Expression Analysis: Data, Methodology and Challenge

Facial micro-expressions indicate brief and subtle facial movements that appear during emotional communication. In comparison to macro-expressions, micro-expressions are more challenging to be analyzed due to the short span of time and the fine-grained changes. In recent years, micro-expression recognition (MER) has drawn much attention because it can benefit a wide range of applications, e.g. police interrogation, clinical diagnosis, depression analysis, and business negotiation. In this survey, we offer a fresh overview to discuss new research directions and challenges these days for MER tasks. For example, we review MER approaches from three novel aspects: macro-to-micro adaptation, recognition based on key apex frames, and recognition based on facial action units. Moreover, to mitigate the problem of limited and biased ME data, synthetic data generation is surveyed for the diversity enrichment of micro-expression data. Since micro-expression spotting can boost micro-expression analysis, the state-of-the-art spotting works are also introduced in this paper. At last, we discuss the challenges in MER research and provide potential solutions as well as possible directions for further investigation.

A full-resolution convolutional network with a dynamic graph cut algorithm for skin cancer classification and detection

A robust medical decision support system for classifying skin lesions from dermoscopy images is a crucial instrument for determining skin cancer prognosis. In recent years, full resolution convolutional network has made significant progress in recognizing skin cancer types from Dermoscopic images despite their fine-grained changes in appearance. Recently, full resolution convolutional network have gained popularity as a solution to semantic segmentation issues. However, the hyper-parameters it chooses are what determine how well it performs, and manually fine-tuning these hyper-parameters takes time. Therefore, a hyper-parameter optimized full resolution convolutional network is suggested for dermoscopy picture segmentation in this research. The network’s hyper-parameters are optimized by a brand-new dynamic graph cut algorithm method. Hyper-parameters emphasize the proper balance between exploration and exploitation by combining the wolves’ individual haunting tactics with their global haunting strategies to generate a neighborhood-based searching strategy. The fundamental objective of this study is to develop a hyper-parameter-optimized Full resolution convolutional network-based model capable of reliably diagnosing skin cancer types using dermoscopy images. The computer-aided diagnosis could be more efficient and precise. The segmentation approach is the primary way to identify cancerous tumors with precision. This study introduces a dynamic graph cut algorithm -based method for accurate segmentation and improved skin cancer classification using a full resolution convolutional network. Experiments reveal that the proposed model effectively addresses the frequent over-segmentation and under-segmentation issues in graph cut and the subject of wrongly segmented small sections in the grab cut method. In addition, the results illustrate the utility of data augmentation in training and testing, with enhanced performance compared to the usage of fresh images. Multiple experiments were done using various transferring models, and the results of our recommended model showed superior performance in skin lesion categorization tasks relative to other architectures with an accuracy of 97.986%.

SMNet: Symmetric Multi-Task Network for Semantic Change Detection in Remote Sensing Images Based on CNN and Transformer

Deep learning has achieved great success in remote sensing image change detection (CD). However, most methods focus only on the changed regions of images and cannot accurately identify their detailed semantic categories. In addition, most CD methods using convolutional neural networks (CNN) have difficulty capturing sufficient global information from images. To address the above issues, we propose a novel symmetric multi-task network (SMNet) that integrates global and local information for semantic change detection (SCD) in this paper. Specifically, we employ a hybrid unit consisting of pre-activated residual blocks (PR) and transformation blocks (TB) to construct the (PRTB) backbone, which obtains more abundant semantic features with local and global information from bi-temporal images. To accurately capture fine-grained changes, the multi-content fusion module (MCFM) is introduced, which effectively enhances change features by distinguishing foreground and background information in complex scenes. In the meantime, the multi-task prediction branches are adopted, and the multi-task loss function is used to jointly supervise model training to improve the performance of the network. Extensive experimental results on the challenging SECOND and Landsat-SCD datasets, demonstrate that our SMNet obtains 71.95% and 85.65% at mean Intersection over Union (mIoU), respectively. In addition, the proposed SMNet achieves 20.29% and 51.14% at Separated Kappa coefficient (Sek) on the SECOND and Landsat-SCD datasets, respectively. All of the above proves the effectiveness and superiority of the proposed method.

Self-Supervised Global–Local Contrastive Learning for Fine-Grained Change Detection in VHR Images

Self-supervised contrastive learning (CL) can learn high-quality feature representations that are beneficial to downstream tasks without labeled data. However, most CL methods are for image-level tasks. For the fine-grained change detection (FCD) tasks, such as change or change trend detection of some specific ground objects, it is usually necessary to perform pixel-level discriminative analysis. Therefore, feature representations learned by image-level CL may have limited effects on FCD. To address this problem, we propose a self-supervised global–local contrastive learning (GLCL) framework, which extends the instance discrimination task to the pixel level. GLCL follows the current mainstream CL paradigm and consists of four parts, including data augmentation to generate different views of the input, an encoder network for feature extraction, a global CL head, and a local CL head to perform image-level and pixel-level instance discrimination tasks, respectively. Through GLCL, features belonging to different perspectives of the same instance will be pulled closer, while features of different instances will be alienated, which can enhance the discriminativeness of feature representations from both global and local perspectives, thereby facilitating downstream FCD tasks. In addition, GLCL makes a targeted structural adaptation to FCD, i.e., the encoder network is undertaken by the common backbone networks of FCD, which can accelerate the deployment on downstream FCD tasks. Experimental results on several real datasets show that compared with other parameter initialization methods, the FCD models pretrained by GLCL can obtain better detection performance.

Viewpoint-Adaptive Representation Disentanglement Network for Change Captioning.

Change captioning is to describe the fine-grained change between a pair of images. The pseudo changes caused by viewpoint changes are the most typical distractors in this task, because they lead to the feature perturbation and shift for the same objects and thus overwhelm the real change representation. In this paper, we propose a viewpoint-adaptive representation disentanglement network to distinguish real and pseudo changes, and explicitly capture the features of change to generate accurate captions. Concretely, a position-embedded representation learning is devised to facilitate the model in adapting to viewpoint changes via mining the intrinsic properties of two image representations and modeling their position information. To learn a reliable change representation for decoding into a natural language sentence, an unchanged representation disentanglement is designed to identify and disentangle the unchanged features between the two position-embedded representations. Extensive experiments show that the proposed method achieves the state-of-the-art performance on the four public datasets. The code is available at https://github.com/tuyunbin/VARD.

Neighborhood Contrastive Transformer for Change Captioning

Change captioning is to describe the semantic change between a pair of similar images in natural language. It is more challenging than general image captioning, because it requires capturing fine-grained change information while being immune to irrelevant viewpoint changes, and solving syntax ambiguity in change descriptions. In this paper, we propose a neighborhood contrastive transformer to improve the model's perceiving ability for various changes under different scenes and cognition ability for complex syntax structure. Concretely, we first design a neighboring feature aggregating to integrate neighboring context into each feature, which helps quickly locate the inconspicuous changes under the guidance of conspicuous referents. Then, we devise a common feature distilling to compare two images at neighborhood level and extract common properties from each image, so as to learn effective contrastive information between them. Finally, we introduce the explicit dependencies between words to calibrate the transformer decoder, which helps better understand complex syntax structure during training. Extensive experimental results demonstrate that the proposed method achieves the state-of-the-art performance on three public datasets with different change scenarios. The code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/tuyunbin/NCT</uri> .

Changes in Cultivated Land Loss and Landscape Fragmentation in China from 2000 to 2020

Cultivated land area and fragmentation are two crucial elements that influence food production in China. However, few studies have analyzed the fine-grained changes in both the area and the fragmentation of cultivated land from a national perspective. Using a 30 m annual land-cover dataset with the help of cultivated land area (CA) and a newly constructed coupling landscape fragmentation index (CLFI), this research identified the spatial distribution characteristics of both the area and the landscape fragmentation of cultivated land in Chinese counties and simultaneously analyzed the changes in the above two elements since the beginning of the 21st century. The results showed that the distribution pattern of CA in China’s counties was large in the north and small in the south. During the first two decades of the 21st century, the total CA of China decreased, but the decrease rate decreased from 2.91% in 2000–2010 to 0.41% in 2010–2020; in contrast, the CLFI of China showed an increasing trend, but the increase rate also decreased, from 4.06% in 2000–2010 to 0.51% in 2010–2020. Spatially, from 2000 to 2010, there was an obvious reverse distribution pattern between counties with large CA changes and those with large CLFI changes; however, this pattern became less obvious from 2010 to 2020. Urban expansion, sloping land conversion programs and other ecological protection programs, and land development and consolidation programs were major causes of changes in the CA and CLFI across China. The findings of this research provide a data basis and scientific support for the protection and large-scale utilization of cultivated land, where this information is necessary to ensure food security.

Fast and robust active camera relocalization in the wild for fine-grained change detection

Active camera relocalization (ACR) is an important and challenging task, whose feasibility and success highly depend on illumination consistency and convergence speed. If under varied lighting conditions in outdoor scenes, however, both the convergence and accuracy of ACR cannot be guaranteed. In this paper, we propose a fast and robust ACR scheme, namely rACR, that works well under highly varied illuminations. To achieve robustness to lighting variations, rather than using 2D feature matching, we rely on 3D point clouds, acquired by a visual SLAM engine (VSE), to register the current and reference camera coordinate frames. We present a scale-aware point cloud matching function that is minimized by a two-stage coarse-to-fine method, i.e., fast alignment considering only geometric error at first, followed by fine-grained alignment optimizing both geometric, photometric errors and the poses of VSE keyframes. The two aligned point clouds with equalized scales help to bridge current and reference observations, avoiding 2D feature matching that are sensitive to large lighting variances, and can directly generate effective camera pose adjustments. Moreover, to achieve fast convergence speed, we implement the above algorithm with a parallel scheme, which is specifically composed of an initialization procedure and three parallel threads, i.e., VSE thread, pose alignment thread, and pose adjustment thread. Extensive experiments show that, rACR has much higher robustness to lighting variations and 5× faster convergence rate over state-of-the-art methods, thus significantly improves its feasibility in real-world fine-grained change detection tasks in the wild.

Learning Token-Aligned Representations With Multimodel Transformers for Different-Resolution Change Detection

Different-resolution change detection (DRCD) is now becoming an urgent problem to be solved, which is of great potential in rapid monitoring, such as disaster assessment, urban expansion, etc. In DRCD tasks, bi-temporal inputs are given in the form of different resolutions, thus conventional CD methods cannot be applied directly. Previous studies have attempted to deal with this problem by reconstructing the low-resolution (LR) image into a high-resolution (HR) one, including interpolation and super-resolution (SR). However, these solutions are limited by the availability of training data, making it hard to meet different kinds of needs. Besides, these image-level strategies have also ignored the interaction and alignment of high-level features. Therefore, we propose a new approach based on multi-model Transformers (MM-Trans), which solves the resolution gaps of bi-temporal inputs in DRCD tasks from the perspective of feature alignment. In the MM-Trans, a weight-unshared feature extractor is first utilized to precisely capture the features of the different-resolution inputs; then a spatial-aligned Transformer (sp-Trans) is introduced to align the LR-image features to the same size of the HR-image ones, which can be optimized in a learnable way by an auxiliary token loss; after that, a semantic-aligned Transformer (se-Trans) is adopted, in which the bi-temporal features can be further interacted and aligned semantically; finally, a prediction head is employed to obtain fine-grained change results. Experiments conducted on three common CD datasets, CDD, S2Looking, and HTCD dataset, have shown the advancement of the MM-Trans and fully demonstrated its potential in DSCD tasks.

An intelligent in-situ visual inspection system for preventive conservation of cultural heritages

Preventive conservation is an advanced protection scheme that is widely recognized and constantly pursued by the worldwide conservation community of cultural heritages. Following the minimum interruption principle, preventive protection can effectively inhibit the deterioration of cultural relics by finely inspecting the symptoms, inferring the causes of diseases, and accurately controlling relevant pathogenesis factors. Reliable inspection of symptoms is the starting point and basis of preventive conservation. At present, many technologies and data have been accumulated for environmental factors. Despite the plenty techniques and accumulated data for the perception of environmental factors, the feasible, accurate, and reliable in-situ inspection of the status of cultural heritages still remains an open problem due to slow changing speed, subtle change scale, and versatile occurrence environments, which seriously hinders the development and promotion of preventive protection. In this paper, we present the first intelligent in-situ visual inspection system for the preventive conservation of cultural heritages. By physically reproducing the cameras 6-DoF pose, effective image-guided detection of fine-grained changes for cultural heritages is realized. Till now, this system has been used in more than 10 cultural heritage sites in China, such as the Dunhuang Mogao Grottoes, the Imperial Palace, and the Summer Palace.

The role of governmental weapons procurements in forecasting monthly fatalities in intrastate conflicts: a semiparametric hierarchical hurdle model

ABSTRACT Accurate and interpretable forecasting models predicting spatially and temporally fine-grained changes in the numbers of intrastate conflict casualties are of crucial importance for policymakers and international non-governmental organizations (NGOs). Using a count data approach, we propose a hierarchical hurdle regression model to address the corresponding prediction challenge at the monthly PRIO-grid level. More precisely, we model the intensity of local armed conflict at a specific point in time as a three-stage process. Stages one and two of our approach estimate whether we will observe any casualties at the country- and grid-cell-level, respectively, while stage three applies a regression model for truncated data to predict the number of such fatalities conditional upon the previous two stages. Within this modeling framework, we focus on the role of governmental arms imports as a processual factor allowing governments to intensify or deter from fighting. We further argue that a grid cell’s geographic remoteness is bound to moderate the effects of these military buildups. Out-of-sample predictions corroborate the effectiveness of our parsimonious and theory-driven model, which enables full transparency combined with accuracy in the forecasting process.

Robust Damage Estimation of Typhoon Goni on Coconut Crops with Sentinel-2 Imagery

Typhoon Goni crossed several provinces in the Philippines where agriculture has high socioeconomic importance, including the top-3 provinces in terms of planted coconut trees. We have used a computational model to infer coconut tree density from satellite images before and after the typhoon’s passage, and in this way estimate the number of damaged trees. Our area of study around the typhoon’s path covers 15.7 Mha, and includes 47 of the 87 provinces in the Philippines. In validation areas our model predicts coconut tree density with a Mean Absolute Error of 5.9 Trees/ha. In Camarines Sur we estimated that 3.5 M of the 4.6 M existing coconut trees were damaged by the typhoon. Overall we estimated that 14.1 M coconut trees were affected by the typhoon inside our area of study. Our validation images confirm that trees are rarely uprooted and damages are largely due to reduced canopy cover of standing trees. On validation areas, our model was able to detect affected coconut trees with 88.6% accuracy, 75% precision and 90% recall. Our method delivers spatially fine-grained change maps for coconut plantations in the area of study, including unchanged, damaged and new trees. Beyond immediate damage assessment, gradual changes in coconut density may serve as a proxy for future changes in yield.