Global Semantics Research Articles

Continuous frame motion sensitive self-supervised collaborative network for video representation learning

Motion, as a feature of video that changes in temporal sequences, is crucial to visual understanding. The powerful video representation and extraction models are typically able to focus attention on motion features in challenging dynamic environments to complete more complex video understanding tasks. However, previous approaches discriminate mainly based on similar features in the spatial or temporal domain, ignoring the interdependence of consecutive video frames. In this paper, we propose the motion sensitive self-supervised collaborative network, a video representation learning framework that exploits a pretext task to assist feature comparison and strengthen the spatiotemporal discrimination power of the model. Specifically, we first propose the motion-aware module, which extracts consecutive motion features from the spatial regions by frame difference. The global–local contrastive module is then introduced, with context and enhanced video snippets being defined as appropriate positive samples for a broader feature similarity comparison. Finally, we introduce the snippet operation prediction module, which further assists contrastive learning to obtain more reliable global semantics by sensing changes in continuous frame features. Experimental results demonstrate that our work can effectively extract robust motion features and achieve competitive performance compared with other state-of-the-art self-supervised methods on downstream action recognition and video retrieval tasks.

Global-and-Local Collaborative Learning for Co-Salient Object Detection.

The goal of co-salient object detection (CoSOD) is to discover salient objects that commonly appear in a query group containing two or more relevant images. Therefore, how to effectively extract interimage correspondence is crucial for the CoSOD task. In this article, we propose a global-and-local collaborative learning (GLNet) architecture, which includes a global correspondence modeling (GCM) and a local correspondence modeling (LCM) to capture the comprehensive interimage corresponding relationship among different images from the global and local perspectives. First, we treat different images as different time slices and use 3-D convolution to integrate all intrafeatures intuitively, which can more fully extract the global group semantics. Second, we design a pairwise correlation transformation (PCT) to explore similarity correspondence between pairwise images and combine the multiple local pairwise correspondences to generate the local interimage relationship. Third, the interimage relationships of the GCM and LCM are integrated through a global-and-local correspondence aggregation (GLA) module to explore more comprehensive interimage collaboration cues. Finally, the intra and inter features are adaptively integrated by an intra-and-inter weighting fusion (AEWF) module to learn co-saliency features and predict the co-saliency map. The proposed GLNet is evaluated on three prevailing CoSOD benchmark datasets, demonstrating that our model trained on a small dataset (about 3k images) still outperforms 11 state-of-the-art competitors trained on some large datasets (about 8k-200k images).

An Online Hashing Algorithm for Image Retrieval Based on Optical-Sensor Network.

Online hashing is a valid storage and online retrieval scheme, which is meeting the rapid increase in data in the optical-sensor network and the real-time processing needs of users in the era of big data. Existing online-hashing algorithms rely on data tags excessively to construct the hash function, and ignore the mining of the structural features of the data itself, resulting in a serious loss of the image-streaming features and the reduction in retrieval accuracy. In this paper, an online hashing model that fuses global and local dual semantics is proposed. First, to preserve the local features of the streaming data, an anchor hash model, which is based on the idea of manifold learning, is constructed. Second, a global similarity matrix, which is used to constrain hash codes is built by the balanced similarity between the newly arrived data and previous data, which makes hash codes retain global data features as much as possible. Then, under a unified framework, an online hash model that integrates global and local dual semantics is learned, and an effective discrete binary-optimization solution is proposed. A large number of experiments on three datasets, including CIFAR10, MNIST and Places205, show that our proposed algorithm improves the efficiency of image retrieval effectively, compared with several existing advanced online-hashing algorithms.

Bishift Networks for Thick Cloud Removal with Multitemporal Remote Sensing Images

Because of the presence of clouds, the available information in optical remote sensing images is greatly reduced. These temporal-based methods are widely used for cloud removal. However, the temporal differences in multitemporal images have consistently been a challenge for these types of methods. Towards this end, a bishift network (BSN) model is proposed to remove thick clouds from optical remote sensing images. As its name implies, BSN is combined of two dependent shifts. Moment matching (MM) and deep style transfer (DST) are the first shift to preliminarily eliminate temporal differences in multitemporal images. In the second shift, an improved shift net is proposed to reconstruct missing information under cloud covers. It introduces multiscale feature connectivity with shift connections and depthwise separable convolution (DSC), which can capture local details and global semantics effectively. Through experiments with Sentinel-2 images, it has been demonstrated that the proposed BSN has great advantages over traditional methods and state-of-the-art methods in cloud removal.

Adaptive Text Denoising Network for Image Caption Editing

Image caption editing, which aims at editing the inaccurate descriptions of the images, is an interdisciplinary task of computer vision and natural language processing. As the task requires encoding the image and its corresponding inaccurate caption simultaneously and decoding to generate an accurate image caption, the encoder-decoder framework is widely adopted for image caption editing. However, existing methods mostly focus on the decoder, yet ignore a big challenge on the encoder: the semantic inconsistency between image and caption. To this end, we propose a novel A daptive T ext D enoising Net work (ATD-Net) to filter out noises at the word level and improve the model’s robustness at sentence level. Specifically, at the word level, we design a cross-attention mechanism called Textual Attention Mechanism (TAM), to differentiate the misdescriptive words. The TAM is designed to encode the inaccurate caption word by word based on the content of both image and caption. At the sentence level, in order to minimize the influence of misdescriptive words on the semantic of an entire caption, we introduce a Bidirectional Encoder to extract the correct semantic representation from the raw caption. The Bidirectional Encoder is able to model the global semantics of the raw caption, which enhances the robustness of the framework. We extensively evaluate our proposals on the MS-COCO image captioning dataset and prove the effectiveness of our method when compared with the state-of-the-arts.

Human Co-Parsing Guided Alignment for Occluded Person Re-identification.

Occluded person re-identification (ReID) is a challenging task due to more background noises and incomplete foreground information. Although existing human parsing-based ReID methods can tackle this problem with semantic alignment at the finest pixel level, their performance is heavily affected by the human parsing model. Most supervised methods propose to train an extra human parsing model aside from the ReID model with cross-domain human parts annotation, suffering from expensive annotation cost and domain gap; Unsupervised methods integrate a feature clustering-based human parsing process into the ReID model, but lacking supervision signals brings less satisfactory segmentation results. In this paper, we argue that the pre-existing information in the ReID training dataset can be directly used as supervision signals to train the human parsing model without any extra annotation. By integrating a weakly supervised human co-parsing network into the ReID network, we propose a novel framework that exploits shared information across different images of the same pedestrian, called the Human Co-parsing Guided Alignment (HCGA) framework. Specifically, the human co-parsing network is weakly supervised by three consistency criteria, namely global semantics, local space, and background. By feeding the semantic information and deep features from the person ReID network into the guided alignment module, features of the foreground and human parts can then be obtained for effective occluded person ReID. Experiment results on two occluded and two holistic datasets demonstrate the superiority of our method. Especially on Occluded-DukeMTMC, it achieves 70.2% Rank-1 accuracy and 57.5% mAP.

A Dual-Path Multi-Scale Feature Fusion Decoder for SegFormer

The encoder-decoder structure is the basic structure of most semantic segmentation models and is adopted by a large number of segmentation models. How to effectively extract image features and achieve high-precision mapping through the optimal design of encoder and decoder is the key issue of current research. SegFormer designs an encoder with excellent performance, which fully extracts the feature information of different semantic granularity in the image with a large receptive field. Even if a simple fully connected layer decoder is used, excellent segmentation results can also be achieved. However, this simplified decoder does not make full use of the advantages of the SegFormer encoder. Therefore, a decoder structure with dual-path multi-scale feature fusion is designed in this paper, and the decoder is redesigned according to the characteristics of the SegFormer encoder. The decoder adopts a dual-path structure, one path passes the abstract global information layer by layer to the local detail information through the layer-by-layer upsampling fusion module (LFM), and gradually upsamples the feature maps obtained from the encoder, and then use the channel fusion module to learn the importance of different channels in the deep abstract semantic feature map and the shallow local detail feature map, and perform dynamic fusion to obtain a feature map containing both abstract semantic information and local details. The other path takes advantage of the large receptive field of the feature map output by the SegFormer encoder, and uses the weighted hybrid multi-scale feature extraction module (WMF) to extract multi-scale features containing global semantics from the deep semantic feature map finally output by the encoder. Finally, the Deep Feature Fusion Module (DFM) is used to fuse the outputs of the first two modules, fully mining the multi-scale global information in the encoder, and obtaine the feature maps with rich semantic information, which effectively improves the algorithm model performance.

An Unsupervised Domain Adaptation Model Based on Multi-Level Joint Alignment for Multi-Modal Cardiac Image Segmentation

Unsupervised Domain Adaptation has greatly boosted the performance of multi-modal medical segmentation when there are only source domain labels and no labels in the target domain. Many previous work relies on Convolutional Neural Networks for distribution or instance alignment, however, the receptive field of CNN makes it overly concerned with texture and local semantic information of images, while losing the global style information and semantic information and lacking relationships between local and global semantics. In order to address these issues, we propose a two-stage, multi-level framework for unsupervised domain adaptation, which consists of an image translation network and a Transformer-based domain adaptation segmentation network, jointly aligning the data distribution in the source and target domains from the image, feature and output level through adversarial learning. Experimental results indicate that our method can achieve satisfactory results and outperform other state-of-the-art medical image segmentation methods.

VCGAN: Video Colorization With Hybrid Generative Adversarial Network

We propose a hybrid recurrent Video Colorization with Hybrid Generative Adversarial Network (VCGAN), an improved approach to video colorization using end-to-end learning. The VCGAN addresses two prevalent issues in the video colorization domain: Temporal consistency and unification of colorization network and refinement network into a single architecture. To enhance colorization quality and spatiotemporal consistency, the mainstream of generator in VCGAN is assisted by two additional networks, i.e., global feature extractor and placeholder feature extractor, respectively. The global feature extractor encodes the global semantics of grayscale input to enhance colorization quality, whereas the placeholder feature extractor acts as a feedback connection to encode the semantics of the previous colorized frame in order to maintain spatiotemporal consistency. If changing the input for placeholder feature extractor as grayscale input, the hybrid VCGAN also has the potential to perform image colorization. To improve the consistency of far frames, we propose a dense long-term loss that smooths the temporal disparity of every two remote frames. Trained with colorization and temporal losses jointly, VCGAN strikes a good balance between color vividness and video continuity. Experimental results demonstrate that VCGAN produces higher-quality and temporally more consistent colorful videos than existing approaches.

Основные языковые источники современной терминологии

Frequently encountered sources and methods of term formation in the Russian language are presented in the article. The main forms for their construction are basic terms, generalscientific, general-technical, and global semantics terms. The article presents the importance of the language system and forms, which have a great impact on the creation and formation of terminology.

WaterHRNet: A multibranch hierarchical attentive network for water body extraction with remote sensing images

• Multi-resolution feature extractor for strong-semantic feature encoding. • Dual-attention module for semantic-relevant feature promotion. • Semantic enhancement principle for target-oriented feature representation. • Hierarchical attentive network for high-quality water body extraction. Water is a kind of vital natural resource, which acts as the lifeblood of the ecosystem and the energy source for the living and production activities of humans. Regularly mapping the conditions of water resources and taking effective measures to prevent them from pollutions and shortages are very important and necessary to maintain the sustainability of the ecosystem. As a preliminary step for image-based water resource analysis, the complete recognition and accurate extraction of water bodies are important prerequisites in many applications. Nevertheless, due to the issues of topology diversities, appearance variabilities, and land cover interferences, there is still a large gap to achieve the human-level water bodies interpretation quality. This paper presents a hierarchical attentive high-resolution network, abbreviated as WaterHRNet, for extracting water bodies from remote sensing imagery. First, by building a multibranch high-resolution feature extractor integrated with global feature semantics aggregation, the WaterHRNet behaves laudably to supply high-quality, strong-semantic feature representations. Furthermore, by inlaying an effective feature attention scheme with the comprehensive exploitation of both the spatial and channel feature significances, the WaterHRNet is forced to strengthen the semantic-determinate, task-aware feature encodings. In addition, by designing a hierarchical processing principle with the progressive enhancement of category-attentive feature semantics, the WaterHRNet performs effectively to export semantic-discriminative, target-oriented feature representations for precise water body segmentation. The WaterHRNet is elaborately verified both quantitatively and qualitatively on three remote sensing datasets. Evaluation results show that the WaterHRNet achieves an average precision of 98.44%, average recall of 97.84%, average IoU of 96.35%, and average F 1 -score of 98.14%. Comparative analyses also demonstrate the superior performance and excellent feasibility of the WaterHRNet in segmenting water bodies.

Components.js: Semantic dependency injection

A common practice within object-oriented software is using composition to realize complex object behavior in a reusable way. Such compositions can be managed by Dependency Injection (DI), a popular technique in which components only depend on minimal interfaces and have their concrete dependencies passed into them. Instead of requiring program code, this separation enables describing the desired instantiations in declarative configuration files, such that objects can be wired together automatically at runtime. Configurations for existing DI frameworks typically only have local semantics, which limits their usage in other contexts. Yet some cases require configurations outside of their local scope, such as for the reproducibility of experiments, static program analysis, and semantic workflows. As such, there is a need for globally interoperable, addressable, and discoverable configurations, which can be achieved by leveraging Linked Data. We created Components.js as an open-source semantic DI framework for TypeScript and JavaScript applications, providing global semantics via Linked Data-based configuration files. In this article, we report on the Components.js framework by explaining its architecture and configuration, and discuss its impact by mentioning where and how applications use it. We show that Components.js is a stable framework that has seen significant uptake during the last couple of years. We recommend it for software projects that require high flexibility, configuration without code changes, sharing configurations with others, or applying these configurations in other contexts such as experimentation or static program analysis. We anticipate that Components.js will continue driving concrete research and development projects that require high degrees of customization to facilitate experimentation and testing, including the Comunica query engine and the Community Solid Server for decentralized data publication.

Transformer-Based Model with Dynamic Attention Pyramid Head for Semantic Segmentation of VHR Remote Sensing Imagery.

Convolutional neural networks have long dominated semantic segmentation of very-high-resolution (VHR) remote sensing (RS) images. However, restricted by the fixed receptive field of convolution operation, convolution-based models cannot directly obtain contextual information. Meanwhile, Swin Transformer possesses great potential in modeling long-range dependencies. Nevertheless, Swin Transformer breaks images into patches that are single-dimension sequences without considering the position loss problem inside patches. Therefore, Inspired by Swin Transformer and Unet, we propose SUD-Net (Swin transformer-based Unet-like with Dynamic attention pyramid head Network), a new U-shaped architecture composed of Swin Transformer blocks and convolution layers simultaneously through a dual encoder and an upsampling decoder with a Dynamic Attention Pyramid Head (DAPH) attached to the backbone. First, we propose a dual encoder structure combining Swin Transformer blocks and reslayers in reverse order to complement global semantics with detailed representations. Second, aiming at the spatial loss problem inside each patch, we design a Multi-Path Fusion Model (MPFM) with specially devised Patch Attention (PA) to encode position information of patches and adaptively fuse features of different scales through attention mechanisms. Third, a Dynamic Attention Pyramid Head is constructed with deformable convolution to dynamically aggregate effective and important semantic information. SUD-Net achieves exceptional results on ISPRS Potsdam and Vaihingen datasets with 92.51%mF1, 86.4%mIoU, 92.98%OA, 89.49%mF1, 81.26%mIoU, and 90.95%OA, respectively.

Triplet Contrastive Learning for Aspect Level Sentiment Classification

The domain of Aspect Level Sentiment Classification, in which the sentiment toward a given aspect is analyzed, attracts much attention in NLP. Recently, the state-of-the-art Aspect Level Sentiment Classification methods are devised by using the Graph Convolutional Networks to deal with both the semantics and the syntax of the sentence. Generally, the parsing of syntactic structure inevitably incorporates irrelevant information toward the aspect. Besides, the syntactic and semantic alignment and uniformity that contribute to the sentiment delivery is currently neglected during processing. In this work, a Triplet Contrastive Learning Network is developed to coordinate the syntactic information and the semantic information. To start with, the aspect-oriented sub-tree is constructed to replace the syntactic adjacency matrix. Further, a sentence-level contrastive learning scheme is proposed to highlight the features of sentiment words. Based on The Triple Contrastive Learning, the syntactic information and the semantic information are thoroughly interacted and coordinated whilst the global semantics and syntax can be exploited. Extensive experiments are performed on three benchmark datasets and achieve accuracies (BERT-based) of 87.40, 82.80, 77.55 on Rest14, Lap14, and Twitter datasets, which demonstrate that our approach achieves state-of-the-art results in Aspect Level Sentiment Classification task.

IAF-LG: An Interactive Attention Fusion Network With Local and Global Perspective for Aspect-Based Sentiment Analysis

One of the interesting trending phenomena in sentiment analysis is the prediction of sentiment given by the user towards an aspect term. Till today, a considerable number of researchers have proposed varying methodologies for predicting aspect-based sentiments. But they mostly encapsulate the semantic information by manifesting themselves within a local boundary around each aspect term and overlook capturing the semantic concept that is conveyed within the entire review (global). Therefore, this study proposes a model, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">IAF-LG</i> , that performs semantic learning at both local and global scales to discover aspect-based sentiments. IAF-LG first encodes the local semantics by fusing contextual-semantic dependencies between tokens and computing relational semantics between inter-aspects. Next, it develops the global semantics by formulating interactions between local semantics and review-based sentiment learning. Lastly, it conjoins the local and global interactive learning to earn credible semantics for predicting the accurate sentiment of aspect terms. Extensive experiments on publicly available datasets demonstrate the significantly improved performance of IAF-LG than competitive baselines.

Further Exploration of Deep Aggregation for Shadow Detection

Shadow detection is a fundamental challenge in the field of computer vision. It requires the network to understand the global semantics and local details of the image. All existing methods depend on the aggregation of the features of a multi-stage pre-trained convolution neural network but in comparison to high-level capabilities, low-level capabilities provide less to the detection performance. Using low-level features not only increases the complex difficulty of the network but also reduces the time efficiency. In this article, we propose a new shadow detector, which only uses high-level features and explores the complementary information between adjacent feature layers. Experiments show that the technique in this paper can accurately detect shadows and perform well compared with the most advanced methods. The detailed experiments performed in three public shadow detection datasets SUB, UCF, and ISTD demonstrate that the suggested method is efficient and stable.

Subgraph-based feature fusion models for semantic similarity computation in heterogeneous knowledge graphs

Semantic similarity is a fundamental task in natural language processing that determines the similarity between two concepts within a taxonomy. For example, a pair of words (e.g., car and bike) appear similar because they share the same category (e.g., vehicle). Numerous computation methods, such as distance-based and feature-based approaches, are proposed to precisely depict this similarity. As knowledge graphs become heterogeneous (e.g., DBpedia), existing methods have limitations on utilizing multi-view features (e.g., abstract, structure, and categories). On the one hand, some features are incomplete for various reasons, reducing the effectiveness of embedding methods. On the other hand, the hidden connections among multi-view features are omitted by existing approaches. To address the problems mentioned above, we first extract three subgraphs from a heterogeneous knowledge graph and then combine various embedding approaches to capture the global semantics of each concept. Next, we offer subgraph-based feature fusion models that improve concept representation by fusing multi-view features. Finally, we devise mixed computation methods to calculate the semantic similarity between the two concepts. Experiment results show that multi-view features, particularly the abstract feature, can effectively improve the performance of the proposed methods. Compared to existing approaches, our methods significantly improve the Pearson correlation coefficient by about 7%. The source code of this paper is available at: https://github.com/fiego/SubgraphSS.

Self-supervised graph representation learning via positive mining

Inspired by the recent success of self-supervised methods applied on images, self-supervised learning on graph structured data has seen rapid growth especially centered on augmentation-based contrastive methods. However, these methods pay little attention to the inherent distinction between images and graphs: while augmentation is well defined on images, it may behave arbitrarily on graphs. That is, without carefully designed augmentation techniques, augmentations on graphs may behave arbitrarily in that the underlying semantics of graphs can drastically change. As a consequence, the performance of existing augmentation-based methods is highly dependent on the choice of augmentation scheme, i.e., hyperparameters associated with augmentations. In this paper, we propose a novel augmentation-free self-supervised learning method for graphs, named AFGRL, which learns graph representations without using any arbitrary augmentation scheme. Specifically, we generate an alternative view of a graph by discovering nodes that share the local structural information and the global semantics within the graph. Extensive experiments toward various node-level tasks (i.e., node classification, clustering, and similarity search) and graph-level tasks (i.e. graph classification) on various real-world datasets demonstrate the superiority of AFGRL.

Semantic-aware network embedding via optimized random walk and paragaraph2vec

The Deepwalk-based network embedding algorithm, which was inspired by the word2vec model, can quickly and efficiently learn a low-dimensional vector representation for each node in a complex network. However, the success of word2vec in natural language environments relies heavily on ubiquitous contextual semantics, and the Deepwalk model is less accurate than other deep models due to ignoring contextual semantics in complex networks. To solve this challenge, a semantic-aware network embedding algorithm is proposed in this paper, simply called SANE, via optimized random walk and paragaraph2vec. In the SANE, a semantic-aware random walk strategy is designed to unsupervised generate many semantic-bearing node sequences from a complex network as a training set. Then, based on the training set, the paragraph2vec model is optimized by considering both global semantics and local semantics for learning more accurate low-dimensional node feature representations. Experiments on multiple tasks and multiple networks show the advantages of our SANE algorithm.

Global Types and Event Structure Semantics for Asynchronous Multiparty Sessions

We propose an interpretation of multiparty sessions with asynchronous communication as Flow Event Structures. We introduce a new notion of global type for asynchronous multiparty sessions, ensuring the expected properties for sessions, including progress. Our global types, which reflect asynchrony more directly than standard global types and are more permissive, are themselves interpreted as Prime Event Structures. The main result is that the Event Structure interpretation of a session is equivalent, when the session is typable, to the Event Structure interpretation of its global type.