Unified Semantic Space Research Articles

CFNet: Automatic multi-modal brain tumor segmentation through hierarchical coarse-to-fine fusion and feature communication

Automatic segmentation of brain tumors employing images from multi-modalities is important for preoperative diagnosis and prognostic assessment. The rich complementary information contained within multi-modal images allows for improved brain tumor segmentation performance when models are trained on multi-modal data. However, accurate segmentation of small lesion regions from medical images remains challenging, due to the irregular shapes and low boundary contrast of brain gliomas. To address these challenges, we propose a coarse-to-fine feature fusion network (CFNet) that effectively incorporates multi-modal image features through modal interaction, semantic perception, and feature fusion. Specifically, the modality cross-attention fusion module is proposed introducing complementary features and mapping them into a unified semantic space to learn complementary representations between modalities. Furthermore, the multi-scale context perception module enables the model to focus on multi-scale lesion fine-grained information while capturing deep semantics. Moreover, the further fusion of fine-grained features with coarse-grained features enhances edge features and the complementary of features. We evaluate the proposed CFNet on two public brain tumor segmentation datasets BraTS2019 and BraTS2020. Experimental results show that the proposed framework outperforms state-of-the-art methods in Dice score, HD95, Sensitivity, demonstrating the effectiveness of CFNet on brain tumor segmentation. Code will be available at https://github.com/YaruC/CFNet.

Unifying Visual and Vision-Language Tracking via Contrastive Learning

Single object tracking aims to locate the target object in a video sequence according to the state specified by different modal references, including the initial bounding box (BBOX), natural language (NL), or both (NL+BBOX). Due to the gap between different modalities, most existing trackers are designed for single or partial of these reference settings and overspecialize on the specific modality. Differently, we present a unified tracker called UVLTrack, which can simultaneously handle all three reference settings (BBOX, NL, NL+BBOX) with the same parameters. The proposed UVLTrack enjoys several merits. First, we design a modality-unified feature extractor for joint visual and language feature learning and propose a multi-modal contrastive loss to align the visual and language features into a unified semantic space. Second, a modality-adaptive box head is proposed, which makes full use of the target reference to mine ever-changing scenario features dynamically from video contexts and distinguish the target in a contrastive way, enabling robust performance in different reference settings. Extensive experimental results demonstrate that UVLTrack achieves promising performance on seven visual tracking datasets, three vision-language tracking datasets, and three visual grounding datasets. Codes and models will be open-sourced at https://github.com/OpenSpaceAI/UVLTrack.

Hierarchical visual-semantic interaction for scene text recognition

Proper interaction between visual and semantic features is crucial to obtain a powerful feature representation for scene text recognition (STR). The existing interaction methods usually treat visual and semantic features as distinct tokens and use transformers to learn contextual information and prior language knowledge, and they achieve promising performance for STR task. However, there still remain several issues needed to be further addressed such as the imbalance number and mis-alignment between visual and semantic features, and the necessarily of stacking transformers to a progressive improvement in accuracy. To this aim, this paper proposes a novel interaction manner namely hierarchical visual-semantic interaction (HVSI) which contains three novel modules including a hierarchical visual-semantic interaction module, fusion module, and visual-semantic alignment module. The hierarchical visual-semantic interaction module employs multiple visual-semantic interaction blocks in various scales to enhance the representation power of visual features and semantic features. To better exploit multi-scale visual and semantic features, the fusion module is introduced to fuse multiple semantic features based on attention mechanisms. Furthermore, our HVSI presents a simple plug-in block named visual-semantic alignment module to alleviate mis-alignment of semantic features by mapping them into a unified semantic space, which helps improve the performance of HVSI. Extensive experiments on multiple benchmarks including English and Chinese text recognition datasets show that our method obtains state-of-the-art or competitive performances.

Solving Geometry Problems via Feature Learning and Contrastive Learning of Multimodal Data

This paper presents an end-to-end deep learning method to solve geometry problems via feature learning and contrastive learning of multimodal data. A key challenge in solving geometry problems using deep learning is to automatically adapt to the task of understanding single-modal and multimodal problems. Existing methods either focus on single-modal or multimodal problems, and they cannot fit each other. A general geometry problem solver should obviously be able to process various modal problems at the same time. In this paper, a shared feature-learning model of multimodal data is adopted to learn the unified feature representation of text and image, which can solve the heterogeneity issue between multimodal geometry problems. A contrastive learning model of multimodal data enhances the semantic relevance between multimodal features and maps them into a unified semantic space, which can effectively adapt to both single-modal and multimodal downstream tasks. Based on the feature extraction and fusion of multimodal data, a proposed geometry problem solver uses relation extraction, theorem reasoning, and problem solving to present solutions in a readable way. Experimental results show the effectiveness of the method.

Average Approximate Hashing-Based Double Projections Learning for Cross-Modal Retrieval.

Cross-modal retrieval has attracted considerable attention for searching in large-scale multimedia databases because of its efficiency and effectiveness. As a powerful tool of data analysis, matrix factorization is commonly used to learn hash codes for cross-modal retrieval, but there are still many shortcomings. First, most of these methods only focus on preserving locality of data but they ignore other factors such as preserving reconstruction residual of data during matrix factorization. Second, the energy loss of data is not considered when the data of cross-modal are projected into a common semantic space. Third, the data of cross-modal are directly projected into a unified semantic space which is not reasonable since the data from different modalities have different properties. This article proposes a novel method called average approximate hashing (AAH) to address these problems by: 1) integrating the locality and residual preservation into a graph embedding framework by using the label information; 2) projecting data from different modalities into different semantic spaces and then making the two spaces approximate to each other so that a unified hash code can be obtained; and 3) introducing a principal component analysis (PCA)-like projection matrix into the graph embedding framework to guarantee that the projected data can preserve the main energy of data. AAH obtains the final hash codes by using an average approximate strategy, that is, using the mean of projected data of different modalities as the hash codes. Experiments on standard databases show that the proposed AAH outperforms several state-of-the-art cross-modal hashing methods.

Deep objectness hashing using large weakly tagged photos

CNN-based hashing methods have greatly boosted the performance of image retrieval, under the strong supervision of large amounts of manually annotated labels. In recent years, a large number of social media images with user tags have been generated on the Internet. These images can be regarded as weakly labeled training data, which can provide rich samples for training hash network, and greatly reduce the cost of obtaining training data. However, there are noise and visual irrelevant tags in user tags, and different tags may describe different objects in the image. In the previous CNN-based hashing method, a training image usually corresponds to a manual label and generates a hash code. These methods are difficult to use the image described by user tags with noise. For solving the above problem, we propose a CNN-based objectness hash learning method using user tags as a guide for training. First of all, the user tags are roughly filtered to remove noise tags that are not related to the visual content of images. Secondly, we quantify user tags into a unified semantic space and extract the highest-frequency words of the semantic space from similar objectness areas as their labels. Then, these objectness areas with their labels are grouped into a series of triple units as training data. So that the generated hash code can inherit the semantic similarity of the objectness areas well, that is, the Hamming distance between hash codes generated by similar objectness areas is closer, the reverse is the farther. Experimental results on NUS-WIDE and Flickr datasets show that our method can effectively extract object-level semantic information from weak user tags, and improve the accuracy of image retrieval.

A Study on Cross-Media Teaching Model for College English Classroom Based on Output-Driven Hypothetical Neural Network.

In the field of education, the development of educational big data has become an important strategic choice to promote the construction of the digital campus and educational reform, and educational big data has become a new driving force in the field of education that cannot be ignored. Based on the theoretical basis of output-driven hypothesis neural network, and combining the media spanning of contemporary art and cross-media association effect, this study changes the status quo of English teaching through traditional methods such as grammar-translation method and deductive method and constructs a new cross-media university English teaching model. Based on the existing feature learning model of two-way attention, combined with existing techniques such as generative adversarial networks and semantic hashing, the semantic association between different media data is deeply mined, and feature learning is integrated with adversarial learning and hash learning to build a unified semantic space for different media data. In this paper, we focus on the structure and characteristics of convolutional neural networks through the study of deep learning theory, discuss three classical convolutional neural network models, such as AlexNet, VGG, and GoogLeNet, and propose a convolutional neural network model applicable to cross-media teaching in college English classroom and carry out experimental validation, and the results show that the proposed neural network model is based on output-driven hypothesis. The following research has been added to the abstract: to address the key problem of the semantic gap that is difficult to cross in cross-media semantic learning, a cross-media supervised adversarial hashing model based on two-way attentional features is proposed. Based on the existing two-way attention-based feature learning model, we combine existing techniques such as generative adversarial networks and semantic hashing to deeply explore the semantic association between different media data and integrate feature learning with adversarial learning and hashing to build a unified semantic space for different media data. The results show that the proposed neural network model of cross-media teaching in college English classrooms based on the output-driven hypothesis can not only promote the improvement of students' English literacy skills but also have a certain promotion effect on their overall performance improvement.

A Survey of Entity Alignment of Knowledge Graph Based on Embedded Representation

This paper summarizes the main methods of knowledge representation learning. Representation learning represents the entity information of the knowledge graph as a low dimensional vector. Its vector dimension is low, which helps to improve the computational efficiency and make full use of the semantic information between entities. In order to embed two KGs into a unified semantic space, joint embedding is used to achieve this goal. With the development of research, there are many improved embedding methods, such as iteration, multi view embedding, knowledge graph entity alignment based on graph neural network and so on.

Oracle Bone Inscriptions information processing based on multi-modal knowledge graph

To solve the problems of the great learning difficulty, the long learning period, wide range of knowledge points but weak knowledge connection, and low sharing of Oracle Bone Studies (OBS), a solution of constructing a multi-modal knowledge graph is proposed. Because OBS research involves various kinds of modal data, and these modalities need to be combined together to solve some problems. The OBS multi-modal knowledge graph can provide a unified semantic space for multi-source heterogeneous data. Through multi-modal fusion and information complementation, the defects of a single modality in information processing can be resolved. This multi-modal knowledge graph organizes and manages the basic data better to serve Oracle Bone Inscriptions (OBI) information processing research. Taking OBI detection and recognition as examples, we studied the applications of OBS multi-modal knowledge graph. The experimental results demonstrate that the proposed method reaches 81.3% accuracy in detection and 80.43% accuracy in recognition, and it has 3.7% in detection and 14.8% in recognition improved to the conventional methods.

A Multi-RNN Research Topic Prediction Model Based on Spatial Attention and Semantic Consistency-Based Scientific Influence Modeling.

Computer science discipline includes many research fields, which mutually influence and promote each other's development. This poses two great challenges of predicting the research topics of each research field. One is how to model fine-grained topic representation of a research field. The other is how to model research topic of different fields and keep the semantic consistency of research topics when learning the scientific influence context from other related fields. Unfortunately, the existing research topic prediction approaches cannot handle these two challenges. To solve these problems, we employ multiple different Recurrent Neural Network chains which model research topics of different fields and propose a research topic prediction model based on spatial attention and semantic consistency-based scientific influence modeling. Spatial attention is employed in field topic representation which can selectively extract the attributes from the field topics to distinguish the importance of field topic attributes. Semantic consistency-based scientific influence modeling maps research topics of different fields to a unified semantic space to obtain the scientific influence context of other related fields. Extensive experiment results on five related research fields in the computer science (CS) discipline show that the proposed model is superior to the most advanced methods and achieves good topic prediction performance.

Стихотворение И. Бунина «Айя-София»: архитектурный «космизм» в структуре лирического сюжета

The article is devoted to the poetics of the poem “Hagia Sophia” (1906) by I. A. Bunin in the aspect of the plot representation of artistic ontology. The ontological basis of I. Bunin’s poetry and prose is the cosmic worldview, which determines the specifics of the construction of his artistic universe. The formation of the ideology of “cosmism” in Bunin’s work occurs at the beginning of the 20 th century, when the poet intensively learns the experience of other religions and cultures. The feeling of being as a cosmic unity of the world order is affirmed in his “oriental” lyrics of 1903–1907. At the same time, the East appears as a unified semantic space that equalizes various religious and historical and cultural formations in terms of values. The poem “Hagia Sophia” is included in the corpus of “oriental” poems by the poet and reveals his creative perception of the Constantinople Cathedral of Hagia Sophia (mosque). Structural-semiotic analysis of this poetic text shows that its plot construction explicates the logic of Bunin’s “grasping” of the being essence of the temple space. In the process of unfolding the lyrical plot of the poem, the contemplated process of the “evening” Muslim worship, the “morning” solar pacification of the temple and the “dove” appeal to the life forces of the universe are combined in an integral system of the world order. The convergence of these vital manifestations of the universe in the space of the temple becomes the central event in the plot structure of the text, which explicates the lyric subject’s awareness of the cosmic unity of the created world. The reception of the lyrical subject of Hagia Sophia is aimed at understanding not so much the Islamic religious and cultural identity as the fullness of life that testifies to itself in the sacred space of the temple. The eternity of the cosmos, which appears in the architectural locus, is thought of as an axiological peak in self-determination on the axis “man – universe”. The unfolding of the text’s plot structure, which represents through the oppositions “evening – morning”, “darkness – light”, “speech – silence”, “man – dove” the events of striving for the transformation of being, confirms the idea of the ontological unity of the vital manifestations of the world order. It is concluded that in the poem “Hagia Sophia” by I. Bunin the architectural world of the temple embodies the ideology of “cosmism” as a convergence of anthropological and natural principles, revealing one of the aspects of the movement of Bunin’s poetics to the concept of “unified soul of the universe”.

Zero-shot learning with Multi-Battery Factor Analysis

Zero-shot learning (ZSL) extends the conventional image classification technique to a more challenging situation where the testing image categories are not seen in the training samples. Most studies on ZSL utilize side information such as attributes or word vectors to bridge the relations between the seen classes and the unseen classes. However, existing approaches on ZSL typically exploit a shared space for each type of side information independently, which cannot make full use of the complementary knowledge of different types of side information. To this end, this paper presents an MBFA-ZSL approach to embed different types of side information as well as the visual feature into one shared space. Specifically, we first develop an algorithm named Multi-Battery Factor Analysis (MBFA) to build a unified semantic space, and then employ multiple types of side information in it to achieve the ZSL. The close-form solution makes MBFA-ZSL simple to implement and efficient to run on large datasets. Extensive experiments on the popular AwA, CUB, and SUN datasets show its superiority over the state-of-the-art approaches.

Nasari: Integrating explicit knowledge and corpus statistics for a multilingual representation of concepts and entities

Owing to the need for a deep understanding of linguistic items, semantic representation is considered to be one of the fundamental components of several applications in Natural Language Processing and Artificial Intelligence. As a result, semantic representation has been one of the prominent research areas in lexical semantics over the past decades. However, due mainly to the lack of large sense-annotated corpora, most existing representation techniques are limited to the lexical level and thus cannot be effectively applied to individual word senses. In this paper we put forward a novel multilingual vector representation, called Nasari, which not only enables accurate representation of word senses in different languages, but it also provides two main advantages over existing approaches: (1) high coverage, including both concepts and named entities, (2) comparability across languages and linguistic levels (i.e., words, senses and concepts), thanks to the representation of linguistic items in a single unified semantic space and in a joint embedded space, respectively. Moreover, our representations are flexible, can be applied to multiple applications and are freely available at http://lcl.uniroma1.it/nasari/. As evaluation benchmark, we opted for four different tasks, namely, word similarity, sense clustering, domain labeling, and Word Sense Disambiguation, for each of which we report state-of-the-art performance on several standard datasets across different languages.

SOCIOLOGICAL UNDERSTANDING OF INTERNET: THEORETICAL APPROACHES TO THE NETWORK ANALYSIS

The network is an efficient way of social structure analysis for contemporary sociologists. It gives broad opportunities for detailed and fruitful research of different patterns of ties and social relations by quantitative analytical methods and visualization of network models. The network metaphor is used as the most representative tool for description of a new type of society. This new type is characterized by flexibility, decentralization and individualization. Network organizational form became the dominant form in modern societies. The network is also used as a mode of inquiry. Actually three theoretical network approaches in the Internet research case are the most relevant: social network analysis, “network society” theory and actor-network theory. Every theoretical approach has got its own notion of network. Their special methodological and theoretical features contribute to the Internet studies in different ways. The article represents a brief overview of these network approaches. This overview demonstrates the absence of a unified semantic space of the notion of “network” category. This fact, in turn, points out the need for detailed analysis of these approaches to reveal their theoretical and empirical possibilities in application to the Internet studies.