Semantic Level Research Articles

Graph pattern detection and structural redundancy reduction to compress named graphs

The flexible paradigm of Resource Description Framework (RDF) has accelerated the raw data published on the web. Therefore, the volume of generated RDF data has increased impressively in the last decade promoting compression to manage and reduce the size of RDF datasets. Universal RDF compressors can be able to detect and remove redundancy at symbolic, syntactic, or semantic levels. However, these compressors rarely exploit the graph patterns as well as structural regularities in real-world datasets. An efficient approach for compressing the RDF datasets based on the structural properties is HDT (Header-Dictionary-Triple). However, it cannot manage the RDF datasets with named graphs, the regularities of the graph structure, and structural redundancies. Because HDT considers all the triples to reside in the same default graph. Though, the triples of an RDF dataset belong to various named graphs. In this study, we have proposed a novel approach to deal with the above-mentioned challenges. We introduce hybrid TI-GI (Triple Interpreter-Graph Interpreter) to manage the RDF datasets with named graphs and use compact RDF serialization. We also propose RDF-RR (RDF-Redundancy Reducer) and object mapping that detects and removes structural redundancies by identifying the common patterns related to the predicates and objects in the RDF datasets. We employ a differential compressor to discover the frequent graph pattern in a single pass by using the data structure-oriented approach of the dataset. Evaluation of real-world datasets affirms that our proposed approach can substantially reduce the size of the experimental RDF datasets at approximately 30.52%, 24.92%, and 26.96% when compared with the existing HDT, HDT-FoQ (HDT-Focused on Querying) and the 2Tp (two Triple Predicate based index) approaches. Moreover, the indexing time of our proposed system is also reduced at approximately 17.89%, 13.70%, and 9.32% when compared with the HDT, HDT-FoQ, and 2Tp approaches.

The Indoor Positioning Method Time Difference of Arrival with Conic Curves Utilizing a Novel Networking RFID System

At present, the demand for accurate indoor positioning at a low cost is increasing. Based on the architecture of networking passive radio frequency identification (RFID) systems, research into passive location algorithms is important for finding a location solution with ultra-low cost, easy implementation, and no required maintenance. In this paper, TDACC (time difference of arrival with conic curves) based on signal propagation time is proposed, which breaks down the positioning problem into solving the intersection of an ellipse and a hyperbola. The results indicate that this method has a positioning error of 0 m in the absence of signal interference. When the time delay fluctuates to 1 ns and 2 ns, the average errors of TDACC are 0.19 m and 0.33 m, respectively. Different from other time-based localization methods, the proposed method only requires two distribution nodes without time synchronization, which reduces the system cost. These results will help to promote the deeper semantic communication level fusion of passive RFID. By improving the coordinate positioning in the semantic prior knowledge base, this method will lead to more efficient and accurate industry applications.

Research on Personalized Video Matching Algorithm Based on Implicit Feature Transfer and PTransE

With the rapid development of internet plus education, it is increasingly important to quickly and accurately match personalized videos for learners from massive learning videos. However, the existing resource recommendation methods have two problems. On the one hand, they do not make full use of the implicit interaction data of learners in the learning process, i.e., they rarely reflect the comprehensive consideration of learners' interest preferences and cognitive level when describing learners. On the other hand, they are mostly based on collaborative filtering algorithms, using the similarity between learners or videos to recommend, ignoring the impact of semantic relations between videos on the recommendation results. In view of this, we proposed a personalized learning video matching method (IFT‐PTransE) based on heterogeneous feature data transfer and knowledge reasoning from two aspects of learning behavior data mining and knowledge graph representation learning. In this method, firstly, a learner model is constructed. By analyzing and quantifying various implicit interaction behavior data of learners, the sparse video scoring matrix is filled as auxiliary data to complete the transfer of target scoring data. Secondly, the semantic close relationship between videos is introduced. PTransE algorithm is used to mine the multi‐path relationship between entities. All entity relationships are embedded into the low dimensional vector space, so that the semantic similarity between videos can be calculated using the distance between vectors. Finally, video score similarity and semantic similarity between frequencies are fused. video sorting is performed based on the improved collaborative filtering algorithm and then recommend the top N videos to the students. Through simulation and analysis, the effectiveness of this method in personalized video matching is proved. This method makes up for the shortcomings of collaborative filtering algorithm in using implicit information, enhances the recommendation effect at the semantic level, and solves the data sparse and cold start problems to a certain extent. © 2023 Institute of Electrical Engineers of Japan. Published by Wiley Periodicals LLC.

An ontology-based methodology to establish city information model of digital twin city by merging BIM, GIS and IoT

With the development of digital city and smart city construction, the City Information Model (CIM) has played a critical role as a container of spatial–temporal data to establish the Digital Twin City. For a digital twin city, a virtual high-fidelity CIM model that corresponds closely to the real physical world is the premise and cornerstone of its construction. Therefore, the integration of BIM, GIS and IoT has become the preferred topic for researchers and has received much more attention from a wide academic circle. However, traditional integration mainly focuses on the conversion of both IFC and CityGML, and IoT data are also often used as visualizations. More importantly, the underlying data formats of GIS, BIM and IoT are still independent of each other without a unified data structure expression, so real data-driven analysis and decision-making cannot be implemented. This study aims to establish a general CIM ontology to integrate heterogeneous BIM, GIS and IoT data. First, the related work of BIM, GIS and IoT integration is studied and analyzed. A comparison of three mainstream approaches, data conversion, standard extension and data linking, is conducted, and it illustrates the advantages of ontology techniques in solving data interoperability problems. Second, a technical framework of BIM, GIS and IoT data integration based on ontology technology is proposed. The approach is mainly divided into five steps: geometry processing, data instantiation, ontology construction, ontology mapping and querying application. On the basis of the CIM ontology, an application ontology is built for a specific application domain to illustrate rule-based mapping, querying and inferring. Finally, the case study shows that the Ontology-based methodology in this paper has contributed to establish a general pattern for CIM data integration by mapping and linking concepts from the semantic level. It avoids changes in the original data sources and the missing data problem.

Enhanced LDR Detail Rendering for HDR Fusion by TransU- Fusion Network

High Dynamic Range (HDR) images are widely used in automotive, aerospace, AI, and other fields but are limited by the maximum dynamic range of a single data acquisition using CMOS image sensors. High dynamic range images are usually synthesized through multiple exposure techniques and image processing techniques. One of the most challenging task in multiframe Low Dynamic Range (LDR) images fusion for HDR is to eliminate ghosting artifacts caused by motion.In traditional algorithms, optical flow is generally used to align dynamic scenes before image fusion, which can achieve good results in cases of small-scale motion scenes but causes obvious ghosting artifacts when motion magnitude is large. Recently, attention mechanisms have been introduced during the alignment stage to enhance the network’s ability to remove ghosts. However, significant ghosting artifacts still occur in some scenarios with large-scale motion or oversaturated areas. We proposea novel Distilled Feature TransformerBlock (DFTB) structure to distill and re-extract information from deep image features obtained after U-Net downsampling, achieving ghost removal at the semantic level for HDR fusion. We introduce a Feature Distillation Transformer Block (FDTB), based on the Swin-Transformer and RFDB structure. FDTB uses multiple distillation connections to learn more discriminative feature representations. For the multiexposure moving scene image fusion HDR ghost removal task, in the previous method, the use of deep learning to remove the ghost effect in the composite image has been perfect, and it is almost difficult to observe the ghost residue of moving objects in the composite HDR image. The method in this paper focuses more on how to save the details of LDR image more completely after removing the ghost to synthesize high-quality HDR image. After using the proposed FDTB, the edge texture details of the synthesized HDR image are saved more perfectly, which shows that FDTB has a better effect in saving the details of image fusion. Futhermore, we propose a new depth framework based on DFTB for fusing and removing ghosts from deep image features, called TransU-Fusion. First of all, we use the encoder in U-Net to extract image features of different exposures and map them to different dimensional feature spaces. By utilizing the symmetry of the U-Net structure, we can ultimately output these feature images as original size HDR images. Then, we further fuse high-dimensional space features using Dilated Residual Dense Block (DRDB) to expand the receptive field, which is beneficial for repairing over-saturated regions. We use the transformer in DFTB to perform low-pass filtering on low-dimensional space features and interact with global information to remove ghosts. Finally, the processed features are merged and output as an HDR image without ghosting artifacts through the decoder. After testing on datasets and comparing with benchmark and state-of-the-art models, the results demonstrate our model’s excellent information fusion ability and stronger ghost removal capability.

ADscreen: A speech processing-based screening system for automatic identification of patients with Alzheimer's disease and related dementia

Alzheimer's disease and related dementias (ADRD) present a looming public health crisis, affecting roughly 5 million people and 11 % of older adults in the United States. Despite nationwide efforts for timely diagnosis of patients with ADRD, >50 % of them are not diagnosed and unaware of their disease. To address this challenge, we developed ADscreen, an innovative speech-processing based ADRD screening algorithm for the protective identification of patients with ADRD. ADscreen consists of five major components: (i) noise reduction for reducing background noises from the audio-recorded patient speech, (ii) modeling the patient's ability in phonetic motor planning using acoustic parameters of the patient's voice, (iii) modeling the patient's ability in semantic and syntactic levels of language organization using linguistic parameters of the patient speech, (iv) extracting vocal and semantic psycholinguistic cues from the patient speech, and (v) building and evaluating the screening algorithm. To identify important speech parameters (features) associated with ADRD, we used the Joint Mutual Information Maximization (JMIM), an effective feature selection method for high dimensional, small sample size datasets. Modeling the relationship between speech parameters and the outcome variable (presence/absence of ADRD) was conducted using three different machine learning (ML) architectures with the capability of joining informative acoustic and linguistic with contextual word embedding vectors obtained from the DistilBERT (Bidirectional Encoder Representations from Transformers). We evaluated the performance of the ADscreen on an audio-recorded patients' speech (verbal description) for the Cookie-Theft picture description task, which is publicly available in the dementia databank. The joint fusion of acoustic and linguistic parameters with contextual word embedding vectors of DistilBERT achieved F1-score = 84.64 (standard deviation [std] = ±3.58) and AUC-ROC = 92.53 (std = ±3.34) for training dataset, and F1-score = 89.55 and AUC-ROC = 93.89 for the test dataset. In summary, ADscreen has a strong potential to be integrated with clinical workflow to address the need for an ADRD screening tool so that patients with cognitive impairment can receive appropriate and timely care.

An Evidence Accumulation Account of Masked Translation Priming in Two Bilingual Populations.

This manuscript addresses the phenomenon of masked priming and the cognitive process of switching from Spanish to English while reading in sequential bilingual texts compared to heritage speakers. A lexical decision task was employed in the present study with masked translation priming, which serves as a valuable tool for elucidating the orthographic and lexical processes involved in the initial stages of reading. This study builds upon previous research conducted on monolingual masked priming, which consistently demonstrates shifts in the response time (RT) distributions when comparing related and unrelated primes. Within the framework of a diffusion model, we implemented two theoretical positions. First, we posited that translation priming operates at the orthographic level, resulting in enhanced efficiency during the encoding process. Second, we explored the possibility that translation priming operates at the semantic level, influencing the accumulation of evidence during the lexical decision task. The findings of the present study indicate that translation priming elicits outcomes similar to those observed in monolingual priming paradigms. Specifically, we observed that translation priming facilitation is manifested as shifts in the RT distributions. These findings are interpreted to suggest that the benefits derived from the encoding process are not specific to the accessed lexicon following a brief stimulus presentation.

RUSSIAN-SOVIET EXPERIENCE OF TRANSLATING THE POEM “ANNABELLE LEE” BY EDGAR ALLAN POE

This article is devoted to the study of the tradition of translating the romantic poetry of Edgar Allan Poe into Russian. The work of Edgar Allan Poe is distinguished by the grace of the Gothic landscape, a skillful combination of the mystical and romantic aspects of the inner experiences of the individual, the acute struggle of the lofty ideal with social reality. “Madness as art” is a distinctive feature of the search for love of Poe’s lyrical heroes, a feature of their perception of the world. The poetry of the author conveys his inner experiences, which, as a rule, are the personal motives of the lyrical heroes. This specificity significantly complicates the translation of poetry. The author’s last poem “Annabel Lee” generalizes the image of “eternal love”, and the stylistic and lexical-semantic features of the poem testify to the fullness of the artistic world with cultural, mystical-religious, emotional, religious and cult-ritual concepts. Despite the fact that Poe has been criticized for his penchant for formalism, his poetic skill is considered the ideal type of maintaining the unity and integrity of artistic impression. The mechanical preservation of form, characteristic of Poe’s early work, was embodied in his last poem. For the linguistic embodiment of “eternal love” the author contrasts archetypes and forms of interaction between the earthly and heavenly worlds. Adhering to the position of the need for deep, gradual and methodical work on the poem, Poe created a unique synthetic world saturated with the spirit of Antiquity, the Middle Ages and the New Time. The image of Annabel Lee has retained the features of Lenore, Helena, Ligeia and other female characters that were created by the author in poetry and prose. The translations made by K. Balmont, V. Bryusov and V. Fedorov are considered to be classical ones. We aim to analyze other well-known translations by S. Andreevsky, D. Sadovnikov, L. Umants, V. Zhabotinsky, A. Olenich-Gnenenko. The result of the research was the identification of the translation features of the poem at the compositional, stylistic, lexical and semantic levels.

Specific Features of Operations of Semantic Syntaxing of Separate and Coherent Statements in oral and Written Speech in Younger Schoolchildren with Severe Speech Disorders

The article is devoted to the problem of formation of semantic level of syntaxing in the oral and written speech of younger schoolchildren with severe speech disorders. The purpose of the article is to present the results of the research of the semantic level of the syntactic component of the language ability of younger students with severe speech disorders. For a long time, the essence of the syntactic component of language ability was considered from the position of surface structure. In most cases, existing studies describe the external manifestations of syntactic disorders and do not reveal their mechanisms. At present, the issue of searching for the unformed mechanisms of the syntactic component of language ability has become relevant. According to T. V. Akhutina`s point of view, there are three multi-level syntax operations, that are carried out as an actualization of the frame, the slots of which are filled with the help of operations of selecting elements from the semantic-lexical paradigms of the appropriate level: meaningful, semantic and the level of formal-grammatical structuring (T. V. Akhutina, A. R. Luria, A. A. Leontiev and others). The lack of formation of one of the components prevents from successful making the syntactic component and the language ability in general. The topicality of identifying the features of the formation of the semantic level of syntax of primary school students is due, on the one hand, to the importance of its formation for the development of language ability, and on the other hand, to the insufficient development of the content and methods of speech therapy work in this direction. Errors that indicate the lack of formation of the semantic level of syntax of younger students with severe speech disorders consist in violation of the process of structuring a non-speech situation, the impossibility of calculating its components, which caused errors in the formation of the program of a separate and coherent statement in oral and written speech even at the pre-linguistic stage.

Фреймовое моделирование предметной области ЗДОРОВЫЙ ОБРАЗ ЖИЗНИ в арабском медиадискурсе

The work studies the content of the HEALTHY LIFESTYLE domain in the Arabic linguistic consciousness through the use of methods of frame, conceptual and contextual analysis of the vocabulary belonging to the domain under consideration. The collective, culturally and nationally conditioned understanding of this domain is examined. The aim of the paper is to build a frame model of the HEALTHY LIFESTYLE domain in the Arabic-language media discourse. Frames, subframes and slots revealing the content and structure of the vocabulary of the HEALTHY LIFESTYLE domain (نمط الحياة الصحية) in the studied worldview, as well as the features of its verbalization in the Arabic-language media discourse are identified and analyzed. The paper is novel in that it is the first to categorize and parameterize the HEALTHY LIFESTYLE domain in media texts in Arabic linguoculture based on frame analysis. As a result, the research revealed the objective essential features of the HEALTHY LIFESTYLE domain at the semantic level, classified its nuclear and peripheral features and provided a description of the experience of modeling the HEALTHY LIFESTYLE domain using frame structure.

Поэтическая трансляция как герменевтический диалог переводчика с исходным текстом (опыт интерпретации разновременных переводов «Казачьей колыбельной песни» М.Ю. Лермонтова)

The article discusses the comparative hermeneutic analysis of the semantic-derivational worldview of the translators who render the classical poetic text in different historical periods. The relevance of the study is stipulated by the increased attention to the sociocultural determination of meaning generation within the framework of the cognitive-and-discursive paradigm that dominates in the modern community of linguists and translators. Cossack Cradle Song and its five translations into German carried out in different periods served as the practical material. The article aims to establish the features of multi-temporal Russian-German translations of the poem and the source text as a single semantic space that reflects the worldview of the author of the original text, variably explicated by different translators through the language of the target text. According to the research hypothesis, poetic translation reflects the translator's hermeneutic interaction with the original text, which generates the personal meaning of the translated text. The findings of the linguo-stylistic analysis of the Cossack Cradle Song at the semantic, meta-semiotic, and meta-metasemiotic levels, reveal that the compensation technique plays the leading role in the process of translating the poetic text studied in this article, contributing to the achievement of expressive and impressive equivalence of the original and the translation in the process of explication of invariant content-factual and content-conceptual meanings, as well as the subjective-personal determinism of translation decisions, which does not contradict the intentions implied by the author of the source text.

ANALISIS KESALAHAN BERBAHASA DALAM TATARAN SEMANTIK PADA POSTINGAN AKUN INSTAGRAM @demauinsurakarta

This research is motivated by the use of the Indonesian language where there are still many errors found, especially at the semantic level. The purpose of this study is to find out language errors at the semantic level contained in the @demauinsurakarta account. This study uses a qualitative descriptive method approach. The data source in this study is the account caption @demauinsurakarta. This study uses data analysis methods in the form of references from the Kamus Besar Bahasa Indonesia (KBBI) and Pedoman Umum Ejaan Bahasa Indonesia (PUEBI). The results of this study found semantic level errors in the field of hypercorrection found 7 errors, pleonasm found 8 errors, inappropriate word choice or diction found 3 errors and word selection which resulted in ambiguity meaning found 3 errors.

The Semiotics of the Thresholds in (Sunset Garden) Diwan of the Poet Ghazi Al-Qasaibi

This study is concerned with revealing the semiotics of thresholds, and its role in building textual coherence at the form and semantic level in Sunset Garden Diwan of the Saudi poet Ghazi Al-Qasaibi. This research aims to spotlight on the semiotic theme of the title threshold and its relation to other thresholds. As the death phenomenon and absence dominated the poetic achievement space through intertwined semiotic relationships that lead to each other through the parallel text semiotics.This study is based on the semiotic method which concerns with searching for the interpretive connotations and revealing the profound symbols in the threshold structure and its role in the whole formation of the Diwan. Among the most prominent conclusions for this study is that; AlQasaibi was concerned with the thresholds poetics in his book Sunset Garden and he was able to transfer the meaning of thresholds from the margin to the centre when he revealed the significance of the symbols and connotations of the thresholds in the Diwan, which contributed to the space and coherence of the text.It also balanced between the centrality of receiving. Moreover, the poetic thresholds and their semiotic symbols contributed to the manifestation of the Diwan purposes and its contents, forming a unified texture starting with the front cover and ending with the back cover of the Diwan. The semiotic analysis of the poetic threshold in Sunset Garden Diwan showedAlQasaibi special vision of death and absence merged with nostalgia, eulogy and reconciliation with the concept and reality of death. The title also represents the first thresholds of the text that cannot be crossed.That is because it has the codes and symbols that help the reader and recipient to introspect the entire work.

National and Cultural Features Representations of the Concept of “War” in the Russian and Chuvash language Pictures of the World

The article is devoted to a comparative study of the national and cultural features of the representation of the concept of “war” in the Russian and Chuvash pictures of the world. Based on the material of Russian and Chuvash proverbs and sayings selected from lexicographic and folklore sources, a comprehensive comparative analysis of the concept of “war” at the semantic, pragmatic and culturological levels was conducted for the first time, the features of its verbalization were considered. For the first time, the national-cultural features of the representation of the concept of “war” at the cognitive-discursive level are revealed, which determines the relevance of our research. The analysis made it possible to reveal that the concept of “war” is a universal unit included in the spiritual code of the nation, has a dynamic semantic structure and is multidimensional in its content. As a result of a comparative analysis, it was found that the fundamental values of the moral order in the study of the concept of “war” are the same for both peoples. The speech embodiment of this concept in the Russian and Chuvash language pictures of the world is defined as a process that contradicts the human mind.

Syntactic independent sentence without illocutive force

The article deals with the factors which influence the illocutionary force of the sentence. The syntactically independent sentences are in the focus of the discussion. It is shown that illocutionary force is formed by three factors: communicative binary, semantics of the sentence and temporal reference. They are universal. At the level of the deep structure, sentences, which have illocutionary force include three components: topic, focus and modality. The presence of modal component is not possible without the topic. Sentences without any illocutionary force do not have topicalized components and are incompatible with modal indicators. At the level of the surface structure the absence of illocutionary force is marked by the displacement of the predicate to the initial position, the position of the subject after the predicate or the monomial structure of a sentence. Sentences devoid of illocutionary force in the Russian language are always marked. In German the word order is fixed, therefore, in communicatively monomial sentences, the “shift” of the verb to the first position and the displacement of the referential subject is carried out by using the formal subject es in the preposition. In English, in this case, the initial elements there or it can be used. In proposals of this type, local and time indicators are often shifted to the initial position. Neither local nor temporal indicators can be the subject of a statement: at the beginning of a sentence they mark the absence of illocutionary force. Sentences devoid of illocutionary force serve as the background of the main narrative in the text or initiate the text. At the level of semantics, the sentences of this type are characterized by providing descriptions of nature, weather, situation, state, or act as introductory sentences in microtext or text. When describing states and characteristics, we mostly refer to permanent signs or properties of objects/objects/phenomena in the environment. The factors that shape the illocutionary force of a sentence are universal.

Unlocking Efficiency in Fine-Grained Compositional Image Synthesis: A Single-Generator Approach

The use of Generative Adversarial Networks (GANs) has led to significant advancements in the field of compositional image synthesis. In particular, recent progress has focused on achieving synthesis at the semantic part level. However, to enhance performance at this level, existing approaches in the literature tend to prioritize performance over efficiency, utilizing separate local generators for each semantic part. This approach leads to a linear increase in the number of local generators, posing a fundamental challenge for large-scale compositional image synthesis at the semantic part level. In this paper, we introduce a novel model called Single-Generator Semantic-Style GAN (SSSGAN) to improve efficiency in this context. SSSGAN utilizes a single generator to synthesize all semantic parts, thereby reducing the required number of local generators to a constant value. Our experiments demonstrate that SSSGAN achieves superior efficiency while maintaining a minimal impact on performance.

Learning Semantic Degradation-Aware Guidance for Recognition-Driven Unsupervised Low-Light Image Enhancement

Low-light images suffer severe degradation of low lightness and noise corruption, causing unsatisfactory visual quality and visual recognition performance. To solve this problem while meeting the unavailability of paired datasets in wide-range scenarios, unsupervised low-light image enhancement (ULLIE) techniques have been developed. However, these methods are primarily guided to alleviate the degradation effect on visual quality rather than semantic levels, hence limiting their performance in visual recognition tasks. To this end, we propose to learn a Semantic Degradation-Aware Guidance (SDAG) that perceives the low-light degradation effect on semantic levels in a self-supervised manner, which is further utilized to guide the ULLIE methods. The proposed SDAG utilizes the low-light degradation factors as augmented signals to degrade the low-light images, and then capture their degradation effect on semantic levels. Specifically, our SDAG employs the subsequent pre-trained recognition model extractor to extract semantic representations, and then learns to self-reconstruct the enhanced low-light image and its augmented degraded images. By constraining the relative reconstruction effect between the original enhanced image and the augmented formats, our SDAG learns to be aware of the degradation effect on semantic levels in a relative comparison manner. Moreover, our SDAG is general and can be plugged into the training paradigm of the existing ULLIE methods. Extensive experiments demonstrate its effectiveness for improving the ULLIE approaches on the downstream recognition tasks while maintaining a competitive visual quality. Code will be available at https://github.com/zheng980629/SDAG.

ROIFormer: Semantic-Aware Region of Interest Transformer for Efficient Self-Supervised Monocular Depth Estimation

The exploration of mutual-benefit cross-domains has shown great potential toward accurate self-supervised depth estimation. In this work, we revisit feature fusion between depth and semantic information and propose an efficient local adaptive attention method for geometric aware representation enhancement. Instead of building global connections or deforming attention across the feature space without restraint, we bound the spatial interaction within a learnable region of interest. In particular, we leverage geometric cues from semantic information to learn local adaptive bounding boxes to guide unsupervised feature aggregation. The local areas preclude most irrelevant reference points from attention space, yielding more selective feature learning and faster convergence. We naturally extend the paradigm into a multi-head and hierarchic way to enable the information distillation in different semantic levels and improve the feature discriminative ability for fine-grained depth estimation. Extensive experiments on the KITTI dataset show that our proposed method establishes a new state-of-the-art in self-supervised monocular depth estimation task, demonstrating the effectiveness of our approach over former Transformer variants.

BridgeTower: Building Bridges between Encoders in Vision-Language Representation Learning

Vision-Language (VL) models with the Two-Tower architecture have dominated visual-language representation learning in recent years. Current VL models either use lightweight uni-modal encoders and learn to extract, align and fuse both modalities simultaneously in a deep cross-modal encoder, or feed the last-layer uni-modal representations from the deep pre-trained uni-modal encoders into the top cross-modal encoder. Both approaches potentially restrict vision-language representation learning and limit model performance. In this paper, we propose BridgeTower, which introduces multiple bridge layers that build a connection between the top layers of uni-modal encoders and each layer of the cross-modal encoder. This enables effective bottom-up cross-modal alignment and fusion between visual and textual representations of different semantic levels of pre-trained uni-modal encoders in the cross-modal encoder. Pre-trained with only 4M images, BridgeTower achieves state-of-the-art performance on various downstream vision-language tasks. In particular, on the VQAv2 test-std set, BridgeTower achieves an accuracy of 78.73%, outperforming the previous state-of-the-art model METER by 1.09% with the same pre-training data and almost negligible additional parameters and computational costs. Notably, when further scaling the model, BridgeTower achieves an accuracy of 81.15%, surpassing models that are pre-trained on orders-of-magnitude larger datasets. Code and checkpoints are available at https://github.com/microsoft/BridgeTower.

RealityMedia: immersive technology and narrative space

In this paper, we treat VR as a new writing space in the long tradition of inscription. Constructing Virtual Reality (VR) narratives can then be understood as a process of inscribing text in space, and consuming them as a process of “reading” the space. Our research objective is to explore the meaning-making process afforded by spatial narratives—to test whether VR facilitates traditional ways of weaving complex, multiple narrative strands and provides new opportunities for leveraging space. We argue that, as opposed to the linear space of a printed book, a VR narrative space is similar to the physical space of a museum and can be analyzed on three distinct levels: (1) the architecture of the space itself, (2) the collection, and (3) the individual artifacts. To provide a deeper context for designing VR narratives, we designed and implemented a testbed called RealityMedia to explore digital remediations of traditional narrative devices and the spatial, immersive, and interactive affordances of VR. We conducted task-based user study using a VR headset and follow-up qualitative interviews with 20 participants. Our results highlight how the three semantic levels (space, collection, and artifacts) can work together to constitute meaningful narrative experiences in VR.