Additional Semantic Information Research Articles

Semantic Segmentation for Change Detection in Satellite Imaging

Change detection is a common and actual problem in the field of remote sensing. The classical approaches using raw pixel information are very sensitive to noise. In this study we propose the usage of additional semantic information for change detection. We use the semantic segmentation methods like geospatial Segment Anything Model and encoder based U-Net to evaluate the predictions and tracing the semantic information as well as raw information in change detection. Later the multidimensional time series data is used via the Vector Autoregression model to predict the future changes in the landscape. The observations which fall out of the prediction interval are considered as the changes in the landscape. The proposed method is evaluated on the dataset of the random locations across the Baltic region. The research is accompanied by the data and reproducible code at Github repository.

Generalized zero-shot learning for classifying unseen wafer map patterns

A data-driven approach for classifying defect patterns in wafer maps is crucial for automating the quality assurance process during semiconductor fabrication. Existing works have proposed machine learning-based methods to accomplish this important problem. However, many of these approaches assume model training from a supervised learning perspective, in which a substantial number of annotated wafer maps, each with its actual defect pattern, need to be available. Due to the cost and time required to generate such an annotated dataset, recently a few works started to explore the problem of identifying previously unseen defect patterns. However, the methods proposed in those works are able to only detect the presence of unseen defect patterns without identifying what those defect patterns are. To address this issue, this paper introduces a generalized zero-shot learning (GZSL)-based framework capable of classifying and identifying both seen and unseen defect patterns. It consists of a visual feature extraction phase based on semi-supervised contrastive learning (CL), a clustering phase using distance-based initialization, and an alignment phase based on the Wasserstein-Distance-based Visual Structure Constraint method. Our framework further reduces the need for expensive annotated datasets previously required to train supervised learning-based models, thereby improving the practicality of the data-driven approach for wafer map pattern classification. Furthermore, our framework introduces an effective way of incorporating additional semantic information in wafer map pattern classification that traditionally relied on training models solely from wafer maps. To the best of the authors’ knowledge, our work is the first work to adopt GZSL in wafer map pattern classification.

Cross‐modal knowledge learning with scene text for fine‐grained image classification

AbstractScene text in natural images carries additional semantic information to aid in image classification. Existing methods lack full consideration of the deep understanding of the text and the visual text relationship, which results in the difficult to judge the semantic accuracy and the relevance of the visual text. This paper proposes image classification based on Cross modal Knowledge Learning of Scene Text (CKLST) method. CKLST consists of three stages: cross‐modal scene text recognition, text semantic enhancement, and visual‐text feature alignment. In the first stage, multi‐attention is used to extract features layer by layer, and a self‐mask‐based iterative correction strategy is utilized to improve the scene text recognition accuracy. In the second stage, knowledge features are extracted using external knowledge and are fused with text features to enhance text semantic information. In the third stage, CKLST realizes visual‐text feature alignment across attention mechanisms with a similarity matrix, thus the correlation between images and text can be captured to improve the accuracy of the image classification tasks. On Con‐Text dataset, Crowd Activity dataset, Drink Bottle dataset, and Synth Text dataset, CKLST can perform significantly better than other baselines on fine‐grained image classification, with improvements of 3.54%, 5.37%, 3.28%, and 2.81% over the best baseline in mAP, respectively.

Synergistic Detection of Multimodal Fake News Leveraging TextGCN and Vision Transformer

In today's digital age, the rapid spread of fake news is a pressing concern. Fake news, whether intentional or inadvertent, manipulates public sentiment and threatens the integrity of online information. To address this, effective detection and prevention methods are vital. Detecting and addressing this multimodal fake news is an intricate challenge as, unlike traditional news articles that predominantly rely on textual content, multimodal fake news leverages the persuasive power of visual elements, making its identification a formidable task. Manipulated images can significantly sway individuals' perceptions and beliefs, making the detection of such deceptive content complex. Our research introduces an innovative approach to multimodal fake news identification by presenting a fusion-based methodology that harnesses the capabilities of Text Graph Convolutional Neural Networks (TextGCN) and Vision Transformers (ViT) to effectively utilise both text and image modalities. The proposed Methodology starts with preprocessing textual content using TextGCN, allowing for the capture of intricate structural dependencies among words and phrases. Simultaneously, visual features are extracted from associated images using ViT. Through a fusion mechanism, these modalities seamlessly integrate, yielding superior embeddings. The primary contributions encompass an in-depth exploration of multimodal fake news detection through a fusion-based approach. What sets our approach apart from existing techniques is its integration of graph-based feature extraction through TextGCN. While previous methods predominantly rely on text or image features, our approach harnesses the additional semantic information and intricate relationships within a graph structure, in addition to image embeddings. This enables our method to capture more comprehensive understanding of the data, resulting in increased accuracy and reliability. Our experiments demonstrate the exceptional performance of our fusion-based approach, which leverages multiple modalities and incorporates graph-based representations and semantic relationships. This method outperformed single modalities of text or image, achieving an impressive accuracy of 94.17% using a neural network after fusion. By seamlessly integrating graph-based representations and semantic relationships, our fusion-based technique represents a significant stride in addressing the challenges posed by multimodal fake news.

Curriculum Consistency Learning and Multi-Scale Contrastive Constraint in Semi-Supervised Medical Image Segmentation.

Data scarcity poses a significant challenge in medical image segmentation, thereby highlighting the importance of leveraging sparse annotation data. In addressing this issue, semi-supervised learning has emerged as an effective approach for training neural networks using limited labeled data. In this study, we introduced a curriculum consistency constraint within the context of semi-supervised medical image segmentation, thus drawing inspiration from the human learning process. By dynamically comparing patch features with full image features, we enhanced the network's ability to learn. Unlike existing methods, our approach adapted the patch size to simulate the human curriculum process, thereby progressing from easy to hard tasks. This adjustment guided the model toward improved convergence optima and generalization. Furthermore, we employed multi-scale contrast learning to enhance the representation of features. Our method capitalizes on the features extracted from multiple layers to explore additional semantic information and point-wise representations. To evaluate the effectiveness of our proposed approach, we conducted experiments on the Kvasir-SEG polyp dataset and the ISIC 2018 skin lesion dataset. The experimental results demonstrated that our method surpassed state-of-the-art semi-supervised methods by achieving a 9.2% increase in the mean intersection over union (mIoU) for the Kvasir-SEG dataset. This improvement substantiated the efficacy of our proposed curriculum consistency constraint and multi-scale contrastive loss.

Facial first impressions following a prison sentence: Negative shift in trait ratings but the same underlying structure

The first impressions we form of unfamiliar others can often guide many important decisions such as whether someone is guilty of a crime or the severity of their sentence, even in the presence of more relevant information. While most of the current work in this context has focused on their impact during trial proceedings and sentencing, little is known about the potential impact of first impressions following a guilty sentence and the success of the subsequent reintegration into society. Here, we used a data-driven approach to address this question by first collecting unconstrained spontaneous impressions from two groups of perceivers – one group believed that the identities they were presented with had received a prison sentence, whereas the other received no additional semantic information (Study 1). This then allowed us to establish the most prevalent traits people refer to when describing their first impressions in this context and to reveal the underlying structure of these impressions using an Exploratory Factor Analysis (Study 2). We find a substantial negative shift in social evaluation following the knowledge of a prison sentence, both in terms of spontaneous descriptions and specific trait ratings. However, this additional contextual information did not affect the underlying structure of first impressions. These findings support social evaluation theories arguing for a more complex interplay between bottom-up visual and top-down semantic or contextual cues during the formation of facial first impressions but also reveal important constraints to the impact of such cues on the core impression formation processes.

0071 Individual Differences in Sleep Duration and Creative Thinking on the Remote Associates Test

Abstract Introduction Short sleep has been shown to affect attention, processing speed, and memory. Creative cognition, including convergent thinking (e.g., identifying remote connections), has received less attention in the sleep literature. In four studies, we examined individual differences in sleep duration relative to performance on a convergent thinking task in which participants view three words (sore-should-sweat) and have to identify a target word that connects them (cold). Based on the literature on sleep deprivation and attention/memory, we hypothesized that short sleepers would show lower convergent thinking performance. By contrast, based on the view that convergent thinking is reflective of fluid intelligence—a higher-order cognitive construct that is resilient to sleep deprivation—short sleep should be unrelated to convergent thinking performance. Methods Participants were 551 adults aged 18-30 who were recruited across four experiments. Each experiment compared performance on a traditional 3-item compound Remote Associates Test to performance on a modified approach that provided additional semantic information (4-item Remote-Associates Test). After each experiment, participants were asked to indicate the duration of their previous night of sleep. Results Participants performed 27% better on the 4-item version, relative to the traditional 3-item Remote Associates Test (p<.001). There were no statistically significant associations between sleep duration and performance accuracy on the 3-item or 4-item compound Remote Associates Tests (rs<.10, ps > .05). This pattern was observed within each experiment, as well as when the data were collapsed across experiments. In addition, there were no significant associations when comparing extreme sleep durations (< 5 hours to ≥8 hours; all ps > .05). Conclusion Individual differences in recent sleep duration were not related to individual differences in convergent thinking abilities. These findings are consistent with the view of convergent thinking as a marker of fluid intelligence. Experimental work is needed to determine whether convergent thinking is fully resistant to sleep deprivation and to examine generalizability to other creative processes (e.g., divergent thinking, imagination). Support (if any) National Science Foundation (1920730)

Knowledge Graph-Based Integration of Autonomous Driving Datasets

Autonomous Driving (AD) datasets, when used in combination with deep learning techniques, have enabled significant progress on difficult AD tasks such as perception, trajectory prediction, and motion planning. These datasets represent the content of driving scenes as captured by various sensors, including cameras, RADAR, and LiDAR, along with 2D/3D annotations of traffic participants. Such datasets, however, often fail to capture and to represent the spatial, temporal, and semantic relations between entities in a scene. This lack of knowledge leads to a shallow understanding of the true complexity and dynamics inherent in a driving scene. In this paper, we argue that a Knowledge Graph (KG)-based representation of driving scenes, that provides a richer structure and semantics, will lead to further improvements in AD. Towards this goal, we developed a layered architecture and ontologies for specific AD datasets and a fundamental ontology of shared concepts. We also built KGs for three different AD datasets. We perform an analysis with respect to information contained in the AD KGs and outline how the additional semantic information contained in the KGs could improve the performance of different AD tasks. Moreover, example queries are provided to retrieve relevant information that can be exploited for augmenting the AD pipelines. All artifacts needed for reproducibility purposes are provided via a GitHub repository. ( https://github.com/boschresearch/dskg-constructor — Note that we removed our internal namespaces of reused ontologies, because of confidentiality and to provide self-contained ontologies. As the original datasets are under specific licences, the KGs are not published, but we provide the scripts to generate them.)

Learning multi-view and centerline topology connectivity information for pulmonary artery–vein separation

Background:Automatic pulmonary artery–vein separation has considerable importance in the diagnosis and treatment of lung diseases. However, insufficient connectivity and spatial inconsistency have always been the problems of artery–vein separation. Methods:A novel automatic method for artery–vein separation in CT images is presented in this work. Specifically, a multi-scale information aggregated network (MSIA-Net) including multi-scale fusion blocks and deep supervision, is proposed to learn the features of artery–vein and aggregate additional semantic information, respectively. The proposed method integrates nine MSIA-Net models for artery–vein separation, vessel segmentation, and centerline separation tasks along with axial, coronal, and sagittal multi-view slices. First, the preliminary artery–vein separation results are obtained by the proposed multi-view fusion strategy (MVFS). Then, centerline correction algorithm (CCA) is used to correct the preliminary results of artery–vein separation by the centerline separation results. Finally, the vessel segmentation results are utilized to reconstruct the artery–vein morphology. In addition, weighted cross-entropy and dice loss are employed to solve the class imbalance problem. Results:We constructed 50 manually labeled contrast-enhanced computed CT scans for five-fold cross-validation, and experimental results demonstrated that our method achieves superior segmentation performance of 97.7%, 85.1%, and 84.9% on ACC, Pre, and DSC, respectively. Additionally, a series of ablation studies demonstrate the effectiveness of the proposed components. Conclusion:The proposed method can effectively solve the problem of insufficient vascular connectivity and correct the spatial inconsistency of artery–vein.

No semantic information is necessary to evoke general neural signatures of face familiarity: evidence from cross-experiment classification

Recent theories on the neural correlates of face identification stressed the importance of the available identity-specific semantic and affective information. However, whether such information is essential for the emergence of neural signal of familiarity has not yet been studied in detail. Here, we explored the shared representation of face familiarity between perceptually and personally familiarized identities. We applied a cross-experiment multivariate pattern classification analysis (MVPA), to test if EEG patterns for passive viewing of personally familiar and unfamiliar faces are useful in decoding familiarity in a matching task where familiarity was attained thorough a short perceptual task. Importantly, no additional semantic, contextual, or affective information was provided for the familiarized identities during perceptual familiarization. Although the two datasets originate from different sets of participants who were engaged in two different tasks, familiarity was still decodable in the sorted, same-identity matching trials. This finding indicates that the visual processing of the faces of personally familiar and purely perceptually familiarized identities involve similar mechanisms, leading to cross-classifiable neural patterns.

Baby steps towards few-shot learning with multiple semantics

• We propose to consider a new, closer to ‘infant learning’, setting of Few-Shot Learning with Multiple and Complex Semantics. • In this context we propose a new benchmark for FSL-MCS, and an associated training and evaluation protocol. • A new multi-branch architecture that provides the first batch of encouraging results for the proposed FSL-MCS benchmark. Learning from one or few visual examples is one of the key capabilities of humans since early infancy, but is still a significant challenge for modern AI systems. While considerable progress has been achieved in few-shot learning from a few image examples, much less attention has been given to the verbal descriptions that are usually provided to infants when they are presented with a new object. In this paper, we focus on the role of additional semantics that can significantly facilitate few-shot visual learning. Building upon recent advances in few-shot learning with additional semantic information , we demonstrate that further improvements are possible by combining multiple and richer semantics (category labels, attributes, and natural language descriptions). Using these ideas, we offer the community new results on the popular mini ImageNet and CUB few-shot benchmarks, comparing favorably to the previous state-of-the-art results for both visual only and visual plus semantics-based approaches. We also performed an ablation study investigating the components and design choices of our approach. Code available on github.com/EliSchwartz/mutiple-semantics .

SEMANTIC URBAN MESH SEGMENTATION BASED ON AERIAL OBLIQUE IMAGES AND POINT CLOUDS USING DEEP LEARNING

Abstract. The use of deep machine learning methods for semantic classification of city mesh models is one of the current trends in geoscience development. Thanks to the thriving development of Convolutional Neural Networks (CNNs) it is now achievable to conduct fully automated process of building aforementioned 3D model by means of photogrammetric techniques and supplement it with additional semantic information obtained by Artificial Intelligence (AI) algorithms. In order to guarantee the comprehensiveness of said information it is essential to use an extensive range of 3D data including oblique aerial imagery and aerial laser scanning (ALS). Such comprehensive 3D mesh models may be later implemented in many Digital Twin class solutions additionally supported with modern GIS systems and its algorithms. To proof the validity of this thesis, the article showcases results of research conducted using deep learning based solutions tested on two datasets - real-world data in the form of oblique aerial images and ALS point clouds acquired in Bordeaux, France using novel Leica CityMapper-1 multisensoral system and large-scale dataset from SUM: A Benchmark Dataset of Semantic Urban Meshes. Both subalgorithms make use of CNNs as its core-feature. To perform accurate classification of oblique aerial scenes PSP-Net architecture accelerated by techniques of transfer learning has been used. Second algorithm destined for ALS point clouds utilizes CNN as well, but in this case implementation is based on proprietary architecture. The results of the experiments demonstrate that the utilizing these two mutually complementary solutions to extract new semantic information for city mesh models in proposed manner compared with the state-of-the-art methods achieves competitive classification performance.

A Literature Review on Additional Semantic Information Conveyed from Driving Automation Systems to Drivers through Advanced In-Vehicle HMI Just Before, During, and Right After Takeover Request

In-vehicle human-machine interface (HMI) plays a significant role in accomplishing effective interactions between driving automation systems and drivers, especially during the transition of control. For this reason, different in-vehicle HMIs have been designed to convey additional semantic information from the driving automation systems to the drivers to realize safer, smoother, and better control transitions. This review summarizes and analyses 86 previously published studies that researched the effects of additional semantic information delivered through in-vehicle HMIs just before, during and right after takeover request (TOR). The additional semantic information mentioned in this review refer to the information beyond simple alerts to not only gain drivers’ attention but also additionally communicate contextual content and explanation to the drivers regarding its own purpose. In this review, the additional semantic information are categorized according to their purposes and effects into three aspects: mode awareness enhancement, situation awareness enhancement, and takeover maneuver assistance. The specificities and the corresponding concerns when applying additional semantic information to in-vehicle HMIs have been detailed analyzed throughout the entire article. Further suggestions are proposed for what should be carefully considered when adding additional information for better takeover. Prospects into future in-vehicle HMI possibilities are also raised that could be applied in both academic research and industry.

Identifying informative tweets during a pandemic via a topic-aware neural language model.

Every epidemic affects the real lives of many people around the world and leads to terrible consequences. Recently, many tweets about the COVID-19 pandemic have been shared publicly on social media platforms. The analysis of these tweets is helpful for emergency response organizations to prioritize their tasks and make better decisions. However, most of these tweets are non-informative, which is a challenge for establishing an automated system to detect useful information in social media. Furthermore, existing methods ignore unlabeled data and topic background knowledge, which can provide additional semantic information. In this paper, we propose a novel Topic-Aware BERT (TABERT) model to solve the above challenges. TABERT first leverages a topic model to extract the latent topics of tweets. Secondly, a flexible framework is used to combine topic information with the output of BERT. Finally, we adopt adversarial training to achieve semi-supervised learning, and a large amount of unlabeled data can be used to improve inner representations of the model. Experimental results on the dataset of COVID-19 English tweets show that our model outperforms classic and state-of-the-art baselines.

MFPMiner: Mining Meaningful Frequent Patterns from Spatio-textual Trajectories

In the second decade of this century, technical progress has led to a worldwide proliferation of devices for tracking the movement behavior of a person, a vehicle, or another kind of entity. One of the consequences of this development is a massive and still growing amount of movement and movement-related data recorded by cellphones, automobiles, vessels, aircraft, and further GPS-enabled entities. As a result, the requirements for managing and analyzing movement records also increase, serving commercial, administrative, or private purposes. Since the development of hardware components cannot keep pace with the data growth, exploring methods of analyzing such trajectory datasets has become a very active and influential research field. For many application scenarios, besides the spatial trajectory of an entity, it is desirable to take additional semantic information into consideration. These descriptions also change with time and may represent, e.g., the course of streets passed by a bus, the sequence of region names traversed by an aircraft, or the points of interest in proximity of the positions of a taxi. Such data may be directly recorded by a sensor (such as the altitude of an aircraft) or computed from the spatial trajectory combined with some underlying information (for example, street names). It is often helpful or even necessary to focus on such semantic information for efficient analyses, as changes usually occur less frequently than it is the case for the spatial trajectory, where data points usually arrive in very close temporal distances. However, any kind of querying requires a deep semantic knowledge of the dataset at hand, particularly for retrieving the set of trajectories that match a certain mobility pattern, that is, a sequence of temporal, spatial, and semantic specifications. In this article, we introduce a framework named MFPMiner 1 for retrieving all mobility patterns fulfilling a user-specified frequency threshold from a spatio-textual trajectory dataset. The resulting patterns and their relative frequency can be regarded as a knowledge base of the considered data. They may be directly visualized or applied for a pattern matching query yielding the set of matching trajectories. We demonstrate the functionality of our approach in an application scenario and provide an experimental evaluation of its performance on real and synthetic datasets by comparing it to three competitive methods. The framework has been fully implemented in a DBMS environment and is freely available open source software.

Pseudo Segmentation for Semantic Information-Aware Stereo Matching

Stereo matching plays an important role in computer vision and robotics. Though substantial progress has been made on deep learning-based algorithms, the inherent semantic information within the ground truth of the training data for stereo matching has not been well explored. In this letter, we propose to use a pseudo segmentation sub-network to extract additional semantic information. More specifically, we divide the disparity label into groups and let each group correspond to a class for pseudo segmentation. To assist stereo matching with the semantic information obtained from pseudo segmentation, we inject the feature maps at the end of the pseudo segmentation sub-network into the cost volume that is used to infer the pixel-level disparity. To validate the effectiveness of the proposed approach, we select PSMNet (Chang and Chen, 2018)and GwcNet (Guo <i>et al.</i>, 2019) as baselines and enhance them with the pseudo segmentation sub-network. Comprehensive experiments are carried out on the Scene Flow, KITTI 2015, and KITTI 2012 datasets, and the results show that our proposed method can improve the performance notably.

Spatial-wise and channel-wise feature uncertainty for occluded person re-identification

Occluded person re-identification is a challenging task since the available data often suffers from information incompleteness and spatial misalignment. Most state-of-the-art occluded models rely on the external model to provide additional semantic information. However, for the time being, external models, such as the human parsing model and the pose estimation model cannot provide accurate semantic information under a complex occlusion environment and may introduce errors to the Re-ID model instead. In this paper, we propose an occluded person Re-ID model that mines the latent recognizable information of the person image itself, without the help of external models. Feature/Data uncertainty can reduce the influence of noisy samples in datasets and has been discussed in person Re-ID and face recognition, we extend the uncertainty to the micro feature level, and propose the spatial-wise and channel-wise feature uncertainty to constantly refine the features in the spatial domain and the channel domain respectively during feature construction by weakening the influence of noise features. Extensive experiments on the occluded datasets and holistic datasets have proved the effectiveness of our proposed methods.

INFERRING ROOF SEMANTICS FOR MORE ACCURATE SOLAR POTENTIAL ASSESSMENT

Abstract. In guiding the energy transition efforts towards renewable energy sources, 3D city models were shown to be useful tools when assessing the annual solar energy generation potential of urban landscapes. However, the simplified roof geometry included in these 3D city models and the lack of additional semantic information about the buildings’ roof often yield less accurate solar potential evaluations than desirable. In this paper we propose three different methods to infer and store additional information into 3D city models, namely on physical obstacles present on the roof and existing solar panels. Both can be used to increase the accuracy of roof solar panel retrofit potential. These methods are developed and tested on the open datasets available in the Netherlands, specifically AHN3 lidar point-cloud and PDOK aerial photography. However, we believe they can be adapted to different environments as well, based on the available datasets and their precision locally available.

Investigating Semantic Augmentation in Virtual Environments for Image Segmentation Using Convolutional Neural Networks.

Collecting real-world data for the training of neural networks is enormously time-consuming and expensive. As such, the concept of virtualizing the domain and creating synthetic data has been analyzed in many instances. This virtualization offers many possibilities of changing the domain, and with that, enabling the relatively fast creation of data. It also offers the chance to enhance necessary augmentations with additional semantic information when compared with conventional augmentation methods. This raises the question of whether such semantic changes, which can be seen as augmentations of the virtual domain, contribute to better results for neural networks, when trained with data augmented this way. In this paper, a virtual dataset is presented, including semantic augmentations and automatically generated annotations, as well as a comparison between semantic and conventional augmentation for image data. It is determined that the results differ only marginally for neural network models trained with the two augmentation approaches.

A Feature Percolation Convention of Affixation Processes in Urhobo

This paper seeks to examine headedness in affixation processes in Urhobo using a feature percolation approach. Affixation is a rich source of word formation process in the Urhobo language. The specific objectives are to identify affixation processes in Urhobo, investigate how they are used to derive words and determine headedness in the Urhobo language using a feature percolation theory as a framework. This paper reveals three affixation processes in Urhobo, which are prefixes, circumfixes and suffixes. Affixation is a derivational aspect of morphology and it brings about change in the grammatical class of the word or rather, provides additional semantic information to a word. The paper discovers that new words are derived in the language by attaching an affix either at the beginning (prefix), or to the end (suffix) of a word. This process is very productive in the language. The paper reveals that the head of a word using prefix is left-right branching while for suffixes, it is right-left branching. The prefix and suffix attached to a root to form the circumfix project to be the head of a word. The study observes circumfixes to have two heads. Feature percolation theory is also used to determine the head of a word in Urhobo. The paper concludes by recommending further researches on the use of theories especially morphological theories in analysing morphological processes in Urhobo.