Semantic Labeling Research Articles

Standardizing imaging findings representation: harnessing Common Data Elements semantics and Fast Healthcare Interoperability Resources structures.

Designing a framework representing radiology results in a standards-based data structure using joint Radiological Society of North America/American College of Radiology Common Data Elements (CDEs) as the semantic labels on standard structures. This allows radiologist-created report data to integrate with artificial intelligence-generated results for use throughout downstream systems. We developed a framework modeling radiology findings as Health Level 7 (HL7) Fast Healthcare Interoperability Resources (FHIR) observations using CDE set/element identifiers as standardized semantic labels. This framework deploys CDE identifiers to specify radiology findings and attributes, providing consistent labels for radiology report concepts-diagnoses, recommendations, tabular/quantitative data-with built-in integration with RadLex, SNOMED CT, LOINC, and other ontologies. Observation structures fit within larger HL7 FHIR DiagnosticReport resources, providing output including both nuanced text and structured data. Labeling radiology findings as discrete data for interchange between systems requires two components: structure and semantics. CDE definitions provide semantic identifiers for findings and their component values. The FHIR observation resource specifies a structure for associating identifiers with radiology findings in the context of reports, with CDE-encoded observations referring to definitions for CDE identifiers in a central repository. The discussion includes an example of encoding pulmonary nodules on a chest CT as CDE-labeled observations, demonstrating the application of this framework to exchange findings throughout the imaging workflow, making imaging data available to downstream clinical systems. CDE-labeled observations establish a lingua franca for encoding, exchanging, and consuming radiology data at the level of individual findings, facilitating use throughout healthcare systems. CDE-labeled FHIR observation objects can increase the value of radiology results by facilitating their use throughout patient care.

Label-semantics enhanced multi-layer heterogeneous graph convolutional network for Aspect Sentiment Quadruplet Extraction

Aspect-based sentiment analysis (ABSA) is a pivotal area within natural language processing (NLP) that focuses on extracting fine-grained sentiment information from text. A particularly demanding task within ABSA is Aspect Sentiment Quadruplet Extraction (ASQE), which aims to identify quadruplets comprising aspect terms, their corresponding opinion terms, sentiment polarity, and categories. This level of analysis is crucial for downstream applications such as sentiment monitoring and user experience research, offering nuanced insights into textual data. However, current methodologies fall short of fully leveraging the rich linguistic features of sentences and the semantic information of labels embedded within sentences. To address this gap, this paper introduces a novel framework, the Label-Semantics Enhanced Multi-layer Heterogeneous Graph Convolutional Network (LSEMH-GCN), specifically designed for ASQE. Our model integrates sentence linguistic features and label semantics to construct a graph neural network tailored for this task. It employs a multi-layer graph convolutional network that synergizes various linguistic features, and utilizes Biaffine attention to enrich the label probability distribution for token pairs with semantic label information. Furthermore, our approach introduces a token pair vector concatenation strategy alongside an advanced asymmetric label tagging scheme to enhance quadruplet extraction. Comprehensive evaluations on benchmark datasets reveal that LSEMH-GCN significantly surpasses existing state-of-the-art models, establishing a new benchmark for ASQE. Our model achieves an average F1 score improvement of 15.52% and 12.30% on the Restaurant-ACOS and Laptop-ACOS datasets, respectively.

Discrete online cross-modal hashing with consistency preservation

Online cross-modal hashing has attracted widespread attention with the rapid expansion of large-scale streaming data, which can reduce storage requirements and enhance efficiency for online cross-modal retrieval. However, despite promising progress, existing methods still suffer from defective accuracy in a way, primarily attributed to two issues: insufficient semantic information exploitation and mismatched training-retrieval process. To address these challenges, we propose a novel supervised hashing method with dual consistency preservation, called Discrete Online Cross-Modal Hashing (DOCMH). On the one hand, we design more informative continuous semantic labels and fine-grained similarity graphs to preserve semantic consistency across different streaming data chunks and modality representations. On the other hand, we propose an effective modality deviation calibration mechanism for preserving learning process consistency between the training and retrieval phases. Extensive experiments on three widely used benchmark datasets demonstrate the superior performance of the proposed DOCMH under various scenarios.

Approximate ground truth generation for semantic labeling of historical documents with minimal human effort

Deep learning approaches have shown high performance for layout analysis of historical documents, provided that enough labeled data is available. This is not an issue for generic tasks such as image binarization, text graphics separation, or text line and text block detection but can become an impediment for more specialized tasks specific to one or a few books only. This paper addresses layout analysis of medieval books with rich and complex layouts, for which no labeled data is initially available. The proposed strategy consists of training an initial model with artificial data created to reflect the rules a deep neural network should learn. Then, the model is iteratively fine-tuned by mixing the artificial data with real data obtained by previous predictions, post-processed, and manually selected by an expert user. Such a strategy needs less human effort than manual ground truthing. The approach is qualitatively and quantitatively assessed and shows that the system converges to an accurate model that finally produces approximate ground truth stable and good enough to train a final model to solve the targeted task with high accuracy.

Efficient Calculation of Multi-Scale Features for MMS Point Clouds

Abstract. Point clouds acquired by Mobile Mapping System (MMS) are useful for creating 3D maps that can be used for autonomous driving and infrastructure development. However, many applications require semantic labels to each point of the point clouds, and the manual labeling process is very time consuming and expensive. Therefore, there is a strong need to develop a method to automatically assigning semantic labels. For automatic labeling tasks, classification methods using multiscale features are effective because multiscale features include features of various scales of roadside objects. Multiscale features are calculated using points inside spheres of multiscale radii centered at each point in a point cloud. When calculating multiscale features that are useful for classifying MMS point clouds, it is necessary to calculate features using relatively large radii. However, when calculating multiscale features using wide range of neighbor points, existing methods, such as kd-tree, require unacceptably long computation time for neighbor search. In this paper, we propose a method to calculate multiscale features in practical time for semantic labeling of large-scale point clouds. In our method, an MMS point cloud is first divided into small spherical regions. Then, radius search using multiscale radii is performed, and multiscale features are calculated using those neighbor points. Our experimental results showed that our method achieved significantly faster computational performance than conventional methods, and multiscale features could be calculated from large-scale point clouds in practical time.

Depth-Aware Panoptic Segmentation

Abstract. Panoptic segmentation unifies semantic and instance segmentation and thus delivers a semantic class label and, for so-called thing classes, also an instance label per pixel. The differentiation of distinct objects of the same class with a similar appearance is particularly challenging and frequently causes such objects to be incorrectly assigned to a single instance. In the present work, we demonstrate that information on the 3D geometry of the observed scene can be used to mitigate this issue: We present a novel CNN-based method for panoptic segmentation which processes RGB images and depth maps given as input in separate network branches and fuses the resulting feature maps in a late fusion manner. Moreover, we propose a new depth-aware dice loss term which penalises the assignment of pixels to the same thing instance based on the difference between their associated distances to the camera. Experiments carried out on the Cityscapes dataset show that the proposed method reduces the number of objects that are erroneously merged into one thing instance and outperforms the method used as basis by +2.2% in terms of panoptic quality.

UG-schematic Annotation for Event Nominals: A Case Study in Mandarin Chinese

Abstract Divergence of languages observed at the surface level is a major challenge encountered by multilingual data representation, especially when typologically distant languages are involved. Drawing inspiration from a formalist Chomskyan perspective towards language universals, Universal Grammar (UG), this article uses deductively pre-defined universals to analyze a multilingually heterogeneous phenomenon, event nominals. In this way, deeper universality of event nominals beneath their huge divergence in different languages is uncovered, which empowers us to break barriers between languages and thus extend insights from some synthetic languages to a non-inflectional language, Mandarin Chinese. Our empirical investigation also demonstrates this UG-inspired schema is effective: With its assistance, the inter-annotator agreement (IAA) for identifying event nominals in Mandarin grows from 88.02% to 94.99%, and automatic detection of event-reading nominalizations on the newly-established data achieves an accuracy of 94.76% and an F1 score of 91.3%, which significantly surpass those achieved on the pre-existing resource by 9.8% and 5.2%, respectively. Our systematic analysis also sheds light on nominal semantic role labeling. By providing a clear definition and classification on arguments of event nominal, the IAA of this task significantly increases from 90.46% to 98.04%.

A Comparative Review on Enhancing Visual Simultaneous Localization and Mapping with Deep Semantic Segmentation.

Visual simultaneous localization and mapping (VSLAM) enhances the navigation of autonomous agents in unfamiliar environments by progressively constructing maps and estimating poses. However, conventional VSLAM pipelines often exhibited degraded performance in dynamic environments featuring mobile objects. Recent research in deep learning led to notable progress in semantic segmentation, which involves assigning semantic labels to image pixels. The integration of semantic segmentation into VSLAM can effectively differentiate between static and dynamic elements in intricate scenes. This paper provided a comprehensive comparative review on leveraging semantic segmentation to improve major components of VSLAM, including visual odometry, loop closure detection, and environmental mapping. Key principles and methods for both traditional VSLAM and deep semantic segmentation were introduced. This paper presented an overview and comparative analysis of the technical implementations of semantic integration across various modules of the VSLAM pipeline. Furthermore, it examined the features and potential use cases associated with the fusion of VSLAM and semantics. It was found that the existing VSLAM model continued to face challenges related to computational complexity. Promising future research directions were identified, including efficient model design, multimodal fusion, online adaptation, dynamic scene reconstruction, and end-to-end joint optimization. This review shed light on the emerging paradigm of semantic VSLAM and how deep learning-enabled semantic reasoning could unlock new capabilities for autonomous intelligent systems to operate reliably in the real world.

Unsupervised Scene Image Text Segmentation Based on Improved CycleGAN

Scene image text segmentation is an important task in computer vision, but the complexity and diversity of backgrounds make it challenging. All supervised image segmentation tasks require paired semantic label data to ensure the accuracy of segmentation, but semantic labels are often difficult to obtain. To solve this problem, we propose an unsupervised scene image text segmentation model based on the image style transfer model cyclic uniform Generation Adversarial network (CycleGAN), which is trained by partial unpaired label data. Text segmentation is achieved by converting a complex background to a simple background. Since the images generated by CycleGAN cannot retain the details of the text content, we also introduced the Atrous spatial Pyramid pool module (ASPP) to obtain the features of the text from multiple scales. The resulting image quality is improved. The proposed method is verified by experiments on a synthetic data set, the IIIT 5k word data set and the MACT data set, which effectively segments the text and preserves the details of the text content.

Topological 3D reconstruction of multiple anatomical structures from volumetric medical data

In the medical field, usually, practitioners mainly base their analysis on 2D slices produced from MRI or CT-scans that correspond to restricted views of a pathology. To facilitate the work of doctors, increase diagnostic accuracy and cross-reference multimodal data, a 3D reconstruction is required. However, most of the time, reconstruction methods fail at visualizing complex and noisy data made up of several tissues. Indeed, these methods often build each tissue independently so that the consistency of the global model is not ensured: overlaps may appear between segments whereas some disjointed volumes exhibit empty spaces. This paper presents a complete topologically consistent reconstruction system from 3D medical acquisitions such as MRI or CT-scans. Compared to other methods, our system offers a single volumetric representation of an organ corresponding to a 3D space partition, where a semantic label is associated to each volume to identify the represented tissue and adjacency between volumes is explicitly and precisely defined. This partition is controlled and free from topological and geometric defects usually found in other 3D reconstruction approaches. Experimental studies were conducted on MRI datasets of brains resulting in consistent reconstructions. An application of the model for calculating the distribution of physiological data in brain tissue is also shown.

YS-SLAM: YOLACT++ based semantic visual SLAM for autonomous adaptation to dynamic environments of mobile robots

Aiming at the problem of poor autonomous adaptability of mobile robots to dynamic environments, this paper propose a YOLACT++ based semantic visual SLAM for autonomous adaptation to dynamic environments of mobile robots. First, a light-weight YOLACT++ is utilized to detect and segment potential dynamic objects, and Mahalanobis distance is combined to remove feature points on active dynamic objects, also, epipolar constraint and clustering are employed to eliminate feature points on passive dynamic objects. Then, in terms of the semantic labels of dynamic and static components, the global semantic map is divided into three parts for construction. The semantic overlap and uniform motion model are chose to track moving objects and the dynamic components are added to the background map. Finally, a 3D semantic octree map is constructed that is consistent with the real environment and updated in real time. A series of simulations and experiments demonstrated the feasibility and effectiveness of the proposed approach.

A novel self-supervised sentiment classification approach using semantic labeling based on contextual embeddings

AbstractSentiment Analysis (SA) is a domain or context-oriented task since the sentiment words convey different sentiments in various domains. As a result, the domain-independent lexicons cannot correctly recognize the sentiment of domain-dependent words. To address this problem, this paper proposes a novel self-supervised SA method based on semantic similarity, contextual embedding, and Deep Learning Techniques. It introduces a new Pseudo-label generator that estimates the pseudo-labels of samples using semantic similarity between the samples and their sentiment words. It proposes two new concepts to calculate semantic similarity: The Soft-Cosine Similarity of a sample with its Positive words (SCSP) and the Soft-Cosine Similarity of a document with its Negative words (SCSN). Then, the Pseudo-label generator uses these concepts and the number of sentiment words to estimate the label of each sample. Later on, a novel method is proposed to find the samples with highly accurate pseudo-labels. Finally, a hybrid classifier, composed of a Convolutional Neural Network (CNN) and a Gated Recurrent Unit (GRU), is trained using these highly accurate pseudo-labeled data to predict the label of unseen data. The comparison of the proposed method with the lexicons and other similar existing methods demonstrates that the proposed method outperforms them in terms of accuracy, precision, recall, and F1 score.

More Than Syntaxes: Investigating Semantics to Zero-shot Cross-lingual Relation Extraction and Event Argument Role Labelling

Syntactic dependency structures are commonly utilized as language-agnostic features to solve the word order difference issues in zero-shot cross-lingual relation and event extraction tasks. However, while sentences in multiple forms can be employed to express the same meaning, the syntactic structure may vary considerably in specific scenarios. To fix this problem, we find semantics are rarely considered, which could provide a more consistent semantic analysis of sentences and be served as another bridge between different languages. Therefore, in this article, we introduce Syntax and Semantic Driven Network (SSDN) to equip syntax and semantic knowledge across languages simultaneously. Specifically, predicate–argument structures from semantic role labelling are explicitly incorporated into word representations. Then, a semantic-aware relational graph convolutional network and a transformer-based encoder are utilized to model both semantic dependency and syntactic dependency structures, respectively. Finally, a fusion module is introduced to integrate output representations adaptively. We conduct experiments on the widely used Automatic Content Extraction 2005 English, Chinese, and Arabic datasets. The evaluation results demonstrate that the proposed method achieves the state-of-the-art performance. Further study also indicates SSDN could produce robust representations that facilitate the transfer operations across languages.

SAROS: A dataset for whole-body region and organ segmentation in CT imaging

The Sparsely Annotated Region and Organ Segmentation (SAROS) dataset was created using data from The Cancer Imaging Archive (TCIA) to provide a large open-access CT dataset with high-quality annotations of body landmarks. In-house segmentation models were employed to generate annotation proposals on randomly selected cases from TCIA. The dataset includes 13 semantic body region labels (abdominal/thoracic cavity, bones, brain, breast implant, mediastinum, muscle, parotid/submandibular/thyroid glands, pericardium, spinal cord, subcutaneous tissue) and six body part labels (left/right arm/leg, head, torso). Case selection was based on the DICOM series description, gender, and imaging protocol, resulting in 882 patients (438 female) for a total of 900 CTs. Manual review and correction of proposals were conducted in a continuous quality control cycle. Only every fifth axial slice was annotated, yielding 20150 annotated slices from 28 data collections. For the reproducibility on downstream tasks, five cross-validation folds and a test set were pre-defined. The SAROS dataset serves as an open-access resource for training and evaluating novel segmentation models, covering various scanner vendors and diseases.

Two-stage fine-tuning with ChatGPT data augmentation for learning class-imbalanced data

Classification of long-tailed distributed data is a challenging problem, which suffers from serious class imbalance and hence poor performance on tail classes, which have only a few samples. Owing to this paucity of samples, learning on the tail classes is especially challenging for fine-tuning when transferring a pretrained model to a downstream task. In this work, we present a simple modification of standard fine-tuning to cope with these challenges. Specifically, we propose a two-stage fine-tuning. In Stage 1, we fine-tune the final layer of the pretrained model with class-balanced augmented data, generated using ChatGPT. As a large generative language model, ChatGPT is capable of generating novel and contextually similar responses to a given prompt, which makes it an excellent candidate for data augmentation. In Stage 2, we perform the standard fine-tuning. Our modification has several benefits: (1) it leverages pretrained representations by only fine-tuning a small portion of the model parameters while keeping the rest untouched; (2) it allows the model to learn an initial representation of the specific task; and importantly (3) it protects the learning of tail classes from being at a disadvantage during the model updating. We conduct extensive experiments on synthetic datasets of both two-class and multi-class tasks of text classification as well as a real-world application to ADME (i.e., absorption, distribution, metabolism, and excretion) semantic drug labeling. The experimental results show that the proposed two-stage fine-tuning outperforms vanilla fine-tuning and state-of-the-art methods on the above datasets.

Weakly supervised semantic segmentation based on superpixel affinity

Weakly supervised semantic segmentation (WSSS) usually employs the method of modifying and extending class activation map (CAM) seeds to achieve semantic segmentation. However, the down-sampling of the network weakens the edge awareness of CAMs, leading to under- or over-activation problems, which affects the segmentation quality. Considering the excellent contour attachment property of superpixels and the high semantic similarity between pixels within the same superpixel, we propose a superpixel affinity-based method that uses multi-scale features to aggregate superpixels with the same semantics, providing complete localization supervision for the generation of CAMs. In order to improve the accuracy of semantic labels for superpixels, we utilize a method of deep feature reorganization to improve the quality of network-generated CAM seeds. The experimental results indicate that the proposed method has achieved satisfactory performance on PASCAL VOC 2012 and MS COCO 2014 datasets.

Visual Semantic Segmentation Based on Few/Zero-Shot Learning: An Overview

Visual semantic segmentation aims at separating a visual sample into diverse blocks with specific semantic attributes and identifying the category for each block, and it plays a crucial role in environmental perception. Conventional learning-based visual semantic segmentation approaches count heavily on large-scale training data with dense annotations and consistently fail to estimate accurate semantic labels for unseen categories. This obstruction spurs a craze for studying visual semantic segmentation with the assistance of few/zero-shot learning. The emergence and rapid progress of few/zero-shot visual semantic segmentation make it possible to learn unseen categories from a few labeled or even zero-labeled samples, which advances the extension to practical applications. Therefore, this paper focuses on the recently published few/zero-shot visual semantic segmentation methods varying from 2D to 3D space and explores the commonalities and discrepancies of technical settlements under different segmentation circumstances. Specifically, the preliminaries on few/zero-shot visual semantic segmentation, including the problem definitions, typical datasets, and technical remedies, are briefly reviewed and discussed. Moreover, three typical instantiations are involved to uncover the interactions of few/zero-shot learning with visual semantic segmentation, including image semantic segmentation, video object segmentation, and 3D segmentation. Finally, the future challenges of few/zero-shot visual semantic segmentation are discussed.

EAD-Net: Efficiently Asymmetric Network for Semantic Labeling of High-Resolution Remote Sensing Images with Dynamic Routing Mechanism

Semantic labeling of high-resolution remote sensing images (HRRSIs) holds a significant position in the remote sensing domain. Although numerous deep-learning-based segmentation models have enhanced segmentation precision, their complexity leads to a significant increase in parameters and computational requirements. While ensuring segmentation accuracy, it is also crucial to improve segmentation speed. To address this issue, we propose an efficient asymmetric deep learning network for HRRSIs, referred to as EAD-Net. First, EAD-Net employs ResNet50 as the backbone without pooling, instead of the RepVGG block, to extract rich semantic features while reducing model complexity. Second, a dynamic routing module is proposed in EAD-Net to adjust routing based on the pixel occupancy of small-scale objects. Concurrently, a channel attention mechanism is used to preserve their features even with minimal occupancy. Third, a novel asymmetric decoder is introduced, which uses convolutional operations while discarding skip connections. This not only effectively reduces redundant features but also allows using low-level image features to enhance EAD-Net’s performance. Extensive experimental results on the ISPRS 2D semantic labeling challenge benchmark demonstrate that EAD-Net achieves state-of-the-art (SOTA) accuracy performance while reducing model complexity and inference time, while the mean Intersection over Union (mIoU) score reaching 87.38% and 93.10% in the Vaihingen and Potsdam datasets, respectively.

Wildfire response of forest species from multispectral LiDAR data. A deep learning approach with synthetic data

Forests play a crucial role as the lungs and life-support system of our planet, harbouring 80% of the Earth's biodiversity. However, we are witnessing an average loss of 480 ha of forest every hour because of destructive wildfires spreading across the globe. To effectively mitigate the threat of wildfires, it is crucial to devise precise and dependable approaches for forecasting fire dynamics and formulating efficient fire management strategies, such as the utilisation of fuel models.The objective of this study was to enhance forest fuel classification that considers only structural information, such as the Prometheus model, by integrating data on the fire responses of various tree species and other vegetation elements, such as ground litter and shrubs. This distinction can be achieved using multispectral (MS) Light Detection and Ranging (LiDAR) data in mixed forests. The methodology involves a novel approach in semantic classifications of forests by generating synthetic data with semantic labels regarding fire responses and reflectance information at different spectral bands, as a real MS scanner device would detect. Forests, which are highly intricate environments, present challenges in accurately classifying point clouds. To address this complexity, a deep learning (DL) model for semantic classification was trained on synthetic point clouds in different formats to achieve the best performance when leveraging MS data.Forest plots in the study region were scanned using different Terrestrial Laser Scanning sensors at wavelengths of 905 and 1550 nm. Subsequently, an interpolation process was applied to generate the MS point clouds of each plot, and the trained DL model was applied to classify them. These classifications surpassed the average thresholds of 90% and 75% for accuracy and intersection over union, respectively, resulting in a more precise categorisation of fuel models based on the distinct responses of forest elements to fire. The results of this study reveal the potential of MS LiDAR data and DL classification models for improving fuel model retrieval in forest ecosystems and enhancing wildfire management efforts.

A construction waste landfill dataset of two districts in Beijing, China from high resolution satellite images

Construction waste is unavoidable in the process of urban development, causing serious environmental pollution. Accurate assessment of municipal construction waste generation requires building construction waste identification models using deep learning technology. However, this process requires high-quality public datasets for model training and validation. This study utilizes Google Earth and GF-2 images as the data source to construct a specific dataset of construction waste landfills in the Changping and Daxing districts of Beijing, China. This dataset contains 3,653 samples of the original image areas and provides mask-labeled images in the semantic segmentation domains. Each pixel within a construction waste landfill is classified into 4 categories of the image areas, including background area, vacant landfillable area, engineering facility area, and waste dumping area. The dataset contains 237,115,531 pixels of construction waste and 49,724,513 pixels of engineering facilities. The pixel-level semantic segmentation labels are provided to quantify the construction waste yield, which can serve as the basic data for construction waste extraction and yield estimation both for academic and industrial research.