Semantic Parsing Research Articles

Two-stage Text-to-BIMQL semantic parsing for building information model extraction using graph neural networks

With the increasing complexity of the building process, it is difficult for project stakeholders to retrieve large and multi-disciplinary building information models (BIMs). A natural language interface (NLI) is beneficial for users to query BIM models using natural language. However, parsing natural language queries (NLQs) is challenging due to ambiguous name descriptions and intricate relationships between entities. To address these issues, this study proposes a graph neural network (GNN)-based semantic parsing method that automatically maps NLQs into executable queries. Firstly, ambiguous mentions are collectively linked to referent ontological entities via a GNN-based entity linking model. Secondly, the logical forms of NLQs are interpreted through a GNN-based relation extraction model, which predicts links between mentioned entities in a heterogeneous graph fusing ontology and NLQ texts. The experiment based on 786 queries shows its outstanding performance. Moreover, a real-world case verifies the practicability of the proposed method for BIM model retrieval.

Light RAT-SQL: A RAT-SQL with More Abstraction and Less Embedding of Pre-existing Relations

RAT-SQL is among the popular framework used in the Text-To-SQL challenges for jointly encoding the database relations and questions in a way to improve the semantic parser. In this work, we propose a light version of the RAT-SQL where we dramatically reduced the number of the preexisting relations from 55 to 7 (Light RAT-SQL-7) while preserving the same parsing accuracy. To ensure the effectiveness of our approach, we trained a Light RAT-SQL-2, (with 2 embeddings) to show that there is a statistically significant difference between RAT-SQL and Light RAT-SQL-2 while Light RAT-SQL-7 can compete with RAT-SQL. Keywords: Deep learning, Natural Language Processing, Neural Semantic Parsing, Relation Aware Transformer, RAT-SQL, Text-To-SQL, Transformer.

QuEHRy: a question answering system to query electronic health records.

We propose a system, quEHRy, to retrieve precise, interpretable answers to natural language questions from structured data in electronic health records (EHRs). We develop/synthesize the main components of quEHRy: concept normalization (MetaMap), time frame classification (new), semantic parsing (existing), visualization with question understanding (new), and query module for FHIR mapping/processing (new). We evaluate quEHRy on 2 clinical question answering (QA) datasets. We evaluate each component separately as well as holistically to gain deeper insights. We also conduct a thorough error analysis for a crucial subcomponent, medical concept normalization. Using gold concepts, the precision of quEHRy is 98.33% and 90.91% for the 2 datasets, while the overall accuracy was 97.41% and 87.75%. Precision was 94.03% and 87.79% even after employing an automated medical concept extraction system (MetaMap). Most incorrectly predicted medical concepts were broader in nature than gold-annotated concepts (representative of the ones present in EHRs), eg, Diabetes versus Diabetes Mellitus, Non-Insulin-Dependent. The primary performance barrier to deployment of the system is due to errors in medical concept extraction (a component not studied in this article), which affects the downstream generation of correct logical structures. This indicates the need to build QA-specific clinical concept normalizers that understand EHR context to extract the "relevant" medical concepts from questions. We present an end-to-end QA system that allows information access from EHRs using natural language and returns an exact, verifiable answer. Our proposed system is high-precision and interpretable, checking off the requirements for clinical use.

SP-NLG: A Semantic-Parsing-Guided Natural Language Generation Framework

We propose SP-NLG: A semantic-parsing-guided natural language generation framework for logical content generation with high fidelity. Prior studies adopt large pretrained language models and coarse-to-fine decoding techniques to generate text with logic; while achieving considerable results on automatic evaluation metrics, they still face challenges in terms of logical fidelity based on human evaluation. Inspired by semantic parsing, which translates natural language utterances into executable logical forms, we propose to guide the generation process with a semantic parser. Different from semantic parsing and QA tasks, of which the logical correctness can be evaluated based on the execution result, the logic information of the generated content is implicit. To leverage a semantic parser for generation, we propose a slot-tied back-search algorithm. We follow the coarse-to-fine generation scheme, but instead of filling the slots with model predictions, which is less uncontrolled, the slot values are offline searched by the algorithm. The slot-tied back-search algorithm ties the parameters of a logic form with the slots of a template in one-to-one correspondence. We back-search the arguments that correctly execute the logic form and fill the arguments (as slot values) into the textual template to generate the final target, which ensures the logical correctness. Experiment results of a model built on SP-NLG demonstrates that our framework achieves high fidelity on logical text generation.

Server-side Cross-site Scripting Detection Powered by HTML Semantic Parsing Inspired by XSS Auditor

Cross-site Scripting attacks have been a perennial threat to web applications for many years. Conventional practices to prevent cross-site scripting attacks revolve around secure programming and client-side prevention techniques. However, client-side preventions are still prone to bypasses as the inspection is done on the user’s browser, so an adversary can alter the inspection algorithm to come up with the bypasses or even manipulate the victim to turn off the security measures. This decreases the effectiveness of the protection and leads to many web applications are still vulnerable to cross-site scripting attacks. We believe that XSS Auditor, which was pre-installed in Google Chrome browser for more than 9 years, is a great approach in combating and preventing XSS attacks. Hence, in this paper, we proposed a novel approach to thoroughly identify two types of cross-site scripting attacks through server-side filter implementation. Our proposed approach follows the original XSS Auditor mechanism implemented in Google Chrome. However, instead of placing the detection system on the client side, we design a detection mechanism that checks HTTP requests and responses as well as database responses for possible XSS attacks from the server side. From 500 payloads used to evaluate the proposed method, 442 payloads were classified correctly, thus showing that the proposed method was able to reach 88.4% accuracy. This work showed that the proposed approach is very promising in protecting users from devastating Cross-site Scripting attacks.

ReCoMIF: Reading comprehension based multi-source information fusion network for Chinese spoken language understanding

Spoken language understanding (SLU) plays a crucial role in the performance of dialogue systems. It usually includes slot filling and intent detection (SFID) tasks aiming at semantic parsing of utterances. At present, researchers focus mainly on English SLU tasks, while such investigations on Chinese utterances are not sufficient. In this paper, we first propose a reading comprehension based multi-source information fusion network, called ReCoMIF for Chinese SFID tasks by transforming the SLU task into a multi-turn question answering procedure comprising multiple choice for intent detection and span extraction for slot filling. Moreover, we present a TCM_CLS module with a concise architecture composed of TextCNN, MaxPooling, and feed forward network plus the [CLS] representation. Such three TCM_CLS modules are stacked in the proposed ReCoMIF network that can serve as sufficient integration of multi-source information originating from contexts, reading comprehension based queries, and hidden representations concerning intent and slot semantics. Finally, experimental results and ablation studies on three Chinese SLU datasets show that our proposed model can effectively fuse intent and slot information achieved by\\ state-of-the-art performance compared with other baseline methods.

Design considerations for a hierarchical semantic compositional framework for medical natural language understanding.

Medical natural language processing (NLP) systems are a key enabling technology for transforming Big Data from clinical report repositories to information used to support disease models and validate intervention methods. However, current medical NLP systems fall considerably short when faced with the task of logically interpreting clinical text. In this paper, we describe a framework inspired by mechanisms of human cognition in an attempt to jump the NLP performance curve. The design centers on a hierarchical semantic compositional model (HSCM), which provides an internal substrate for guiding the interpretation process. The paper describes insights from four key cognitive aspects: semantic memory, semantic composition, semantic activation, and hierarchical predictive coding. We discuss the design of a generative semantic model and an associated semantic parser used to transform a free-text sentence into a logical representation of its meaning. The paper discusses supportive and antagonistic arguments for the key features of the architecture as a long-term foundational framework.

Complementary networks for person re-identification

Recently, person re-identification (re-ID) has gained much attention for its extensive applications in public security. In order to mine more detailed information, some researches have attempted to extract local representations from important regions through human semantic parsing. However, some valuable visual cues beyond human body, such as backpacks and handbags, which can provide complementary information for recognition, may be misclassified as background noise. In addition, when meeting low-quality captured images or severe occlusions, the semantic regions generated by parsing model are not accurate enough, even resulting in disturbance. In this paper, we present complementary networks for person re-ID, which aims to extract more discriminative and robust local representations with complementary clues. Our model contains two branches and one of which aims to mine potential discriminative information in the background that is useful for identifying a person, as a complement for foreground semantics of human body parts. Another branch is designed to capture the salient regions in global scope so as to make up for the problem of poor discrimination of representations caused by the inaccurate semantic confidence maps of former branch. Experiments show that our model achieves competitive performance in many cases.

MSTIL: Multi-cue Shape-aware Transferable Imbalance Learning for effective graphic API recommendation

Application Programming Interface (API) recommendation based on graphs is a valuable task in the fields of data visualization and software engineering. However, this task was previously undefined until a recently published paper coining the task as Plot2API and utilizing a deep learning-based method named SPGNN. Compared to general image classification methods, this dedicated approach uses semantic parsing to exploit deep features and yields better performance. However, its performance declines sharply in unbalanced datasets, thus limiting its generalizability. To address this issue, we propose a method named Multi-cue Shape and software engineering-aware Transferable Imbalance Learning (MSTIL), consisting of three major components: Cross-Language Shape-Aware Plot Transfer Learning (CLSAPTL), Cross-Language API Semantic Similarity-based Data Augmentation (CLASSDA), and Imbalance Plot2API Learning (IPL). Motivated by the hierarchical classification of the graphs, CLSAPTL guides the model to learn the graphs’ class hierarchy and thereby enabling the model to learn more transferable visual features. Given that a graph can be associated with multiple APIs and motivated by the fact that many APIs that exert similar functions in different languages have semantically similar names, CLASSDA leverages the samples of APIs with semantically similar names to assist in feature learning. Finally, inspired by the essence of softmax cross entropy loss, IPL alleviates the imbalances between positive and negative samples during training. We conduct our experiments on two public datasets. Extensive experimental results shows that MSTIL improves the performance of classic CNNs along with the state-of-the-art method, demonstrating its effectiveness. Specifically, MSTIL has an average relative mAP improvement of 12.94% across the models on all datasets.

Analyzing Geographic Questions Using Embedding-based Topic Modeling

Recently, open-domain question-answering systems have achieved tremendous progress because of developments in large language models (LLMs), and have successfully been applied to question-answering (QA) systems, or Chatbots. However, there has been little progress in open-domain question answering in the geographic domain. Existing open-domain question-answering research in the geographic domain relies heavily on rule-based semantic parsing approaches using few data. To develop intelligent GeoQA agents, it is crucial to build QA systems upon datasets that reflect the real users’ needs regarding the geographic domain. Existing studies have analyzed geographic questions using the geographic question corpora Microsoft MAchine Reading Comprehension (MS MARCO), comprising real-world user queries from Bing in terms of structural similarity, which does not discover the users’ interests. Therefore, we aimed to analyze location-related questions in MS MARCO based on semantic similarity, group similar questions into a cluster, and utilize the results to discover the users’ interests in the geographic domain. Using a sentence-embedding-based topic modeling approach to cluster semantically similar questions, we successfully obtained topic models that could gather semantically similar documents into a single cluster. Furthermore, we successfully discovered latent topics within a large collection of questions to guide practical GeoQA systems on relevant questions.

Controllable Image Synthesis With Attribute-Decomposed GAN.

This paper proposes Attribute-Decomposed GAN (ADGAN) and its enhanced version (ADGAN++) for controllable image synthesis, which can produce realistic images with desired attributes provided in various source inputs. The core ideas of the proposed ADGAN and ADGAN++ are both to embed component attributes into the latent space as independent codes and thus achieve flexible and continuous control of attributes via mixing and interpolation operations in explicit style representations. The major difference between them is that ADGAN processes all component attributes simultaneously while ADGAN++ utilizes a serial encoding strategy. More specifically, ADGAN consists of two encoding pathways with style block connections and is capable of decomposing the original hard mapping into multiple more accessible subtasks. In the source pathway, component layouts are extracted via a semantic parser and the segmented components are fed into a shared global texture encoder to obtain decomposed latent codes. This strategy allows for the synthesis of more realistic output images and the automatic separation of un-annotated component attributes. Although the original ADGAN works in a delicate and efficient manner, intrinsically it fails to handle the semantic image synthesizing task when the number of attribute categories is huge. To address this problem, ADGAN++ employs the serial encoding of different component attributes to synthesize each part of the target real-world image, and adopts several residual blocks with segmentation guided instance normalization to assemble the synthesized component images and refine the original synthesis result. The two-stage ADGAN++ is designed to alleviate the massive computational costs required when synthesizing real-world images with numerous attributes while maintaining the disentanglement of different attributes to enable flexible control of arbitrary component attributes of the synthesized images. Experimental results demonstrate the proposed methods' superiority over the state of the art in pose transfer, face style transfer, and semantic image synthesis, as well as their effectiveness in the task of component attribute transfer. Our code and data are publicly available at https://github.com/menyifang/ADGAN.

Meta-Learning a Cross-lingual Manifold for Semantic Parsing

Abstract Localizing a semantic parser to support new languages requires effective cross-lingual generalization. Recent work has found success with machine-translation or zero-shot methods, although these approaches can struggle to model how native speakers ask questions. We consider how to effectively leverage minimal annotated examples in new languages for few-shot cross-lingual semantic parsing. We introduce a first-order meta-learning algorithm to train a semantic parser with maximal sample efficiency during cross-lingual transfer. Our algorithm uses high-resource languages to train the parser and simultaneously optimizes for cross-lingual generalization to lower-resource languages. Results across six languages on ATIS demonstrate that our combination of generalization steps yields accurate semantic parsers sampling ≤10% of source training data in each new language. Our approach also trains a competitive model on Spider using English with generalization to Chinese similarly sampling ≤10% of training data.1

Schema dependency-enhanced curriculum pre-training for table semantic parsing

Large pre-trained models exhibit improved table-semantic-parsing performances by leveraging large-scale corpora to enhance the representation learning ability of semantic parsers. However, existing table pre-training methods do not sufficiently consider the explicit interactions among natural language (NL) questions, SQL queries, and the corresponding database schemas, which are essential for table semantic parsing. To overcome the aforementioned limitation, this study designs a novel schema dependency prediction (SDP) objective to incorporate SQL-aware schema linking information into table pre-training. Specifically, SDP aims to predict the schema dependency connecting the NL question, table schema, and triggered SQL operations. We further propose a schema-aware curriculum learning approach (SAC) to mitigate the impact of noise present in the pre-training data and train the table pre-training model in an easy-to-hard manner. We evaluate the effectiveness of our pre-training framework by fine-tuning it on three downstream benchmarks (Spider, Spider-DK, and SQUALL). Experimental results demonstrate that the proposed pre-training method outperforms existing methods considered for comparison.

Local-to-Global Framework for Multi-target Frame Identification

Frame identification is a crucial and challenging task in frame semantic parsing, where the objective is to determine the most appropriate frame for a given target within its context. Existing approaches typically classify each target individually, overlooking the potential interactions among different targets. To address these limitations, we propose a novel two-step prediction framework that encompasses both local and global perspectives for target identification. Additionally, we leverage the frame ontology graph to enhance the interactions among multiple targets by incorporating rich frame ontology knowledge. Moreover, we introduce a dynamic masking strategy during model training. This strategy encourages our model to adopt a global view during prediction, avoiding local optimization. Experimental results demonstrate the superiority of our model compared to previous approaches on FrameNet1.5, and it achieves competitive performance on FrameNet1.7. Furthermore, supplementary experiments and analyses provide additional evidence of the effectiveness of our proposed model.

Biphasic Face Photo-Sketch Synthesis via Semantic-Driven Generative Adversarial Network With Graph Representation Learning.

Biphasic face photo-sketch synthesis has significant practical value in wide-ranging fields such as digital entertainment and law enforcement. Previous approaches directly generate the photo-sketch in a global view, they always suffer from the low quality of sketches and complex photograph variations, leading to unnatural and low-fidelity results. In this article, we propose a novel semantic-driven generative adversarial network to address the above issues, cooperating with graph representation learning. Considering that human faces have distinct spatial structures, we first inject class-wise semantic layouts into the generator to provide style-based spatial information for synthesized face photographs and sketches. In addition, to enhance the authenticity of details in generated faces, we construct two types of representational graphs via semantic parsing maps upon input faces, dubbed the intraclass semantic graph (IASG) and the interclass structure graph (IRSG). Specifically, the IASG effectively models the intraclass semantic correlations of each facial semantic component, thus producing realistic facial details. To preserve the generated faces being more structure-coordinated, the IRSG models interclass structural relations among every facial component by graph representation learning. To further enhance the perceptual quality of synthesized images, we present a biphasic interactive cycle training strategy by fully taking advantage of the multilevel feature consistency between the photograph and sketch. Extensive experiments demonstrate that our method outperforms the state-of-the-art competitors on the CUHK Face Sketch (CUFS) and CUHK Face Sketch FERET (CUFSF) datasets.

OccluMix: Towards De-Occlusion Virtual Try-on by Semantically-Guided Mixup

Image Virtual try-on aims at replacing the cloth on a personal image with a garment image (in-shop clothes), which has attracted increasing attention from the multimedia and computer vision communities. Prior methods successfully preserve the character of clothing images, however, occlusion remains a pernicious effect for realistic virtual try-on. In this work, we first present a comprehensive analysis of the occlusions and categorize them into two aspects: i) Inherent-Occlusion: the ghost of the former cloth still exists in the try-on image; ii) Acquired-Occlusion: the target cloth warps to the unreasonable body part. Based on the in-depth analysis, we find that the occlusions can be simulated by a novel semantically-guided mixup module, which can generate semantic-specific occluded images that work together with the try-on images to facilitate training a de-occlusion try-on (DOC-VTON) framework. Specifically, DOC-VTON first conducts a sharpened semantic parsing on the try-on person. Aided by semantics guidance and pose prior, various complexities of texture are selectively blending with human parts in a copy-and-paste manner. Then, the Generative Module (GM) is utilized to take charge of synthesizing the final try-on image and learning to de-occlusion jointly. In comparison to the state-of-the-art methods, DOC-VTON achieves better perceptual quality by reducing occlusion effects.

Multiple POS Dependency-Aware Mixture of Experts for Frame Identification

Frame identification, which is finding the exact evoked frame for a target word in a given sentence, is a fundamental and crucial prerequisite for frame semantic parsing. It is generally seen as a classification task for target words, whose contextual representations are usually obtained using a neural network like BERT as an encoder, and enriched with a joint learning model or the knowledge of FrameNet. However, the distinction at a fine-grained level, such as the delicate differences in the information of syntax and PropBank roles caused by different parts-of-speech (POS) of target words, is neglected. We propose a Multiple POS Dependency-aware Mixture of Experts(MPDaMoE) network that integrates five types of information, consisting of the syntactic information of target words whose POS are nominal, adjectival, adverbial, or prepositional, and the PropBank role information of target words whose POS are only verbal. To better learn such information, a Mixture of Experts network is employed, in which every expert is a Graph Convolutional Network, to incorporate the different dependency information of target words. Our model outperforms state-of-the-art models in experiments on two benchmark datasets, which shows its effectiveness.

PhraseTransformer: an incorporation of local context information into sequence-to-sequence semantic parsing

PhraseTransformer: an incorporation of local context information into sequence-to-sequence semantic parsing

ImageSchemaNet: A framester graph for embodied commonsense knowledge

Commonsense knowledge is a broad and challenging area of research which investigates our understanding of the world as well as human assumptions about reality. Deriving directly from the subjective perception of the external world, it is intrinsically intertwined with embodied cognition. Commonsense reasoning is linked to human sense-making, pattern recognition and knowledge framing abilities. This work presents a new resource that formalizes the cognitive theory of image schemas. Image schemas are dynamic conceptual building blocks originating from our sensorimotor interactions with the physical world, and enable our sense-making cognitive activity to assign coherence and structure to entities, events and situations we experience everyday. ImageSchemaNet is an ontology that aligns pre-existing resources, such as FrameNet, VerbNet, WordNet and MetaNet from the Framester hub, to image schema theory. This article describes an empirical application of ImageSchemaNet, combined with semantic parsers, on the task of annotating natural language sentences with image schemas.

Compositionality in Computational Linguistics

Neural models greatly outperform grammar-based models across many tasks in modern computational linguistics. This raises the question of whether linguistic principles, such as the Principle of Compositionality, still have value as modeling tools. We review the recent literature and find that while an overly strict interpretation of compositionality makes it hard to achieve broad coverage in semantic parsing tasks, compositionality is still necessary for a model to learn the correct linguistic generalizations from limited data. Reconciling both of these qualities requires the careful exploration of a novel design space; we also review some recent results that may help in this exploration.