Annotated Research Articles

Semantic automatic annotation method based on artificial intelligence for electric power internet of things

AbstractThe development of the power Internet of Things is currently underway, and a proposal for a semantic Internet of Things based on artificial intelligence algorithms is made to address the challenges in obtaining prior knowledge for heterogeneous data fusion, improving the real‐time performance of the ontology library, and enhancing the efficiency of manual labeling of instance object data in the power field. This proposal introduces an Automatic Semantic Annotation Method to provide an effective knowledge organization model for sensor systems. Data mining knowledge is utilized to drive ontology update and improvement, resulting in more accurate semantic annotation and enhanced machine understanding. Experimental results show that artificial intelligence algorithms can automatically extract concepts from sensory data and achieve automatic semantic annotation during ontology instantiation.

On Undisputed Sets in Abstract Argumentation

We introduce the notion of an undisputed set for abstract argumentation frameworks, which is a conflict-free set of arguments, such that its reduct contains no non-empty admissible set. We show that undisputed sets, and the stronger notion of strongly undisputed sets, provide a meaningful approach to weaken admissibility and deal with the problem of attacks from self-attacking arguments, in a similar manner as the recently introduced notion of weak admissibility. We investigate the properties of our new semantical notions and show certain relationships to classical semantics, in particular that undisputed sets are a generalisation of preferred extensions and strongly undisputed sets are a generalisation of stable extensions. We also investigate the computational complexity of standard reasoning tasks with these new notions and show that they lie on the second and third level of the polynomial hierarchy, respectively.

KnowGL: Knowledge Generation and Linking from Text

We propose KnowGL, a tool that allows converting text into structured relational data represented as a set of ABox assertions compliant with the TBox of a given Knowledge Graph (KG), such as Wikidata. We address this problem as a sequence generation task by leveraging pre-trained sequence-to-sequence language models, e.g. BART. Given a sentence, we fine-tune such models to detect pairs of entity mentions and jointly generate a set of facts consisting of the full set of semantic annotations for a KG, such as entity labels, entity types, and their relationships. To showcase the capabilities of our tool, we build a web application consisting of a set of UI widgets that help users to navigate through the semantic data extracted from a given input text. We make the KnowGL model available at https://huggingface.co/ibm/knowgl-large.

AN APPROACH TO BUILD A COMPLETE DIGITAL REPORT OF THE NOTRE DAME CATHEDRAL AFTER THE FIRE, USING THE AIOLI PLATFORM

Abstract. With the development of digital technologies in the documentation methods for cultural and architectural heritage, platforms and tools have emerged as solutions to collect, manage and produce data for multidisciplinary monitoring and studying purposes. In the context of an incident such as the fire of the Notre-Dame de Paris in April 2019, the considerable mobilization of the architects, scientists and researchers, has led the "digital data" working group to conceive a digital ecosystem capable to collect and integrate existing data, produce new data, share and archive, and finally structure and semantically enrich. One of the particular aspects of this works involves an approach to build a complete digital report of the Notre-Dame de Paris cathedral. With the help of photogrammetric acquisitions of the cathedral, leading to point clouds generation, projects were created in the aïoli platform. The structuring of the collected data (condition report, architectural descriptions, chemical and physical analysis…) has then led to the construction of semantic annotations, accompanied by description sheets and attachments. The vast multidisciplinary studies conducted on the cathedral as well as the large number of projects and annotations built over the course of 4 years constitute a singular collection of data, that will be the object of cross comparison and data interpretation in the future years.

ENABLING SCAN-TO-BIM WORKFLOW FOR HERITAGE CONSERVATION AND MANAGEMENT PROCESS

Abstract. The paper stresses the potential of digital technology in the field of Cultural Heritage (CH), focusing on Architectural Heritage (AH) conservation and management process. It presents a workflow based on non-parametric automatic 3D modeling and Industry Foundation Classes (IFC) authoring to combine geometry and semantic information. The goal of such a methodology is to overcome time-consuming manual tasks that still limit the Historical Building Information Modelling (HBIM), which nevertheless proves to be one of the most effective digital tools for the conservation and management of AH in the medium to long term. The proposed workflow consists of five steps: the 3D survey campaign and raw data processing, the semantic annotation of images according to different information layers, the annotation transfer from images to the point cloud representing the studied architecture, the obtained Point Cloud Information Model (PCIM) automatic segmentation and 3D mesh wrapping, and finally the IFC generation for the HBIM representation. This unconventional digital chain was tested to support the knowledge representation of Villa Leonardi, an historical house in the Marche Region (Italy). The final output is a 3D model of the building, automatically processed from digital survey data, and semantically enriched thanks to the material consistency and decay analysis carried out and annotated on spherical panoramas. The effectiveness of this approach was proven in the conservation project for the façades of the house: enriched data on materials and decay supported an action plan of restoration works, which was then recorded in the HBIM model, thus permitting time and cost control.

Application of machine learning methods for filling and updating nuclear knowledge bases

The paper deals with issues of designing and creating knowledge bases in the field of nuclear science and technology. The authors present the results of searching for and testing optimal classification and semantic annotation algorithms applied to the textual network content for the convenience of computer-aided filling and updating of scalable semantic repositories (knowledge bases) in the field of nuclear physics and nuclear power engineering and, in the future, for other subject areas, both in Russian and English. The proposed algorithms will provide a methodological and technological basis for creating problem-oriented knowledge bases as artificial intelligence systems, as well as prerequisites for the development of semantic technologies for acquiring new knowledge on the Internet without direct human participation. Testing of the studied machine learning algorithms is carried out by the cross-validation method using corpora of specialized texts. The novelty of the presented study lies in the application of the Pareto optimality principle for multi-criteria evaluation and ranking of the studied algorithms in the absence of a priori information about the comparative significance of the criteria. The project is implemented in accordance with the Semantic Web standards (RDF, OWL, SPARQL, etc.). There are no technological restrictions for integrating the created knowledge bases with third-party data repositories as well as metasearch, library, reference or information and question-answer systems. The proposed software solutions are based on cloud computing using DBaaS and PaaS service models to ensure the scalability of data warehouses and network services. The created software is in the public domain and can be freely replicated.

Semi-automatic classification of digital heritage on the Aïoli open source 2D/3D annotation platform via machine learning and deep learning

Today, digital tools represent a privileged means of describing and understanding of cultural heritage. However, existing hybrid methods for semantic annotation on digital models are mostly manual and require considerable human intervention. This may cause data dispersion or data alteration issues, preventing the digital collection of multidisciplinary heritage information in collaborative valorization and conservation projects. Tackling this problem, this work presents three different methods for more automated 2D/3D data annotation, making the most of Machine Learning or Deep Learning and of the collaborative, web-based annotation platform Aïoli. The proposed approaches are designed to assist heritage experts in the hybrid annotation of architectural objects, e.g., to identify building components, degradation patterns, mapping of alterations, materials etc. and to share and access 2D/3D information via the web. The work aims at providing public and private stakeholders in charge of restoration and conservation activities with a set of more automated annotation tools, allowing to: i) preserve data in the transition to different representation types; ii) propagate analytically relevant information via digital models and/or images; iii) share heritage-related data on the web with an open source strategy. The validated workflows sketch the outlines of a dedicated and user-friendly toolbar for semi-automatic annotation, directly integrated into the Aïoli user interface, where automated annotation tools are readily available to the community of heritage professionals to properly support documentation and restoration activities.

Large-scale Unsupervised Semantic Segmentation.

Empowered by large datasets, e.g., ImageNet and MS COCO, unsupervised learning on large-scale data has enabled significant advances for classification tasks. However, whether the large-scale unsupervised semantic segmentation can be achieved remains unknown. There are two major challenges: i) we need a large-scale benchmark for assessing algorithms; ii) we need to develop methods to simultaneously learn category and shape representation in an unsupervised manner. In this work, we propose a new problem of large-scale unsupervised semantic segmentation (LUSS) with a newly created benchmark dataset to help the research progress. Building on the ImageNet dataset, we propose the ImageNet-Sdataset with 1.2 million training images and 50k high-quality semantic segmentation annotations for evaluation. Our benchmark has a high data diversity and a clear task objective. We also present a simple yet effective method that works surprisingly well for LUSS. In addition, we benchmark related un/weakly/fully supervised methods accordingly, identifying the challenges and possible directions of LUSS. The benchmark and source code is publicly available at https://github.com/LUSSeg.

Unified building change detection pre-training method with masked semantic annotations

Building change detection (CD) using remote sensing images plays a vital role in urban development, and deep learning models attracted attention for their potential to accomplish CD tasks automatically. However, most methods are still facing challenges, such as the costly and time-consuming process of constructing CD datasets and the severely imbalanced distribution of positive and negative samples preventing loss functions from functioning as desired in the training process. Inspired by weak supervision have demonstrated excellent performance in solving the above-mentioned problems, a unified change detection pre-training paradigm is proposed to accomplish the CD task using a small number of samples and improve the inference accuracy of building change detection. The keys of this proposed method are as follows. First, the pre-training paradigm detects building changes using pseudo-labels and them generated by highly available semantic segmentation datasets. Second, a balanced sample distribution is ensured by using the proposed method for semantic masked building areas and controlling the proportion of the areas. Third, multi-task networks for simultaneous building extraction and change detection are used in the proposed unified paradigm, owing to the semantic information can be employed as an effective supervision signal to assist with the CD training to solve the problem that pseudo-labels adversely affect the ability of the algorithm to converge. In particular, experiments were performed on three challenging datasets. For aerial WHU-CD and satellite Gaofen Challenge-CD datasets, our pre-trained weights generated with pseudo-bitemporal samples were applied to subsets containing different proportions of ground truth for fine-tuning, respectively. Notably, 10% of ground truth for fine-tuning with our pre-trained weights obtained intersection over union (IoU) comparable to that obtained using 100% of the CD ground truth without our pre-trained weights, whereas an even greater IoU was achieved using 30% of ground truth with our pre-trained weights. Experiments demonstrated that the performance of our method based on a small number of samples was superior to that of conventional supervised learning methods. When conducting experiments with 100% ground truth, the results showed that the use of our pre-trained weights yields IoU that substantially exceeds that of conventional supervised learning methods. The results of experiments conducted on the LEVIR-CD dataset proved the excellent transferable performance of our method. The proposed method can greatly reduce the need for high-quality CD labels and alleviate bottlenecks caused by an imbalanced distribution. Moreover, our method can improve CD accuracy and has excellent generalization capabilities.

Semantic Annotation Based Effective and Quality Oriented Web Service Discovery

The main objective of this work is to design an effective web service discovery engine that retrieves the best matching results based on the measure of syntactic cum semantic similarity between the user query and the web service to be fetched. The idea is to draw bridges across the broad spectrum of lexical entities based on their relativeness. The essence of this work could be extended to support a wide range of applications from ‘being inculcated in search engines to fetch user-relevant data’ to ‘being used for training robots and AI based devices to respond/adhere appropriately to the different phrases of human commands’.With the accelerated revolution of internet, enterprises and organizations highly rely on Service oriented computing. Web services support inter-operation of distributed applications. Humongous amount of web services present on the internet the user is searching for. The paper focuses ondesigninga practical means of fast and relevant retrieval of web services.The phrase used for searching any given web service differs from one person to another. This work deals with the syntactic cum semantic mechanism proposed for retrieving web services based on the measure of similarity between the name of the web service and the search query entered by the user to discover that service. A list of the Web Service Description Language (WSDL) files is taken as the dataset. Protégé is the tool used for semantic annotation of WSDL files for converting them to Semantically Annotated Web Service Description Language (SAWSDL) files. ‘Wordnet’ is used as the lexical dictionary and Java language has been used to build the rest of the package for implementing the search discovery mechanism.Netbeans is used as IDE.Wamp server with PHPMyAdmin was used for managing the database of SAWSDL files. The degree of similarity is measured by evaluating performance of the engine using metrics such as ‘Precision’, ‘Recall’, ‘Accuracy’ and ‘F-measure’.While the syntactic approach is easier to implement, it suffers from keyword polysemy issues. The proposed search discovery mechanism is based on semantically annotating WSDL files and retrieving the files based on a novel syntactic cum semantic discovery algorithm which usesLeocockChordorow function for computing the similarity. The effectiveness of the proposed algorithm is tested experimentally by building a desktop application using Java.The WSDL files from various domains were semantically annotated by tagging related concepts using ontology. Themachine learning algorithm that best classifies the web services on the basis of their performance metrics is identified. The related services are retrieved by the application of the proposed LCH based Syntactic cum Semantic discovery algorithm (LCH based SSDA).A ranking system is proposed to rank the results by evaluating various QoS attributes. The results of the experiment showed that the proposed system yielded high precision and recall value.The solution has been found to be effective in minimizing the execution time and in improving the degree of relevancy. With an impending need for constructing a semantic context based secure structure,the proposed solution will- help in refining the search results and in minimizing users’ cognitive load during search formulation and execution.

Construction of a 3D Model Knowledge Base Based on Feature Description and Common Sense Fusion

Three-dimensional models represent the shape and appearance of real-world objects in a virtual manner, enabling users to obtain a comprehensive and accurate understanding by observing their appearance from multiple perspectives. The semantic retrieval of 3D models is closer to human understanding, but semantic annotation for describing 3D models is difficult to automate, and it is still difficult to construct an easy-to-use 3D model knowledge base. This paper proposes a method for building a 3D model knowledge base to enhance the ability to intelligently manage and reuse 3D models. The sources of 3D model knowledge are obtained from two aspects: on the one hand, constructing mapping rules between the 3D model features and semantics, and on the other hand, extraction from a common sense database. Firstly, the viewpoint orientation is established, the semantic transformation rules of different feature values are established, and the representation degree of different features is divided to describe the degree of the contour approximating the regular shape under different perspectives through classification. An automatic output model semantic description of the contour is combined with spatial orientation. Then, a 3D model visual knowledge ontology is designed from top to bottom based on the upper ontology of the machine-readable comprehensive knowledge base and the relational structure of the ConceptNet ontology. Finally, using a weighted directed graph representation method with a sparse-matrix-integrated semantic dictionary as a carrier, an entity dictionary and a relational dictionary are established, covering attribute names and attribute value data. The sparse matrix is used to record the index information of knowledge triplets to form a three-dimensional model knowledge base. The feasibility of this method is demonstrated by semantic retrieval and reasoning on the label meshes dataset and the cultural relics dataset.

STO2Vec: A Multiscale Spatio-Temporal Object Representation Method for Association Analysis

Spatio-temporal association analysis has attracted attention in various fields, such as urban computing and crime analysis. The proliferation of positioning technology and location-based services has facilitated the expansion of association analysis across spatio-temporal scales. However, existing methods inadequately consider the scale differences among spatio-temporal objects during analysis, leading to suboptimal precision in association analysis results. To remedy this issue, we propose a multiscale spatio-temporal object representation method, STO2Vec, for association analysis. This method comprises of two parts: graph construction and embedding. For graph construction, we introduce an adaptive hierarchical discretization method to distinguish the varying scales of local features. Then, we merge the embedding method for spatio-temporal objects with that for discrete units, establishing a heterogeneous graph. For embedding, to enhance embedding quality for homogeneous and heterogeneous data, we use biased sampling and unsupervised models to capture the association strengths between spatio-temporal objects. Empirical results using real-world open-source datasets show that STO2Vec outperforms other models, improving accuracy by 16.25% on average across diverse applications. Further case studies indicate STO2Vec effectively detects association relationships between spatio-temporal objects in a range of scenarios and is applicable to tasks such as moving object behavior pattern mining and trajectory semantic annotation.

Terminology and ontology development for semantic annotation: A use case on sepsis and adverse events

Annotations enrich text corpora and provide necessary labels for natural language processing studies. To reason and infer underlying implicit knowledge captured by labels, an ontology is needed to provide a semantically annotated corpus with structured domain knowledge. Utilizing a corpus of adverse event documents annotated for sepsis-related signs and symptoms as a use case, this paper details how a terminology and corresponding ontology were developed. The Annotated Adverse Event NOte TErminology (AAENOTE) represents annotated documents and assists annotators in annotating text. In contrast, the complementary Catheter Infection Indications Ontology (CIIO) is intended for clinician use and captures domain knowledge needed to reason and infer implicit information from data. The approach taken makes ontology development understandable and accessible to domain experts without formal ontology training.

Enhancing Biomedical Semantic Annotations through a Knowledge Graph-Based Approach

An abundance of biomedical data is generated in the form of clinical notes, reports, and research articles available online. This data holds valuable information that requires extraction, retrieval, and transformation into actionable knowledge. However, this information has various access challenges due to the need for precise machine-interpretable semantic metadata required by search engines. Despite search engines' efforts to interpret the semantics information, they still struggle to index, search, and retrieve relevant information accurately. To address these challenges, we propose a novel graph-based semantic knowledge-sharing approach to enhance the quality of biomedical semantic annotation by engaging biomedical domain experts. In this approach, entities in the knowledge-sharing environment are interlinked and play critical roles. Authorial queries can be posted on the "Knowledge Cafe," and community experts can provide recommendations for semantic annotations. The community can further validate and evaluate the expert responses through a voting scheme resulting in a transformed "Knowledge Cafe" environment that functions as a knowledge graph with semantically linked entities. We evaluated the proposed approach through a series of scenarios, resulting in precision, recall, F1-score, and accuracy assessment matrices. Our results showed an acceptable level of accuracy at approximately 90%. The source code for "Semantically" is freely available at: https://github.com/bukharilab/Semantically

Resolving Lexical Verb Ambiguity in Yoruba through Semantic Web Annotations

Lexical verb ambiguity refers to the potential for multiple interpretations of units of speech in languages that renders it difficult or impossible to understand a lexical item without additional information. The goal of semantic web research is to allow the vast range of web-accessible information and services to be more effectively exploited by both humans and automated tools. Technological advancement has given rise to scholars developing automated translation systems from English to the Yorùbá language. However, the problem arises that these tools are not coping easily with semantic nuances found especially in Yorùbá verbs. To explore this problem, a number of Yorùbá lexical ambiguous verbs which may pose serious challenges to the existing machine translation tools were generated and analyzed. To facilitate a solution to the problem, the identified verbs are annotated following the Qualia Structure (QS) of the Generative Lexicon. Protégé 4.5 is employed as the editing tool to map the semantic interpretations of the QS to the semantic web. With the aid of the unique formal specification implemented for each of the lexically ambiguous verbs, the existing automated translation system can now utilize the Internationalized Resource Identifier (IRI) of the Unified Resource Locator (URL) found in the model developed to link the loop and thereby alleviate the problem of ambiguity for the automated translation system. This paper, therefore recommends that more efforts be geared towards ontological annotation in order to foster more automated tools that will be built for the Yorùbá language.

Beyond rankings: Learning (more) from algorithm validation.

Challenges have become the state-of-the-art approach to benchmark image analysis algorithms in a comparative manner. While the validation on identical data sets was a great step forward, results analysis is often restricted to pure ranking tables, leaving relevant questions unanswered. Specifically, little effort has been put into the systematic investigation on what characterizes images in which state-of-the-art algorithms fail. To address this gap in the literature, we (1) present a statistical framework for learning from challenges and (2) instantiate it for the specific task of instrument instance segmentation in laparoscopic videos. Our framework relies on the semantic meta data annotation of images, which serves as foundation for a General Linear Mixed Models (GLMM) analysis. Based on 51,542 meta data annotations performed on 2,728 images, we applied our approach to the results of the Robust Medical Instrument Segmentation Challenge (ROBUST-MIS) challenge 2019 and revealed underexposure, motion and occlusion of instruments as well as the presence of smoke or other objects in the background as major sources of algorithm failure. Our subsequent method development, tailored to the specific remaining issues, yielded a deep learning model with state-of-the-art overall performance and specific strengths in the processing of images in which previous methods tended to fail. Due to the objectivity and generic applicability of our approach, it could become a valuable tool for validation in the field of medical image analysis and beyond.

Relationship Prediction in a Knowledge Graph Embedding Model of the Illicit Antiquities Trade

ABSTRACTThe transnational networks of the illicit and illegal antiquities trade are hard to perceive. We suggest representing the trade as a knowledge graph with multiple kinds of relationships that can be transformed by a neural architecture into a “knowledge graph embedding model.” The result is that the vectorization of the knowledge represented in the graph can be queried for missing “knowledge” of the trade by virtue of the various entities’ proximity in the multidimensional embedding space. In this article, we build a knowledge graph about the antiquities trade using a semantic annotation tool, drawing on the series of articles in the Trafficking Culture Project's online encyclopedia. We then use the AmpliGraph package, a series of tools for supervised machine learning (Costabello et al. 2019) to turn the graph into a knowledge graph embedding model. We query the model to predict new hypotheses and to cluster actors in the trade. The model suggests connections between actors and institutions hitherto unsuspected and not otherwise present in the original knowledge graph. This approach could hold enormous potential for illuminating the hidden corners of the illicit antiquities trade. The same method could be applied to other kinds of archaeological knowledge.

An Entity Extraction and Categorization Technique on Twitter Streams

As social media platforms have gained huge momentum in recent years, the amount of information generated from the social media sites is growing exponentially and gives the information retrieval systems a great challenge to extract the potential named entities. Researchers have utilized the semantic annotation mechanism to retrieve the entities from the unstructured documents, but the mechanism returns with too many ambiguous entities. In this work, the DBpedia knowledge base is adopted for entity extraction and categorization. To achieve the entity extraction task precisely, a two-step process is proposed: (a) train the unstructured datasets with Word2Vec and classify the entities into their respective categories. (b) crawl the web pages, forums, and other web sources to identifying the entities that are not present in the DBpedia. The evaluation shows the results with more precision and promising F1 score.

ODP-Transformer: Interpretation of pest classification results using image caption generation techniques

Pest image classification systems are key tools to identify pests in time. However, existing image classification systems can only predict the labels of pest images and lack the interpretation of image content. In this paper, image caption generation techniques are introduced to interpret the results of pest image classification. Specifically, we proposed the ODP-Transformer by imitating the three basic actions in the diagnostic process of agricultural experts, which are Observation, Description and Prediction. ODP-Transformer is a two-stage model, the first stage is a pest part detector based on the faster R-CNN framework. And the second stage contains three modules: Parts Sequence Encoder, Description Decoder and Classification Decoder, which are used for image caption generation tasks and classification tasks. At the same time, a prior knowledge matrix is introduced to guide the optimization direction of the attention mechanism in the Description Decoder, which is used to learn the concept correspondences in images and texts. Additionally, an agricultural pest textual and visual dataset (APTV-99) is collected, which contains not only the semantic annotations of images but also the textual descriptions of corresponding parts. Extensive experiments are implemented on APTV-99 to evaluate the performance of ODP-Transformer. In the pest image classification task, ODP-Transformer is 12.91% higher in accuracy than the 8 commonly used CNN models. In the image captioning generation task, compared with the other 6 methods, ODP-Transformer improves by 1.62, 8.08, and 1.08 for Bleu1, CiderD and Rouge indicators, respectively.

TME: Tree-guided Multi-task Embedding Learning towards Semantic Venue Annotation

The prevalence of location-based services has generated a deluge of check-ins, enabling the task of human mobility understanding. Among the various types of information associated with the check-in venues, categories (e.g., Bar and Museum ) are vital to the task, as they often serve as excellent semantic characterization of the venues. Despite its significance and importance, a large portion of venues in the check-in services do not have even a single category label, such as up to 30% of venues in the Foursquare system lacking category labels. We, therefore, address the problem of semantic venue annotation, i.e., labeling the venue with a semantic category. Existing methods either fail to fully exploit the contextual information in the check-in sequences, or do not consider the semantic correlations across related categories. As such, we devise a Tree-guided Multi-task Embedding model (TME for short) to learn effective representations of venues and categories for the semantic annotation. TME jointly learns a common feature space by modeling multi-contexts of check-ins and utilizes the predefined category hierarchy to regularize the relatedness among categories. We evaluate TME over the task of semantic venue annotation on two check-in datasets. Experimental results show the superiority of TME over several state-of-the-art baselines.