Semi-structured Datasets Research Articles

A Largely Unsupervised Domain-Independent Qualitative Data Extraction Approach for Empirical Agent-Based Model Development

Agent-based model (ABM) development needs information on system components and interactions. Qualitative narratives contain contextually rich system information beneficial for ABM conceptualization. Traditional qualitative data extraction is manual, complex, and time- and resource-consuming. Moreover, manual data extraction is often biased and may produce questionable and unreliable models. A possible alternative is to employ automated approaches borrowed from Artificial Intelligence. This study presents a largely unsupervised qualitative data extraction framework for ABM development. Using semantic and syntactic Natural Language Processing tools, our methodology extracts information on system agents, their attributes, and actions and interactions. In addition to expediting information extraction for ABM, the largely unsupervised approach also minimizes biases arising from modelers’ preconceptions about target systems. We also introduce automatic and manual noise-reduction stages to make the framework usable on large semi-structured datasets. We demonstrate the approach by developing a conceptual ABM of household food security in rural Mali. The data for the model contain a large set of semi-structured qualitative field interviews. The data extraction is swift, predominantly automatic, and devoid of human manipulation. We contextualize the model manually using the extracted information. We also put the conceptual model to stakeholder evaluation for added credibility and validity.

Cyber Threat Intelligence Discovery using Machine Learning from the Dark Web

Cyber threat intelligence (CTI) is an actionable information or insight an organization uses to understand potential vulnerabilities it does have and threats it is facing. One important CTI for proactive cyber defense is exploit type with possible values system, web, network, website or Mobile. This study compares the performance of machine learning algorithms in predicating exploit types using form posts in the dark web, which is a semi- structured dataset collected from dark web. The study uses the CRISP data science approach. The results of the study show that machine learning algorithms which are function-based including support vector machine and deep-learning using artificial neural network are more accurate than those algorithms which are based on tree including Random Forest and Decision-Tree for CTI discovery from semi-structured dataset. Future research will include the use of high-performance computing and advanced deep-learning algorithms.

A Novel Approach for Contextual Clustering and Retrieval of Behavior Trees to Enrich the Behavior of Social Intelligent Agents

Recently, many works have been carried out to find effective ways that can allow for plausibly effective interactions of social intelligent agents (SIAs) in unpredictable environments in a reasonable time. Behavior trees (BTs) allow for knowledge to be modeled as a graph representation and provide a way for SIAs to effectively interact with the received information. BTs can store past social experiences that can then be used by SIAs to provide adequate human-like interactions when facing new social situations (query). One challenge appears when a social agent with vast past experiences—represented as a forest of BTs—tries to retrieve a similar BT to learn from in order to provide plausible interactions in the current situation in a cost-effective manner. Cognitive scripts with their inherent temporal structure can address this challenge where they can facilitate the use of contextual retrieval techniques on BTs represented as cognitive scripts. This paper introduces novel hybrid retrieval techniques that use agglomerative hierarchical clustering (H-clustering) and similarity-based algorithms: map-and-reduce and least common parent (LCP) to effectively retrieve similar BTs to a specific query BT in a reasonable time. The model groups BTs, represented as cognitive scripts, into compact clusters that can then be used to retrieve the most similar BT to a query one in real time without noticeable delay. A comparison was done between the performance of the proposed hybrid-retrieval techniques using a semi-structured dataset of cognitive scripts. The results showed that H-clustering-map-and-reduce is more cost-effective than H-clustering-LCP as it allowed for a low average retrieval time of 8 × 10−3 s compared to 3.1 s, respectively.

Deep transformation models for functional outcome prediction after acute ischemic stroke.

In many medical applications, interpretable models with high prediction performance are sought. Often, those models are required to handle semistructured data like tabular and image data. We show how to apply deep transformation models (DTMs) for distributional regression that fulfill these requirements. DTMs allow the data analyst to specify (deep) neural networks for different input modalities making them applicable to various research questions. Like statistical models, DTMs can provide interpretable effect estimates while achieving the state-of-the-art prediction performance of deep neural networks. In addition, the construction of ensembles of DTMs that retain model structure and interpretability allows quantifying epistemic and aleatoric uncertainty. In this study, we compare several DTMs, including baseline-adjusted models, trained on a semistructured data set of 407 stroke patients with the aim to predict ordinal functional outcome three months after stroke. We follow statistical principles of model-building to achieve an adequate trade-off between interpretability and flexibility while assessing the relative importance of the involved data modalities. We evaluate the models for an ordinal and dichotomized version of the outcome as used in clinical practice. We show that both tabular clinical and brain imaging data are useful for functional outcome prediction, whereas models based on tabular data only outperform those based on imaging data only. There is no substantial evidence for improved prediction when combining both data modalities. Overall, we highlight that DTMs provide a powerful, interpretable approach to analyzing semistructured data and that they have the potential to support clinicaldecision-making.

Dichotomic Pattern Mining Integrated With Constraint Reasoning for Digital Behavior Analysis.

Sequential pattern mining remains a challenging task due to the large number of redundant candidate patterns and the exponential search space. In addition, further analysis is still required to map extracted patterns to different outcomes. In this paper, we introduce a pattern mining framework that operates on semi-structured datasets and exploits the dichotomy between outcomes. Our approach takes advantage of constraint reasoning to find sequential patterns that occur frequently and exhibit desired properties. This allows the creation of novel pattern embeddings that are useful for knowledge extraction and predictive modeling. Based on dichotomic pattern mining, we present two real-world applications for customer intent prediction and intrusion detection. Overall, our approach plays an integrator role between semi-structured sequential data and machine learning models, improves the performance of the downstream task, and retains interpretability.

Named Entity Recognition for Addresses: An Empirical Study

In this paper, several model architectures are explored in order to design a high-performing named entity recognition model for addresses which deals with challenges such as diversity, ambiguity and complexity of the address entity. Different types of neural networks are used for training the classifier, including the bidirectional LSTM network in combination with a convolutional layer, a conditional random field layer and different word embeddings. Experiments are conducted on two types of corpora specifically constructed and tagged for tackling this challenge: unstructured and semi-structured datasets. For model evaluation, two versions of the unstructured dataset are used that are tagged differently based on the granularity of address entity: entire address, and address consisting of subparts. For both types of corpora, the best results are achieved on a BiLSTM-CRF architecture model with a single RNN layer trained with BERT embeddings.

Large-bodied birds are over-represented in unstructured citizen science data

Citizen science platforms are quickly accumulating hundreds of millions of biodiversity observations around the world annually. Quantifying and correcting for the biases in citizen science datasets remains an important first step before these data are used to address ecological questions and monitor biodiversity. One source of potential bias among datasets is the difference between those citizen science programs that have unstructured protocols and those that have semi-structured or structured protocols for submitting observations. To quantify biases in an unstructured citizen science platform, we contrasted bird observations from the unstructured iNaturalist platform with that from a semi-structured citizen science platform—eBird—for the continental United States. We tested whether four traits of species (body size, commonness, flock size, and color) predicted if a species was under- or over-represented in the unstructured dataset compared with the semi-structured dataset. We found strong evidence that large-bodied birds were over-represented in the unstructured citizen science dataset; moderate evidence that common species were over-represented in the unstructured dataset; strong evidence that species in large groups were over-represented; and no evidence that colorful species were over-represented in unstructured citizen science data. Our results suggest that biases exist in unstructured citizen science data when compared with semi-structured data, likely as a result of the detectability of a species and the inherent recording process. Importantly, in programs like iNaturalist the detectability process is two-fold—first, an individual organism needs to be detected, and second, it needs to be photographed, which is likely easier for many large-bodied species. Our results indicate that caution is warranted when using unstructured citizen science data in ecological modelling, and highlight body size as a fundamental trait that can be used as a covariate for modelling opportunistic species occurrence records, representing the detectability or identifiability in unstructured citizen science datasets. Future research in this space should continue to focus on quantifying and documenting biases in citizen science data, and expand our research by including structured citizen science data to understand how biases differ among unstructured, semi-structured, and structured citizen science platforms.

Unsupervised DNF Blocking for Efficient Linking of Knowledge Graphs and Tables

Entity Resolution (ER) is the problem of identifying co-referent entity pairs across datasets, including knowledge graphs (KGs). ER is an important prerequisite in many applied KG search and analytics pipelines, with a typical workflow comprising two steps. In the first ’blocking’ step, entities are mapped to blocks. Blocking is necessary for preempting comparing all possible pairs of entities, as (in the second ‘similarity’ step) only entities within blocks are paired and compared, allowing for significant computational savings with a minimal loss of performance. Unfortunately, learning a blocking scheme in an unsupervised fashion is a non-trivial problem, and it has not been properly explored for heterogeneous, semi-structured datasets, such as are prevalent in industrial and Web applications. This article presents an unsupervised algorithmic pipeline for learning Disjunctive Normal Form (DNF) blocking schemes on KGs, as well as structurally heterogeneous tables that may not share a common schema. We evaluate the approach on six real-world dataset pairs, and show that it is competitive with supervised and semi-supervised baselines.

A Survey of Bio Inspired Algorithms for Web Information Extraction and Optimization for Big Data Analytics

Information extraction is systematic process of extracting structured information from documents which has both unstructured and semi structured data set. Data available over the web is unstructured which is processed and delivered that may be challenging due to massive data over web. Bigdata analytics approach is used in the computation field where massive data is managed and processed as information. Data from various sources like industries, institutes are processed using algorithms in efficient means employing web of things or Internet of things used to mine such a large data. Bio inspired algorithms have evolved from application of heuristic approaches to meta-heuristic and hyper-heuristic methodologies. Bio inspired techniques are categorized into human inspired algorithms, Swarm Intelligence algorithms, evolutionary algorithms and ecology based algorithms. Genetic algorithms are purely heuristic in nature and are employed for computation and extracting information and from big data. This improves the computation speed effectively for extracting web related information as evolutionary algorithm resolves information extraction problems. The Ant colony and Particle Swarm Intelligence algorithms are of meta-heuristic in nature. The Cuckoo search, Artificial Bee Colony, Firefly algorithm and Bat algorithms are of hyper heuristic in nature i.e., they employ a combination of methods. Web information extraction using bio inspired concepts and genetic operators increases efficiency, capability to search particular information in massive data in web. Some of the tools that are available for data extraction and mining are DataMelt, Apache Mahout, Weka, Orange and Rapid Miner for enhancing web data extraction efficiency. This survey on bio inspired methodologies can be extended to parameter tuning and controlling is another big strategy that can be implemented, in addition to convergence speed up.

An Improved NER Methodology to the Portuguese Language

The text mining process typically involves the application of natural language processing (NLP) techniques, in order to obtain important information and extract insights from texts. This is achieved by detecting patterns, which are not explicitly a priori in this unstructured or semi-structured dataset. One of the most significant NLP tasks is Named Entity Recognition (NER). The NER process seeks to extract and classify mentioned entities detected in a text written in natural language. These categories are predefined and can be names of people or organizations, locations, dates, monetary values, specific codes, etc. A wide range of algorithms based on LSTM (Long-Short Term Memory) architecture has being proposed to enhance the NER accuracy. However, a key component to a successful information extraction is the corpora used for NER training. Another key issue concerns the language being worked on, since the vast majority of algorithms were designed to work with English. According to the literature, while the NER process applied to the English language reaches about 90% accuracy, when it is applied to the Portuguese language, this precision reaches a maximum of 83.38%. This paper proposes a methodology to improve the Portuguese-based NER, which uses journalistic corpora as a basis for text corpora training. We believe the journalistic writting has the best adherence to the contemporaneity of any language, since it preserves features such as objectivity, simplicity, impartiality, and is a reference of transmitting the information without ambiguity. The proposed methodology provides a model to extract entities and assess the obtained results with the use of Recurrent Neural Network architectures. At the best of our knowledge, the proposed methodology applied to the Portuguese language not only overcomes the average accuracy found in the literature by increasing it from 83.38% to 85.64%, but also could decrease the computational costs related to the NER processing tasks.

Towards Predicting Risk of Coronary Artery Disease from Semi-Structured Dataset.

Many kinds of disease-related data are now available and researchers are constantly attempting to mine useful information out of these. Medical data are not always homogeneous and in structured form, and mostly they are time-stamped data. Thus, special care is required to prevent any kind of information loss during mining such data. Mining medical data is challenging as predicting the non-accurate result is often not acceptable in this domain. In this paper, we have analyzed a partially annotated coronary artery disease (CAD) dataset which was originally in a semi-structured form. We have created a set of some well-defined features from the dataset, and then build predictive models for CAD risk identification using different supervised learning algorithms. We then further enhanced the performances of the models using a feature selection technique. Experiments show that results are quite interesting, and are expected to help medical practitioners for investigating CAD risk in patients.

Streaming Big Data Analytics- Current Status, Challenges and Connection of unbounded data Processing platforms

A strategy of examining immense dimensions of structured, un-structured, Semi-Structured data sets is referred as Big data Analytics. Streaming Big Data refers to data generated continuously from number of data sources like Internet-of-Things (IoT) devices, mobile applications, Embedded Sensors, web clicks and many more are needed to be store, processed and analyzed in a tiny interval of time in order to extract meaningful insights and take proper decisions in a timely fashion as the necessity arises. However analyzing streaming big data (continuous flow or unbounded data) is a very challenging problem. Continuous data streams have become essential prerequisite for numerous industrial and scientific applications, the current existing technology Hadoop-MapReduce is not appropriate for stream processing of big data. This paper discusses the challenges and benefits of streaming big data along with its architecture, and focuses on different open source streaming processing platforms that are existed to process the huge data at a high speed

Extracting common sense knowledge via triple ranking using supervised and unsupervised distributional models

In this paper we are concerned with developing information extraction models that support the extraction of common sense knowledge from a combination of unstructured and semi-structured datasets. Our motivation is to extract manipulation-relevant knowledge that can support robots' action planning. We frame the task as a relation extraction task and, as proof-ofconcept, validate our method on the task of extracting two types of relations: locative and instrumental relations. The locative relation relates objects to the prototypical places where the given object is found or stored. The second instrumental relation relates objects to their prototypical purpose of use. While we extract these relations from text, our goal is not to extract specific textual mentions, but rather, given an object as input, extract a ranked list of locations and uses ranked by `prototypicality'. We use distributional methods in embedding space, relying on the well-known skip-gram model to embed words into a low-dimensional distributional space, using cosine similarity to rank the various candidates. In addition, we also present experiments that rely on the vector space model NASARI, which compute embeddings for disambiguated concepts and are thus semantically aware. While this distributional approach has been published before, we extend our framework by additional methods relying on neural networks that learn a score to judge whether a given candidate pair actually expresses a desired relation. The network thus learns a scoring function using a supervised approach. While we use a ranking-based evaluation, the supervised model is trained using a binary classification task. The resulting score from the neural network and the cosine similarity in the case of the distributional approach are both used to compute a ranking. We compare the different approaches and parameterizations thereof on the task of extracting the above mentioned relations. We show that the distributional similarity approach performs very well on the task. The best performing parameterization achieves an NDCG of 0.913, a Precision@ 1 of 0.400 and a Precision@ 3 of 0.423. The performance of the supervised learning approach, in spite of having being trained on positive and negative examples of the relation in question, is not as good as expected and achieves an NCDG of 0.908, a Precision@ 1 of 0.454 and a Precision@3 of 0.387, respectively.

Cost-effective conceptual design using taxonomies

It is known that annotating entities in unstructured and semi-structured datasets by their concepts improves the effectiveness of answering queries over these datasets. Ideally, one would like to annotate entities of all relevant concepts in a dataset. However, it takes substantial time and computational resources to annotate concepts in large datasets, and an organization may have sufficient resources to annotate only a subset of relevant concepts. Clearly, it would like to annotate a subset of concepts that provides the most effective answers to queries over the dataset. We propose a formal framework that quantifies the amount by which annotating entities of concepts from a taxonomy in a dataset improves the effectiveness of answering queries over the dataset. Because the problem is $$\mathbf {NP}$$ -hard, we propose efficient approximation and pseudo-polynomial time algorithms for several cases of the problem. Our extensive empirical studies validate our framework and show accuracy and efficiency of our algorithms.

Big Data Analytics Initiatives Using Business Intelligence Maturity Model Approach in Public Sector

Big Data requirement in Public Sector is gaining focus in many countries. Described in the Eleventh Malaysia Plan 2016–2020, Big Data Analytics (BDA) has been identified as key focus for Information and Communications Technology (ICT). Especially with the recent participation of Trans-Pacific Partnership Agreement, Malaysia will be competing amongst other nations to make quick and accurate decision contributing to economic and GDP growth. With good understanding and guided implementation framework of Business Intelligence, right and agile decision can be made using the various available information. Malaysia Public Sector requires ICT involvement in Business Intelligence domain to perform Big Data Analytics. With evolving of data availability and needs for building correlation with different sets of data, Big-Data offers an emerging space to analyze wealth of information form structure, semi-structured and un-structured data set. Business Intelligence provides an organization the insight required to make the key and critical decision for leading competitive edge.

Dynamic Transform Method for Ontology DB from Semi-structured Datasets

Dynamic Transform Method for Ontology DB from Semi-structured Datasets

Large-scale complex analytics on semi-structured datasets using asterixDB and spark

Large quantities of raw data are being generated by many different sources in different formats. Private and public sectors alike acclaim the valuable information and insights that can be mined from such data to better understand the dynamics of everyday life, such as traffic, worldwide logistics, and social behavior. For this reason, storing, managing, and analyzing "Big Data" at scale is getting a tremendous amount of attention, both in academia and industry. In this paper, we demonstrate the power of a parallel connection that we have built between Apache Spark and Apache AsterixDB (Incubating) to enable complex analytics such as machine learning and graph analysis on data drawn from large semi-structured data collections. The integration of these two systems allows researchers and data scientists to leverage AsterixDB capabilities, including fast ingestion and indexing of semi-structured data and efficient answering of geo-spatial and fuzzy text queries. Complex data analytics can then be performed on the resulting AsterixDB query output in order to obtain additional insights by leveraging the power of Spark's machine learning and graph libraries.

Towards Systematic Parallelization of Graph Transformations Over Pregel

Graphs can be used to model many kinds of data, from traditional datasets to social networks or semi-structured datasets. To process large graphs, many systems have been proposed. The Pregel progra...

TreeScope

Semi-structured data are prevalent on the web, with formats such as XML and JSON soaring in popularity due to their generality, flexibility and easy customization. However, these very same features make semi-structured data prone to a range of data quality errors, from errors in content to errors in structure. While the former has been well studied, little attention has been paid to structural errors.In this demonstration, we present TreeScope, which analyzes semi-structured data sets with the goal of automatically identifyingstructural anomaliesfrom the data. Our techniques learn robust structural models that have high support, to identify potential errors in the structure. Identified structural anomalies are then concisely summarized to provide plausible explanations of the potential errors. The goal of this demonstration is to enable an interactive exploration of the process of identifying and summarizing structural anomalies in semi-structured data sets.

Entity Resolution in a Big Data Framework

Entity Resolution (ER) concerns identifying logically equivalent pairs of entities that may be syntactically disparate. Although ER is a long-standing problem in the artificial intelligence community, the growth of Linked Open Data, a collection of semi-structured datasets published and inter-connected on the Web, mandates a new approach. The thesis is that building a viable Entity Resolution solution for serving Big Data needs requires simultaneously resolving challenges of automation, heterogeneity, scalability and domain independence. The dissertation aims to build such a system and evaluate it on real-world datasets published already as Linked Open Data.