Unstructured Data Sources Research Articles

Extraction of Professional Details from Web-URLs using DeepDive

Manual extraction of data from unstructured data sources like websites is labour intensive and becomes almost in-feasible at large scale. Recent state-of-the-art techniques for the task of information extraction show encouraging results. In this work, we make an attempt to extract professional details like name, email, address, contact number, and specialization from home pages of doctors. The work covers two possible scenarios of websites having these details. One scenario is where a website contains details of a single doctor. Another scenario is where a website may contain multiple information of multiple doctors/professionals at the same time. The problem is attempted to be solved as a relation extraction task for Information Extraction. The proposed solution has been built on top of DeepDive, a tool developed by Stanford. In both scenarios, DeepDive takes pre-processed data sentences as input and constructs entity-relations. For each entity-relation, DeepDive computes a probability that the relationship is a correct match using distance supervision and user-defined heuristic rules. In case of experiment-1, our system achieves 69.14% accuracy for the name, 88.67% accuracy for location and 100% for email, number and specialization. In case of experiment-2, the observed probabilities are not so significant and mostly around 0.5-0.7 but we present some solutions for future work. The techniques presented here can easily be extended to generalize for other types of professionals too and not just doctors.

Mapping the Radical Innovations in Food Industry: A Text Mining Study

The article presents the results of the study of radical innovations in the global food industry which were obtained through semantic analysis of heterogeneous unstructured text data sources by applying innovative big data text mining system. The approach used allows performing rapid, yet comprehensive aggregation of the whole polyphony of existing knowledge of the technology development in any sector for traditional foresight, future oriented technology analysis, and horizon scanning studies. The sources for the analysis include research papers, patent applications with both full-text data and additional structured metadata, analytical reports by main international organizations and national key players, various media and news resources, including all the major technology innovation, disruption and venture capital news websites. Their processing with an introduced approach for trend- and technology-mapping helps to identify ongoing and emerging technology-related trends, weak signals on possible scientific breakthroughs in the global food industry, including most promising startup strategies and food innovation controversies. This kind of analysis can be performed on a regular basis owing to constant accumulation of textual data and serve as a framework for constant science and technology (S&T) monitoring for early warning on changing technology landscape and its implications on agriculture and food markets.

A Fast Image Retrieval Method Designed for Network Big Data

In the field of big data applications, image information is widely used. The value density of information utilization in big data is very low, and how to extract useful information quickly is very important. So we should transform the unstructured image data source into a form that can be analyzed. In this paper, we proposed a fast image retrieval method which designed for big data. First of all, the feature extraction method is necessary and the feature vectors can be obtained for every image. Then, it is the most important step for us to encode the image feature vectors and make them into database, which can optimize the feature structure. Finally, the corresponding similarity matching is used to determined the retrieval results. There are three main contributions for image retrieval in this paper. New feature extraction method, reasonable elements ranking, and appropriate distance metric can improve the algorithm performance. Experiments show that our method has a great improvement in the effective performance of feature extraction and can also get better search matching results.

Using structured and unstructured data to identify patients’ need for services that address the social determinants of health

IntroductionIncreasingly, health care providers are adopting population health management approaches that address the social determinants of health (SDH). However, effectively identifying patients needing services that address a SDH in primary care settings is challenging. The purpose of the current study is to explore how various data sources can identify adult primary care patients that are in need of services that address SDH. MethodsA cross-sectional study described patients in need of SDH services offered by a safety-net hospital’s federally qualified health center clinics. SDH services of social work, behavioral health, nutrition counseling, respiratory therapy, financial planning, medical-legal partnership assistance, patient navigation, and pharmacist consultation were offered on a co-located basis and were identified using structured billing and scheduling data, and unstructured electronic health record data. We report the prevalence of the eight different SDH service needs and the patient characteristics associated with service need. Moreover, characteristics of patients with SDH services need documented in structured data sources were compared with those documented by unstructured data sources. ResultsMore than half (53%) of patients needed SDH services. Those in need of such services tended to be female, older, more medically complex, and higher utilizers of services. Structured and unstructured data sources exhibited poor agreement on patient SDH services need. Patients with SDH services need documented by unstructured data tended to be more complex. DiscussionThe need for SDH services among a safety-net population is high. Identifying patients in need of such services requires multiple data sources with structured and unstructured data.

QUERY-BASED SUMMARIZATION METHODS FOR CONVERSATIONAL AGENTS: AN OVERVIEW

Summarization is a topic that will be of a great important in the coming age since intelligent assistants especially the ones in the form of conversational agents will have to sift through the abundance of raw unstructured text data to provide relevant information. The data will be in the form of Social media posts, content websites and other user generated text content from which the user shall require tailored information from and about the data. The paper hence explores various methods for summarization and focuses particularly on extracting the gist from the perspective of a given keyword i.e. query based summarization from raw unstructured text data sources available at scale. Along with that, the need for a proper framework to mine relevant knowledge from the said data is acknowledged and the challenges that a conversational agent would hence face are identified. Various approaches that contribute to building a framework and solve the identified challenges are explored as well. It is hoped that the approaches discussed in the paper will be of use to researchers building algorithms in areas of knowledge mining and understanding, such as summarization, that deal with the challenges that are expected to arise.

Automated extraction of fragments of Bayesian networks from textual sources

Mining large amounts of unstructured data for extracting meaningful, accurate, and actionable information, is at the core of a variety of research disciplines including computer science, mathematical and statistical modelling, as well as knowledge engineering. In particular, the ability to model complex scenarios based on unstructured datasets is an important step towards an integrated and accurate knowledge extraction approach. This would provide a significant insight in any decision making process driven by Big Data analysis activities. However, there are multiple challenges that need to be fully addressed in order to achieve this, especially when large and unstructured data sets are considered.In this article we propose and analyse a novel method to extract and build fragments of Bayesian networks (BNs) from unstructured large data sources. The results of our analysis show the potential of our approach, and highlight its accuracy and efficiency. More specifically, when compared with existing approaches, our method addresses specific challenges posed by the automated extraction of BNs with extensive applications to unstructured and highly dynamic data sources.The aim of this work is to advance the current state-of-the-art approaches to the automated extraction of BNs from unstructured datasets, which provide a versatile and powerful modelling framework to facilitate knowledge discovery in complex decision scenarios.

Spatial Context from Open and Online Processing (SCOOP): Geographic, Temporal, and Thematic Analysis of Online Information Sources

The Internet is increasingly a source of data for geographic information systems, as more data becomes linked, available through application programing interfaces (APIs), and more tools become available for handling unstructured web data. While many web data extraction and structuring methods exist, there are few examples of comprehensive data processing and analysis systems that link together these tools for geographic analyses. This paper develops a general approach to the development of spatial information context from unstructured and informal web data sources through the joint analysis of the data’s thematic, spatial, and temporal properties. We explore the utility of this derived contextual information through a case study into maritime surveillance. Extraction and processing techniques such as toponym extraction, disambiguation, and temporal information extraction methods are used to construct a semi-structured maritime context database supporting global scale analysis. Geographic, temporal, and thematic content were analyzed, extracted and processed from a list of information sources. A geoweb interface is developed to allow user visualization of extracted information, as well as to support space-time database queries. Joint keyword clustering and spatial clustering methods are used to demonstrate extraction of documents that relate to real world events in official vessel information data. The quality of contextual geospatial information sources is evaluated in reference to known maritime anomalies obtained from authoritative sources. The feasibility of automated context extraction using the proposed framework and linkage to external data using standard clustering tools is demonstrated.

Cognitive Analytics for Making Better Evidence-Based Decisions

The actions associated with business decisions are guided by a range of variables, that include: opportunities, funding types, customer categories, competencies, proposal status, resource feasibility, technical feasibility, capability increase, risk level and commitment. The analysis of these decision variables or more likely the data associated with them is based on using descriptive, predictive, or prescriptive analytics as a means of “searching for answers” to the business problems and issues confronting the enterprise. Cognitive analytics embodies a fourth area of decision support that facilitates the analysis of structured and unstructured data sources and the use of natural language processing, learning and reasoning capabilities to enhance hypothesis generation. In short, cognitive analytics enables the enterprise to “ask the right questions” surrounding the evidence. This research highlights the impact of cognitive analytics in making evidence-based decision actions – specifically by modeling “what if” scenarios concerning the impact of resource and schedule on project risk associated with the development of a new product using IBM SPSS Modeler and IBM Watson Analytics.

DeepDive

The dark data extraction or knowledge base construction (KBC) problem is to populate a relational database with information from unstructured data sources, such as emails, webpages, and PDFs. KBC is a long-standing problem in industry and research that encompasses problems of data extraction, cleaning, and integration. We describe DeepDive, a system that combines database and machine learning ideas to help to develop KBC systems. The key idea in DeepDive is to frame traditional extract-transform-load (ETL) style data management problems as a single large statistical inference task that is declaratively defined by the user. DeepDive leverages the effectiveness and efficiency of statistical inference and machine learning for difficult extraction tasks, whereas not requiring users to directly write any probabilistic inference algorithms. Instead, domain experts interact with DeepDive by defining features or rules about the domain. DeepDive has been successfully applied to domains such as pharmacogenomics, paleobiology, and antihuman trafficking enforcement, achieving human-caliber quality at machine-caliber scale. We present the applications, abstractions, and techniques used in DeepDive to accelerate the construction of such dark data extraction systems.

Obtaining structured clinical data from unstructured data using natural language processing software

ABSTRACT
 BackgroundFree text documents in healthcare settings contain a wealth of information not captured in electronic healthcare records (EHRs). Epilepsy clinic letters are an example of an unstructured data source containing a large amount of intricate disease information. Extracting meaningful and contextually correct clinical information from free text sources, to enhance EHRs, remains a significant challenge. SCANR (Swansea University Collaborative in the Analysis of NLP Research) was set up to use natural language processing (NLP) technology to extract structured data from unstructured sources.
 IBM Watson Content Analytics software (ICA) uses NLP technology. It enables users to define annotations based on dictionaries and language characteristics to create parsing rules that highlight relevant items. These include clinical details such as symptoms and diagnoses, medication and test results, as well as personal identifiers.
 
 ApproachTo use ICA to build a pipeline to accurately extract detailed epilepsy information from clinic letters.
 MethodsWe used ICA to retrieve important epilepsy information from 41 pseudo-anonymized unstructured epilepsy clinic letters. The 41 letters consisted of 13 ‘new’ and 28 ‘follow-up’ letters (for 15 different patients) written by 12 different doctors in different styles. We designed dictionaries and annotators to enable ICA to extract epilepsy type (focal, generalized or unclassified), epilepsy cause, age of onset, investigation results (EEG, CT and MRI), medication, and clinic date. Epilepsy clinicians assessed the accuracy of the pipeline.
 ResultsThe accuracy (sensitivity, specificity) of each concept was: epilepsy diagnosis 98% (97%, 100%), focal epilepsy 100%, generalized epilepsy 98% (93%, 100%), medication 95% (93%, 100%), age of onset 100% and clinic date 95% (95%, 100%).
 Precision and recall for each concept were respectively, 98% and 97% for epilepsy diagnosis, 100% each for focal epilepsy, 100% and 93% for generalized epilepsy, 100% each for age of onset, 100% and 93% for medication, 100% and 96% for EEG results, 100% and 83% for MRI scan results, and 100% and 95% for clinic date.
 Conclusions ICA is capable of extracting detailed, structured epilepsy information from unstructured clinic letters to a high degree of accuracy. This data can be used to populate relational databases and be linked to EHRs. Researchers can build in custom rules to identify concepts of interest from letters and produce structured information. We plan to extend our work to hundreds and then thousands of clinic letters, to provide phenotypically rich epilepsy data to link with other anonymised, routinely collected data.

Automating Clinical Score Calculation within the Electronic Health Record. A Feasibility Assessment.

Evidence-based clinical scores are used frequently in clinical practice, but data collection and data entry can be time consuming and hinder their use. We investigated the programmability of 168 common clinical calculators for automation within electronic health records. We manually reviewed and categorized variables from 168 clinical calculators as being extractable from structured data, unstructured data, or both. Advanced data retrieval methods from unstructured data sources were tabulated for diagnoses, non-laboratory test results, clinical history, and examination findings. We identified 534 unique variables, of which 203/534 (37.8%) were extractable from structured data and 269/534 (50.4.7%) were potentially extractable using advanced techniques. Nearly half (265/534, 49.6%) of all variables were not retrievable. Only 26/168 (15.5%) of scores were completely programmable using only structured data and 43/168 (25.6%) could potentially be programmable using widely available advanced information retrieval techniques. Scores relying on clinical examination findings or clinical judgments were most often not completely programmable. Complete automation is not possible for most clinical scores because of the high prevalence of clinical examination findings or clinical judgments - partial automation is the most that can be achieved. The effect of fully or partially automated score calculation on clinical efficiency and clinical guideline adherence requires further study.

Towards an integrated food safety surveillance system: a simulation study to explore the potential of combining genomic and epidemiological metadata.

Foodborne infection is a result of exposure to complex, dynamic food systems. The efficiency of foodborne infection is driven by ongoing shifts in genetic machinery. Next-generation sequencing technologies can provide high-fidelity data about the genetics of a pathogen. However, food safety surveillance systems do not currently provide similar high-fidelity epidemiological metadata to associate with genetic data. As a consequence, it is rarely possible to transform genetic data into actionable knowledge that can be used to genuinely inform risk assessment or prevent outbreaks. Big data approaches are touted as a revolution in decision support, and pose a potentially attractive method for closing the gap between the fidelity of genetic and epidemiological metadata for food safety surveillance. We therefore developed a simple food chain model to investigate the potential benefits of combining ‘big’ data sources, including both genetic and high-fidelity epidemiological metadata. Our results suggest that, as for any surveillance system, the collected data must be relevant and characterize the important dynamics of a system if we are to properly understand risk: this suggests the need to carefully consider data curation, rather than the more ambitious claims of big data proponents that unstructured and unrelated data sources can be combined to generate consistent insight. Of interest is that the biggest influencers of foodborne infection risk were contamination load and processing temperature, not genotype. This suggests that understanding food chain dynamics would probably more effectively generate insight into foodborne risk than prescribing the hazard in ever more detail in terms of genotype.

Desain dan Implementasi Platform Manajemen Historis Harga Saham dengan Kurasi Data dan Analisis Teknikal

Saat ini penelitian terkait saham didominasi oleh topik prediksi harga saham. Untuk melakukan prediksi harga saham dibutuhkan data historis harga saham. Banyak provider yang menyediakan data tersebut namun tidak semuanya gratis. Beberapa provider yang menyediakan data historis harga saham secara gratis adalah Yahoo Finance, Google Finance, Stooq, dan National Stock Exchange (NSE) India. Namun, seringkali terdapat perbedaan data antar provider baik dari sisi ketersediaan, konten, format data, dan sebagainya. Sehingga, investor perlu mengambil data dari beberapa provider untuk diperbandingkan guna mendapatkan hasil analisis yang optimal. Oleh karena itu, penelitian ini membangun platform manajemen data historis harga saham yang melakukan kurasi data dari keempat sumber data tersebut sekaligus dilengkapi dengan analisis teknikal untuk memudahkan analisis investasi. Dalam penelitian ini ditemukan bahwa kurasi data historis harga saham untuk keempat sumber data tersebut sudah dapat dilakukan dengan teknologi RDBMS sebagai databasenya. Namun, terdapat kelemahan yaitu kurang fleksibel apabila diperlukan penambahan sumber data dengan data yang tidak terstruktur atau memiliki kolom berbeda-beda. Meskipun demikian, dengan data dari keempat sumber data yang telah terintegrasi, analisis teknikal dapat memberikan gambaran yang lebih luas terhadap tren pergerakan harga dengan memperbandingkan hasil analisis untuk setiap sumber data. 
 Kata kunci – historis harga saham, kurasi data, rdbms, analisis teknikal, moving average

Development of a Total Environment Data Science Approach in a Big Data Scale

We use the Big Data paradigm, as a driving mechanism of an integrated research framework. As a case study, we consider analysing various ecological systems and their connectivity in the framework. An unknown coexistence among different species and lack of knowledge on their sustainability motivate us for undertaking the current research. For describing the recycling systems in nature, an articulated design science research (DSR) framework is necessary for which we have constructed data models for composite lithosphere-atmosphere-biosphere-hydrosphere ecosystem (LABHE). The unstructured big-size environmental data sources and their anomalies existing in nature are taken advantage of, to describe various constructs, compute models and validate them by DSR guidelines. For this purpose, the domain ontology artefacts are drawn and integrated into a warehouse approach to compute an environmental metadata and interpret it in different knowledge domains. The data models and the proposed integrated framework facilitate the environment explorers for planning and management of environmental resources worldwide. The Big Data associated LABHE bring out new knowledge and its interpretation in a variety of environmental data map and plot views. The constructs, models and methodologies used in the current domain application are research deliverables for the total environment researchers and explorers.

An Upstream Business Data Science in a Big Data Perspective

The rugged geographies, geomorphologies and complex geological environments make the explorers more challenging exploration and production (E & P). Despite challenges, many sedimentary basins, associated oil & gas fields and E & P Ventures are productive and commercially viable. The difficulty in understanding the connectivity among multiple reservoirs is due to lack of knowledge of multidisciplinary data of petroleum systems, complicating the data integration and interpretation process. The geological and geophysical data of an upstream business are vital assets of any oil & gas industry, in particular in E & P perspective. The data are often unstructured with a variety of anomalous attributes, mingling with volumes of spatial-temporal dimension attributes and instances. In recent years, the concepts of Big Data have taken different hype in petroleum industries, because of involvement of big sized data in the data integration process. Because of the unstructured data sources, a new direction in the database organization is needed. Investigating the science behind the Big Data and their integrated interpretation of the upstream project is a principal objective of the research. In this context, various constructs and models are articulated with different artefacts. Opportunities of Big Data are explored with exploration data and business analytics, supporting sustainable E & P systems. Petroleum management information systems (PMIS) and digital petroleum ecosystems (PDE) are developed to establish a connectivity among various data sources in multiple domains and systems. The implementation of robust methodologies ascertains the significance of the integrated upstream business in the oil and gas industries that comply with the characteristics of the Big Data.

KIWI: A technology for public health event monitoring and early warning signal detection.

Objectives: To introduce the Canadian Network for Public Health Intelligence’s new Knowledge Integration using Web-based Intelligence (KIWI) technology, and to pefrom preliminary evaluation of the KIWI technology using a case study. The purpose of this new technology is to support surveillance activities by monitoring unstructured data sources for the early detection and awareness of potential public health threats.Methods: A prototype of the KIWI technology, adapted for zoonotic and emerging diseases, was piloted by end-users with expertise in the field of public health and zoonotic/emerging disease surveillance. The technology was assessed using variables such as geographic coverage, user participation, and others; categorized by high-level attributes from evaluation guidelines for internet based surveillance systems. Special attention was given to the evaluation of the system’s automated sense-making algorithm, which used variables such as sensitivity, specificity, and predictive values. Event-based surveillance evaluation was not applied to its full capacity as such an evaluation is beyond the scope of this paper.Results: KIWI was piloted with user participation = 85.0% and geographic coverage within monitored sources = 83.9% of countries. The pilots, which focused on zoonotic and emerging diseases, lasted a combined total of 65 days and resulted in the collection of 3243 individual information pieces (IIP) and 2 community reported events (CRE) for processing. Ten sources were monitored during the second phase of the pilot, which resulted in 545 anticipatory intelligence signals (AIS). KIWI’s automated sense-making algorithm (SMA) had sensitivity = 63.9% (95% CI: 60.2-67.5%), specificity = 88.6% (95% CI: 87.3-89.8%), positive predictive value = 59.8% (95% CI: 56.1-63.4%), and negative predictive value = 90.3% (95% CI: 89.0-91.4%).Discussion: Literature suggests the need for internet based monitoring and surveillance systems that are customizable, integrated into collaborative networks of public health professionals, and incorporated into national surveillance activities. Results show that the KIWI technology is well posied to address some of the suggested challenges. A limitation of this study is that sample size for pilot participation was small for capturing overall readiness of integrating KIWI into regular surveillance activities.Conclusions: KIWI is a customizable technology developed within an already thriving collaborative platform used by public health professionals, and performs well as a tool for discipline-specific event monitoring and early warning signal detection.

Enhancing Delirium Case Definitions in Electronic Health Records Using Clinical Free Text

Delirium is an acute confusional state, associated with morbidity and mortality in diverse medically ill populations. Delirium is preventable and treatable when diagnosed but the diagnosis is often missed. This important and difficult diagnosis is an attractive candidate for computer-aided decision support if it can be reliably identified at scale. Here, using an electronic health record-based case definition of delirium, we characterize incidence of this highly morbid condition in 2 large academic medical centers. Using the electronic health record of 2 large New England academic medical centers, we calculated and compared the rate of the diagnosis of delirium using a range of administrative and discharge summary text-based case definitions over an 8-year period. Depending on case definitions, the overall delirium rate ranged from 2.0-5.4% of 809,512 admissions identified. The identified rate of delirium increased between 2005 and 2013, such that by the final year of the study, one of the two sites reported delirium in 7.0% of cases. The concordance between case definitions was low; only half of the cases identified by text analysis were captured by administrative data. Delirium may be better captured by composite outcomes, including both administrative claims data and elements drawn from unstructured data sources. That the rate of delirium observed in this study is far lower than the current literature estimates suggests that further work on case definitions, identification, and documented diagnosis is required.

A new approach to distinguish migraine from stroke by mining structured and unstructured clinical data-sources

Distinguishing migraine from stroke is a challenge due to many common signs and symptoms. It is important to consider the cost of hospitalization and the time spent by neurologists and stroke nurses to visit, diagnose, and assign appropriate care to the patients; therefore, devising new ways to distinguish stroke, migraine and other types of mimics can help in saving time and cost, and improve decision-making. In this study, we utilized text and data mining methods to extract the most important predictors from clinical reports in order to establish a migraine detection model and distinguish migraine patients from stroke or other types of mimic (non-stroke) cases. The available data for this study was a heterogeneous mix of free-text fields, such as triage main-complaints and specialist final-impressions, as well as numeric data about patients, such as age, blood-pressure, and so on. After a careful combination of these sources, we obtained a highly imbalanced dataset where the migraine cases were only about 6 % of the dataset. Our main challenge was tackling this data imbalance. Using the dataset in its original form to build classifiers led to a learning bias towards the majority class and against the minority (migraine) class. We used a sampling method to address the imbalance problem. First, different sources of data were preprocessed and balanced datasets were generated; second, attribute selection algorithms were used to reduce the dimensionality of the data; third, a novel combination of data mining algorithms was employed in order to effectively distinguish migraine from other cases. We achieved a sensitivity and specificity of about 80 and 75 %, respectively, which is in contrast to a sensitivity and specificity of 15.7 and 97 % when using the original imbalanced data for building classifiers.

A retrospective of knowledge graphs

Information on the Internet is fragmented and presented in different data sources, which makes automatic knowledge harvesting and understanding formidable for machines, and even for humans. Knowledge graphs have become prevalent in both of industry and academic circles these years, to be one of the most efficient and effective knowledge integration approaches. Techniques for knowledge graph construction can mine information from either structured, semi-structured, or even unstructured data sources, and finally integrate the information into knowledge, represented in a graph. Furthermore, knowledge graph is able to organize information in an easy-to-maintain, easy-to-understand and easy-to-use manner. In this paper, we give a summarization of techniques for constructing knowledge graphs. We review the existing knowledge graph systems developed by both academia and industry. We discuss in detail about the process of building knowledge graphs, and survey state-of-the-art techniques for automatic knowledge graph checking and expansion via logical inferring and reasoning. We also review the issues of graph data management by introducing the knowledge data models and graph databases, especially from a NoSQL point of view. Finally, we overview current knowledge graph systems and discuss the future research directions.

DeepDive

The dark data extraction or knowledge base construction (KBC) problem is to populate a SQL database with information from unstructured data sources including emails, webpages, and pdf reports. KBC is a long-standing problem in industry and research that encompasses problems of data extraction, cleaning, and integration. We describe DeepDive, a system that combines database and machine learning ideas to help develop KBC systems. The key idea in DeepDive is that statistical inference and machine learning are key tools to attack classical data problems in extraction, cleaning, and integration in a unified and more effective manner. DeepDive programs are declarative in that one cannot write probabilistic inference algorithms; instead, one interacts by defining features or rules about the domain. A key reason for this design choice is to enable domain experts to build their own KBC systems. We present the applications, abstractions, and techniques of DeepDive employed to accelerate construction of KBC systems.