Unstructured Datasets Research Articles

An Entity Extraction and Categorization Technique on Twitter Streams

As social media platforms have gained huge momentum in recent years, the amount of information generated from the social media sites is growing exponentially and gives the information retrieval systems a great challenge to extract the potential named entities. Researchers have utilized the semantic annotation mechanism to retrieve the entities from the unstructured documents, but the mechanism returns with too many ambiguous entities. In this work, the DBpedia knowledge base is adopted for entity extraction and categorization. To achieve the entity extraction task precisely, a two-step process is proposed: (a) train the unstructured datasets with Word2Vec and classify the entities into their respective categories. (b) crawl the web pages, forums, and other web sources to identifying the entities that are not present in the DBpedia. The evaluation shows the results with more precision and promising F1 score.

Myasthenia Gravis Treatments Are Associated with Negative Patient Sentiment in Digital Conversation Data (P7-8.013)

<h3>Objective:</h3> To explore patient experiences with treatments for Myasthenia Gravis (MG) via digital conversations with a focus on corticosteroids and Immunoglobulin (Ig) therapies. <h3>Background:</h3> MG is a rare, chronic autoantibody disease that requires long-term treatment and is associated with high disease and treatment burden and reduced quality of life. <h3>Design/Methods:</h3> One year (8/2021–8/2022) of conversations focused on MG and treatments, specifically corticosteroids and Ig, originating from US internet protocol addresses were tagged based on self-identification in the conversations or on public profiles. Advanced search techniques together with artificial intelligence powered algorithms were used to extract and organize data by topics into a large, unstructured dataset. Sentiment analysis via natural language processing was conducted to categorize conversations as positive, negative or neutral and analyzed to derive the most frequent drivers of each sentiment. <h3>Results:</h3> Among 13,234 conversations extracted from topical sites, message boards, social networks and blogs, 19% were related to MG treatments. Most conversations had negative sentiment (59%), 39% were neutral, and 1% were positive. Among corticosteroid and Ig treated individuals, respectively, posts with negative (69% and 61%) and neutral (31% and 39%) sentiments were the norm, with virtually no “positive” conversations. “Treatment issues” was the dominant driver of negative sentiment among the corticosteroid and Ig related conversations. When posting about “barriers to treatment”, corticosteroid related conversations were most frequently about lack of effectiveness and side effects, while patients discussing Ig treatments most frequently focused on side effects and cost/insurance. <h3>Conclusions:</h3> Digital conversations reveal a high degree of negativity and concern related to MG treatment, specifically, corticosteroids and Immunoglobulin therapies. There is a need for new treatments that provide effectiveness through symptom control with few side effects. By capturing patient voice, this study reinforces limitations of current treatments and highlights the need for improved options for individuals with MG. <b>Disclosure:</b> Dr. Pesa has received personal compensation for serving as an employee of Janssen. Dr. Pesa has stock in J&amp;J. Dr. Chaudhry has received personal compensation for serving as an employee of Johnson &amp; Johnson. Dr. Chaudhry has received personal compensation for serving as an employee of Takeda Pharmaceuticals. Mrs. Brethenoux has received personal compensation for serving as an employee of Cien+ . Mr. Furey has nothing to disclose. The institution of Mrs. Gil Valletta has received personal compensation in the range of $50,000-$99,999 for serving as a Consultant for Janssen. The institution of Mrs. Gil Valletta has received personal compensation in the range of $10,000-$49,999 for serving on a Speakers Bureau for A- Speakers. The institution of Mrs. Gil Valletta has received personal compensation in the range of $10,000-$49,999 for serving on a Speakers Bureau for The Cleveland Clinic. Mrs. Gil Valletta has received personal compensation in the range of $500-$4,999 for serving as an officer or member of the Board of Directors for United Health Group. Mrs. Gil Valletta has received personal compensation in the range of $100,000-$499,999 for serving as an officer or member of the Board of Directors for Zumiez. Miss Gonzalez Quijano has nothing to disclose. Mr. Lorenzo has nothing to disclose.

Correlation Evaluation Scale Through Text Mining Algorithms and Implementation on the Kurdish Language - A Review

In recent times, because of the many articles that are found on the web, text extraction has become an interesting area of research. Text mining is a technique that can be used to extract useful information or knowledge from text documents that are not typically in an unstructured form. There are some studies conducted to use different techniques of text production for unstructured data sets. This study will provide an overview of the different methods and algorithms that are related to Text mining and also some studies on the mining of Kurdish web documentation. In addition, a collection of research problems and research methodologies will assist scholars in tracking their future research.

AIDA: Analytic isolation and distance-based anomaly detection algorithm

Many unsupervised anomaly detection algorithms rely on the concept of nearest neighbours to compute the anomaly scores. Such algorithms are popular because there are no assumptions about the data, making them a robust choice for unstructured datasets. However, the number (k) of nearest neighbours, which critically affects the model performance, cannot be tuned in an unsupervised setting. Hence, we propose the new and parameter-free Analytic Isolation and Distance-based Anomaly (AIDA) detection algorithm, that combines the metrics of distance with isolation. Based on AIDA, we also introduce the Tempered Isolation-based eXplanation (TIX) algorithm, which identifies the most relevant features characterizing an outlier, even in large multi-dimensional datasets, improving the overall explainability of the detection mechanism. Both AIDA and TIX are thoroughly tested and compared with state-of-the-art alternatives, proving to be useful additions to the existing set of tools in anomaly detection.

A new data integration framework for Covid-19 social media information

The Covid-19 pandemic presents a serious threat to people’s health, resulting in over 250 million confirmed cases and over 5 million deaths globally. To reduce the burden on national health care systems and to mitigate the effects of the outbreak, accurate modelling and forecasting methods for short- and long-term health demand are needed to inform government interventions aiming at curbing the pandemic. Current research on Covid-19 is typically based on a single source of information, specifically on structured historical pandemic data. Other studies are exclusively focused on unstructured online retrieved insights, such as data available from social media. However, the combined use of structured and unstructured information is still uncharted. This paper aims at filling this gap, by leveraging historical and social media information with a novel data integration methodology. The proposed approach is based on vine copulas, which allow us to exploit the dependencies between different sources of information. We apply the methodology to combine structured datasets retrieved from official sources and a big unstructured dataset of information collected from social media. The results show that the combined use of official and online generated information contributes to yield a more accurate assessment of the evolution of the Covid-19 pandemic, compared to the sole use of official data.

Quantum Pattern Classification in a Three-Qubit System

The problem of pattern classification in quantum data has been of great importance over the past few years. This study investigates the effect of deploying Grover’s, the partial diffusion, and the fixed-phase algorithms separately to amplify the amplitudes of a desired pattern in an unstructured dataset. These quantum search operators were applied to symmetric and antisymmetric input superpositions on a three-qubit system for 20 iterations each. After each iteration, different probabilities of classification were calculated in order to determine the accuracy of classification for each of the three quantum search operators. The results indicated that, in the case of applying the three quantum search operators to incomplete superposition input states, the partial diffusion operator outperformed the other operators with a probability of correct classification that reached 100% in certain iterations. It also showed that the classification accuracy of the fixed-phase operator exceeded the accuracy of the other two operators by 40% in most cases when the input state was a uniform superposition, and some of the basis states were phase-inverted.

데이터 경제 발전을 위한 데이터베이스권의 보완 및 개선

In order to support the transition to a data economy where data functions as an important product and commodity, it is necessary to establish an institution to encourage producing and opening quality datasets. Database right under the Copyright Act and the prohibition of unauthorized use of other’s achievements and illegal use of data under the Unfair Competition Prevention Act serves as potential institution for protection of datasets. Each of these has its own strengths and weaknesses and can play a role, but the prospects of the database right stand out in particular. This is because it has excellent clarity and predictability, protects against infringement by non-competitors, and separates the domains of monopoly and sharing appropriately and clearly. However, database right is not without room for improvement. First, there are unstructured datasets without searchability among training data for AI, and it is necessary to remove the element of searchability from the concept of database in order to include them in the scope of protection. Second, in order to compensate for the uncertainty of protection requirements, it is necessary to introduce the provision for presuming the establishment of ‘substantial part’ of the database when structural information on the arrangement or composition of materials is included. Third, it is necessary to reexamine the boundary between monopoly and sharing. For example, if the database is the sole source for a certain data, it may be desirable to introduce a compulsory licensing regime according to the FRAND standard.

A fusion of structured and unstructured datasets in curating fire damage

Accurate documentation and mitigation of wildfires is significant in addressing Sustainable Goal eleven (11) as well as the African Unions Agenda 2063 Goal one (1) which resonate an intention to preserve sustainable environments, communities, and cities for future generations. Poor and unscientific documentation of wildfire leads to error prone and quickly outdated event curation, making it difficult to create comprehensive overviews of wildfire events. Although several geospatial studies have recorded advances in wildfire data curation, many of these rely on structured spatial datasets which enjoy numerous advantages but can be prone to temporal or time-voids in data. Voids in data can be mitigated through interpolation techniques, but these are not without challenges for the modellers tasked with optimally filling the voids. On this backdrop, this paper therefore aims to test and extend strategies of wildfire reporting by investigating the strengths of fusing mainstream spatial data (including high resolution unmanned aerial vehicle (UAV) footage) with unstructured Twitter (tweets) data in order to disrupt common practise and contribute to the global agenda. The novelty of this contribution lies in its unusual hybrid data approach used to statistically establishes long- and short-term pre-fire conditions tested on the African landscape. It proceeds to illustrate how the social media data can create fascinating and yet accurate visual timelines of the events that surprisingly compared strongly to the results from structured sources. Finally, the study detects the extent of damage from the wildfire using supervised classification, burn indices and three-dimensional (3D) reconstruction. A 95% positive detection rate was reported, and it affirmed the place of unstructured data in mainstream scientific approaches such as wildfire documentation. The work presented therefore contributes towards meeting African Union's Agenda 2063 and the United Nations Sustainable Development Goals (2030) in curation, documentation and mitigation of sustainable community and environments.

Assessment of a pandemic emergency financing facility

The pandemic bond issued by the World Bank (WB) in 2017 is a financial innovation enabling the transfer of the pandemic risk from the underdeveloped/developing countries to the financial market. It covers perils of various diseases that could overwhelm the global health systems and adversely impact the world economy. If all the triggers are activated, the bond’s principal and coupons are used to finance coordinated, swift and resilient medical response to safeguard the well-being of the populace. This product, however, is criticised for its onerous trigger requirements. We examine the WB’s pandemic-bond pricing framework, which requires inputs that are only partially available. From a rather unstructured COVID-19 data set, an information database is created and customised for pandemic-bond valuation. A vector auto-regressive moving average model is utilised to jointly describe the triggers dynamics. Our modelling simulations of risk triggers reveal that the bond payout could be made in less than half of the WB’s earliest opportunity of 85 days.

Data-centric artificial intelligence in oncology: a systematic review assessing data quality in machine learning models for head and neck cancer

Machine learning models have been increasingly considered to model head and neck cancer outcomes for improved screening, diagnosis, treatment, and prognostication of the disease. As the concept of data-centric artificial intelligence is still incipient in healthcare systems, little is known about the data quality of the models proposed for clinical utility. This is important as it supports the generalizability of the models and data standardization. Therefore, this study overviews the quality of structured and unstructured data used for machine learning model construction in head and neck cancer. Relevant studies reporting on the use of machine learning models based on structured and unstructured custom datasets between January 2016 and June 2022 were sourced from PubMed, EMBASE, Scopus, and Web of Science electronic databases. Prediction model Risk of Bias Assessment (PROBAST) tool was used to assess the quality of individual studies before comprehensive data quality parameters were assessed according to the type of dataset used for model construction. A total of 159 studies were included in the review; 106 utilized structured datasets while 53 utilized unstructured datasets. Data quality assessments were deliberately performed for 14.2% of structured datasets and 11.3% of unstructured datasets before model construction. Class imbalance and data fairness were the most common limitations in data quality for both types of datasets while outlier detection and lack of representative outcome classes were common in structured and unstructured datasets respectively. Furthermore, this review found that class imbalance reduced the discriminatory performance for models based on structured datasets while higher image resolution and good class overlap resulted in better model performance using unstructured datasets during internal validation. Overall, data quality was infrequently assessed before the construction of ML models in head and neck cancer irrespective of the use of structured or unstructured datasets. To improve model generalizability, the assessments discussed in this study should be introduced during model construction to achieve data-centric intelligent systems for head and neck cancer management.

Mining Competitors and Finding Winning Plans Using Feature Scoring and Ranking-Based CMiner++ Algorithm

For a business to succeed, it is very important to make things speaking more to clients than to rivals. It is more critical to decide on the significant feature of an item which influences its competency. In spite of the works that have been done already, a few algorithms gained efficient solution. This paper proposes the CMiner++ Algorithm to assess the competitive relationship among items in unstructured dataset and finding the Top-K competitors of a given item. Definitively, the nature of the outcomes and the versatility of this methodology utilizing numerous datasets from various areas are assessed, and the efficiency and adaptability of this algorithm on various data sets are improved when compared to existing algorithms. In today's busy world, automatic recommendation systems are emerging because people are looking for the products best suited for them. So, it is very important to analyse the behaviour of people, make a review on large and large unstructured data sets, and make the fully automated deep learning system to ensure the accurate outcome.

Quantum density peak clustering

Clustering algorithms are of fundamental importance when dealing with large unstructured datasets and discovering new patterns and correlations therein, with applications ranging from scientific research to medical imaging and marketing analysis. In this work, we introduce a quantum version of the density peak clustering algorithm, built upon a quantum routine for minimum finding. We prove a quantum speedup for a decision version of density peak clustering depending on the structure of the dataset. Specifically, the speedup is dependent on the heights of the trees of the induced graph of nearest-highers, i.e. the graph of connections to the nearest elements with higher density. We discuss this condition, showing that our algorithm is particularly suitable for high-dimensional datasets. Finally, we benchmark our proposal with a toy problem on a real quantum device.

Data Driven Classification of Opioid Patients Using Machine Learning–An Investigation

The opioid crisis has led to an increased number of drug overdoses in recent years. Several approaches have been established to predict opioid prescription by health practitioners. However, due to the complex nature of the problem, the accuracy of such methods is not yet satisfactory. Dependable and reliable classification of opioid dependent patients from well-grounded data sources is essential. Majority of the previous studies do not focus on the users’ mental health association for opioid intake classification. These studies do not also employ the latest deep learning based techniques such as attention and knowledge distillation mechanism to find better insights. This paper investigates the opioid classification problem by using machine learning and deep learning based techniques. We used structured and unstructured data from the MIMIC-III database to identify intentional and unintentional intake of opioid drugs. We selected 455 patient instances and used traditional machine learning and deep learning to predict intentional and accidental users. We obtained 95% and 64% test accuracy to predict the intentional and accidental users from the structured and unstructured datasets, respectively. We also achieve a distilled knowledge based test accuracy of 76.44% from the integrated above two models. Our research includes an ablation analysis and new insights related to opioid patients are extracted.

Analyzing lives information of empty nesters in data visualization method

This paper analyzes the information of 39,276 empty nesters living in Ziyang city. Using the Neo4j Graph database and data visualization tools, Pyecharts. It takes Ziyang City as a case. By collecting an unstructured dataset containing specific details of 39,276 empty nesters, released by the local Bureau of Civil Affairs. The final result of this paper proves that the practical method is achievable. Besides, the final analysis result could be a trustable reference cited by local government or municipal decision-makers to perfect laws and policies in the local elderly care industry.

Semantic similarity measure for topic modeling using latent Dirichlet allocation and collapsed Gibbs sampling

Automatically extracting topics from large amounts of text is one of the main uses of natural language processing (NLP). The latent Dirichlet allocation (LDA) technique is frequently used to extract topics from pre-processed materials based on word frequency. One of the main problems of LDA is that the topics extracted are of poor quality if the document does not coherently belong to a single topic. However, Gibbs sampling operates on a word-by-word basis, which allows it to be used on documents with a variety of topics and modifies the topic assignment of a single word. To improve the quality of topics extracted, this paper developed a hybrid-based semantic similarity measure for topic modeling combining LDA and Gibbs sampling to maximize the coherence score. To verify the effectiveness of the suggested model, an unstructured dataset was taken from a public repository. The evaluation carried out shows that the proposed LDA-Gibbs had a coherence score of 0.52650 as against the LDA coherence score of 0.46504. The proposed multi-level model provides better quality of topics extracted.

External Validation of Natural Language Processing Algorithms to Extract Common Data Elements in THA Operative Notes

BackgroundNatural language processing (NLP) systems are distinctive in their ability to extract critical information from raw text in electronic health records (EHR). We previously developed three algorithms for total hip arthroplasty (THA) operative notes with rules aimed at capturing (1) operative approach, (2) fixation method, and (3) bearing surface using inputs from a single institution. The purpose of this study was to externally validate and improve these algorithms as a prerequisite for broader adoption in automated registry data curation. MethodsThe previous NLP algorithms developed at Mayo Clinic were deployed and refined on EHRs from OrthoCarolina, evaluating 39 randomly selected primary THA operative reports from 2018 to 2021. Operative reports were available only in PDF format, requiring conversion to “readable” text with Adobe software. Accuracy statistics were calculated against manual chart review. ResultsThe operative approach, fixation technique, and bearing surface algorithms all demonstrated perfect accuracy of 100%. By comparison, validated performance at the developing center yielded an accuracy of 99.2% for operative approach, 90.7% for fixation technique, and 95.8% for bearing surface. ConclusionNLP algorithms applied to data from an external center demonstrated excellent accuracy in delineating common elements in THA operative notes. Notably, the algorithms had no functional problems evaluating scanned PDFs that were converted to “readable” text by common software. Taken together, these findings provide promise for NLP applied to scanned PDFs as a source to develop large registries by reliably extracting data of interest from very large unstructured data sets in an expeditious and cost-effective manner.

Computer Vision and Internet Meme Genealogy: An Evaluation of Image Feature Matching as a Technique for Pattern Detection

ABSTRACT Internet memes are a fundamental aspect of digital culture. Despite being individual expressions, they vastly transcend the individual level as windows into and vehicles for wide-stretching social, cultural, and political narratives. Empirical research into meme culture is thriving, yet particularly compartmentalized. In the humanities and social sciences, most efforts involve in-depth linguistic and visual analyses of mostly handpicked examples of memes, begging the question on the origins and meanings of those particular expressions. In technical disciplines, such as computer science, efforts are focused on the large-scale identification and classification of meme images, as well as patterns of “viral” spread at scale. This contribution aims to bridge the chasm between depth and scale by introducing a three-step approach suitable for “computational grounded theoretical” studies in which (1) an automated procedure establishes formal links between meme images drawn from a large-scale corpus paving the way for (2) network analysis to infer patterns of relatedness and spread, and (3) practically classifying visually related images in file folders for the purpose of further local, hermeneutic analysis. The procedure is demonstrated and evaluated on two datasets: an artificially constructed, structured dataset and a naturally harvested unstructured dataset. Future horizons and domains of application are discussed.

Exploring natural language processing techniques to extract semantics from unstructured dataset which will aid in effective semantic interlinking

Designing efficacious semantics for the dynamic interaction and searches has proven to be concretely challenging because of the dynamically of the semantic searches, method of browsing and visualization interfaces for high volume information. This has a direct impact on enhancing the capabilities of the web. To surmount the challenges of providing meaning to high volume unstructured datasets, Natural language processing techniques and implements have been proven to be propitious, however, the reactivity of these techniques should be studied and predicated on the objective of providing meaning to the unstructured data. This paper demonstrates the working of five NLP techniques namely, bag-of-words, TF-IDF, NER, LSA, and LDA. The experiment provides the kindred attribute accomplishment or the identification of the meaning of this unstructured data varies from one technique to another. However, NLP techniques can be efficient as they provide insights into the data and make it human-readable. This will in turn avail in building better human–machine intractable browsing and applications.

Utilizing Deep Learning for Detecting Adverse Drug Events in Structured and Unstructured Regulatory Drug Data Sets.

The US Food and Drug Administration (FDA) collects and retains several data sets on post-market drugs and associated adverse events (AEs). The FDA Adverse Event Reporting System (FAERS) contains millions of AE reports submitted by the public when a medication is suspected to have caused an AE. The FDA monitors these reports to identify drug safety issues that were undetected during the premarket evaluation of these products. These reports contain patient narratives that provide information regarding the AE that needs to be coded using standardized terminology to enable aggregation of reports for further review. Additionally, the FDA collects structured drug product labels (SPLs) that facilitate standardized distribution of information regarding marketed medical products. Manufacturers are currently not required to code labels with associated AEs. Approaches for automated classification of reports by preferred terminology could enhance regulatory efficiency. The goal of this work was to assess the suitability of manually annotated FDA FAERS and SPL data sets to be subjected to predictive modeling. A recurrent neural network (RNN) was proposed as a proof-of-concept model for automated extraction of preferred AE terminology. A separate RNN was fit and cross-validated on two regulatory data sets with varying properties. First, the researchers trained and cross-validated a model on 325 annotated FAERS patient narratives for a sample of AE terms. A model was then trained and validated on a data set of 100 SPLs. Model cross-validation results for product labels demonstrated that the model performed at least as well as more conventional models for all but one of the terms selected based on F1-score. Model results for the FAERS data set were mixed. This work successfully demonstrated a proof-of-concept machine learning approach to automatically detect AEs in several textual regulatory data sets to support post-market regulatory activities. Limited instances of each AE class likely prohibited models from generalizing data effectively. Additional data may permit more robust validation.

Privacy‐preserving clustering of unstructured big data for cloud‐based enterprise search solutions

Cloud-based enterprise search services (e.g., Amazon Kendra) are enchanting to big data owners by providing them with convenient search solutions over their enterprise big datasets. However, individuals and businesses dealing with confidential big data (e.g., criminal reports) are reluctant to fully embrace such cloud services due to valid data privacy concerns. Solutions based on client-side encryption have been developed to mitigate these concerns. Nonetheless, such solutions hinder data processing, especially, data clustering, which is pivotal in applications such as real-time search on large corpora (e.g., big datasets). To cluster encrypted big data, we propose privacy-preserving clustering schemes, called ClusPr, for three forms of unstructured datasets, namely static, semi-dynamic, and dynamic. ClusPr functions based on statistical characteristics of the datasets to: (A) determine the suitable number of clusters; (B) populate the clusters with topically relevant tokens; and (C) adapt the cluster set based on the dynamism of the underlying dataset. Experimental results, obtained from evaluating ClusPr against other schemes in the literature, on three different test datasets demonstrate between 30 % $$ 30\% $$ and 60 % $$ 60\% $$ improvement on the cluster coherency. Moreover, we notice that employing ClusPr within a privacy-preserving enterprise search system can reduce the search time by up to 78 % $$ 78\% $$ , while improving the search accuracy by up to 35 % $$ 35\% $$ .