Data Extraction Algorithm Research Articles

Automatic data extraction from 24 hour blood pressure measurement reports of a large multicenter clinical trial

Background and objectivesAmbulatory blood pressure monitoring (ABPM) is usually reported in descriptive values such as circadian averages and standard deviations. Making use of the original, individual blood pressure measurements may be advantageous, particularly for research purposes, as this increases the flexibility of the analytical process, enables alternative statistical analyses and provide novel insights. Here we describe the development of a new multistep, hierarchical data extraction algorithm to collect raw data from .pdf reports and text files as part of a large multi-center clinical study. MethodsOriginal reports were saved in a nested file system, from which they were automatically extracted, read and saved into databases with custom made programs written in Python 3. Data were further processed, cleaned and relevant descriptive statistics such as averages and standard deviations calculated according to a variety of definitions of day- and night-time. Additionally, data control mechanisms for manual review of the data and programmatic auto-detection of extraction errors was implemented as part of the project. ResultsThe developed algorithm extracted 97% of the data automatically, the missing data consisted mostly of reports that were saved incorrectly or not formatted in the specified way. Manual checks comparing samples of the extracted data to original reports indicated a high level of accuracy of the extracted data, no errors introduced due to flaws in the extraction software were detected in the extracted dataset. ConclusionsThe developed multistep, hierarchical data extraction algorithm facilitated collection from different file formats and paired with database cleaning and data processing steps led to an effective and accurate assembly of raw ABPM data for further and adjustable analyses. Manual work was minimized while data quality was ensured with standardized, reproducible procedures.

Analysis of WEKA data mining algorithms Bayes net, random forest, MLP and SMO for heart disease prediction system: A case study in Iraq

Data mining is defined as a search through large amounts of data for valuable information. The association rules, grouping, clustering, prediction, sequence modeling is some essential and most general strategies for data extraction. The processing of data plays a major role in the healthcare industry's disease detection. A variety of disease evaluations should be required to diagnose the patient. However, using data mining strategies, the number of examinations should be decreased. This decreased examination plays a crucial role in terms of time and results. Heart disease is a death-provoking disorder. In this recent instance, health issues are immense because of the availability of health issues and the grouping of various situations. Today, secret information is important in the healthcare industry to make decisions. For the prediction of cardiovascular problems, (Weka 3.8.3) tools for this analysis are used for the prediction of data extraction algorithms like sequential minimal optimization (SMO), multilayer perceptron (MLP), random forest and Bayes net. The data collected combine the prediction accuracy results, the receiver operating characteristic (ROC) curve, and the PRC value. The performance of Bayes net (94.5%) and random forest (94%) technologies indicates optimum performance rather than the sequential minimal optimization (SMO) and multilayer perceptron (MLP) methods.

Using electronic health record’s data to assess daily dose of opioids prescribed for outpatients with chronic non-cancer pain

This research intended to examine electronic health record (EHR) based methods for automated estimation of morphine equivalent daily dose (MEDD) of prescribed opioids in primary care research and practice. The...

AUTOMATIC DETERMINATION OF THE NAVIGATORS MOTIVATION MODEL WHEN OPERATING WATER TRANSPORT

Context. The article proposes an approach for automated identification of the navigators motivational model in the control of water transport. Algorithms for data extraction as a result of the man-machine interaction of navigator with the electronic control systems of the vessel during performing navigation operations of increased complexity are proposed.
 Objective. The purpose of research is to apply formal and algorithmic approaches to extracting data on the motivational model of navigator to prevent accidents in water transport. 
 Method. The identification of manifestation determination of navigators’ mental activity by means of the visual concept of the geometric group theory is proposed. This approach delivered the visual systematic-logical combining of diagnostic methods aimed at determining navigators motivational centers and the processes of professional activity like maneuver performing. The key indicator of identification is said to be the parameter of the navigator’s activity as “rpm_port” having an impact on the vessel speed being a marker of intensification of the navigator’s physiological activity. Such an approach is beneficial in time phase identification while maneuvering indicating explicitly at the stepping up of the navigator’s physiological motivational state. It was proven to be correct based on the results due to Ward’s dendrogram, several statistical methods and applied software. The obtained research results encourage the prediction of the navigator’ motivational states in critical situations.
 Results. In order to confirm the proposed formal-algorithmic approach, an experiment was carried out using the navigation simulator Navi Trainer 5000. Automated analysis of experimental ones made it possible to form a motivational map of the navigator and determine the decision-making model affecting in the processes of control vessel in difficult situations.
 Conclusions. The proposed research approaches made it possible to automate the processes of extracting data indicating the principles of decision-making by navigator. The effectiveness of proposed approach was substantiated by the results of experimental data automated processing and the constructed tree-like decision-making spaces.

Sysrev: A FAIR Platform for Data Curation and Systematic Evidence Review.

Well-curated datasets are essential to evidence based decision making and to the integration of artificial intelligence with human reasoning across disciplines. However, many sources of data remain siloed, unstructured, and/or unavailable for complementary and secondary research. Sysrev was developed to address these issues. First, Sysrev was built to aid in systematic evidence reviews (SER), where digital documents are evaluated according to a well defined process, and where Sysrev provides an easy to access, publicly available and free platform for collaborating in SER projects. Secondly, Sysrev addresses the issue of unstructured, siloed, and inaccessible data in the context of generalized data extraction, where human and machine learning algorithms are combined to extract insights and evidence for better decision making across disciplines. Sysrev uses FAIR - Findability, Accessibility, Interoperability, and Reuse of digital assets - as primary principles in design. Sysrev was developed primarily because of an observed need to reduce redundancy, reduce inefficient use of human time and increase the impact of evidence based decision making. This publication is an introduction to Sysrev as a novel technology, with an overview of the features, motivations and use cases of the tool.Methods: Sysrev. com is a FAIR motivated web platform for data curation and SER. Sysrev allows users to create data curation projects called “sysrevs” wherein users upload documents, define review tasks, recruit reviewers, perform review tasks, and automate review tasks.Conclusion: Sysrev is a web application designed to facilitate data curation and SERs. Thousands of publicly accessible Sysrev projects have been created, accommodating research in a wide variety of disciplines. Described use cases include data curation, managed reviews, and SERs.

SPY-BOT: Machine learning-enabled post filtering for Social Network-Integrated Industrial Internet of Things

A far-reaching expansion of advanced information technology enables ease and seamless communications over online social networks, which have been a de facto premium correspondents in the current cyber world. The ever-growing social network data has gained attention in recent years and can be handy for industrial revolution 4.0. With the integration of social networks with the Internet of Things being noticed in different industries to enhance human involvement and increase their productivity, security in such networks is increasingly alarming. Vulnerabilities can be characterized in the form of privacy invasion, leading to hazardous contents, which can be detrimental to social media actors and in turn impact the processes of the overall Social Network-Integrated Industrial Internet of Things (SN-IIoT) ecosystem. Despite this prevalence, the current platforms do not have any significant level of functionality to capture, process, and reveal unhealthy content among the social media actors. To address those challenges by detecting hazardous contents and create a stable social internet environment within IIoT, a statistical learning-enabled trustworthy analytic tool for human behaviors has been developed in this paper. More specifically, this paper proposes a machine learning (ML)-enabled scheme SPY-BOT, which incorporates a hybrid data extraction algorithm to perform post-filtering that arbitrates the users’ behavior polarity. The scheme creates class labels based on the featured keywords from the decision user and classifies suspicious contacts through the aid of ML. The results suggest the potential of the proposed approach to classify the users’ behavior in SN-IIoT.

Study of the pitch behaviour of large‐scale wind turbines based on statistic evaluation

Pitch control is a basic function of large-scale wind turbines to improve energy capture efficiency. Pitch system is also a component system with a high failure rate due to the complex pitch behaviour. It is helpful to the practical design and control if a clear pitch behaviour is obtained. However, pitch behaviour is affected by the structure of wind turbines, control strategies, wind resource characteristics, and so on. So, there is still a lack of sufficient understanding. In this study, the pitch behaviour of a 2 MW wind turbine is investigated based on SCADA data. Four key evaluation indicators describing pitch behaviour are proposed, which include pitch status, pitch time consumption, pitch angle peak, and mean pitch angle. Moreover, the SCADA data extraction algorithm for pitch behaviour investigation is also given. Then, the pitch behaviour is investigated based on statistical evaluation. Some valuable conclusions have been given. For example, for the investigated wind turbine, the time proportions for the four states (non-pitch, increasing pitch, pausing pitch, and decreasing pitch) are 85.80%, 5.97%, 1.45%, and 6.78%, respectively. The pitch cycles, pitch angle peak, and mean pitch angle increase with the increase of the standard deviation of wind speed.

Mother-Infant Interaction Kinect Analysis (MIIKA): An automatic kinematic-based methodology for the investigation of interpersonal distance during early exchanges

Interpersonal distance is a core aspect of mother-child interaction. While conventional measures based on human coders do not fully capture the dynamics of this feature, computational methods provide automatic measures which can detect even small changes and more accurate estimates both spatially and temporally. Using RGB-D sensors (Microsoft Kinect V2), the present study describes a setup to automatically examine interpersonal distance during mother-child interactions, termed Mother-Infant Interaction Kinect Analysis (MIIKA). First, the laboratory setting and the data extraction method are described. By using an ad-hoc algorithm for kinematic data extraction, MIIKA returns three metrics: barycenter position (distance and velocity of approach and separation), movements (number of small, medium and large approaches and separations) and contributions (proportional contributions of mother and child to approaches and separations). Secondly, preliminary MIIKA metrics are described for a non-clinical mother-child dyad as an exemplification of the protocol. As interpersonal distance can be affected by contingent situations, we detected mother-infant full skeleton during three interactional contexts characterized by different kinds of dyadic exchanges: a free play session, a task-oriented activity and an emotionally arousing condition. Results highlighted similarities and differences between the three interactional contexts. MIIKA appears to be a promising setup to automatically examine interpersonal distance in early mother-child interactions.

Panoramic visualization of thermal faults in power transformer based on data mapping technology

For the thermal monitoring of power transformers, contactless infrared technology has advantages of high resolution and fast responding time, and it does not disturb the original thermal profile across the transformer. However, the thermal faults detected by infrared technology can only be visualized on 2-D projection of the region of interest. The precise spatial localization of the thermal faults cannot be evaluated. In this paper, an improved Augmented Reality technology for panoramic visualization of thermal faults in power transformer is presented. Refined power transformer model is made by using 3-D scanning and reconstruction technology. Infrared images for the apparent temperature measurements of power transformers from different point of views and scales are collected, and the temperature matrix for each image are obtained by using a self-developed data extraction algorithm. Further, 2-D to 3-D geometry transformation are applied to do the mapping of temperature data on the meshing of the refined transformer model. Eventually, the thermal profile across the surface of transformer is holographically displayed, and the precise localization of thermal faults on the transformer can be carried out in 3-D space. Research in this paper makes contribution to smarter and efficient condition monitoring of power transformers.

Efficient Methodology for Deep Web Data Extraction

Deep Web substance are gotten to by inquiries submitted to Web information bases and the returned information records are enwrapped in progressively created Web pages (they will be called profound Web pages in this paper). Removing organized information from profound Web pages is a difficult issue because of the fundamental mind boggling structures of such pages. As of not long ago, an enormous number of strategies have been proposed to address this issue, however every one of them have characteristic impediments since they are Web-page-programming-language subordinate. As the mainstream two-dimensional media, the substance on Web pages are constantly shown routinely for clients to peruse. This inspires us to look for an alternate path for profound Web information extraction to beat the constraints of past works by using some fascinating normal visual highlights on the profound Web pages. In this paper, a novel vision-based methodology that is Visual Based Deep Web Data Extraction (VBDWDE) Algorithm is proposed. This methodology basically uses the visual highlights on the profound Web pages to execute profound Web information extraction, including information record extraction and information thing extraction. We additionally propose another assessment measure amendment to catch the measure of human exertion expected to create wonderful extraction. Our investigations on a huge arrangement of Web information bases show that the proposed vision-based methodology is exceptionally viable for profound Web information extraction.

WEIDJ: Development of a new algorithm for semi-structured web data extraction

In the era of industrial digitalization, people are increasingly investing in solutions that allow their process for data collection, data analysis and performance improvement. In this paper, advancing web scale knowledge extraction and alignment by integrating few sources by exploring different methods of aggregation and attention is considered in order focusing on image information. The main aim of data extraction with regards to semi-structured data is to retrieve beneficial information from the web. The data from web also known as deep web is retrievable but it requires request through form submission because it cannot be performed by any search engines. As the HTML documents start to grow larger, it has been found that the process of data extraction has been plagued with lengthy processing time. In this research work, we propose an improved model namely wrapper extraction of image using document object model (DOM) and JavaScript object notation data (JSON) (WEIDJ) in response to the promising results of mining in a higher volume of image from a various type of format. To observe the efficiency of WEIDJ, we compare the performance of data extraction by different level of page extraction with VIBS, MDR, DEPTA and VIDE. It has yielded the best results in Precision with 100, Recall with 97.93103 and F-measure with 98.9547.

A Data Extraction Algorithm from Open Source Software Project Repositories for Building Duration Estimation Models: Case Study of Github

Software project estimation is important for allocating resources and planning a reasonable work schedule. Estimation models are typically built using data from completed projects. While organizations have their historical data repositories, it is difficult to obtaintheir collaboration due to privacy and competitive concerns. To overcome the issue of public access to private data repositories this study proposes an algorithm to extract sufficient data from the GitHub repository for building duration estimation models. More specifically, this study extracts and analyses historical data on WordPress projects to estimate OSS project duration using commits as an independent variable as well as an improved classification of contributors based on the number of active days for each contributor within a release period. The results indicate that duration estimation models using data from OSS repositories perform well and partially solves the problem of lack of data encountered in empirical research in software engineering.

Is there evidence that periodontal diseases are risk factors for coronary arterial disease?-Scoping review.

To determine the scientific evidence regarding the condition of periodontal diseases (PDs) as risk factors of coronary artery diseases. A scoping review was performed through the selection of articles from online databases (Medline via PubMed, LILACS, and Cochrane CENTRAL), using a search algorithm and eligibility criteria for data extraction and data synthesis of included papers. Four hundred forty-one studies from online databases and 1364 from reference lists were initially identified, and 34 articles were selected for inclusion. Articles had to be published between 2015 and 2020 and had to associate PDs to coronary arterial diseases. All studies comprised a sample of 446443 participants. Favorable opinions totaled 70.59%, and the cross-sectional observational study design was more frequently found (32.35%). Biochemical outcomes were showed more frequently, such as inflammatory biomarkers (58.82%). Despite a large advantage for favorable opinions, a statistically significant result was not found when extracted data were combined with studies opinions. The evidence was classified as level II-3, hence, still distant from the best ranks and can therefore be considered as unclear. Moreover, studies with the higher potential to generate evidence showed a high risk of bias.

A parameterized automatic programming solution for composite grinding based on digital image processing

In this paper, a parameterized automatic programming solution whose advantage depends on the automatic feature recognition of digital image is proposed and applied to the development of automatic programming software for composite grinding. This solution can overcome the difficulty of recognizing intersecting feature of complicated rotational part. And a 7-layer CNN classifier is utilized to decide whether the part has the internal features or not, which makes the proposed feature recognition method more intelligent than retrieving the unknown objects blindly. The emphasis of the research is the geometric data extraction algorithm which is the synthesis of border following algorithm, corner detection algorithm and a variety of morphological processing. Under the condition of 12000 × 12000 pixel dimension and 200-dpi resolution of input image, the relative errors between the extracted and actual values of various geometric data are all less than 0.05% for the rotational parts of maximum diameter 500 mm and maximum length of 1500 mm. And all the extracted values of geometric data rounded to integers can fully meet the requirements of NC programming. The automatic programming software based on the proposed solution has excellent portability and practicability, which is independent of any CAD tools or data exchange standards. After the programs generated by the automatic programming software are validated in simulation software NCSIMUL, the software is integrated into HNC-848 CNC system and applied in the prototype of H377 composite grinding center.

Hot Embossing Process Technology Forming Arbitrary Patterns in Real Time

AbstractWith the development of mobile sensors and biosensors, an ultrafine process technology that can generate ultrafine patterns and has mass production capability has garnered attention recently. Unlike the existing hot embossing techniques, the impact print‐type hot embossing process can generate a microscale pattern that has arbitrary pattern width and depth on a polymer film in real time using a single impact header. For this reason, this process can be applied to various industrial fields that require ultrafine patterns. However, it cannot create intricate patterns due to the limitations of the two‐axis stage control system. Furthermore, its application to a mass production that requires pattern repetition is limited because of the single header system. In this paper, a three‐axis impact print‐type hot embossing system using a multiheader and two algorithms, the impacting coordinate‐based data extraction algorithm and independent driving algorithm, to overcome the limitations of previous systems is proposed. Thanks to the multiheader and control algorithm, this process can generate repetitive patterns in real time, and it can adjust the pattern resolution by controlling the pattern extraction interval. The proposed method is thus expected to be widely applicable to industrial fields that require microscale user‐defined patterns such as mobile sensors and biosensors.

Data extraction methods for systematic review (semi)automation: A living review protocol.

Background: Researchers in evidence-based medicine cannot keep up with the amounts of both old and newly published primary research articles. Conducting and updating of systematic reviews is time-consuming. In practice, data extraction is one of the most complex tasks in this process. Exponential improvements in computational processing speed and data storage are fostering the development of data extraction models and algorithms. This, in combination with quicker pathways to publication, led to a large landscape of tools and methods for data extraction tasks. Objective: To review published methods and tools for data extraction to (semi)automate the systematic reviewing process. Methods: We propose to conduct a living review. With this methodology we aim to do monthly search updates, as well as bi-annual review updates if new evidence permits it. In a cross-sectional analysis we will extract methodological characteristics and assess the quality of reporting in our included papers. Conclusions: We aim to increase transparency in the reporting and assessment of machine learning technologies to the benefit of data scientists, systematic reviewers and funders of health research. This living review will help to reduce duplicate efforts by data scientists who develop data extraction methods. It will also serve to inform systematic reviewers about possibilities to support their data extraction.

A Scalable Smartwatch-Based Medication Intake Detection System Using Distributed Machine Learning.

Poor Medication adherence causes significant economic impact resulting in hospital readmission, hospital visits and other healthcare costs. The authors developed a smartwatch application and a cloud based data pipeline for developing a user-friendly medication intake monitoring system that can contribute to improving medication adherence. The developed Android smartwatch application collects activity sensor data using accelerometer and gyroscope. The cloud-based data pipeline includes distributed data storage, distributed database management system and distributed computing frameworks in order to build a machine learning model which identifies activity types using sensor data. With the proposed sensor data extraction, preprocessing and machine learning algorithms, this study successfully achieved a high F1 score of 0.977 with 13.313 seconds of training time and 0.139 seconds for testing.

Analysis of Weka Data Mining Techniques for Heart Disease Prediction System

Data mining is characterized as searching for useful information through very large data sets. Some of the key and most common techniques for data mining are association rules, classification, clustering, prediction, and sequential models. For a wide range of applications, data mining techniques are used. Data mining plays a significant role in disease detection in the health care industry. The patient should be needed to detect a number of tests for the disease. However, the number of tests should be reduced by using data mining techniques. In time and performance, this reduced test plays an important role. Heart disease is a cardiovascular disease that causes death. Health problems are enormous in this recent situation because of the prediction and the classification of health problems in different situations. The data mining area included the prediction and identification of abnormality and its risk rate in these domains. Today the health industry holds hidden information essential for decision-making. For predicting heart problems, data extraction algorithms like K-star, J48, SMO, Naive Bayes, MLP, Random Forest, Bayes Net, and REPTREE are used for this study (Weka 3.8.3) software. The results of the predictive accuracy, the ROC curve, and the AUC value are combined using a standard set of data and a collected dataset. By applying different data mining algorithms, the patient data can be used for diagnosis as training samples. The main drawbacks of the previous studies are that they need accurate and the number of features. This paper surveys recent data mining techniques applied to predict heart diseases. And Identifying the major risk factors of Heart Disease categorizing the risk factors in an order which causes damages to the heart such as high blood cholesterol, diabetes, smoking, poor diet, obesity, hypertension, stress, etc. Data mining functions and techniques are used to identify the level of risk factors to help the patients in taking precautions in advance to save their life.

Post Classification in the Social Networks using the Map-reduce Algorithm

Wrongdoing is increasing through social media. Detecting them requires highlighting the most interesting topics in the posts. This essential part in the characterization of social network users could be done by a classification of posts. For this, we use a tuple of keywords and the Map-reduce algorithm for data collection and extraction. The main purpose is to achieve success on software realization which will establish a network between social networks to extract data and to speed up the classification of posts. The proposed method consists of verifying a sequence of keywords in the posts, following a grammar in order to determine classes. It allows the categorization of posts and monitoring of social networks. The categorization facilitates research of a particular post containing specific words. Thus, we contribute to increase capacity for wrongdoing prevention and strengthening cyber-security.

Improving the performance of precision poverty alleviation based on big data mining and machine learning

Since the reform began in our country, with the rapid economic growth in recent years, the income level has grown extremely unequal, and it is difficult for the low-income poor to benefit from the rapid economic growth. The most important prerequisite for the fight against poverty is the accurate identification of the causes of poverty. To date, our country has not reached the level of maturity required to accurately study the causes of poverty in various households. However, with the rapid development of Internet technology and big data technology in recent years, the application of large-scale data technology and data extraction algorithms to poverty reduction can identify truly poor households faster and more accurately. Compared with traditional machine learning algorithms, there are no machine storage and technical constraints, can use a large amount of data and rely on multiple data samples.