Data Transformation Process Research Articles

Integrating real-world data to assess cardiac ablation device outcomes in a multicenter study using the OMOP common data model for regulatory decisions: implementation and evaluation.

The objective of this study is to describe application of the Observational Medical Outcomes Partnership (OMOP) common data model (CDM) to support medical device real-world evaluation in a National Evaluation System for health Technology Coordinating Center (NESTcc) Test-Case involving 2 healthcare systems, Mercy Health and Mayo Clinic. CDM implementation was coordinated across 2 healthcare systems with multiple hospitals to aggregate both medical device data from supply chain databases and patient outcomes and covariates from electronic health record data. Several data quality assurance (QA) analyses were implemented on the OMOP CDM to validate the data extraction, transformation, and load (ETL) process. OMOP CDM-based data of relevant patient encounters were successfully established to support studies for FDA regulatory submissions. QA analyses verified that the data transformation was robust between data sources and OMOP CDM. Our efforts provided useful insights in real-world data integration using OMOP CDM for medical device evaluation coordinated across multiple healthcare systems.

Artificial Intelligence Model for the Identification of the Personality of Twitter Users through the Analysis of Their Behavior in the Social Network

Currently, social networks have become one of the most used channels by society to share their ideas, their status, generate trends, etc. By applying artificial intelligence techniques and sentiment analysis to the large volume of data found in social networks, it is possible to predict the personality of people. In this work, the development of a data analysis model with machine learning algorithms with the ability to predict the personality of a user based on their activity on Twitter is proposed. To do this, a data collection and transformation process is carried out to be analyzed with sentiment analysis techniques and the linguistic analysis of tweets. Very successful results were obtained by developing a training process for the machine learning algorithm. By generating comparisons of this model, with the related literature, it is shown that social networks today house a large volume of data that contains significant value if your approach is appropriate. Through the analysis of tweets, retweets, and other factors, there is the possibility of creating a virtual profile on the Internet for each person; the uses can vary, from creating marketing campaigns to optimizing recruitment processes.

SPDPOA: Student Psychology Dragonfly Political Optimizer Algorithm-Based Soil Moisture and Heat-Level Prediction for Plant Health Monitoring in Internet of Things

Abstract This article devised an effective Student Psychology-based Dragonfly Political Optimizer (SPDPOA) for predicting heat level and soil moisture to monitor plant health in the Internet of Things (IoT). The developed SPDPOA is modeled by integrating the Student Psychology-based Optimization (SPBO) algorithm, Dragonfly Algorithm (DA) and Political optimizer (PO), respectively. The prediction process is done in the base station (BS), which gathers the IoT nodes’ information through optimal Cluster Head (CH) using Deep Recurrent Neural Network (Deep RNN). Moreover, the CH selection and routing process are established using a developed SPDPOA scheme. The data transformation and feature selection processes are done based on Box-Cox transformation and wrapper model, correspondingly, which helps in the selection of best features. Moreover, the developed SPDPOA scheme attained better performance in Packet Delivery Ratio (PDR), energy and testing accuracy of 0.7232, 0.6342 J and 0.9372, respectively.

A Survey on Symmetrical Neural Network Architectures and Applications

A number of modern techniques for neural network training and recognition enhancement are based on their structures’ symmetry. Such approaches demonstrate impressive results, both for recognition practice, and for understanding of data transformation processes in various feature spaces. This survey examines symmetrical neural network architectures—Siamese and triplet. Among a wide range of tasks having various mathematical formulation areas, especially effective applications of symmetrical neural network architectures are revealed. We systematize and compare different architectures of symmetrical neural networks, identify genetic relationships between significant studies of different authors’ groups, and discuss opportunities to improve the element base of such neural networks. Our survey builds bridges between a large number of isolated studies with significant practical results in the considered area of knowledge, so that the presented survey acquires additional relevance.

Development and validation of the electronic screening index of frailty.

primary care screening for frailty status is recommended in clinical guidelines, but is impeded by doctor and nurse workloads and the lack of valid, easy-to-use and time-saving screening tools. to develop and validate a new electronic tool (the electronic screening index of frailty, e-SIF) using routinely available electronic health data to automatically and massively identify frailty status in the population aged ≥65years. the e-SIF was developed in three steps: selection of clinical conditions; establishment of ICD-10 codes, criteria and algorithms for their definition; and electronic tool design and data extraction, transformation and load processes. The validation phase included an observational cohort study with retrospective data collection from computerised primary care medical records. The study population included inhabitants aged ≥65years corresponding to three primary care centres (n = 9,315). Evaluated was the relationship between baseline e-SIF categories and mortality, institutionalisation, hospitalisation and health resource consumption after 2 years. according to the e-SIF, which includes 42 clinical conditions, frailty prevalence increases with age and is slightly greater in women. The 2-year adjusted hazard ratios for pre-frail, frail and very frail subjects, respectively, were as follows: 2.23 (95% CI: 1.74-2.85), 3.34 (2.44-4.56) and 6.49 (4.30-9.78) for mortality; 2.80 (2.39-3.27), 5.53 (4.59-6.65) and 9.14 (7.06-11.8) for hospitalisation; and 1.02 (0.70-1.49), 1.93 (1.21-3.08) and 2.69 (1.34-5.40) for institutionalisation. the e-SIF shows good agreement with mortality, institutionalisation, hospitalisation and health resource consumption, indicating satisfactory validity. More studies in larger populations are needed to corroborate our findings.

Penyalahgunaan serta proses Penyelesaian Visa Kunjungan oleh Warga Negara Asing

This research is related to the process of resolving the misuse of visas carried out by foreign nationals in Indonesia. The purpose of this study is to find out the Process for the Settlement of Misuse of Visit Visas Performed by Indonesian Citizens in the implementation of the Settlement of Misuse of Visit Visas that have been carried out by Foreign Citizens, hereby implementing and providing the right impact on Foreign Citizens. The method of collecting data is directly conducting interviews with informants related to the completion process of Visit Visa Abuse. The method used is using empirical research methods with the transformation process of research data sourced from directly related informants. In this case, there is the implementation of the Visit Visa Abuse Settlement Process carried out by Foreign Citizens, there are still several obstacles and shortcomings, both in terms of supervision and the role of Immigration office employees in Indonesia.

An Online Estimation Method of State of Health for Lithium-Ion Batteries Based on Constant Current Charging Curve

Accurate estimation of state of health (SOH) is of great significance for the safety and reliability of lithium-ion batteries. In this paper, a novel method to estimate SOH online based on constant current charging curve is presented. In order to incorporate the factor of rates, a simple two-step data transformation process is carried out to make the method suitable for SOH estimation at different charging rates. Then polynomial is used to fit the transformed curve, and the coefficient sets of analytic expression obtained by fitting are taken as the battery aging feature variables. Finally, linear regression algorithm, the simplest machine learning algorithm, is employed to construct the mapping between feature variables and SOH, thus accomplishing the SOH estimation. When estimating SOH, only the charging curve of the whole constant current charging process is needed, regardless of the charging process at whatever rates. This method takes low computational cost, making it suitable for online estimation. The verification results on battery test data show that the method is of high accuracy and effectiveness.

Five common pitfalls in mixed methods systematic reviews: lessons learned

ObjectivesMixed methods systematic reviews (MMSRs) combine quantitative and qualitative evidence within a single review. Since the revision of the JBI methodology for MMSRs in 2020, there has been an increasing number of reviews published that claim to follow this approach. A preliminary examination of these indicated that authors frequently deviated from the methodology. This article outlines five common ‘pitfalls’ associated with undertaking MMSR and provides direction for future reviewers attempting MMSR. MethodsForward citation tracking identified 17 reviews published since the revision of the JBI mixed methods methodological guidance. Methods used in these reviews were then examined against the JBI methodology to identify deviations. ResultsThe issues identified related to the rationale for choosing the methodological approach, an incorrect synthesis and integration approach chosen to answer the review question/s posed, the exclusion of primary mixed methods studies in the review, the lack of detail regarding the process of data transformation, and a lack of ‘mixing’ of the quantitative and qualitative components. ConclusionThis exercise was undertaken to assist systematic reviewers considering conducting an MMSR and MMSR users to identify potential areas where authors tend to deviate from the methodological approach. Based on these findings a series of recommendations are provided.

From Customer Data to Smart Customer Data: The Smart Data Transformation Process

Nowadays, smart data has emerged as a new trend in creating more business value for enterprises that is defined as the data that is gathered and processed to create new insights to support business decisions. However, the transformation from data into actionable insights is still a real challenge for enterprises. For this reason, this paper presents a smart data transformation process, which aims at transforming customer data into smart customer data in order to offer actionable insights. The purpose of the study is to propose a transformation process that can be used to operate a knowledge structure for a smart service system, which can manage and deliver smart data as a service. The process covers the three dimensions of a service system: Data processing corresponding to the engineering dimension, information processing corresponding to the science dimension, and knowledge processing corresponding to the management dimension for knowledge processing. Accordingly, a case study on the smart data transformation process of a customer journey management system as a smart service system is presented to demonstrate the application of the proposed process.

An Efficient Schema Transformation Technique for Data Migration from Relational to Column-Oriented Databases

Data transformation is the core process in migrating database from relational database to NoSQL database such as column-oriented database. However, there is no standard guideline for data transformation from relational database to NoSQL database. A number of schema transformation techniques have been proposed to improve data transformation process and resulted better query processing time when compared to the relational database query processing time. However, these approaches produced redundant tables in the resulted schema that in turn consume large unnecessary storage size and produce high query processing time due to the generated schema with redundant column families in the transformed column-oriented database. In this paper, an efficient data transformation technique from relational database to column-oriented database is proposed. The proposed schema transformation technique is based on the combination of denormalization approach, data access pattern and multiple-nested schema. In order to validate the proposed work, the proposed technique is implemented by transforming data from MySQL database to MongoDB database. A benchmark transformation technique is also performed in which the query processing time and the storage size are compared. Based on the experimental results, the proposed transformation technique showed significant improvement in terms query processing time and storage space usage due to the reduced number of column families in the column-oriented database.

A data transformation process for using Benford’s Law with bounded data

Benford’s Law is an empirical observation about the frequency of digits in a variety of naturally occurring data sets. Auditors and forensic scientists have used Benford’s Law to detect erroneous data in accounting and legal usage. One well-known limitation is that Benford’s Law fails when data have clear minimum and maximum values. Many kinds of education data, including assessment scores, typically include hard maximums and therefore do not meet the parametric assumptions of Benford’s Law. This paper implements a transformation procedure which allows for assessment data to be compared to Benford’s Law. As a case study, a data quality assessment of oral language scores from the Early Childhood Longitudinal Study, Kindergarten (ECLS-K) study is used and higher risk data segments detected. The same method could be used to evaluate other concerns, such as test fraud, or other bounded datasets.

A data transformation process for using Benford's Law with bounded data

Benford's Law is an empirical observation about the frequency of digits in a variety of naturally occurring data sets. Auditors and forensic scientists have used Benford's Law to detect erroneous data in accounting and legal usage. One well-known limitation is that Benford's Law fails when data have clear minimum and maximum values. Many kinds of education data, including assessment scores, typically include hard maximums and therefore do not meet the parametric assumptions of Benford's Law. This paper implements a transformation procedure which allows for assessment data to be compared to Benford's Law. As a case study, a data quality assessment of oral language scores from the Early Childhood Longitudinal Study, Kindergarten (ECLS-K) study is used and higher risk data segments detected. The same method could be used to evaluate other concerns, such as test fraud, or other bounded datasets.

Making EHRs Reusable: A Common Framework of Data Operations.

Reuse of EHRs requires data extraction and transformation processes are based on homogeneous and formalized operations in order to make them understandable, reproducible and auditable. This work aims to define a common framework of data operations for obtaining EHR-derived datasets for secondary use. Thus, 21 operations were identified from different data-driven projects of a 1,300-beds tertiary Hospital. Then, ISO 13606 standard was used to formalize them. This work is the starting point to homogenize ETL processes for the reuse of EHRs, applicable to any condition and organization. In future studies, defined data operations will be implemented and validated in projects of different purposes.

Is South Africa closing the health gaps between districts? Monitoring progress towards universal health service coverage with routine facility data

BackgroundSouth Africa is committed to advancing universal health coverage (UHC). The usefulness and potential of using routine health facility data for monitoring progress towards UHC, in the form of the 16-tracer WHO service coverage index (SCI), was assessed.MethodsAlternative approaches to calculating the WHO SCI from routine data, allowing for disaggregation to district level, were explored. Data extraction, coding, transformation and modelling processes were applied to generate time series for these alternatives. Equity was assessed using socio-economic quintiles by district.ResultsThe UHC SCI at a national level was 46.1 in 2007–2008 and 56.9 in 2016–2017. Only for the latter period, could the index be calculated for all indicators at a district level. Alternative indicators were formulated for 9 of 16 tracers in the index. Routine or repeated survey data could be used for 14 tracers. Apart from the NCD indicators, a gradient of poorer performance in the most deprived districts was evident in 2016–2017.ConclusionsIt is possible to construct the UHC SCI for South Africa from predominantly routine data sources. Overall, there is evidence from district level data of a trend towards reduced inequity in relation to specific categories (notably RMNCH). Progress towards UHC has the potential to overcome fragmentation and enable harmonisation and interoperability of information systems. Private sector reporting of data into routine information systems should be encouraged.

Management of the Documentation Release by the Programmable Electrical Energy Flow-Individually Made Machine Called Documentomat

It is well known that in a turbulent world (full of processed information), information has a particularly high value. Wide access to technology makes the data transformation and transmission processes faster and more vulnerable to leaks and unauthorized access. This publication discusses the concept of information security management based on a development of a machine called documentomat. It is the machine built to automatically release documents with regard to high security standards. The entire flow of the documents in the company has been presented and the formalized process of decision-making to develop and finally build above-mentioned machine was described. The problems and doubts raised during the machine building process were presented along with the solutions adopted. The device has been designed, constructed, and programmed within the resources of one micro-sized limited liability company in Europe. Thus, it is the solution working in a real world that may find application in many similar companies, not the theoretical concept only. The final result exceeded the expectations of the project team, leading to strengthening of the competitive position of the company. This occurred through the following aspects: adding significant advantage to its offer, which is the ability to release the documents at any time with regard to high security standards. Moreover, the long-term level of costs of such a solution seems to be very low. During the one year of operation period, two malfunctions took place, each was repaired in just 3 h (repair and improvement, labor only, no spare parts needed). Any further maintenance, repair, or development can be done at very affordable prices, at non-available on the market labor cost, which will potentially further strengthen the impact of the solution on the competitiveness of the company. Improvement in terms of competitiveness of the company and data safety has been gained. The success of this project proves that, in some cases, it is worth to develop the necessary solution within the resources of the company rather than buying one from an external enterprise.

Implementasi Extract Transform Load Pada Data Akademik Perguruan Tinggi

Each department at university needs to analyze student data and scores. By analyzing student grades, majors can improve lecturer performance in learning; one way to explore student grades is to collect student grade data and then process the data to produce information in the form of learning performance results that can be used to make decisions to improve lecturer performance. The author will analyze a structured programming course student score data; the following collected data are related to student attendance, assignment scores, mid-semester exam scores, and final semester exam scores. After all the data needed has been organized to build the data warehouse, the following process is the ETL process (Extract, Transform, and Load), as the tool used in the ETL process is Pentaho. This data extraction process using 1 data source, namely student value data with excel file type. After carrying out the extraction process, there is a data transformation process, namely making some changes to the extracted data to make it more consistent and uniform according to the needs of the data warehouse.

Three Metric-Based Method for Data Compatibility Calculation

This article analyzes ways of calculating characteristics of data and most common data structure types that allow comparison between them or on a time axis. To achieve this, it studies the key aspects of relational databases, XML, JSON and RDF structure types. These data structure types are compared to multiple isolated approaches to data quality and other data characteristics measurements. The goals of the article are the calculation method itself and a storage structure for calculated values. The article presents a method of characterization of data and data structure types based on the calculation of three metrics: the amount of structuredness, the amount of hierarchicallity and the amount of information. This triad of metrics allows comparison between various data sets (objects), for example evaluating the complexity of the transformation of data from one data object to another, as well as with data structure types (as mentioned above). Based on the vector of three metrics, the calculation method of the compatibility between data and data structure type is proposed. This method can help select the most compatible data format for existing data. The calculated values of metrics can also detect non-optimal storage design and classify data transformations. The method was evaluated on an example case study, which showed its usability on an example demonstration data set. It can be used in the process of data modelling to help select optimal data structure type, to design a data transformation process and to optimize existing data storages.

From Words to Numbers: Getting Started with Text Analysis for Applied Social Scientists

Objectives: With texts as unstructured data everywhere, text analysis or natural language processing (NLP) is a rapidly growing academic field that has great potential for novel research among many applied social scientists and practitioners. This paper presents a practical introduction to NLP using Python as a useful tool for text analysis. Methods: Starting with installation of Python and an external library for NLP, this paper describes a step-by-step process of data preparation, transformation, and summarization for text data using examples. The example texts were obtained from a transcribed business meeting record of a multinational company based in Helsinki. Results: From the initial unstructured text data having numerous irrelevant elements, the data preparation procedures of tokenization, removing stop words, stemming, and lemmatization result in a set of words useful for main analyses. The next step of transforming the words to numbers was conducted using a bag-of-words method by assigning a unique value to each word in a matrix. As the last step, the matrix is computed for frequency summarization using TF-IDF (Term Frequency and Inverse Document Frequency). Conclusions: Unlike structured data, many unstructured text data are not generated for the purpose of data analysis. With numeric data reproduced by the process presented in this paper, communication researchers can perform various statistical methods or use machine learning algorithms. Beyond the scope of this paper, it is strongly recommended to study statistics and computational linguistics as well as have a working knowledge on R and/or Python for advanced text analysis.

Data Reduction in the String Space for Efficient kNN Classification Through Space Partitioning

Within the Pattern Recognition field, two representations are generally considered for encoding the data: statistical codifications, which describe elements as feature vectors, and structural representations, which encode elements as high-level symbolic data structures such as strings, trees or graphs. While the vast majority of classifiers are capable of addressing statistical spaces, only some particular methods are suitable for structural representations. The kNN classifier constitutes one of the scarce examples of algorithms capable of tackling both statistical and structural spaces. This method is based on the computation of the dissimilarity between all the samples of the set, which is the main reason for its high versatility, but in turn, for its low efficiency as well. Prototype Generation is one of the possibilities for palliating this issue. These mechanisms generate a reduced version of the initial dataset by performing data transformation and aggregation processes on the initial collection. Nevertheless, these generation processes are quite dependent on the data representation considered, being not generally well defined for structural data. In this work we present the adaptation of the generation-based reduction algorithm Reduction through Homogeneous Clusters to the case of string data. This algorithm performs the reduction by partitioning the space into class-homogeneous clusters for then generating a representative prototype as the median value of each group. Thus, the main issue to tackle is the retrieval of the median element of a set of strings. Our comprehensive experimentation comparatively assesses the performance of this algorithm in both the statistical and the string-based spaces. Results prove the relevance of our approach by showing a competitive compromise between classification rate and data reduction.

A hybrid model considering cointegration for interval‐valued pork price forecasting in China

Compared with point forecasting, interval forecasting is believed to be more effective and helpful in decision making, as it provides more information about the data generation process. Based on the well‐established “linear and nonlinear” modeling framework, a hybrid model is proposed by coupling the vector error correction model (VECM) with artificial intelligence models which consider the cointegration relationship between the lower and upper bounds (Coin‐AIs). VECM is first employed to fit the original time series with the residual error series modeled by Coin‐AIs. Using pork price as a research sample, the empirical results statistically confirm the superiority of the proposed VECM‐CoinAIs over other competing models, which include six single models and six hybrid models. This result suggests that considering the cointegration relationship is a workable direction for improving the forecast performance of the interval‐valued time series. Moreover, with a reasonable data transformation process, interval forecasting is proven to be more accurate than point forecasting.