Data Preparation Process Research Articles

AI-Driven Data Preparation: The Key to Unlocking Cloud-Based Analytics

The rapid adoption of cloud-based analytics has revolutionized data-driven decision-making across industries. Cloud- based analytics has transformed how businesses make decisions by leveraging vast amounts of data. However, preparing data for analysis—such as cleaning, transforming, and organizing it—can be a complicated and time- consuming process. AI-driven data preparation (AIDP) is a solution that automates these steps, reducing the time and effort needed to prepare data while improving its quality. This paper explains the importance of AI-driven data preparation, discusses how it works, and shows how businesses can benefit from using AI in their data preparation process for cloud analytics. The use of diagrams, flowcharts, and pseudocode helps explain these concepts in a simplified yet technical manner. Keywords AI-driven data preparation, cloud-based analytics, data pipeline, automation, machine learning, data wrangling, ETL, data transformation, big data

Data-Driven Approaches for Predicting Catalyst Performance in CO2 Hydrogenation

This paper explores the critical role of data collection and preparation in leveraging machine learning techniques for predicting catalyst performance in CO2 hydrogenation processes. As the global community seeks sustainable solutions to mitigate carbon emissions, understanding the efficiency of catalysts in converting CO2 into valuable chemicals becomes increasingly important. The study discusses various types of data utilized in this context, including both experimental and simulation data, while highlighting their significance in comprehensively understanding key factors such as reaction rates, selectivity, and catalyst stability. For instance, the ability of a catalyst to selectively produce desired products over others can significantly impact the overall economic viability of CO2 hydrogenation processes. Furthermore, stability data sheds light on the longevity and durability of catalysts, revealing insights into deactivation mechanisms that can occur due to factors like sintering, poisoning, or leaching of active sites. On the other hand, simulation data generated from advanced computational methods such as density functional theory (DFT) and molecular dynamics (MD) provides a deeper understanding of the electronic and structural properties of catalysts. These computational techniques allow researchers to predict reaction pathways, activation energies, and the behavior of intermediate species, thereby complementing experimental findings and guiding the design of more effective catalysts. The paper also emphasizes the importance of utilizing publicly available databases and collaborative research datasets, which serve to enhance data accessibility and foster scientific collaboration among researchers in the field. By pooling resources and sharing findings, the scientific community can accelerate the discovery and optimization of novel catalysts. Additionally, the study outlines essential steps in the data preparation process, including rigorous data cleaning, preprocessing, and feature selection, all of which are crucial for ensuring the quality and reliability of the data used in machine learning models. The paper discusses the implementation of cross-validation techniques and performance metrics that help evaluate and validate model predictions, ensuring that the developed models generalize well to unseen data.

A scalable and transparent data pipeline for AI-enabled health data ecosystems.

Transparency and traceability are essential for establishing trustworthy artificial intelligence (AI). The lack of transparency in the data preparation process is a significant obstacle in developing reliable AI systems which can lead to issues related to reproducibility, debugging AI models, bias and fairness, and compliance and regulation. We introduce a formal data preparation pipeline specification to improve upon the manual and error-prone data extraction processes used in AI and data analytics applications, with a focus on traceability. We propose a declarative language to define the extraction of AI-ready datasets from health data adhering to a common data model, particularly those conforming to HL7 Fast Healthcare Interoperability Resources (FHIR). We utilize the FHIR profiling to develop a common data model tailored to an AI use case to enable the explicit declaration of the needed information such as phenotype and AI feature definitions. In our pipeline model, we convert complex, high-dimensional electronic health records data represented with irregular time series sampling to a flat structure by defining a target population, feature groups and final datasets. Our design considers the requirements of various AI use cases from different projects which lead to implementation of many feature types exhibiting intricate temporal relations. We implement a scalable and high-performant feature repository to execute the data preparation pipeline definitions. This software not only ensures reliable, fault-tolerant distributed processing to produce AI-ready datasets and their metadata including many statistics alongside, but also serve as a pluggable component of a decision support application based on a trained AI model during online prediction to automatically prepare feature values of individual entities. We deployed and tested the proposed methodology and the implementation in three different research projects. We present the developed FHIR profiles as a common data model, feature group definitions and feature definitions within a data preparation pipeline while training an AI model for "predicting complications after cardiac surgeries". Through the implementation across various pilot use cases, it has been demonstrated that our framework possesses the necessary breadth and flexibility to define a diverse array of features, each tailored to specific temporal and contextual criteria.

Mathematical Model of Logistic Regression for Binary Classification. Part 2. Data Preparation, Learning and Testing Processes

This article reviews the theoretical aspects of logistic regression for binary data classification, including data preparation processes, training, testing, and model evaluation metrics. Requirements for input data sets are formulated, methods of coding categorical data are described, methods of scaling input features are defined and substantiated. A scheme for learning logistic regression using the gradient descent method has been developed to minimize the loss function by the appropriate adjustment of the weights of the features of the sample of objects intended for classification. Features of the construction of recurrent methods of classical and stochastic gradient descent are determined. The requirements for the organization of the data sample for the multi-stage learning model in order to avoid overtraining or undertraining of logistic regression are described. The scheme of testing the trained logistic regression is given and the main quality metrics of binary classification are described. The influence of the height of the classification threshold on the efficiency of logistic regression was noted. According to the results of the work, the directions of perspective research of logistic regression are outlined.

Multi-layer cutting path planning for composite enclosed cavity in additive and subtractive hybrid manufacturing

Additive and subtractive hybrid manufacturing (ASHM) refers to the hybrid manufacturing process where in-situ subtractive machining (SM) is introduced during additive manufacturing (AM). Its process characteristics dictate the necessity of planning multi-layer cutting paths in ASHM. Currently, the slice-based planning method cannot plan multi-axis cutting paths, and the machining accuracy is difficult to directly control. Meanwhile, the manual layering planning method is inefficient when dealing with complex models. Consequently, this paper presents an innovative automatic planning method for multi-layer, multi-axis, interference-free cutting paths with controllable precision in ASHM of composite enclosed cavity parts. To enhance the ASHM efficiency, criteria for the recognition of hybrid machining features (HMFs) have been defined to identify HMFs within the model. The identification of interference planes during cavity conversion has been achieved, and these interference planes are then utilized as the conversion planes for the ASHM process. Furthermore, a boundary-guided method is employed to automatically plan the overall cutting path for HMFs. According to the G-code standard, the overall cutting paths are then output to the corresponding cutting path file within the height interval of the conversion planes. Through practical machining, it has been demonstrated that the proposed method can significantly enhance the efficiency and automation of the data preparation process in ASHM, while also improving the surface quality and dimensional accuracy of the AM part.

Data preparation for Deep Learning based Code Smell Detection: A systematic literature review

Code Smell Detection (CSD) plays a crucial role in improving software quality and maintainability. And Deep Learning (DL) techniques have emerged as a promising approach for CSD due to their superior performance. However, the effectiveness of DL-based CSD methods heavily relies on the quality of the training data. Despite its importance, little attention has been paid to analyzing the data preparation process. This systematic literature review analyzes the data preparation techniques used in DL-based CSD methods. We identify 36 relevant papers published by December 2023 and provide a thorough analysis of the critical considerations in constructing CSD datasets, including data requirements, collection, labeling, and cleaning. We also summarize seven primary challenges and corresponding solutions in the literature. Finally, we offer actionable recommendations for preparing and accessing high-quality CSD data, emphasizing the importance of data diversity, standardization, and accessibility. This survey provides valuable insights for researchers and practitioners to harness the full potential of DL techniques in CSD.

Table 0; documenting the steps to go from clinical database to research dataset

ObjectivesData-driven decision support tools have been increasingly recognized to transform health care. However, such tools are often developed on predefined research datasets without adequate knowledge of the origin of this data and how it was selected. How a dataset is extracted from a clinical database can profoundly impact the validity, interpretability and interoperability of the dataset, and downstream analyses, yet is rarely reported. Therefore, we present a case study illustrating how a definitive patient list was extracted from a clinical source database and how this can be reported. Study Design and SettingA single-center observational study was performed at an academic hospital in the Netherlands to illustrate the impact of selecting a definitive patient list for research from a clinical source database, and the importance of documenting this process. All admissions from the critical care database admitted between January 1, 2013, and January 1, 2023, were used. ResultsAn interdisciplinary team collaborated to identify and address potential sources of data insufficiency and uncertainty. We demonstrate a stepwise data preparation process, reducing the clinical source database of 54,218 admissions to a definitive patient list of 21,553 admissions. Transparent documentation of the data preparation process improves the quality of the definitive patient list before analysis of the corresponding patient data. This study generated seven important recommendations for preparing observational health-care data for research purposes. ConclusionDocumenting data preparation is essential for understanding a research dataset originating from a clinical source database before analyzing health-care data. The findings contribute to establishing data standards and offer insights into the complexities of preparing health-care data for scientific investigation. Meticulous data preparation and documentation thereof will improve research validity and advance critical care.

Enhancing Business Sustainability Through Technology-Enabled AI: Forecasting Student Data and Comparing Prediction Models for Higher Education Institutions (HEIs)

This study aims to enhance business sustainability in the context of Higher Education Institutions (HEIs) by utilizing AI and forecasting techniques. It explores the development and comparison of prediction models, including the use of dashboard development, to support decision-making processes within HEIs. The study covers various aspects, including the background of forecasting and prediction models, the use of specific models such as the Prophet Model, Long Short-Term Memory (LSTM) Model, and Polynomial Regression Model, as well as the importance of dashboards for HEIs. The methodology section outlines the data collection and preparation process, model selection, approach, diagrams, functional and non-functional requirements, justification of tools, and libraries and models used. The implementation section delves into the system design and development of the dashboard, including the login page, homepage, forecast page, and insert data page. As for the findings, the LSTM Model has proven to be the most accurate and suitable model to be implemented for forecasting student enrolment data in this study. The dashboard's future enhancements involve adding more faculties, predictive features for resource allocation, refining the visual identity, improving user registration on the login page, and exploring better models for student enrolment predictions. Overall, the study provides valuable insights into the application of AI and forecasting techniques in HEIs, aiming to enhance business sustainability and decision-making processes. It contributes to the growing body of knowledge on the use of technology-enabled AI in higher education institutions, with a focus on forecasting student enrolment data and developing prediction models.

IMPLEMENTATION IN SСAS KIDYM OF SOFTWARE DETERMINATION OF DIMENSIONS OF UNKNOWNS IN PROBLEMS OF MECHANICS BASED ON ENERGY RELATIONS

The paper presents the results of the study of an algorithm for processing measurement units of physical and geometric quantities in the problems of mechanics implemented in the specialized computer algebra system (SCAS) KiDyM. The KiDyM software complex is used for solving problems of mechanics of engineering and scientific orientation of any complexity. For the completeness of the initial data preparation process for such tasks, the use of measurement units in the initial and resulting data is essential. The output data of KiDyM tasks have a rather slow appearance due to their ana- lytical form. To provide assistance to the user, KiDyM programs have a special diagnostic block of output data. Therefore, the implementation of the dimensions used in KiDyM data pursues the additional goal of increasing the diagnostic capabilities of the system. Dimensional formulas are imple- mented here as ordinary analytical expressions, which enables the built-in computer algebra system to form expressions of data measurement units during their computer analytical transformations in the process of solving problems of dynamics, statics and kinematics. Thus, working out the meas- urement units in the SCAS KiDyM includes reading the values of the variables with the dimensions specified in formula form the initial data, diagnos- ing the correctness of dimensionising and compliance of the dimensions with the data, storing them in instances of the "variable" and "element" classes, calculating the dimensions according to data conversion formulas to obtain calculation results. The paper demonstrates how to efficiently es- tablish dimensions of geometrical and physical quantities using energy relations between coordinates and characteristics of elements of the mechanical model. For the demonstration of the logic of the implemented algorithm, simple compact examples from educational problems of kinematics, statics, and dynamics of flat and spatial systems are provided. The article shows how KiDyM constructs the necessary equations for solving tasks, and how units of measurement for equation components and their solutions can be obtained. Since dimensions are represented as formulas here, their simplifica- tion – reduction and substitution with derivative units – can be easily implemented. Moreover, since the variables involved in dimension formulas and the variables of the task itself, within KiDyM, are situated in different data spaces, they can have the same names without any confusion.

Application of Data Mining Combined with K-means Clustering Algorithm in Enterprises' Risk Audit

The financial risk management mechanism of enterprises can be more complete through exploration in the application effect of data mining technology combined with K-means clustering algorithm in enterprise risk audit. Hence, K-means clustering algorithm is introduced to study the paperless status of electronic payment in the trading process of e-commerce enterprises. Additionally, a risk audit model of e-commerce enterprises is implemented based on K-means algorithm combined with Random Forest Light Gradient Boosting Machine (RF-LightGBM). In this model, the actual operation process of data preparation, data preprocessing, model construction, model application and evaluation are implemented to study the payment flow in the transaction process of e-commerce enterprises by using big data analysis technology. Eventually, the performance of the model is evaluated by simulation. The results show that, compared with the models and algorithms proposed by scholars in other related fields, the classification accuracy of the model proposed here reaches 95.46 %. Simultaneously, the data message delivery rate of the model algorithm is basically stable at about 81.54 %, and the data message leakage rate, packet loss rate and average delay are lower than those of other models and algorithms. Therefore, under the premise of ensuring the prediction accuracy, the audit model of e-commerce enterprises can also achieve high data transmission security performance, which can provide experimental basis for the safety improvement and risk control of the audit process in e-commerce enterprises.

Use of superpixels for improvement of inter-rater and intra-rater reliability during annotation of medical images

In the context of automatic medical image segmentation based on statistical learning, raters’ variability of ground truth segmentations in training datasets is a widely recognized issue. Indeed, the reference information is provided by experts but bias due to their knowledge may affect the quality of the ground truth data, thus hindering creation of robust and reliable datasets employed in segmentation, classification or detection tasks. In such a framework, automatic medical image segmentation would significantly benefit from utilizing some form of presegmentation during training data preparation process, which could lower the impact of experts’ knowledge and reduce time-consuming labeling efforts. The present manuscript proposes a superpixels-driven procedure for annotating medical images. Three different superpixeling methods with two different number of superpixels were evaluated on three different medical segmentation tasks and compared with manual annotations. Within the superpixels-based annotation procedure medical experts interactively select superpixels of interest, apply manual corrections, when necessary, and then the accuracy of the annotations, the time needed to prepare them, and the number of manual corrections are assessed. In this study, it is proven that the proposed procedure reduces inter- and intra-rater variability leading to more reliable annotations datasets which, in turn, may be beneficial for the development of more robust classification or segmentation models. In addition, the proposed approach reduces time needed to prepare the annotations.

TT-HEALpix: A New Data Indexing Strategy for Efficient Cross-match of Large-scale Astronomical Catalogs

Cross-matching is an indispensable operation in the data preparation, analysis, and research processes of multi-band astronomy and time-domain astronomy. Multi-catalog time-series data reconstruction is an important part of time-domain astronomy. In the large-scale distributed reconstruction process, boundary problems have always affected the accuracy of time-series data. To optimize these boundary problems and improve data precision, this paper proposes a new hybrid astronomical data indexing method called Translated Transformation based HEALPix Dual Index (TT-HEALPix). Under the reasonable Healpix division level, by translation transformation, the two indexes before and after the transformation form a unique pseudo-hybrid index strategy, which not only retains the advantages of the hybrid index scheme suitable for large-scale parallel computing, but also compensates for its shortage of high omission at the block boundary position. Based on TT-HEALPix, this paper completes the multi-catalog time-series reconstruction process on the Spark platform and compares it with the HEALPix+HTM hybrid indexing strategy. The experiments demonstrate that TT-HEALPix has significant advantages over the traditional HEALPix+HTM hybrid indexing method in terms of data accuracy and cross-matching efficiency. At level 9 of the Healpix index, TT-HEALPix achieves a 6%–19% improvement in cross-matching efficiency in a distributed environment compared to HEALPix+HTM. In terms of data accuracy, for the AST3-II dataset at level 9, TT-HEALPix has 62.2% accuracy improvement over HEALPix and 45.5% improvement over HEALPix+HTM. In conclusion, the proposed novel indexing strategy, TT-HEALPix, is better suited to the efficiency and accuracy requirements of cross-match.

Comprehensive, Open-Source, and Automated Workflow for Multisite λ-Dynamics in Lead Optimization.

Multisite λ-dynamics (MSLD) is a highly efficient binding free energy calculation method that samples multiple ligands in a single round by assigning different λ values to the alchemical part of each ligand. This method holds great promise for lead optimization (LO) in drug discovery. However, the complex data preparation and simulation process limits its widespread application in diverse protein-ligand systems. To address this challenge, we developed a comprehensive, open-source, and automated workflow for MSLD calculations based on the BLaDE dynamics engine. This workflow incorporates the Ligand Internal and Cartesian coordinate reconstruction-based alignment algorithm (LIC-align) and an optimized maximum common substructure (MCS) search algorithm to accurately generate MSLD multiple topologies with ideal perturbation patterns. Furthermore, our workflow is highly modularized, allowing straightforward integration and extension of various simulation techniques, and is highly accessible to nonexperts. This workflow was validated by calculating the relative binding free energies of large-scale congeneric ligands, many of which have large perturbing groups. The agreement between the calculations and experiments was excellent, with an average unsigned error of 1.08 ± 0.47 kcal/mol. More than 57.1% of the ligands had an error of less than 1.0 kcal/mol, and the perturbations of 6 targets were fully connected via the calculations, while those of 2 targets were connected via both calculations and experimental data. The Pearson correlation coefficient reached 0.88, indicating that the MSLD workflow provides accurate predictions that can guide lead optimization in drug discovery. We also examined the impact of single-site versus multisite perturbations, ligand grouping by perturbing group size, and the position of the anchor atom on the MSLD performance. By integrating our proposed LIC-align and optimized MCS search algorithm along with the coping strategies to handle challenging molecular substructures, our workflow can handle many realistic scenarios more reasonably than all previously published methods. Moreover, we observed that our MSLD workflow achieved similar accuracy to free energy perturbation (FEP) while improving computational efficiency by over 1 order of magnitude in speedup. These findings provide valuable insights and strategies for further MSLD development, making MSLD a competitive tool for lead optimization.

The Application of Random Forest to the Classification of Fake News

Fake News is one of the most widespread phenomenon with significant consequences on our daily life, particularly in the political realm. Due to the increasing use of the internet and social media, it is now much simpler to propagate false information. Therefore, the identification of elusive news is a significant issue that must be addressed, mostly due to obstacles such as the limited number of benchmark datasets and the volume of news produced per second. This study suggested using comparative data analysis based on random forest machine learning algorithm to identify bogus 4news. In this study the size of the whole dataset is 20.761 fake news record, whereas the size of it is 4.345 records. The first step in the data preparation process is to remove any unnecessary special characters, numbers, English letters, and whitespace. Before implementing the proposed classification algorithms, the most prevalent feature extraction approach (TF-IDF) is used. The data indicate that the highest level of accuracy attained was 88.24%.

Exploratory Data Preparation and Model Training Process for Raspberry Pi-Based Object Detection Model Deployments

Exploratory Data Preparation and Model Training Process for Raspberry Pi-Based Object Detection Model Deployments

Data Augmentation and Preparation Process of PerInfEx: A Persian Chatbot With the Ability of Information Extraction

Data Augmentation and Preparation Process of PerInfEx: A Persian Chatbot With the Ability of Information Extraction

Optimal Integration of Data Fusion in Solar Power Analytics: Enhancing Efficiency and Accuracy

At the forefront of sustainable energy solutions lies renewable energy, particularly solar power. Nevertheless, the optimization of solar power systems necessitates comprehensive analytics, especially for proactive maintenance fault anticipation. This research evaluates data fusion techniques using both linear and non-linear regression models for predicting faults in solar power plants. The study begins with careful data preparation processes to ensure clean and harmonized data sets that include irradiation, temperature, historical fault records, and yield. Linear regression techniques provide insights into straightforward correlations while non-linear models go deep into complex relationships within the data. The results indicate positive outcomes demonstrating the potential of these fusion techniques as far as improving accuracy in fault prediction is concerned. These findings highlight the importance of refining data preparation prior to any fusion process and recommend further exploration into more advanced fusion methodologies. This paper helps advance proactive maintenance strategies for solar power plants thereby making this source of energy more dependable and resilient.

Online Game Outcome Prediction Model Using Weighted-Based Feature Approach

Recently, the popularity of online games has risen drastically due to the latest technology that can connect players globally. League of Legends (LoL) holds the title of being the most extensively played Multiplayer Online Battle Arena (MOBA) game globally. This issue compels a substantial volume of preceding research that still analyzes and predicts the game outcomes with traditional methods that can be inaccurate and imprecise. Furthermore, these methods are frequently associated with the high rates of both false positive and false negative results. Hence, this paper presents a weighted-based feature predictor model to enhance the prediction accuracy. The approach predicts the game outcome of League of Legends matches in the Latin America North (LAN) and North America (NA) regions. We utilize player mastery and win rate for each summoner as the features. The data preparation process includes a weighted algorithm calculation and then evaluation using Naïve Bayes and Support Vector Machine algorithm. The outcomes illustrate that the weight-based feature approach can predict the outcome of LoL matches with an average accuracy of over 97 percent. This approach can be a valuable technique for players, teams, and coaches to analyze their performance and make strategic decisions.

UNDERSTANDING THE NEW NIH DATA SHARING REQUIREMENTS: WHY THEY IMPROVE THE QUALITY OF RESEARCH

Abstract In 2021, NIH began a long period of planning, drafting, and requesting public comment to create a new data-sharing policy, beginning with a Request For Information (RFI). The new policy, beginning January 25, 2023, will apply broadly to all scientific data produced with NIH funding. NACDA and (ICPSR), drawing upon our expertise and understanding of best practices for data sharing, have been working actively to develop tools and resources to assist NIH grant recipients in complying with federal data-sharing expectations with minimal disruptions to their research. This presentation will provide a detailed overview of the new data-sharing requirements and how to incorporate them into the required data-sharing plan. We will offer examples of successful data-sharing plans, identifying an appropriate repository, budgeting for the data preparation process, and creating informed consent documents consistent with the spirit of the NIH Data Sharing requirements. In addition to explaining the new data-sharing policy, the presentation will discuss the rationale behind the requirements and how they contribute to data equity using the FAIR principles. By increasing access to federally funded data, we contribute to a multidisciplinary approach that enhances cross-disciplinary communication and a broad understanding of different ways of knowing when working collaboratively with other researchers. Team Science has facilitated substantive discourse between qualitative and quantitative specialists. Multidisciplinary research increasingly represents a critical aspect of overall knowledge creation. Finally, the presentation will illustrate how data-sharing increases the impact of studies and enhances the credentials of researchers who engage in data-sharing.

Dual-discriminator conditional generative adversarial network optimized with hybrid manta ray foraging optimization and volcano eruption algorithm for hyperspectral anomaly detection

A significant task on remote sensing field is anomaly detection in hyperspectral images (HSI). Many scientists have created several relevant techniques developing deep network-based ways for hyperspectral anomaly detection (HAD) in the recent years. Dual-discriminator conditional generative adversarial network (DcGAN) optimized with hybrid Manta ray foraging optimization algorithm and Volcano eruption algorithm (Hyb-MRF-VEA) is proposed for anomaly identification in Hyperspectral images (DcGAN-HAD) in this manuscript. The three steps of the proposed methodology are data preparation, reconstruction, and detection. Background purification is a stage in the data preparation process that trains the deep network unsupervised. In the reconstruction step, DcGAN method is executed in spatial, spectral, combined spectral-spatial domains. The novelty of DcGAN method is to produce the hyper spectral images (HSIs) that are closer to real ones. Among original and image pixels the reconstruction error map (REM) is computed. The pixel weights are obtained with respect to REM at the detection step. Finally, to obtain the final detection map, the original with reconstructed image is subject to spatial and spectral joint anomaly detector. For scaling the optimal parameters and make sure exact detection, DcGAN approach not exposes any optimization techniques adoption. Therefore, in this work, a Hyb-MRF-VEA with optimize DcGAN weight parameters. The proposed DcGAN-HAD system is done in PyTorch and its efficacy is calculated with certain performances metrics, like, area under ROC curve (AUC) metric, precision, computation time, accuracy. For hyper spectral remote sensing scenes dataset of PAVIA CENTER ANDUNIVERSITY, the proposed DcGAN-HAD method attains 22.15%, 58.97% and 34.29% higher accuracy compared with the existing methods.