Data Preparation Methods Research Articles

Multivariate data binning and examples generation to build a Diabetic Retinopathy classifier based on temporal clinical and analytical risk factors

In this paper, we explore the possibility of exploiting retrospective clinical data from Electronic Health Records (EHR) for classification tasks in chronic patients. The different intervals, short length and high class imbalance make it unfeasible to use traditional time series techniques. The first contribution of the paper is a preprocessing method to construct a multivariate time series dataset using EHR data, which infers missing data and regularizes the data frequency. The second contribution addresses class imbalance by using domain knowledge and existing short EHR series. We synthetically extrapolate patients’ data by using similar long time series and a fuzzy-based approach. The paper addresses the problem of detection of Diabetic Retinopathy (DR). Expert domain knowledge from ophthalmologists has been used in the proposed techniques to guide the processing of time series. The novelty in that case study consists in not using eye-fundus image analysis. Instead, the proposed methods are based solely on EHR data. Several multivariate multiclass time series classifiers are used to detect the four levels of DR severity from the pre-processed data sequences. Experiments prove the quality of the sequence preprocessing techniques proposed for EHR data. Results indicate that the TapNet classifier is the best one for DR grading. Despite being tested for DR detection, the proposed data preparation methods are applicable to other diseases with similar characteristics.

Skin Disease Recognition Based on Deep Learning Algorithms: A Review

The sharp increase in cases of melanoma and other skin cancers worldwide highlights the urgent need for improved diagnostic methods. Because skin lesions vary widely and access to dermatological knowledge is limited in resource-poor areas, traditional methods - which rely on visual inspection and clinical experience - have difficulty identifying diseases accurately. This situation requires innovative approaches to improve accessibility and diagnostic accuracy. To address these issues, this work uses deep learning (DL) and convolutional neural networks (CNNs). This paper is trying to transform skin cancer diagnosis through the use of large databases of dermoscopic images and advanced artificial intelligence algorithms. In order to evaluate the effectiveness of CNNs and DL in identifying skin diseases, we conducted a comprehensive analysis of the literature, focusing on the accuracy of skin cancer type classification. Our approach focused on model architectures, data preparation methods, and performance indicators while examining existing research using AI algorithms to diagnose skin cancer. With the ultimate goal of improving patient outcomes through early detection and accurate classification of skin conditions, this approach not only underscores the great potential of DL and CNN in transcending traditional diagnostic limitations, but also highlights the continued development of AI-based tools in pathology. Dermatology. Diagnosis.

Improved ACOM pattern matching in 4D-STEM through adaptive sub-pixel peak detection and image reconstruction

The technique known as 4D-STEM has recently emerged as a powerful tool for the local characterization of crystalline structures in materials, such as cathode materials for Li-ion batteries or perovskite materials for photovoltaics. However, the use of new detectors optimized for electron diffraction patterns and other advanced techniques requires constant adaptation of methodologies to address the challenges associated with crystalline materials. In this study, we present a novel image-processing method to improve pattern matching in the determination of crystalline orientations and phases. Our approach uses sub-pixel adaptive image processing to register and reconstruct electron diffraction signals in large 4D-STEM datasets. By using adaptive prominence and linear filters, we can improve the quality of the diffraction pattern registration. The resulting data compression rate of 103 is well-suited for the era of big data and provides a significant enhancement in the performance of the entire ACOM data processing method. Our approach is evaluated using dedicated metrics, which demonstrate a high improvement in phase recognition. Several features are extracted from the registered data to map properties such as the spot count, and various virtual dark fields, which are used to enhance the handling of the results maps. Our results demonstrate that this data preparation method not only enhances the quality of the resulting image but also boosts the confidence level in the analysis of the outcomes related to determining crystal orientation and phase. Additionally, it mitigates the impact of user bias that may occur during the application of the method through the manipulation of parameters.

Predicting insurance claims through a variety of data mining techniques: facing lots of missing values and moderate class-imbalanced levels

Abstract This paper copes with a real-world classification problem related to the management of claims received in an insurance company. The way to obtain the classifier is not easy due to the high amount of missing values as well as the inherent imbalanced scenario within class labels. Once the data partition has been done, the training set is submitted to an intensive double grid search in order to obtain the most promising type of missing value imputation approach and then a step ahead is done using the best method and it starts the next round of data mining strategies which now falls into data rebalancing umbrella. Again, a grid search from an undersampling and oversampling family with different settings is done taking into account only seen data. The training data obtained after the first grid search are now submitted to the second step according the second grid search in order to get the ready training set for the further classifier training. The main objective of the work is to find the best combination of data mining techniques that suits the data set with a pipeline containing two types of data preparation methods coming from different families. As an outcome, first the problem of the presence of missing values has been addressed and then the data rebalancing techniques has been applied. The study focuses on obtaining classifiers based on Bayesian and lazy approaches as well as decision trees, evaluated on metrics such as the area under the ROC curve (AUC), Cohen’s kappa, Accuracy and the F-measure, among others. The imputation by the mean the mode is preferable to the Expectation Maximization Imputation in the scenario faced in this paper taking into account that the amount of missing values is higher than a forty percent for many features.

Beyond the Hype : Building a Personalized Movie Experience with Content-Based Recommendation

This review paper provides a detailed analysis of how a movie recommendation system based on content was planned, executed and evaluated using Streamlit framework. The exponential expansion of digital content has led to the need for efficient recommendation systems, particularly in the realm of movies. In this respect, the proposed recommender system utilizes cosine similarity computations and content-based filtering methods to offer personalized film suggestions relying upon various features such as genres, keywords, castings as well as crews. The author further illustrates rigorous data preparation methods which encompass feature engineering, data collection and cleaning techniques. Furthermore, various other data sources such as TMDb API have been integrated to obtain detailed information about movies. The system, after a thorough study, shows that it has the ability of offering personalized movie recommendations that improve user engagement and experience. This paper examines broader implications of recommendation algorithms and argues their importance for improving happiness and content discovery in digital entertainment platforms. It also raises some common issues encountered during system development and advocates further research work aimed at enhancing recommendation algorithms and making them more flexible and scalable. In this way, it advances our understanding on how personalized content delivery can be applied to shape the landscape of digital entertainment through unmasking the complexities of recommendation systems.

An Introduction to Processing, Fitting, and Interpreting Transient Absorption Data.

Transient absorption (TA) spectroscopy is a powerful time-resolved spectroscopic method used to track the evolution of excited-state processes through changes in the system's absorption spectrum. Early implementations of TA were confined to specialized laboratories, but the evolution of commercial turn-key systems has made the technique increasingly available to research groups across the world. Modern TA systems are capable of producing large datasets with high energetic and temporal resolution that are rich in photophysical information. However, processing, fitting, and interpreting TA spectra can be challenging due to the large number of excited-state features and instrumental artifacts. Many factors must be carefully considered when collecting, processing, and fitting TA data in order to reduce uncertainty over which model or set of fitting parameters best describes the data. The goal of data preparation and fitting is to reduce as many of these extraneous factors while preserving the data for analysis. In this method, beginners are provided with a protocol for processing and preparing TA data as well as a brief introduction to selected fitting procedures and models, specifically single wavelength fitting and global lifetime analysis. Commentary on a number of commonly encountered data preparation challenges and methods of addressing them is provided, followed by a discussion of the challenges and limitations of these simple fitting methods.

A framework to improve churn prediction performance in retail banking

Managing customer retention is critical to a company’s profitability and firm value. However, predicting customer churn is challenging. The extant research on the topic mainly focuses on the type of model developed to predict churn, devoting little or no effort to data preparation methods. These methods directly impact the identification of patterns, increasing the model’s predictive performance. We addressed this problem by (1) employing feature engineering methods to generate a set of potential predictor features suitable for the banking industry and (2) preprocessing the majority and minority classes to improve the learning of the classification model pattern. The framework encompasses state-of-the-art data preprocessing methods: (1) feature engineering with recency, frequency, and monetary value concepts to address the imbalanced dataset issue, (2) oversampling using the adaptive synthetic sampling algorithm, and (3) undersampling using NEASMISS algorithm. After data preprocessing, we use XGBoost and elastic net methods for churn prediction. We validated the proposed framework with a dataset of more than 3 million customers and about 170 million transactions. The framework outperformed alternative methods reported in the literature in terms of precision-recall area under curve, accuracy, recall, and specificity. From a practical perspective, the framework provides managers with valuable information to predict customer churn and develop strategies for customer retention in the banking industry.

Integral and generalized indicators of development and ensuring the national security of the country

Purpose: presentation of the results of research on the development and application of methods for calculating integral and generalized indicators intended for assessing the state and making effective management decisions in the field of development and ensuring national security of Russia.Methods: the following methods were used in the study: methods of statistical data preparation, the method of expert assessments, the T. Saaty method, the method of the Center for political conjuncture, the method of linear convolution.Results: the article provides an overview of approaches to managing development and ensuring national security in Russia and a number of foreign countries using various indicators. The authors have developed a methodology for forming integral and generalized indicators of development and ensuring the national security of Russia. The calculation of generalized indicators of the state of economic security in the field of civil air transportation and aircraft manufacturing in Russia in 2021-2022 is presented. The authors proposed a mechanism for analytical interpretation of the influence of changes in general and specific indicators on the transformation of the integral indicator.Сonclusions and Relevance: when studying complex socio-economic phenomena, there is a need to use generalized and integral assessments. This is due to the large amount of information that needs to be analyzed, summarized and structured. Generalized and integral indicators are necessary for forecasting and planning development while ensuring the national security of the country. Such indicators make it possible to quickly and efficiently assess current socio-economic processes, respond in a timely manner and take measures to solve emerging problems. In the article, the authors propose to analyze the dynamics of generalized and integral indicators characterizing the development and ensuring the national security of the country, which makes it possible to identify potential threats and risks and timely take measures to minimize them.

Design of a Novel Convolutional Deep Network Model for Car Accident Prediction

Real-time collision risk estimation is thought to be essential to a sophisticated traffic management system. To swiftly determine accident probability is the goal of real-time crash risk prediction. However, due to the complex traffic situation on urban arterials, urban arterials were rarely included in previous studies, which mostly focused on highways. This paper suggests using Convolutional Deep Network model (CDNM) to forecast the probability of vascular accidents in real time. This model has the benefit of being able to use both LSTM and CNN. CNN retrieves the time-invariant characteristics, while LSTM captures the data's long-term dependability. To estimate the likelihood of an accident, many sorts of data are used, including weather, traffic, and signal timing data. There are also many other data preparation methods employed. The problem of data imbalance is also addressed by normalization which oversamples the crash cases. Using a variety of measures, the CDNM is enhanced on the training data and assessed on the test data. Five more benchmark models are constructed for model comparison. K-NN, ISVM, ANN, CNN, CNN-EVT and GAN are some of the models in this group. Experimental findings show that the proposed CDNM beats the competition in terms of sensitivity, specificity, accuracy, AUC and G-mean value. The findings of this paper demonstrate that CDNM can real-time prediction of crash risk at arterials.

Development of a Technique for Automatic Lineament Allocation Based on a Neural Network Approach

The purpose of the scientific work is to study the potential of neural network technologies in the field of extracting linear structures from digital terrain models SRTM. Linear structures, also known as lineaments, play an important role in the verification of known faults, the identification of fault-fracture structures, the detailing of the framework of discontinuous faults, as well as in the exploration of minerals. Their accurate and effective extraction in solving the designated tasks is of fundamental importance. The use of neural network technologies provides a number of advantages over sequential algorithms, including the ability to search for universal criteria for selecting lineaments based on a training sample. The paper considers a comprehensive innovative methodology that includes several key stages. The first stage is the author’s method of data preparation, which helps to ensure the quality of the training sample and minimize the impact of noise. The second stage is to develop an algorithm for vectorizing the results of the neural network, which allows you to easily export the results (lineaments) to a geographic information system (GIS). The third stage provides a method for minimizing the noise component of the training sample and optimizing the selection of synaptic weighting coefficients by retraining the neural network using simulated data reflecting various localization conditions of the lineaments. To verify the results obtained, a spatial comparison of linear structures extracted by a neural network and lineaments isolated by the operator was carried out. The results of this comparison demonstrate the high potential of the proposed approach and show that the use of neural network technologies is an actual and promising approach to solving the problem of extracting linear structures from digital terrain models. Positive conclusions are made about the expediency of using the results obtained for their practical application in the field of Earth sciences.

Estimation of Soil Loss by Revised USLE Model using Geospatial Techniques: A Case Study of Sainj Valley, Northwestern Himalaya, India

Soil erosion is a severe environmental problem. The current study is being conducted in Sainj valley of Himachal Pradesh. An increase in soil erosion rates has significant effects on land degradation, biodiversity loss, productivity and other factors. The Revised Universal Soil Loss Equation (RUSLE) model along with, Geographic Information System (GIS) and Remote Sensing (RS), was used in current study to quantify soil loss in the Sainj valley. Five important parameters like, Rainfall erosivity factor (R), Soil erodibility factor (K), Slope length and steepness (LS), Cropping management factor (C), and support practice factor (P) have been used to estimate the amount of soil loss in the study area. All these maps were created in GIS software using a variety of data preparation methods. The rainfall erosivity factor ranged between 501.19 to 1097.23 mt ha-1 cm-1. Spatial distribution of conservation support practice on soil loss indicated the variability (0–1) where lower value represents the higher conservation practice. The results show that average annual predicted soil loss is between 0 to 541.52tons/ha/yr and the predicted Average soil erosion rate was 11.15 tons/ha/Yr. Predicted soil loss was classified into six Erosion intensity classes Negligible (0-5), Low Erosion (5-10), Moderate Erosion (10-25), Moderately high (25-75), High Erosion (75-100), Extremely High Erosion (>100) From soil loss estimation it was observed that 62 % of the area is having very low erosion intensity and 0.37% of the study area having Extremely High erosion intensity. These findings can help the decision makers further in developing a suitable conservation program to prevent soil erosion.

Feature selection using differential evolution for microarray data classification

The dimensions of microarray datasets are very large, containing noise and redundancy. The problem with microarray datasets is the presence of more features compared to the number of samples, which adversely affects algorithm performance. In other words, the number of columns exceeds the number of rows. Therefore, to extract precise information from microarray datasets, a robust technique is required. Microarray datasets play a critical role in detecting various diseases, including cancer and tumors. This is where feature selection techniques come into play. In recent times, feature selection (FS) has gained significant importance as a data preparation method, particularly for high-dimensional data. It is preferable to address classification problems with fewer features while maintaining high accuracy, as not all features are necessary to achieve this goal. The primary objective of feature selection is to identify the optimal subset of features. In this context, we will employ the Differential Evolution (DE) algorithm. DE is a population-based stochastic search approach that has found widespread use in various scientific and technical domains to solve optimization problems in continuous spaces. In our approach, we will combine DE with three different classification algorithms: Random Forest (RF), Decision Tree (DT), and Logistic Regression (LR). Our analysis will include a comparison of the accuracy achieved by each algorithmic model on each dataset, as well as the fitness error for each model. The results indicate that when feature selection was used the results were better compared to the results where the feature selection was not used.

Preparing Data for Artificial Intelligence in Pathology with Clinical-Grade Performance.

The pathology is decisive for disease diagnosis but relies heavily on experienced pathologists. In recent years, there has been growing interest in the use of artificial intelligence in pathology (AIP) to enhance diagnostic accuracy and efficiency. However, the impressive performance of deep learning-based AIP in laboratory settings often proves challenging to replicate in clinical practice. As the data preparation is important for AIP, the paper has reviewed AIP-related studies in the PubMed database published from January 2017 to February 2022, and 118 studies were included. An in-depth analysis of data preparation methods is conducted, encompassing the acquisition of pathological tissue slides, data cleaning, screening, and subsequent digitization. Expert review, image annotation, dataset division for model training and validation are also discussed. Furthermore, we delve into the reasons behind the challenges in reproducing the high performance of AIP in clinical settings and present effective strategies to enhance AIP's clinical performance. The robustness of AIP depends on a randomized collection of representative disease slides, incorporating rigorous quality control and screening, correction of digital discrepancies, reasonable annotation, and sufficient data volume. Digital pathology is fundamental in clinical-grade AIP, and the techniques of data standardization and weakly supervised learning methods based on whole slide image (WSI) are effective ways to overcome obstacles of performance reproduction. The key to performance reproducibility lies in having representative data, an adequate amount of labeling, and ensuring consistency across multiple centers. Digital pathology for clinical diagnosis, data standardization and the technique of WSI-based weakly supervised learning will hopefully build clinical-grade AIP.

Automatic method of preparation of data for setting the model of the electric network mode when constructing the mnemonic diagram of the object for the simulator

The article is devoted to an automatic data preparation method for setting up a distribution network model, which is used to develop training tasks for distribution network personnel. The main goal in creating the method is to reduce the time for developing a simulator task by leveling the human error during the stage of filling in the structural data of the distribution network components. To interact with the automatic method and the already pre-created component library that was used in the previous method, it was necessary to change the internal structure of the double winding and three winding transformer components. Also, a new structural component was added to the library – Connection. The program algorithm of the automatic method consists of three cycles that go through all the components transferred from the library to the distribution network model. For a better understanding and perception of textual information about each cycle, a graphical representation of cycle process diagrams is presented. In conclusion, information is provided on the time spent when creating a task with manual data preparation and the developed automatic one, as well as suggestions to reduce computer resourced used during model run.

Skylight Polarization Pattern Simulator Based on a Virtual-Real-Fusion Framework for Urban Bionic Polarization Navigation.

In a data-driven context, bionic polarization navigation requires a mass of skylight polarization pattern data with diversity, complete ground truth, and scene information. However, acquiring such data in urban environments, where bionic polarization navigation is widely utilized, remains challenging. In this paper, we proposed a virtual-real-fusion framework of the skylight polarization pattern simulator and provided a data preparation method complementing the existing pure simulation or measurement method. The framework consists of a virtual part simulating the ground truth of skylight polarization pattern, a real part measuring scene information, and a fusion part fusing information of the first two parts according to the imaging projection relationship. To illustrate the framework, we constructed a simulator instance adapted to the urban environment and clear weather and verified it in 174 urban scenes. The results showed that the simulator can provide a mass of diverse urban skylight polarization pattern data with scene information and complete ground truth based on a few practical measurements. Moreover, we released a dataset based on the results and opened our code to facilitate researchers preparing and adapting their datasets to their research targets.

Representation Bias in Data: A Survey on Identification and Resolution Techniques

Data-driven algorithms are only as good as the data they work with, while datasets, especially social data, often fail to represent minorities adequately. Representation Bias in data can happen due to various reasons, ranging from historical discrimination to selection and sampling biases in the data acquisition and preparation methods. Given that “bias in, bias out,” one cannot expect AI-based solutions to have equitable outcomes for societal applications, without addressing issues such as representation bias. While there has been extensive study of fairness in machine learning models, including several review papers, bias in the data has been less studied. This article reviews the literature on identifying and resolving representation bias as a feature of a dataset, independent of how consumed later. The scope of this survey is bounded to structured (tabular) and unstructured (e.g., image, text, graph) data. It presents taxonomies to categorize the studied techniques based on multiple design dimensions and provides a side-by-side comparison of their properties. There is still a long way to fully address representation bias issues in data. The authors hope that this survey motivates researchers to approach these challenges in the future by observing existing work within their respective domains.

Predicting Online Customer Purchase using Gradient Boost Classifier

Abstract: Understanding client buying behavior is crucial for firms in order to enhance targeting consumers, maximize marketing efforts, and boost overall sales success. In order to identify whether a client is inclined to make a purchase or not, the Gradient Boosting algorithm is used in this paper's predictive modeling technique. The Gradient Boosting technique is used to create a model for prediction because it can handle complicated connections and detect non-linear patterns. The technique includes a number of steps, such as feature engineering, model training, and performance evaluation. The data preparation methods used include feature encoding, feature selection, and missing value imputation. On the labeled dataset, the Gradient Boosting classifier is trained, with prediction accuracy being optimized.

Pre-Processing Training Data Improves Accuracy and Generalisability of Convolutional Neural Network Based Landscape Semantic Segmentation

Data pre-processing for developing a generalised land use and land cover (LULC) deep learning model using earth observation data is important for the classification of a different date and/or sensor. However, it is unclear how to approach deep learning segmentation problems in earth observation data. In this paper, we trialled different methods of data preparation for Convolutional Neural Network (CNN) training and semantic segmentation of LULC features within aerial photography over the Wet Tropics and Atherton Tablelands, Queensland, Australia. This was conducted by trialling and ranking various training patch selection sampling strategies, patch and batch sizes, data augmentations and scaling and inference strategies. Our results showed: a stratified random sampling approach for producing training patches counteracted class imbalances; a smaller number of larger patches (small batch size) improves model accuracy; data augmentations and scaling are imperative in creating a generalised model able to accurately classify LULC features in imagery from a different date and sensor; and producing the output classification by averaging multiple grids of patches and three rotated versions of each patch produced a more accurate and aesthetic result. Combining the findings from the trials, we fully trained five models on the 2018 training image and applied the model to the 2015 test image. The output LULC classifications achieved an average kappa of 0.84, user accuracy of 0.81, and producer accuracy of 0.87. Future research using CNNs and earth observation data should implement the findings of this project to increase LULC model accuracy and transferability.

Application of Geo-informatics for Soil Erosion Mapping

Soil erosion is a most severe environmental problem in hilly area. The study is carried out on Upper Bagmati River basin, North of Kathmandu valley having an area of 61 Sq.km. (approx). Universal Soil Loss Equation (USLE) model, with Geographic Information System (GIS) has been used to quantify the soil loss. Erosion modelling requires huge amount of information and data, usually coming from different sources and available in different formats and scales and for management of these data, GIS was used, which helped considerably in organizing the spatial data representing the effects of each factor affecting soil erosion. Five essential parameters of USLE Rainfall erosivity factor (R), Soil erodibility Factor (K), Slope length and steepness (LS) factor, Cropping management factor (C) and Support practice factor (P) have been used to estimate soil loss amount in the study area. All of these layers have been prepared in Arc GIS using various data sources and data preparation methods. DEM was prepared from the contour data with the interval of 20m which was used to generate LS factor. The monthly rainfall data (2010) of 17 rain gauge stations within the catchment area have been used to predict the R factor. K, C and P factors in basin area are adopted from the literature. The spatial distribution map of soil loss of the basin has been generated and classified into six categories depending on the calculated soil erosion amount. The annual predicted soil loss ranges between 0 and 292.878 t/ha/y. Low soil loss (mean 9.7 t/ha/y) have been recorded under forested areas. The high rate (mean 40.4 t/ha/y) of soil erosion was found in the cultivation area.

Data Preparation Impact on Semantic Segmentation of 3D Mobile LiDAR Point Clouds Using Deep Neural Networks

Currently, 3D point clouds are being used widely due to their reliability in presenting 3D objects and accurately localizing them. However, raw point clouds are unstructured and do not contain semantic information about the objects. Recently, dedicated deep neural networks have been proposed for the semantic segmentation of 3D point clouds. The focus has been put on the architecture of the network, while the performance of some networks, such as Kernel Point Convolution (KPConv), shows that the way data are presented at the input of the network is also important. Few prior works have studied the impact of using data preparation on the performance of deep neural networks. Therefore, our goal was to address this issue. We propose two novel data preparation methods that are compatible with typical density variations in outdoor 3D LiDAR point clouds. We also investigated two already existing data preparation methods to show their impact on deep neural networks. We compared the four methods with a baseline method based on point cloud partitioning in PointNet++. We experimented with two deep neural networks: PointNet++ and KPConv. The results showed that using any of the proposed data preparation methods improved the performance of both networks by a tangible margin compared to the baseline. The two proposed novel data preparation methods achieved the best results among the investigated methods for both networks. We noticed that, for datasets containing many classes with widely varying sizes, the KNN-based data preparation offered superior performance compared to the Fixed Radius (FR) method. Moreover, this research allowed identifying guidelines to select meaningful downsampling and partitioning of large-scale outdoor 3D LiDAR point clouds at the input of deep neural networks.