Data Transformation Research Articles

Variance Estimation under Some Transformation for Both Symmetric and Asymmetric Data

This article suggests an improved class of efficient estimators that use various transformations to estimate the finite population variance of the study variable. These estimators are particularly helpful in situations where we know about the minimum and maximum values of the auxiliary variable, and the ranks of the auxiliary variable are associated with the study variable. Consequently, these rankings can be applied as an effective tool to improve the accuracy of the estimator. A first-order approximation is used to investigate the properties of the proposed class of estimators, such as bias and mean squared error (MSE) under simple random sampling. A simulation study carried out in order to measure the performance and verify the theoretical results. The suggested class of estimators has a greater percent relative efficiency (PRE) than the other existing estimators in all of the simulated situations, according to the results. Three symmetric and asymmetric datasets are examined in the application section in order to show the superior performance of the proposed class of estimators over the existing estimators.

Exploring Ion Mobility Mass Spectrometry Data File Conversions to Leverage Existing Tools and Enable New Workflows.

Ion mobility (IM) is often combined with LC-MS experiments to provide an additional dimension of separation for complex sample analysis. While highly complex samples are better characterized by the full dimensionality of LC-IM-MS experiments to uncover new information, downstream data analysis workflows are often not equipped to properly mine the additional IM dimension. For many samples the data acquisition benefits of including IM separations are all that is necessary to uncover sample information and the full dimensionality of the data is not required for data analysis. Postacquisition reduction and adaptation of the dimensions of LC-IM-MS and IM-MS experiments into an LC-MS format opens the possibility to use a plethora of existing software tools. In this work, we developed data file conversion tools to reduce the complexity of IM data analysis. Three data file transformations are introduced in the PNNL PreProcessor software: (1) mapping the IM axis to the LC axis for IM-MS data, (2) converting the drift time vs m/z space to CCS/z vs m/z space, and (3) transforming All Ions IM/MS mobility aligned fragmentation data to a standard LC-MS DDA data file format. These new data file conversions are demonstrated with corresponding lipidomics and proteomics workflows that leverage existing LC-MS data analysis software to highlight the benefits of the data transformations.

Science as a bridge to scientific knowledge: literature review

Science is practical and originates applicable knowledge that benefits humanity. The objective of this study was to analyze science as a bridge to scientific knowledge. A bibliographic review was conducted, the technique used was content analysis, fifteen articles (n=15) were chosen from the main databases, such as Scopus, Scielo and Web of Science, following the specified inclusion/exclusion criteria. The bibliographic analysis provided insight into perspectives on the transformation of data into information, combined with abstract reasoning to generate knowledge. It is concluded that science plays a crucial role in human progress, since it allows us to solve problems using various methods, which serve as a basis for new forms of research and social and technological development

Implementation of Density-Based Spatial Clustering of Applications with Noise and Fuzzy C – Means for Clustering Car Sales

This study compares the performance of two clustering algorithms, Density-Based Spatial Clustering of Applications with Noise (DBSCAN) and Fuzzy C-Means (FCM), in clustering car sales data at PT. XYZ. The dataset, comprising sales transactions from 2020 to 2023, includes information about vehicles, customers, and transactions. Preprocessing methods such as data transformation and normalization were applied to prepare the data. The results indicate that DBSCAN produces clusters with better validity, measured using the Silhouette Score, compared to FCM. Specifically, DBSCAN achieves the highest Silhouette Score of 0.7874 in cluster 2, while FCM reaches a maximum score of 0.3666 in cluster 3. Thus, DBSCAN proves to be more optimal for clustering car sales data at PT. XYZ, highlighting its superior performance in terms of cluster validity.

Low‐resolution activity recognition using super‐resolution and model ensemble networks

AbstractIn real‐world video super‐resolution, the complexity and diversity of degradations pose substantial challenges during both training and inference. Videos captured in real‐world settings often depict activities at varying resolutions. Typically, these activities are filmed from a distance that reduces the resolution of imagery, which thus lacks discriminative features. To address this problem, we introduce an activity recognition solution. First, a unique integration of data transformation and attention‐based average discriminator are employed for super‐resolution feature augmentation. This approach mitigates the lack of discriminative cues in low‐resolution videos. Subsequently, high‐resolution features extracted from the recovered data are directly fed into a model ensemble for activity recognition. We evaluate the resulting method on the TinyVIRAT‐v2 and HMDB51 datasets, achieving improved visual quality by leveraging the super‐resolution and model ensemble strategy. The proposed method enhances the quality of textures and boosts activity recognition in low‐resolution videos.

Organizing Foresight Tools

Futures and foresight tools support strategy work by structuring inquiry into- and generating information about the user’s uncertain operating environment. Each tool accomplishes this differently, making their selection non-trivial. Scholars have recommended that tools be organized for selection based on criteria such as the nature of inquiry they generate, the resources needed for their implementation, their input or data transformation affordances, the content they produce, or their temporal directionality. In practice, foresight tools are selected by actors under constrained circumstances. With the aim to improve the impact of foresight investments, a selection schema is developed that shortlists tools by how each generates relevant knowledge for solving the user’s problem across a landscape (locus and level) of uncertainty. The paper reports on initial field tests, the resulting refinements to the schema, and proposes a diagnostic protocol for its operationalization. The paper concludes by offering suggestions for leveraging the schema in firms, consulting, and teaching.

Model for converting data into NoSQL data warehouse for developing a real-time financial data warehouse system

This research introduces a novel model, the Financial Data Warehouses API (FDW-API), developed using PHP, Node.js, and Express.js. The model is designed to transform banking credit dataset information into a data warehouse format using a Non-Only SQL (NoSQL) database, stored in JSON format. Three types of databases were employed: MongoDB Node, MongoDB Serverless, and Cassandra. The study includes a comparative analysis of the data retrieval speed from all three databases. The model's applicability was tested in a real-time credit approval web application, demonstrating its effectiveness in transforming and storing data. Testing involved loading datasets ranging from 200, 300, 400, 500, 600, 800, and 1000 entries. Results indicate that the MongoDB serverless database outperformed others in terms of efficiency. Additionally, the FDW-API model streamlines data transformation and storage, facilitating real-time analysis and decision-making for financial institutions and data-driven businesses. Its flexibility integrates seamlessly with existing systems, reducing development time and costs, while its scalability accommodates growing data volumes and evolving business needs, providing a valuable tool for strategic insights and competitive advantage.

Company Supply Chain Management Performance Seen from The Aspect of Information Sharing and Cooperation (Empirical Study of MSMEs in the Food Industry Sector in Kertajati Regency)

The small food processing industry is an industrial sector that is quite capable of surviving economic shocks, because every year the need for food increases along with increasing population growth. Small and medium industries have an important role in developing processing businesses, especially in the food sector. The Kertajati specialty processed food industry is an industry that has potential to be developed, because each region has its own characteristics and characteristics. Suppliers, manufacturers, distributors, retailers and customers work together in the product creation process through to product distribution to final consumers, giving birth to the concept of supply chain management. This research aims to analyze the influence of information sharing and cooperation on the performance of supply chain management of MSMEs in the Food Industry Sector in the Kertajati District Region, both partially and simultaneously. The population in this research is small and medium industries in Kertajati District, Majalengka Regency which are engaged in processed food production, totaling 92 MSMEs. The sample used 48 MSMEs as respondents from the entire population, by collecting data through questionnaires. The sampling technique in this research uses probability sampling with a simple random sampling method. The type and source of data used is primary. The analysis technique used is correlation, where previously an instrument test was carried out using validity and reliability tests, then data transformation and data normality test, then hypothesis testing was carried out partially with the t test and simultaneously with the F test operated through the SPSS 21 program. The results of this research show that partially information sharing and cooperation have a significant effect on supply chain management performance. Simultaneous testing shows that information sharing and cooperation have a significant effect on supply chain management performance.

Data Quality in the Money Direct to School Program (MDSP): Overcoming the Garbage In, Garbage Out (GIGO) Challenge

Introduction: The investigation presents and discusses the relevance of data quality in Data Science, introducing the concept of "Garbage In, Garbage Out" (GIGO) to the Money Direct to School Program (PDDE). Objectives: The objective was to describe the process of data extraction, transformation, and loading for the generation of information panels by the Collaborative Support Center for Monitoring and Management of Educational Programs (CECAMPE/NE). Methodology: The employed methodology was quantitative and used typical Data Science tools. The data related to the PDDE were collected from transactional systems and surveys with the involved populations. Results: The results showed a satisfactory reduction of GIGO, although challenges were identified such as data update discrepancies, lack of adequate documentation of data tables, and issues with free typing fields in survey forms. Conclusion: The analysis reiterates that the GIGO concept is a significant challenge for the effective use of PDDE data and emphasizes the need to develop better data management practices in the context of the program. It suggests that all professionals who use data as raw material for their professional activities should be aware of these challenges and work towards effective solutions.

Classification of Alzheimer’s Disease Based on Deep Learning Using Medical Images

Neurodegenerative disorders, notably Alzheimer’s, pose an escalating global health challenge. Marked by the degeneration of brain neurons, these conditions lead to a gradual decline in nerve cells. Worldwide, over 55 million people grapple with dementia, with Alzheimer’s prominently impacting the aging demographic. The primary hurdle to early Alzheimer’s detection is the widespread lack of awareness. The main goal is to design and implement an artificial intelligence system using deep learning (DL) to detect Alzheimer’s disease (AD) through medical images and classify them into various stages, such as non-demented, moderate dementia, mild dementia, and very mild dementia. The dataset contains 6400 magnetic resonance images in .jpg format, with standardized dimensions of 176 × 208 pixels. To demonstrate the advantages of data augmentation and transformation techniques, four scenarios were created: two without these techniques, utilizing the Adam and SGD optimizers, and two with these techniques, also employing the Adam and SGD optimizers, respectively. The main results revealed that scenarios utilizing these techniques exhibited more stable performance when validated with a new dataset. Scenario 3, using the Adam optimizer, achieved a weighted average accuracy of 91.83%, whereas scenario 4, employing the SGD optimizer, reached 87.58% accuracy. In contrast, scenarios 1 and 2, which omitted these techniques, obtained low accuracies below 55%. It is concluded that classifying AD with a DL model exceeding 90% accuracy is feasible. This is the importance of utilizing data augmentation and transformation techniques to improve generalizability to input image variations, which is a consistent factor in the healthcare sector.

Predictive Customer Analytics: Machine Learning for Churn Prediction and Retention

Customer churn presents a big challenge in the industry. Businesses have to deal with the problem of customers stopping using their products and services due to dissatisfaction, competitive offers, more affordable alternatives, or changing needs. Churn can be damaging to businesses since it causes revenue loss and higher costs. To address this issue, our research aimed to develop a prediction model that helps predict customer churn. We started with getting the data set about telecommunication. Our analysis and model development were based on this dataset. Then we did data visualization to gain a better understanding of the data through multiple charts. After that, we performed data preparation. First, we did data transformation, data cleansing to address missing values and outliers; feature selection was done, and finally, in this step, the data set was split into testing and training sets. Multiple machine learning algorithms were used for modeling, such as decision trees, random forests, logistic regression, support vector machines, Naïve Bayes, and neural networks. Following model development, we evaluated the model performance with each algorithm using tune model hyperparameters. The decision tree algorithm performed the best with %96.7 accuracy, %96.9 precision, %99.3 recall, and %98.1 F1-score. These findings showed how effective decision tree algorithms are in predicting customer churn. This predictive model will enable telecommunication businesses to predict potential churn, make retention strategies, reduce customer churn and increase customer retention rates.

Simple Data Transformations for Mitigating the Syntactic Similarity to Improve Sentence Embeddings at Supervised Contrastive Learning

Contrastive learning of sentence representations has achieved great improvements in several natural language processing tasks. However, the supervised contrastive learning model trained on the natural language inference (NLI) dataset is insufficient to elucidate the semantics of sentences since it is prone to make a prediction based on heuristics. Herein, by using the ParsEVAL and the word overlap metric, it is shown that sentence pairs in the NLI dataset have strong syntactic similarity and propose a framework to compensate for this problem in two aspects. 1) Apply simple syntactic transformations to the hypothesis and 2) expand the objective to SupCon Loss to leverage variants of sentences. The method is evaluated on semantic textual similarity (STS) tasks and transfer tasks. The proposed methods improve the performance of the BERT‐based baseline in STS Benchmark and SICK Relatedness by 1.48% and 2.2%. Furthermore, the model achieves 82.65% on the HANS benchmark dataset, to the best of our knowledge, which is a state‐of‐the‐art performance demonstrating that our approach is effective in grasping semantics without heuristics in the NLI dataset at supervised contrastive learning. The code is available at https://github.com/whnhch/Break-the-Similarity.

Statistical data transformation in agrarian sciences for variance analysis: a systematic review

In statistical analyses, a common practice for enhancing the validity of variance analysis is the application of data transformation to convert measurements into a different mathematical scale. This technique was first employed in 1898 by Edgeworth and remains relevant in current scientific publications despite the proliferation of more modern and advanced techniques that obviate the need for certain assumptions. Data transformations, when appropriately used, can make the model error terms approximate a normal distribution. It is also possible to use the technique to correct the heterogeneity of variances or to render an additive model, ensuring the validity of the analysis of variances. Given that this technique can be hastily applied, potentially leading to erroneous or invalid results, we conducted a systematic literature review of studies in the field of agrarian sciences that utilized data transformations for the validation of analysis of variances. The aim was to check the transformations employed by the scientific community, the motivation behind their use, and to identify possible errors and inconsistencies in applying the technique in publications. In this study, we identified shortcomings and misconceptions associated with using this method, and we observed incomplete and inadequate utilization of the technique in 94.28 % of the analysed sample, resulting in misguided and erroneous conclusions in scientific research outcomes.

Multi-label feature selection based on nonlinear mapping

Feature selection is one of the important pre-processing methods for dimensionality reduction in multi-label learning tasks, which has attracted extensive attention in recent years. Most of the existing approaches transform feature data into the label space during the feature-label mapping process by assuming linear relationship between the feature and label spaces. However, the linearity assumption does not hold in most cases, especially for high-dimensional spaces. This work proposes a novel dimension reduction model for multi-label data by using nonlinear mapping (NMFS). The model introduces a point-to-point sigmoid function to describe the intrinsic relationship from data space to label space. The proposed method improves the generalization ability of feature selection by limiting the range of data transformation to the interval [0,1] which is consistent with the predicted values of the labels. The feature weight matrix is constrained by the l2,1-norm to ensure its sparsity, which forms the basis of feature selection. The variables in the NMFS model are iteratively updated using the gradient momentum optimization strategy, and a sparse weight-coefficient matrix is obtained for multi-label feature ordering. Experimental results on 14 multi-label data sets verify the effectiveness of the proposed method, and show the proposed method is superior to the most advanced multi-label feature selection methods.

Examining ALS: reformed PCA and random forest for effective detection of ALS

ALS (Amyotrophic Lateral Sclerosis) is a fatal neurodegenerative disease of the human motor system. It is a group of progressive diseases that affects the nerve cells in the brain and spinal cord that control the muscle movement of the body hence, detection and classification of ALS at the right time is considered to be one of the vital aspects that can save the life of humans. Therefore, in various studies, different AI techniques are used for the detection of ALS, however, these methods are considered to be ineffectual in terms of identifying the disease due to the employment of ineffective algorithms. Hence, the proposed model utilizes Modified Principal Component Analysis (MPCA) and Modified Random Forest (MRF) for performing dimensionality reduction of all the potential features considered for effective classification of the ALS presence and absence of ALS causing mutation in the corresponding gene. The MPCA is adapted for capturing all the Low-Importance Data transformation. Furthermore, The MPCA is objected to performing three various approaches: Covariance Matrix Correlation, Eigen Vector- Eigenvalue decomposition, and selecting the desired principal components. This is done in aspects of implying the LI (Lower-Importance) Data Transformation. By choosing these potential components without any loss of features ensures better viability of selecting the attributes for ALS-causing gene classification. This is followed by the classification of the proposed model by using Modified RF by updating the clump detector technique. The clump detector is proceeded by clustering approach using K-means, and the data reduced by their dimension are grouped accordingly. These clustered data are analyzed either for ALS causing or devoid of causing ALS. Finally, the model’s performance is assessed using different evaluation metrics like accuracy, recall, F1 score, and precision, and the proposed model is further compared with the existing models to assess the efficacy of the proposed model.

Implementation of Random Forest Algorithm for Graduation Prediction

University also has responsibility for the period of study taken by students in accordance with the level of education taken. The prediction of student study duration is designed to support the study program in guiding students to graduate on time. In this problem, data mining techniques can be applied to make predictions, namely by using the Random Forest classification method. The stages used in this study are data collecting, namely collecting student data, the data selection stage of 300 students with 5 (five) input data attributes including personal data (gender, age, marital status, job status) and academic data (grade) and 1 (one) attribute as an output containing choices about on time and late. The next stage is preprocessing with the aim of eliminating duplication, noise, and missing values, the stage of data transformation by normalizing age attributes (young and old), grade (large and small). Then the data split stage 3 times, namely 50/50, 40/60, and 30/60, the modeling stage with random forest, and finally, the evaluation stage by analyzing the confusion matrix consisting of accuracy, precision, and recall. The results of the study show that the proposed model can do well with predictions, that is, with the same results for all three data splits. The test value is 100% accuracy, 100% recall, and 100% precision. With this value, the success rate for predicting the timeliness of student graduation will be more accurate

Blockchain Financial Statements: Innovating Financial Reporting, Accounting, and Liquidity Management

The complexity and interconnection within the financial ecosystem demand innovative solutions to improve transparency, security, and efficiency in financial reporting and liquidity management, while also reducing accounting fraud. This paper presents Blockchain Financial Statements (BFS), an innovative accounting system designed to address accounting fraud, reduce data manipulation, and misrepresentation of company financial claims, by enhancing availability of the real-time and tamper-proof accounting data, underpinned by a verifiable approach to financial transactions and reporting. The primary goal of this research is to design, develop, and validate a blockchain-based accounting prototype—the BFS system—that can automate transformation of transactional data, generated by traditional business activity into comprehensive financial statements. Incorporating a Design Science Research Methodology with Domain-Driven Design, this study constructs a BFS artefact that harmonises accounting standards with blockchain technology and business orchestration. The resulting Java implementation of the BFS system demonstrates successful integration of blockchain technology into accounting practices, showing potential in real-time validation of transactions, immutable record-keeping, and enhancement of transparency and efficiency of financial reporting. The BFS framework and implementation signify an advancement in the application of blockchain technology in accounting. It offers a functional solution that enhances transparency, accuracy, and efficiency of financial transactions between banks and businesses. This research underlines the necessity for further exploration into blockchain’s potential within accounting systems, suggesting a promising direction for future innovations in tamper-evident financial reporting and liquidity management.

Permutation-invariant linear classifiers

Invariant concept classes form the backbone of classification algorithms immune to specific data transformations, ensuring consistent predictions regardless of these alterations. However, this robustness can come at the cost of limited access to the original sample information, potentially impacting generalization performance. This study introduces an addition to these classes—the permutation-invariant linear classifiers. Distinguished by their structural characteristics, permutation-invariant linear classifiers are unaffected by permutations on feature vectors, a property not guaranteed by other non-constant linear classifiers. The study characterizes this new concept class, highlighting its constant capacity, independent of input dimensionality. In practical assessments using linear support vector machines, the permutation-invariant classifiers exhibit superior performance in permutation experiments on artificial datasets and real mutation profiles. Interestingly, they outperform general linear classifiers not only in permutation experiments but also in permutation-free settings, surpassing unconstrained counterparts. Additionally, findings from real mutation profiles support the significance of tumor mutational burden as a biomarker.

Comprehensive assessment of machine learning methods for diagnosing gastrointestinal diseases through whole metagenome sequencing data

ABSTRACT The gut microbiome, linked significantly to host diseases, offers potential for disease diagnosis through machine learning (ML) pipelines. These pipelines, crucial in modeling diseases using high-dimensional microbiome data, involve selecting profile modalities, data preprocessing techniques, and classification algorithms, each impacting the model accuracy and generalizability. Despite whole metagenome shotgun sequencing (WMS) gaining popularity for human gut microbiome profiling, a consensus on the optimal methods for ML pipelines in disease diagnosis using WMS data remains elusive. Addressing this gap, we comprehensively evaluated ML methods for diagnosing Crohn’s disease and colorectal cancer, using 2,553 fecal WMS samples from 21 case-control studies. Our study uncovered crucial insights: gut-specific, species-level taxonomic features proved to be the most effective for profiling; batch correction was not consistently beneficial for model performance; compositional data transformations markedly improved the models; and while nonlinear ensemble classification algorithms typically offered superior performance, linear models with proper regularization were found to be more effective for diseases that are linearly separable based on microbiome data. An optimal ML pipeline, integrating the most effective methods, was validated for generalizability using holdout data. This research offers practical guidelines for constructing reliable disease diagnostic ML models with fecal WMS data.

Goats on the Move: Evaluating Machine Learning Models for Goat Activity Analysis Using Accelerometer Data.

Putting sensors on the bodies of animals to automate animal activity recognition and gain insight into their behaviors can help improve their living conditions. Although previous hard-coded algorithms failed to classify complex time series obtained from accelerometer data, recent advances in deep learning have improved the task of animal activity recognition for the better. However, a comparative analysis of the generalizing capabilities of various models in combination with different input types has yet to be addressed. This study experimented with two techniques for transforming the segmented accelerometer data to make them more orientation-independent. The methods included calculating the magnitude of the three-axis accelerometer vector and calculating the Discrete Fourier Transform for both sets of three-axis data as the vector magnitude. Three different deep learning models were trained on this data: a Multilayer Perceptron, a Convolutional Neural Network, and an ensemble merging both called a hybrid Convolutional Neural Network. Besides mixed cross-validation, every model and input type combination was assessed on a goat-wise leave-one-out cross-validation set to evaluate its generalizing capability. Using orientation-independent data transformations gave promising results. A hybrid Convolutional Neural Network with L2-norm as the input combined the higher classification accuracy of a Convolutional Neural Network with the lower standard deviation of a Multilayer Perceptron. Most of the misclassifications occurred for behaviors that display similar accelerometer traces and minority classes, which could be improved in future work by assembling larger and more balanced datasets.