Data Transformation Research Articles

Shrinkage estimation of gene interaction networks in single-cell RNA sequencing data

BackgroundGene interaction networks are graphs in which nodes represent genes and edges represent functional interactions between them. These interactions can be at multiple levels, for instance, gene regulation, protein-protein interaction, or metabolic pathways. To analyse gene interaction networks at a large scale, gene co-expression network analysis is often applied on high-throughput gene expression data such as RNA sequencing data. With the advance in sequencing technology, expression of genes can be measured in individual cells. Single-cell RNA sequencing (scRNAseq) provides insights of cellular development, differentiation and characteristics at the transcriptomic level. High sparsity and high-dimensional data structures pose challenges in scRNAseq data analysis.ResultsIn this study, a sparse inverse covariance matrix estimation framework for scRNAseq data is developed to capture direct functional interactions between genes. Comparative analyses highlight high performance and fast computation of Stein-type shrinkage in high-dimensional data using simulated scRNAseq data. Data transformation approaches also show improvement in performance of shrinkage methods in non-Gaussian distributed data. Zero-inflated modelling of scRNAseq data based on a negative binomial distribution enhances shrinkage performance in zero-inflated data without interference on non zero-inflated count data.ConclusionThe proposed framework broadens application of graphical model in scRNAseq analysis with flexibility in sparsity of count data resulting from dropout events, high performance, and fast computational time. Implementation of the framework is in a reproducible Snakemake workflow https://github.com/calathea24/ZINBGraphicalModel and R package ZINBStein https://github.com/calathea24/ZINBStein.

New Quadratic Discriminant Analysis Algorithms for Correlated Audiometric Data.

Paired organs like eyes, ears, and lungs in humans exhibit similarities, and data from these organs often display remarkable correlations. Accounting for these correlations could enhance classification models used in predicting disease phenotypes. To our knowledge, there is limited, if any, literature addressing this topic, and existing methods do not exploit such correlations. For example, the conventional approach treats each ear as an independent observation when predicting audiometric phenotypes and is agnostic about the correlation of data from the two ears of the same person. This approach may lead to information loss and reduce the model performance. In response to this gap, particularly in the context of audiometric phenotype prediction, this paper proposes new quadratic discriminant analysis (QDA) algorithms that appropriately deal with the dependence between ears. We propose two-stage analysis strategies: (1) conducting data transformations to reduce data dimensionality before applying QDA; and (2) developing new QDA algorithms to partially utilize the dependence between phenotypes of two ears. We conducted simulation studies to compare different transformation methods and to assess the performance of different QDA algorithms. The empirical results suggested that the transformation may only be beneficial when the sample size is relatively small. Moreover, our proposed new QDA algorithms performed better than the conventional approach in both person-level and ear-level accuracy. As an illustration, we applied them to audiometric data from the Medical University of South Carolina Longitudinal Cohort Study of Age-related Hearing Loss. In addition, we developed an R package, PairQDA, to implement the proposed algorithms.

Neural Approach to Coordinate Transformation for LiDAR–Camera Data Fusion in Coastal Observation

The paper presents research related to coastal observation using a camera and LiDAR (Light Detection and Ranging) mounted on an unmanned surface vehicle (USV). Fusion of data from these two sensors can provide wider and more accurate information about shore features, utilizing the synergy effect and combining the advantages of both systems. Fusion is used in autonomous cars and robots, despite many challenges related to spatiotemporal alignment or sensor calibration. Measurements from various sensors with different timestamps have to be aligned, and the measurement systems need to be calibrated to avoid errors related to offsets. When using data from unstable, moving platforms, such as surface vehicles, it is more difficult to match sensors in time and space, and thus, data acquired from different devices will be subject to some misalignment. In this article, we try to overcome these problems by proposing the use of a point matching algorithm for coordinate transformation for data from both systems. The essence of the paper is to verify algorithms based on selected basic neural networks, namely the multilayer perceptron (MLP), the radial basis function network (RBF), and the general regression neural network (GRNN) for the alignment process. They are tested with real recorded data from the USV and verified against numerical methods commonly used for coordinate transformation. The results show that the proposed approach can be an effective solution as an alternative to numerical calculations, due to process improvement. The image data can provide information for identifying characteristic objects, and the obtained accuracies for platform dynamics in the water environment are satisfactory (root mean square error—RMSE—smaller than 1 m in many cases). The networks provided outstanding results for the training set; however, they did not perform as well as expected, in terms of the generalization capability of the model. This leads to the conclusion that processing algorithms cannot overcome the limitations of matching point accuracy. Further research will extend the approach to include information on the position and direction of the vessel.

Meta-analysis accelerator: a comprehensive tool for statistical data conversion in systematic reviews with meta-analysis

BackgroundSystematic review with meta-analysis integrates findings from multiple studies, offering robust conclusions on treatment effects and guiding evidence-based medicine. However, the process is often hampered by challenges such as inconsistent data reporting, complex calculations, and time constraints. Researchers must convert various statistical measures into a common format, which can be error-prone and labor-intensive without the right tools.ImplementationMeta-Analysis Accelerator was developed to address these challenges. The tool offers 21 different statistical conversions, including median & interquartile range (IQR) to mean & standard deviation (SD), standard error of the mean (SEM) to SD, and confidence interval (CI) to SD for one and two groups, among others. It is designed with an intuitive interface, ensuring that users can navigate the tool easily and perform conversions accurately and efficiently. The website structure includes a home page, conversion page, request a conversion feature, about page, articles page, and privacy policy page. This comprehensive design supports the tool’s primary goal of simplifying the meta-analysis process.ResultsSince its initial release in October 2023 as Meta Converter and subsequent renaming to Meta-Analysis Accelerator, the tool has gained widespread use globally. From March 2024 to May 2024, it received 12,236 visits from countries such as Egypt, France, Indonesia, and the USA, indicating its international appeal and utility. Approximately 46% of the visits were direct, reflecting its popularity and trust among users.ConclusionsMeta-Analysis Accelerator significantly enhances the efficiency and accuracy of meta-analysis of systematic reviews by providing a reliable platform for statistical data conversion. Its comprehensive variety of conversions, user-friendly interface, and continuous improvements make it an indispensable resource for researchers. The tool’s ability to streamline data transformation ensures that researchers can focus more on data interpretation and less on manual calculations, thus advancing the quality and ease of conducting systematic reviews and meta-analyses.

Spectral Characteristics and Identification of Degraded Alpine Meadow in Qinghai–Tibetan Plateau Based on Hyperspectral Data

Grassland degradation poses a significant challenge to achieving the Sustainable Development Goals (SDGs) on the Qinghai–Tibetan Plateau (QTP). Effective monitoring of grassland degradation is essential for ecological restoration. Hyperspectral technology offers efficient and accurate identification of degradation. However, the influence of observation time, data analysis methods and classification techniques on the accuracy of identifying alpine grasslands remains unclear. In this study, the spectral reflectance of degraded alpine meadow, alpine meadow, alpine shrub and Tibetan barley was measured from May to September 2023 using a ground spectrometer in the northeastern QTP. First-order derivatives (FDR) and continuum removal were applied to the spectra, and characteristic parameters and vegetation indices were calculated. Support vector machine (SVM), random forest (RF), artificial neural network (ANN) and decision tree (DT) were then used to compare the classification accuracy between different months, transformation methods and characteristic parameters. The results showed that the spectral reflectance peaked in July, with significant differences in the near infrared (NIR) bands between alpine meadow and degraded alpine meadow. Alpine shrub and Tibetan barley showed greater differences in reflectance compared to other vegetation types, especially in the NIR bands. Data transformations improved reflectance and absorption characteristics in the NIR and visible bands. Indices such as DVI, RVI and NDGI effectively differentiated vegetation types. Optimal accuracy for the identification of degraded alpine meadow in July was achieved using FDR transformations and ANN or SVM for classification. This study provides methodological insights for monitoring grassland degradation on the QTP.

ShNFN: Shepard Neuro-Fuzzy Network for Intrusion Detection in Fog Computing

Fog computing is a type of distributed computing that makes data storage and computation closer to the network edge. While fog computing offers numerous advantages, it also introduces several challenges, particularly in terms of security. Intrusion Detection System (IDS) plays a crucial role in securing fog computing environments by monitoring network traffic and system activities for signs of malicious behavior. Several techniques can be employed to enhance intrusion detection in fog computing environments. Accordingly, this paper proposes a Shepard Neuro-Fuzzy Network (ShNFN) for intrusion detection in fog computing. Initially, in the cloud layer, the input data are passed to data transformation to transform the unstructured data into structured form. Here, data transformation is done employing the Box-Cox transformation. Following this, the feature selection is done in terms of information gain and symmetric uncertainty process and it is used to create a relationship between two variables. After that, the data are classified by employing the proposed ShNFN. The ShNFN is attained by fusing two networks, such as Cascade Neuro-Fuzzy Network (Cascade NFN) and Shepard Convolutional Neural Networks (ShCNN). After this, the physical process is executed at the endpoint layer. Finally, intrusion detection is accomplished in the fog layer by the proposed ShNFN method. The performance of the intrusion detection using ShNFN is calculated by the metrics of recall, F-measure and precision. The proposed method achieves the values of 93.3%, 92.5% and 94.8% for recall, F-measure, and precision, respectively.

Have Your Cake and Eat It Too

As datasets have become a more significant aspect of Open Science, attention has turned to the data transformations that drive their creation. Li and Ludäscher have pointed out the importance of identifying data cleaning workflows as a series of modular transformations that can be extracted for reuse. This modular approach aids reproducibility and allows for transparency in data provenance. However, the constantly evolving nature of data science technology means that even once these modules have been identified and implemented, their functionality must be ported to new platforms as old ones become less applicable or less common in a field of study. When these transformations take place, it is important to consider not only practicality and functionality, but also transparency within a data processing team. Clarity of communication within a team is the first step towards providing clear and transparent documentation to the end user. This case study of an updated workflow process for a long-running longitudinal health and well-being study provides practical examples of these principles.

Optimizing Kernel Transformations to Handle Binary Class Imbalanced Dataset Classification

ABSTRACT Imbalanced class distributions pose a prevalent challenge in numerous classification problems, requiring effective strategies for learning from such skewed data. Traditional machine learning algorithms often struggle with imbalanced datasets, as they tend to bias their classification functions toward the majority class, resulting in suboptimal performance for minority classes. In our research, we propose a novel approach to address this challenge specifically tailored for Support Vector Machines (SVM), a well-established family of learning algorithms. Our method leverages a kernel trick to enhance the SVM’s classification capabilities on imbalanced datasets named KTI. It aims to streamline the classification process by incorporating adaptive data transformations within the algorithm itself, offering a more efficient and integrated solution for handling imbalanced data. Experimental evaluations conducted on diverse real-world datasets demonstrate the superior performance of our proposed strategy compared to existing methods, showcasing its potential for practical applications in classification tasks with skewed class distributions.

Internet of Things-Driven Precision in Fish Farming: A Deep Dive into Automated Temperature, Oxygen, and pH Regulation

The research introduces a revolutionary Internet of Things (IoT)-based system for fish farming, designed to significantly enhance efficiency and cost-effectiveness. By integrating the NodeMcu12E ESP8266 microcontroller, this system automates the management of critical water quality parameters such as pH, temperature, and oxygen levels, essential for fostering optimal fish growth conditions and minimizing mortality rates. The core of this innovation lies in its intelligent monitoring and control mechanism, which not only supports accelerated fish development but also ensures the robustness of the farming process through automated adjustments whenever the monitored parameters deviate from desired thresholds. This smart fish farming solution features an Arduino IoT cloud-based framework, offering a user-friendly web interface that enables fish farmers to remotely monitor and manage their operations from any global location. This aspect of the system emphasizes the importance of efficient information management and the transformation of sensor data into actionable insights, thereby reducing the need for constant human oversight and significantly increasing operational reliability. The autonomous functionality of the system is a key highlight, designed to persist in adjusting the environmental conditions within the fish farm until the optimal parameters are restored. This capability greatly diminishes the risks associated with manual monitoring and adjustments, allowing even those with limited expertise in aquaculture to achieve high levels of production efficiency and sustainability. By leveraging data-driven technologies and IoT innovations, this study not only addresses the immediate needs of the fish farming industry but also contributes to solving the broader global challenge of protein production. It presents a scalable and accessible approach to modern aquaculture, empowering stakeholders to maximize output and minimize risks associated with fish farming, thereby paving the way for a more sustainable and efficient future in the global food supply.

PREDIKSI PENYAKIT JANTUNG MENGGUNAKAN METODE RANDOM FOREST DAN PENERAPAN PRINCIPAL COMPONENT ANALYSIS (PCA)

Heart disease is a significant public health issue and the leading cause of death worldwide. Risk factors such as hypertension, diabetes, obesity, sedentary lifestyle, smoking, and genetic factors contribute to the development of heart disease. This study aims to develop a heart disease prediction model using the Random Forest method. The dataset used comes from the UCI Machine Learning Repository, containing data from 1026 patients with various health features. The methods used include the stages of knowledge discovery in databases (KDD), namely data selection, preprocessing, transformation, data mining, and evaluation. The study results show that the model with 100 decision trees achieved an accuracy of 0.9823. Further evaluation using the confusion matrix and classification report indicates that the Random Forest method provides 98% accuracy, 100% precision, 96% recall, and a 98% F1-score. In conclusion, the Random Forest method is effective in predicting heart disease, with features such as thal having a significant impact on model accuracy.

Advanced Supply Chain Management Using Adaptive Serial Cascaded Autoencoder with LSTM and Multi-Layered Perceptron Framework

Supply chain management is essential for businesses to handle uncertainties, maintain efficiency, and stay competitive. Financial risks can arise from various internal and external sources, impacting different supply chain stages. Companies that effectively manage these risks gain a deeper understanding of their procurement activities and implement strategies to mitigate financial threats. This paper explores financial risk assessment in supply chain management using advanced deep learning techniques on big data. The Adaptive Serial Cascaded Autoencoder (ASCA), combined with Long Short-Term Memory (LSTM) and Multi-Layered Perceptron (MLP), is used to evaluate financial risks. A data transformation process is used to clean and prepare financial data for analysis. Additionally, Sandpiper Galactic Swarm Optimization (SGSO) is employed to optimize the deep learning model’s performance. The SGSO-ASCALSMLP-based financial risk prediction model demonstrated superior accuracy compared to traditional methods. It outperformed GRU (gated recurrent unit)-ASCALSMLP by 3.03%, MLP-ASCALSMLP by 7.22%, AE-LSTM-ASCALSMLP by 10.7%, and AE-LSTM-MLP-ASCALSMLP by 10.9% based on F1-score performance. The SGSO-ASCALSMLP model is highly efficient in predicting financial risks, outperforming conventional prediction techniques and heuristic algorithms, making it a promising approach for enhancing financial risk management in supply chain networks.

IMPROVEMENT OF THE TEST METHODOLOGY OF AIRCRAFT VIDEO RECORDING EQUIPMENT UNDER THE CONDITIONS OF A SPECIAL PERIOD

The introduction of weapons and military equipment at the stage of Russia's armed aggression requires the rational use of human resources, standard means of measurement, registration and indication as part of measurement systems and further processing of the results obtained in automated systems for collecting and processing information for its use during tests of serial (modernized) samples of OVT. The rapid development of technical means of measurement, transformation and exchange of data within and between systems require the improvement of test methods for video recording of aircraft, which take into account the peculiarities of the development of electronic equipment of aviation equipment and are based on a unified approach to the problem of optimal use of resources. Modern means of video registration have quite justifiably been widely used in the Armed Forces of Ukraine. The research was conducted using the methods of the theory of system analysis, synthesis, comparison, and generalization. The purpose of the article is to develop a typical methodology for testing aircraft video recording equipment in the conditions of a special period. To achieve the goal, the following was done: to analyze the existing methods of testing domestic and foreign aircraft video registration means; to consider an improved model of functioning and the technology of conducting tests of means of video registration of aircraft with a reduction in volume; to develop a typical methodology for conducting tests of aircraft video registration devices in the conditions of a special period. The results of the analysis should be taken into account when justifying ways to reduce the amount of tests in the conditions of a special

The Duck’s Brain

AbstractAlthough database systems perform well in data access and manipulation, their relational model hinders data scientists from formulating machine learning algorithms in SQL. Nevertheless, we argue that modern database systems perform well for machine learning algorithms expressed in relational algebra. To overcome the barrier of the relational model, this paper shows how to transform data into a coordinate relational representation for training neural networks in SQL: We first describe building blocks for data transformation, model training and inference in SQL-92 and their counterparts using an extended array data type. Then, we compare the implementation for model training and inference using array data types to the one using a coordinate relational representation in SQL-92 only. The evaluation in terms of runtime and memory consumption proves the suitability of modern database systems for matrix algebra, although specialised array data types perform better than matrices in coordinate relational representation.

Responding to COVID-19: an exploration of EU country responses and directions for further research.

During COVID-19, scientists advising policymakers were forced to deal with high uncertainty and risks in an environment of unknowns. Evidence on which policies and measures were effective in responding to the pandemic remains underdeveloped to answer the key question 'what worked and why?'. This study aims to provide a basis for studies to go further to answer this critical question, by starting to look efficacy or how countries ensured that health services remained available and what measures were enacted to protect and treat their populations and workers. We applied a three-phase sequential mixed methods design. In phase one, we started with a qualitative content analysis of the EU Country Profile reports to retrieve and analyse data on COVID-19 responses taken by 29 countries in the European region. Phase two is the step of data transformation, converting qualitative data into numerical codes that can be statistically analysed, which are then used in a quantitative cross-national comparative analysis that comprises phase three. The quantifying process resulted in a numerical indicator to measure the 'response efficacy' of the 29 countries, which is used in phase three's association of the response measure with country performance indicators that were derived from European Centre for Disease Control (ECDC) COVID-19 case and death rate data. Through comparing the frequency of COVID-19 measures taken, we found that many countries in the European region undertook similar actions but with differing effects. The cross-national analysis revealed an expected relationship: a lower COVID-19 response efficacy appeared to be related to a higher case and death rates. Still, marked variation for countries with similar response efficacy indicators was found, signalling that the combination and sequence of implementation of COVID-19 responses is possibly just as important as their efficacy in terms of which response measures were implemented. Many European countries employed similar COVID-19 measures but still had a wide variation in their case and death rates. To unravel the question 'what worked and why?', we suggest directions from which more refined research can be designed that will eventually contribute to mitigate the impact of future pandemics and to be better prepared for their economic and human burden.

Building a Data Lake on AWS: From Data Migration to AI-Driven Insights

As organizations generate and process increasing amounts of data, building data lakes on cloud platforms like AWS has become crucial to managing large datasets efficiently. This paper outlines the key steps in constructing a scalable data lake on AWS, starting from data migration to leveraging AI for insights. It explores how AWS services like S3, Glue, and SageMaker work together to facilitate data storage, transformation, and machine learning. In addition, it highlights the importance of orchestrating data pipelines with automation tools like AWS Lambda and Apache Airflow to ensure smooth, scalable, and efficient workflows. This paper explores the end-to-end process of migrating data to AWS, constructing scalable data lakes, and leveraging AI capabilities to drive actionable insights. Through practical examples, diagrams, and pseudocode, this paper provides a comprehensive guide to implementing data lakes with AWS services such as S3, Glue, and SageMaker, highlighting key considerations around data migration, storage, processing, and analytics. The role of automation tools like AWS Lambda and Airflow in orchestrating these pipelines is also discussed. Keywords AWS, Data Lake, AI-driven Insights, Data Migration, Amazon S3, AWS Glue, Amazon SageMaker, Cloud Analytics, Data Pipeline, ETL, Machine Learning

AI-Driven Data Preparation: The Key to Unlocking Cloud-Based Analytics

The rapid adoption of cloud-based analytics has revolutionized data-driven decision-making across industries. Cloud- based analytics has transformed how businesses make decisions by leveraging vast amounts of data. However, preparing data for analysis—such as cleaning, transforming, and organizing it—can be a complicated and time- consuming process. AI-driven data preparation (AIDP) is a solution that automates these steps, reducing the time and effort needed to prepare data while improving its quality. This paper explains the importance of AI-driven data preparation, discusses how it works, and shows how businesses can benefit from using AI in their data preparation process for cloud analytics. The use of diagrams, flowcharts, and pseudocode helps explain these concepts in a simplified yet technical manner. Keywords AI-driven data preparation, cloud-based analytics, data pipeline, automation, machine learning, data wrangling, ETL, data transformation, big data

HERALD: A domain-specific query language for longitudinal health data analytics

BackgroundLarge-scale health data has significant potential for research and innovation, especially with longitudinal data offering insights into prevention, disease progression, and treatment effects. Yet, analyzing this data type is complex, as data points are repeatedly documented along the timeline. As a consequence, extracting cross-sectional tabular data suitable for statistical analysis and machine learning can be challenging for medical researchers and data scientists alike, with existing tools lacking balance between ease-of-use and comprehensiveness. ObjectiveThis paper introduces HERALD, a novel domain-specific query language designed to support the transformation of longitudinal health data into cross-sectional tables. We describe the basic concepts, the query syntax, a graphical user interface for constructing and executing HERALD queries, as well as an integration into Informatics for Integrating Biology and the Bedside (i2b2). MethodsThe syntax of HERALD mimics natural language and supports different query types for selection, aggregation, analysis of relationships, and searching for data points based on filter expressions and temporal constraints. Using a hierarchical concept model, queries are executed individually for the data of each patient, while constructing tabular output. HERALD is closed, meaning that queries process data points and generate data points. Queries can refer to data points that have been produced by previous queries, providing a simple, but powerful nesting mechanism. ResultsThe open-source implementation consists of a HERALD query parser, an execution engine, as well as a web-based user interface for query construction and statistical analysis. The implementation can be deployed as a standalone component and integrated into self-service data analytics environments like i2b2 as a plugin. HERALD can be valuable tool for data scientists and machine learning experts, as it simplifies the process of transforming longitudinal health data into tables and data matrices. ConclusionThe construction of cross-sectional tables from longitudinal data can be supported through dedicated query languages that strike a reasonable balance between language complexity and transformation capabilities.

Real Time Data Ingestion and Transformation in Azure Data Platforms

Real Time Data Ingestion and Transformation in Azure Data Platforms

Positioning of the test surface in a CGH null test by cat's eye interference

Interferometric null test of optical aspheres and freeforms is one of the most established methods, in spite of recognized limitations related to positioning of the test surface. Slight misalignment of the surface in the null test system can introduce remarkable wave aberrations. Current approaches for positioning the test surface are based on a series of fiducial marks or retroreflectors set at given spots around the test surface, which consequently suffer the problem of datum transformation. We propose a method for positioning the test surface in a null test by cat's eye interference without any fiducial marks or retroreflectors. A computer-generated hologram (CGH) is used as null optics fabricated with null test pattern, alignment pattern and positioning pattern. A part of test beam with relatively small f/number is diffracted through the positioning pattern and then focuses on certain spots of the test surface. The cat's eye reflection from the test surface returns to the interferometer and interferes with the reference beam. It is then possible to precisely position the test surface by measuring the wavefront error of the cat's eye interference. The design method for such a CGH positioning pattern is presented, following the physical constraint that the surface normal at the cay's eye reflection is right the angular bisector of the positioning beam. Analysis on the positioning performance is then presented to show the sensitivity to misalignment including defocus, lateral shift and tip-tilt, which at last is experimentally verified by measuring an even asphere with a CGH.

Optimal designs for linear regression models with skew-normal errors

This paper investigates optimal designs for linear regression models with skew-normal errors which provide an alternative to traditional models with symmetric normal errors and eliminate the need for data transformation. Using the maximum likelihood estimation approach, we consider various design criteria, including D-, A-, A s -optimality. Equivalence theorems are provided to determine the optimality of the designs. Optimal designs for a centered parametrization model are also considered. Two examples are provided to illustrate the theoretical results. In addition, the efficiencies of the optimal designs based on skew-normal errors are compared with those based on normal errors.