Data Transformation Research Articles

A general image classification model for agricultural machinery trajectory mode recognition

Field-road trajectory classification is a crucial task for agricultural machinery behavior mode recognition, aiming to distinguish field operation mode and road driving mode automatically. However, the imbalanced distribution of agricultural machine trajectories brings challenges for the field-road trajectory classification task. Additionally, most existing field-road trajectory classification methods have certain shortcomings. For instance, they encounter difficulties in accurately representing the state of agricultural machinery movement using the current features. The data transformation process often leads to information loss, and the model’s generalization capabilities are limited. The performance of the models is constrained by each of these elements. To address these shortcomings, this paper introduces a general image classification model for agricultural machinery trajectory mode recognition named ATRNet. First, to address the issue of imbalanced field-road proportions in agricultural machinery trajectory data, a Conditional Tabular Generative Adversarial Network (CTGAN) is employed to generate quasi trajectories, balancing the distribution of positive and negative samples in the data. This step aims to eliminate biases during the model training process. Second, to accurately characterize the motion status of agricultural machinery, we propose a multiangle feature enhancement method to extract rich spatiotemporal features from trajectory data. Finally, different from conventional field-road trajectory classification models that primarily rely on spatial and temporal information for identifying trajectories, we present a lossless trajectory data representation paradigm. This paradigm maps each trajectory point into a “feature map” and uses an image classification model to capture latent feature representations of trajectory points for the recognition of different behavior modes of agricultural machinery. This paradigm can generalize image classification networks to the field-road trajectory classification task, providing a general vision model solution for agricultural machinery trajectory mode recognition. To validate the effectiveness of the ATRNet model, experiments were conducted on real corn and wheat harvester trajectory datasets. The results demonstrate that the proposed model achieves remarkable performance improvements over the state-of-the-art (SOTA) models. In the corn harvester trajectory dataset, ATRNet achieves an accuracy of 92.36% and an F1-score of 92.34%, surpassing existing SOTA models by 3.12% and 12.46%, respectively. Similarly, in the wheat harvester trajectory dataset, ATRNet achieves an accuracy of 92.36% and an F1-score of 92.33%, outperforming the existing optimal algorithm by 4.76% and 18.18%, respectively.

A Study on Alzhemier Disease in Takum, Taraba State, Nigeria: An ARIMA Model Approach

Alzheimer’s disease, a progressive neurodegenerative disorder, represents a significant and growing public health burden, particularly as life expectancy increases worldwide. In sub-Saharan Africa, including Nigeria, the disease's prevalence is rising due to aging populations and urbanized lifestyles that elevate risk factors for cognitive decline. This study investigates the trends and projected incidence of Alzheimer’s disease in Takum, Taraba State, Nigeria. The research utilizes patient records from General Hospital Takum, spanning from 2012 to 2021. Following diagnostic tests and data transformations, the ARIMA(1,2,0) model was selected as the best fit for predicting future case counts. The findings reveal a steady increase in Alzheimer’s cases, consistent with global patterns, highlighting the need for proactive measures in healthcare planning. The study thus recommends the need for increased public awareness, investment in diagnostic infrastructure, and support systems for caregivers.

Streamlining Patient Data Processing for Payer Compliance with Scalable Big Data Automation

In the fast-changing environment of value-based healthcare, providers must prepare and submit diagnosis files for patients to payers with ever-changing requirements from CMS for contracts. Changing specifications from payers repeatedly requires a flexible system to automatically process items with little reliance on human intervention. It provides a scalable data engineering framework that makes use of PySpark and Apache Airflow to automate and streamline the generation of patient diagnosis files. Its design allows it to be dynamically adapted to changes in specifications, which enhances both operational efficiency and compliance with the automation of complex data transformation processes and large-scale data processing. This framework not only optimizes the reimbursement process but also strengthens the foundations of value-based care by supporting improved patient outcomes and ensuring financial alignment with payer requirements through improving accuracy, timeliness, and reducing manual coding efforts. The proposed framework significantly enhances data processing speed, achieving a 50% reduction in processing time compared to the traditional ETL approach. It offers scalability improvements, allowing data handling from 1 TB to 5 TB—an increase of four times the previous limit—while minimizing manual effort through automation with Airflow, resulting in a 70% reduction in manual intervention.

A Novel Data Transformation for Improving Predictive Accuracy of Online Missing Value Imputation During Reefer Container Monitoring

A Novel Data Transformation for Improving Predictive Accuracy of Online Missing Value Imputation During Reefer Container Monitoring

The Role of Middleware in Integrating Legacy Systems with Modern Policy Administration

This comprehensive article examines the critical role of middleware integration in modernizing insurance industry legacy systems. The article reveals that approximately 75% of insurers rely on systems that are 20-40 years old, consuming up to 70% of IT budgets in maintenance. Through a detailed examination of implementation strategies, benefits, and challenges, the research demonstrates how middleware solutions serve as a crucial bridge between legacy infrastructure and modern Policy Administration Systems (PAS). The article covers key aspects, including protocol translation capabilities, service orchestration frameworks, and data transformation excellence, while providing quantitative metrics on cost reduction (20-30%), operational efficiency improvements (40-50%), and system performance enhancements (99.9% reliability). The article also presents a structured framework for implementing best practices, incorporating assessment, architecture design, security considerations, and testing strategies validated through industry case studies and expert analysis.

QLTI-07. COLLABORATIVE RESEARCH SUITE FOR GENOMIC ANALYSIS, DATA REPORTING AND WORKFLOW CUSTOMISATION

Abstract The genomic analysis pipeline involves multiple stages, including data extraction, transformation, and loading (ETL), exploratory data analysis (EDA), machine learning (ML) modeling, and reporting. The ETL stage involves data profiling and quality control measures to ensure data integrity and consistency. The EDA stage employs statistical and visualization methods to understand data distribution, identify patterns, and detect anomalies. Machine learning modelling involves supervised classification techniques, such as logistic regression and support vector machines (SVMs), to predict tumour classification and develop gene signatures. Unsupervised classification techniques, such as K-means clustering and Hierarchical clustering, identify patterns and relationships in genomic data. Quality metrics evaluate data completeness, accuracy, and consistency, ensuring data reliability and validity. The integration of data science techniques into a comprehensive framework can be achieved with the development of Application Dashboards that provide real-time insights into genomic data across every stage of the analysis pipeline. This enables clinical researchers and bioinformaticians to make informed decisions quickly, improving turn-around time and accelerating the discovery of new insights. In contrast to conventional methods of using stand-alone scripts, these dashboards offer a reusable and scalable solution, allowing for easy modification and adaptation to new datasets and analysis pipelines, thereby reducing development effort and increasing productivity. The dashboards constitute a centralized platform for data visualization, exploration, and analysis, facilitating collaboration and communication among research team members. With customizable and interactive visualizations, users can explore genomic data from multiple dimensions and the integration of multiple sources, including genomic and transcriptomic data. The technologies used to power this Research suite include Streamlit for Web App development, Python for data analysis, Opensearch for data storage and other utility libraries for reporting.

A stochastic algorithm for quantile regression models with fixed effects

In this paper, we present a stochastic algorithm for parameter estimation based on panel quantile regression model. We propose an easy-to-implement estimator based on the proposed algorithm. We profile the quantile-specific fixed effects as functions of the parameters of interest based on the Gaussian mixture representation of the asymmetric Laplace (AL) likelihood and eliminate the fixed effects through a data transformation. Parameters of interest can be estimated via quantile regression. Under a set of sufficient conditions, the proposed estimator is consistent and asymptotically normal when n and T both go to infinity. The proposed estimator is illustrated via both simulations and real data examples.

Collectively Working towards Plant-Pollinator Interactions Data Interoperability and Reuse: Lessons Learned from the WorldFAIR Project

The WorldFAIR project worked with eleven case studies to advance the implementation of the FAIR data principles (Findable, Accessible, Interoperable, and Reusable; Wilkinson et al. 2016). The Case Study on Agricultural Biodiversity (Drucker et al. 2022) addressed the challenges of advancing interoperability and mobilizing plant-pollinator interaction data for reuse. In the first phase of the project we performed a landscape analysis and found that a significant amount of data on plant-pollinator interaction was available as supplementary files of research articles, in a range of formats such as PDFs, Excel® spreadsheets, and text files. The diversity of approaches and the lack of appropriate data vocabularies led to confusion, information loss, and the need for complex data interpretation and transformation. In the second phase of the project, we performed six studies to adopt standards recommended in the first phase. Our approach for plant-pollinator data standardization was based on Darwin Core (Darwin Core Task Group 2009) and Ecological Data Language (EML, Jones et al. 2019), in conjunction with a data model and vocabulary proposed by the Brazilian Network of Plant-Pollinator Interactions (REBIPP). The studies underwent a process of “FAIRification” (i.e., transforming data into a format that adheres to the FAIR data principles) using the Global Biotic Interactions (GloBI, Poelen et al. 2014) platform. Two studies also explored the publishing model for Biotic Interactions as part of the GBIF New Data Model. The standardization strategy was successful and increased the interoperability of plant-pollinator interaction data, resulting in a process that allows tracking the provenance of the data, as well as facilitates the reuse of datasets (Drucker et al. 2024). This effort led to the development of "FAIR best practices" and guidelines for sharing plant-pollinator interaction data, as well as a FAIR assessment tool designed to help researchers and institutions evaluate adherence to the FAIR principles (Drucker et al. 2024). A tutorial to standardize plant-pollinator datasets (Gonzalez-Vaquero et al. 2024) and a cookbook with guidelines and recommendations for publishing agriculture-related plant-pollinator data (Salim et al. 2024) were developed to help different actors standardize plant-pollinator interaction data in accordance with the FAIR principles. Moving from diverse approaches and siloed initiatives to widely available FAIR plant-pollination interaction data for scientists and decision-makers will enable the development of integrative studies that enhance our understanding of species biology, behavior, ecology, phenology, and evolution.

Cloud-Native ETL: Integrating Databricks and Azure Data Factory for Scalable Data Processing in Enterprise Environments

This article examines the implementation of cloud-native ETL solutions leveraging Databricks and Azure Data Factory (ADF) for scalable data processing in enterprise environments. The article presents a comprehensive analysis of the architectural design, integration strategies, and performance optimization techniques for combining Databricks' powerful data transformation capabilities with ADF's robust workflow orchestration. Through a series of case studies and empirical evaluations, we demonstrate how this integrated approach addresses the challenges of big data processing, including scalability, flexibility, and cost-effectiveness. Our findings reveal significant improvements in processing efficiency and resource utilization, with observed reductions in ETL pipeline execution times by up to 40% and overall cloud infrastructure costs by 25%. The article also highlights best practices for data governance, security, and quality management within this framework. These insights provide valuable guidance for data engineers and IT professionals seeking to modernize their data processing infrastructure and harness the full potential of cloud-native ETL solutions.

Multi-Domain Data Integration for Plasma Diagnostics in Semiconductor Manufacturing Using Tri-CycleGAN

The precise monitoring of chemical reactions in plasma-based processes is crucial for advanced semiconductor manufacturing. This study integrates three diagnostic techniques—Optical Emission Spectroscopy (OES), Quadrupole Mass Spectrometry (QMS), and Time-of-Flight Mass Spectrometry (ToF-MS)—into a reactive ion etcher (RIE) system to analyze CF4-based plasma. To synchronize and integrate data from these different domains, we developed a Tri-CycleGAN model that utilizes three interconnected CycleGANs for bi-directional data transformation between OES, QMS, and ToF-MS. This configuration enables accurate mapping of data across domains, effectively compensating for the blind spots of individual diagnostic techniques. The model incorporates self-attention mechanisms to address temporal misalignments and a direct loss function to preserve fine-grained features, further enhancing data accuracy. Experimental results show that the Tri-CycleGAN model achieves high consistency in reconstructing plasma measurement data under various conditions. The model’s ability to fuse multi-domain diagnostic data offers a robust solution for plasma monitoring, potentially improving precision, yield, and process control in semiconductor manufacturing. This work lays a foundation for future applications of machine learning-based diagnostic integration in complex plasma environments.

M-Net based Stacked Autoencoder for Ransomware Detection using Blockchain Data

Ransomware is a kind of malevolent program software that encrypts the items on the hard disc and prevents the clients from accessing them until they are paid a ransom. Associations like monetary establishments and medical care areas (i.e., smart medical care) are mostly targeted by ransomware attacks. Ransomware assaults are crucial holes still in blockchain technology and prevent effective data communication in networks. This study aims to introduce an efficient system, named M-Net-based Stacked Autoencoder (M-Net_SA) for ransomware detection using blockchain data. Initially, the input data is taken from a dataset and then sent to the feature extraction process, which utilizes sequence-based statistical features. After that, data transformation is completed using the Yeo-Johnson transformation to transform the data into a usable format. After that, feature fusion is executed using a Deep Q-network (DQN) with Lorentzian similarity to enhance the representativeness of the target features. Finally, ransomware detection is accomplished by the proposed M-Net_SA, which is the integration of MobileNet and Deep Stacked Autoencoder (DSAE). The experimental validation of the proposed M-Net_SA is compared with other conventional techniques and the proposed model attained maximum accuracy, sensitivity, and specificity of 0.959, 0.967, and 0.957 respectively.

Volumetric Drag Coefficients for Generic Urban Configurations: Insights from Canopy Flow Analysis

Alternative drag approaches for representing unresolved buildings were proposed in literature for computational fluid dynamics (CFD) simulation of macroscopic urban airflow. As a contribution, the present work derives the volumetric drag coefficient (Cd*) through canopy drag and velocity analysis and provides appropriate correlations for Cd*against urban morphological parameters. A total of 72 cases across various urban configurations are investigated, categorized by building typology, horizontal layout, height variability, and plan area density (λp, from 0.0625 to 0.57). Reynolds-Averaged Navier-Stokes (RANS) simulations with periodic boundary conditions are performed to model fully developed flows. Results for the normalized drag force and superficial velocity and their relations with λp are evaluated. Subsequent evaluation of the profiles for the sectional coefficients (Cd*(Z)) reveals four distinct types with variations in uniform-height cases and combinations in varying-height cases. A throughout correlations analysis, facilitated by data transformation, identifies the straightforward relations between Cd* and frontal area density (λf) and tortuosity (τ). The followed stepwise regression provides a recommended formula for Cd*, demonstrating a proper fit with the simulated values. These findings facilitate the understanding and appropriate estimation of Cd* and Cd*(Z), promoting the application of macroscopic turbulence models, for neighborhood-scale wind and air quality studies.

Comparing soil colour using colour chart and colour app: A semi-quantitative approach with data transformation Using pedometric methods

Comparing soil colour using colour chart and colour app: A semi-quantitative approach with data transformation Using pedometric methods

Translation, Reliability and Development of a Calculator for the Brazilian Portuguese Version of the Family Empowerment Scale (FES) in Caregivers of Individuals With Cerebral Palsy.

Care for patients with cerebral palsy (CP) poses significant challenges, with emphasis on the crucial role of family involvement in patients' well-being. The Family Empowerment Scale (FES) assesses parents/caregivers' perception of their ability to promote and influence their children's development in three domains: family, service system and community. The objectives of this study were to translate the FES into Brazilian Portuguese, to develop a calculator for data transformation and analysis and to evaluate the psychometric properties of the scale with Brazilian caregivers of children and adolescents with CP. Translation occurred in four stages: translation, synthesis, back-translation and review of the translated version. A calculator was then developed in the Excel program, to present data in absolute values and graphically for each subscale. Statistical analysis included Bland-Altman, Intraclass Correlation Coefficient (ICC), Internal Consistency, and Floor and Ceiling Effect. In total, 50 parents and caregivers participated to the measurement properties investigation, and the scale demonstrated good reliability for the family (ICC = 0.80) and services (ICC = 0.81) subscales, as well as for the overall scale (ICC = 0.88). The community subscale showed moderate reliability (ICC = 0.68), with an adequate Cronbach's alpha (0.89). The FES showed no ceiling or floor effects and a small limit of agreement. The data of all participants were analysed using the calculator. The FES-Brazilian Portuguese presented good reliability for assessing family empowerment in caregivers of children and adolescents with CP in Brazil, offering insights into this construct and guiding family-centred interventions. In addition, the calculator could be viable for adequate management and dissemination of data.

Vibration Signal Analysis for Intelligent Rotating Machinery Diagnosis and Prognosis: A Comprehensive Systematic Literature Review

Many industrial processes, from manufacturing to food processing, incorporate rotating elements as principal components in their production chain. Failure of these components often leads to costly downtime and potential safety risks, further emphasizing the importance of monitoring their health state. Vibration signal analysis is now a common approach for this purpose, as it provides useful information related to the dynamic behavior of machines. This research aimed to conduct a comprehensive examination of the current methodologies employed in the stages of vibration signal analysis, which encompass preprocessing, processing, and post-processing phases, ultimately leading to the application of Artificial Intelligence-based diagnostics and prognostics. An extensive search was conducted in various databases, including ScienceDirect, IEEE, MDPI, Springer, and Google Scholar, from 2020 to early 2024 following the PRISMA guidelines. Articles that aligned with at least one of the targeted topics cited above and provided unique methods and explicit results qualified for retention, while those that were redundant or did not meet the established inclusion criteria were excluded. Subsequently, 270 articles were selected from an initial pool of 338. The review results highlighted several deficiencies in the preprocessing step and the experimental validation, with implementation rates of 15.41% and 10.15%, respectively, in the selected prototype studies. Examination of the processing phase revealed that time scale decomposition methods have become essential for accurate analysis of vibration signals, as they facilitate the extraction of complex information that remains obscured in the original, undecomposed signals. Combining such methods with time–frequency analysis methods was shown to be an ideal combination for information extraction. In the context of fault detection, support vector machines (SVMs), convolutional neural networks (CNNs), Long Short-Term Memory (LSTM) networks, k-nearest neighbors (KNN), and random forests have been identified as the five most frequently employed algorithms. Meanwhile, transformer-based models are emerging as a promising venue for the prediction of RUL values, along with data transformation. Given the conclusions drawn, future researchers are urged to investigate the interpretability and integration of the diagnosis and prognosis models developed with the aim of applying them in real-time industrial contexts. Furthermore, there is a need for experimental studies to disclose the preprocessing details for datasets and the operational conditions of the machinery, thereby improving the data reproducibility. Another area that warrants further investigation is differentiation of the various types of fault information present in vibration signals obtained from bearings, as the defect information from the overall system is embedded within these signals.

Guide wire Insurance suite: Integration and Deployment Strategies

This article explores two critical aspects of Guidewire InsuranceSuite that are pivotal for insurance companies navigating digital transformation: the Integration Gateway and Blue/Green deployment. The Integration Gateway serves as a powerful middleware solution, facilitating seamless communication between Guidewire's core applications and external systems, thereby enabling insurers to create a more connected and responsive technology ecosystem. Blue/Green deployment, on the other hand, is presented as an advanced release management technique that minimizes downtime and reduces risks associated with software updates. The article discusses the key features of the Integration Gateway, including standardized interfaces, data transformation capabilities, and enhanced security measures. It also delves into the Blue/Green deployment process, highlighting its benefits such as minimal downtime, reduced risk, improved testing, and easy rollback. By examining these technologies, the article aims to provide insurance professionals and technology leaders with insights into how Guidewire InsuranceSuite can be leveraged to create more agile, efficient, and resilient insurance operations in an increasingly digital marketplace.

A data-driven operations management model: Implementing MIS for strategic decision making in tech businesses

In today’s rapidly evolving technological landscape, data-driven decision-making is critical for business growth and competitiveness. This paper presents a comprehensive operations management model that integrates Management Information Systems (MIS) to facilitate strategic decision-making in technology firms. The proposed model emphasizes the transformation of raw data into actionable insights, enabling organizations to optimize their operations and respond effectively to market dynamics. By systematically incorporating MIS into operational frameworks, the model provides a structured approach to data collection, analysis, and dissemination. It identifies key performance indicators (KPIs) that are essential for monitoring business performance and aligning operational activities with strategic objectives. Through case studies of leading tech companies, the model illustrates how data-driven methodologies can lead to improved forecasting, resource allocation, and overall operational efficiency. The framework is designed to be adaptable, allowing technology firms to customize the MIS components based on their unique business needs and market conditions. Key features of the model include real-time data analytics, predictive modeling, and user-friendly dashboards that enhance visibility across departments. These components empower decision-makers with timely and relevant information, fostering a culture of informed decision-making at all organizational levels. Moreover, the model addresses common challenges faced by tech businesses, such as data silos and integration issues. By promoting a centralized data repository and standardized reporting mechanisms, it enhances collaboration among teams and ensures consistency in data interpretation. The implications of this model extend beyond operational efficiency, influencing strategic initiatives such as product development, market expansion, and customer engagement. In conclusion, this data-driven operations management model not only supports strategic decision-making in technology firms but also positions them for sustainable growth in a competitive marketplace. By leveraging MIS effectively, organizations can harness the power of data to drive innovation and enhance overall business performance.

Federated Learning's Dynamic Defense Against Byzantine Attacks: Integrating SIFT-Wavelet and Differential Privacy for Byzantine Grade Levels Detection

Federated learning, which enables decentralized training across multiple devices while maintaining data privacy, is susceptible to Byzantine poisoning attacks. This paradigm reduces the need for centralized data storage and transmission, thereby mitigating privacy risks associated with traditional data aggregation methods. However, FL introduces new challenges, notably susceptibility to Byzantine poisoning attacks, where rogue participants can tamper with model updates, threatening the consistency and security of the aggregated model. Our approach addresses this vulnerability by implementing robust aggregation methods, sophisticated pre-processing techniques, and a novel Byzantine grade-level detection mechanism. We introduce a federated aggregation operator designed to mitigate the impact of malicious clients. Our pre-processing includes data loading and transformation, data augmentation, and feature extraction using SIFT and wavelet transforms. Additionally, we employ differential privacy and model compression to improve the robustness and performance of the federated learning framework. Our approach is assessed using a tailored neural network model applied to the MNIST dataset, achieving 97% accuracy in detecting Byzantine attacks. Our results demonstrate that robust aggregation significantly improves the resilience and performance. This comprehensive approach ensures the integrity of the federated learning process, effectively filtering out adversarial influences and sustaining high accuracy even when faced with adversarial Byzantine clients.

Ai-Driven Big Data Transformation And Personally Identifiable Information Security In Financial Data: A Systematic Review

This systematic review explores the impact of adopting artificial intelligence (AI) to analyze and transform big data in financial and economic contexts, with a specific focus on the privacy and security of personally identifiable information (PII). By examining 37 articles spanning the latest advancements in AI-driven big data technologies, the review identifies both opportunities and challenges in safeguarding PII during financial data transformation. Key findings reveal that while AI enhances data processing capabilities—enabling faster insights and predictive accuracy in economic trends—PII faces increased risks due to sophisticated data aggregation and correlation techniques. The review categorizes major AI methodologies used, including machine learning algorithms, natural language processing, and predictive analytics, highlighting how each can affect data privacy. The findings suggest that, to maintain trust, organizations must adopt AI responsibly, integrating privacy-by-design principles and adhering to data governance standards. This review contributes to a clearer understanding of the interplay between AI and PII protection, offering practical insights for stakeholders in the financial sector aiming to harness AI while prioritizing ethical data handling.

Pengaruh Firm Size, Audit Fee, Audit Tenure, Kompetensi dan Independensi Auditor Terhadap Kualitas Audit pada Perusahaan Perbankan yang Terdaftar di Bursa Pengaruh Indonesia 2018-2022

This research aims to determine the influence of firm size, audit fees, audit tenure, auditor competency and independence on audit quality in banking companies listed on the Indonesia Stock Exchange for the 2018-2022 period. This research population used a purposive sampling technique to obtain a sample of 150 data from 30 companies, but data transformation and outliers were applied so that the sample size became 120 data. The data analysis applied in this research is classical assumption testing, multiple linear regression analysis, and hypothesis testing. The research results show that simultaneously there is an influence of the variables firm size, audit fee, audit tenure, competence and independence of the auditor on audit quality, but partially the variables of competence and independence of the auditor do not have a significant effect on the quality of audits in banking companies listed on the Indonesia Stock Exchange for the period 2018-2022.