Complex Data Processing Research Articles

Application of perceptual graph neural network intelligent algorithm in software testing platform

In the research on algorithm application in software testing platform, the traditional perceptual graph neural network (PGNN) intelligent algorithm has high computational complexity when processing large-scale and complex software testing data. At the same time, it lacks sufficient ability to automatically select and optimize the code’s syntax structure, execution path, test coverage and other features, resulting in poor test case generation effect. This paper constructed an improved PGNN model, which enhanced the modeling ability of complex code structures and execution paths by adding graph convolution layers and introducing attention mechanisms. It also used PGNNs to integrate and optimize features such as grammatical structure, execution path, and test coverage in software codes. Matrix decomposition and sparsification techniques are used to optimize the computational complexity of the PGNN model and achieve rapid processing of large-scale software test data. Based on the improved PGNN model, training is performed on the software test data set, and the model is tuned in combination with a variety of optimization algorithms to improve the overall performance of the test platform. The experimental results show that in terms of the effectiveness of test case generation, the coverage rate, error detection rate, missed rate and false positive rate are 0.87, 0.92, 0.01 and 0.02 respectively.

Machine Learning Algorithms for Intrusion Detection in WSNs

Machine learning (ML) has emerged as a potent tool for optimizing network performance and extending network lifespans. By automating complex data processing tasks, ML algorithms enable real-time solutions that optimize resource allocation and utilization. ML's ability to process vast, intricate datasets with speed and precision empowers networks to adapt dynamically to evolving demands. [13]. Wireless Sensor Networks (WSNs), composed of interconnected sensor and sink nodes, exemplify the transformative potential of ML. These distributed, decentralized networks inherently possess self-organization and self-healing capabilities. By integrating ML techniques, WSNs can significantly enhance their efficiency, reliability, and scalability, enabling a wide array of applications, from environmental monitoring to military operations. As advancements in electronics and wireless communication continue to drive WSN evolution, the fusion of ML and WSNs promises to unlock new horizons in network intelligence and performance. However, WSNs face several challenges, including resource constraints such as limited memory, processing power, and energy. Additionally, the physical infrastructure of WSNs must be secured to protect sensitive data, especially in privacy-critical applications. To address these challenges, the integration of machine learning offers a promising solution. By leveraging ML techniques, WSNs can adapt to dynamic environments, optimize energy consumption, and detect and mitigate potential security threats. This enables the deployment of WSNs in diverse applications, from environmental monitoring to smart cities, while ensuring data privacy and system security. Keywords: Intrusion Detection System (IDS), Security, Wireless Sensor Network (WSN), Attacks, Reinforcement Learning (RL), Denial-of-Service (DoS), Networks, Machine Learning (ML)

Machinery fault diagnosis based on a transfer learning model with local sparse structure of PMDCNN-LSTM

Aiming at the fault diagnosis problem in practical applications where the performance of the model degrades when applied to new domains or tasks due to limited and approximate training data, a transfer learning (TL) model with a local sparse structure based on a parallel multichannel convolutional neural network (CNN) and a long-short-term memory network (LSTM) (PMDCNN-LSTM) is proposed. The model is combined with CNN, LSTM, and TL to achieve parameter sharing by training learning parameters in the source domain and transferring them to the target domain. At the same time, the model is transformed from training data to real-world applications by integrating samples from the target domain, which improves generalization while maintaining the ability to identify faults in the training set. The advantages of CNN and LSTM are combined in this model compared with the traditional TL model, which can efficiently process data containing spatio-temporal features while maintaining sensitivity to time-series variations, exhibiting higher performance and advantages in complex data processing. It is shown by the experimental results that excellent performance and generalization ability can still be maintained by the model in high-noise environments, which is considered to be of great significance for research on bearing and gear fault diagnosis.

Assessing river discharge dynamics through relative surface water extent changes in river basins.

Surface water in rivers is vital for human society. However, our current understanding of the dynamics and drivers of river flows relies predominantly on stream gauging data, which are limited in spatial coverage and involve significant costs. Remote sensing techniques have emerged as complementary tools for monitoring river discharge, but these satellite-based methods often require complex data processing. This paper introduces a simple, yet effective and robust methodology to understand river discharge, using Australian rivers as a case study. Our findings reveal that changes in relative surface water extent within river basins can effectively capture river discharge dynamics. Moreover, we observe a linear relationship between relative surface water extent and river discharge in hilly basins and a quadratic relationship in flat basins. The new approach helps monitor river flows in ungauged river basins, and it has the potential to bridge the gap between local and regional understandings of water dynamics.

Effective data visualization strategies in untargeted metabolomics.

Covering: 2014 to 2023 for metabolomics, 2002 to 2023 for information visualizationLC-MS/MS-based untargeted metabolomics is a rapidly developing research field spawning increasing numbers of computational metabolomics tools assisting researchers with their complex data processing, analysis, and interpretation tasks. In this article, we review the entire untargeted metabolomics workflow from the perspective of information visualization, visual analytics and visual data integration. Data visualization is a crucial step at every stage of the metabolomics workflow, where it provides core components of data inspection, evaluation, and sharing capabilities. However, due to the large number of available data analysis tools and corresponding visualization components, it is hard for both users and developers to get an overview of what is already available and which tools are suitable for their analysis. In addition, there is little cross-pollination between the fields of data visualization and metabolomics, leaving visual tools to be designed in a secondary and mostly ad hoc fashion. With this review, we aim to bridge the gap between the fields of untargeted metabolomics and data visualization. First, we introduce data visualization to the untargeted metabolomics field as a topic worthy of its own dedicated research, and provide a primer on cutting-edge visualization research into data visualization for both researchers as well as developers active in metabolomics. We extend this primer with a discussion of best practices for data visualization as they have emerged from data visualization studies. Second, we provide a practical roadmap to the visual tool landscape and its use within the untargeted metabolomics field. Here, for several computational analysis stages within the untargeted metabolomics workflow, we provide an overview of commonly used visual strategies with practical examples. In this context, we will also outline promising areas for further research and development. We end the review with a set of recommendations for developers and users on how to make the best use of visualizations for more effective and transparent communication of results.

UNCA: A Neutrosophic-Based Framework for Robust Clustering and Enhanced Data Interpretation

Accurately representing the complex linkages and inherent uncertainties included in huge datasets is still a major difficulty in the field of data clustering. We address these issues with our proposed Unified Neutrosophic Clustering Algorithm (UNCA), which combines a multifaceted strategy with Neutrosophic logic to improve clustering performance. UNCA starts with a full-fledged similarity examination via a λ-cutting matrix that filters meaningful relationships between each two points of data. Then, we initialize centroids for Neutrosophic K-Means clustering, where the membership values are based on their degrees of truth, indeterminacy and falsity. The algorithm then integrates with a dynamic network visualization and MST (Minimum Spanning Tree) so that a visual interpretation of the relationships between the clusters can be clearly represented. UNCA employs Single-Valued Neutrosophic Sets (SVNSs) to refine cluster assignments, and after fuzzifying similarity measures, guarantees a precise clustering result. The final step involves solidifying the clustering results through defuzzification methods, offering definitive cluster assignments. According to the performance evaluation results, UNCA outperforms conventional approaches in several metrics: it achieved a Silhouette Score of 0.89 on the Iris Dataset, a Davies-Bouldin Index of 0.59 on the Wine Dataset, an Adjusted Rand Index (ARI) of 0.76 on the Digits Dataset, and a Normalized Mutual Information (NMI) of 0.80 on the Customer Segmentation Dataset. These results demonstrate how UNCA enhances interpretability and resilience in addition to improving clustering accuracy when contrasted with Fuzzy C-Means (FCM), Neutrosophic C-Means (NCM), as well as Kernel Neutrosophic C-Means (KNCM). This makes UNCA a useful tool for complex data processing tasks.

High‐Accuracy Recognition Triboelectric Nanogenerator System for Shooting Report and Ballistic Analysis

AbstractThe accuracy and efficiency of shooting reports constitute indispensable cornerstones in modern military training and shooting fields. However, traditional shooting report systems struggle to offer precise and sensitive instant feedback, especially in rare scenarios, such as multiple piercing and shots on the ring boundaries. Simultaneously, ballistic analysis as a pivotal approach in criminal investigations, confronts similar challenges including high costs, complex data processing, and strong professional dependency. Thus, a flexible, high‐temperature resistant, and high‐accuracy recognition intelligent target reporting and ballistic analysis (ITBA) system is designed based on the mechanism of triboelectric nanogenerator (TENG), which recognizes multiple bullet piercings and delivers bullet velocity through a time‐difference algorithm combined with shooting angle to accurately determine the shooting position. Thanks to the novel structural design and preparation method, the ITBA system achieves remarkable integration, lightweight, and industrialization, ensuring exceptional stability even in harsh environments. Furthermore, the ITBA system integrates machine learning techniques to achieve 100% accuracy in bullet material recognition and 97% accuracy in determining shooting angles, providing a solid foundation for enhancing the precision of military training and the scientific rigor of forensic investigations.

Automating Data Pipelines with AI for Scalable, Real-Time Process Optimization in the Cloud

Modern data processing environments demand efficient, scalable solutions for handling massive data streams in real-time, yet traditional Extract, Transform, Load (ETL) pipelines face significant limitations in processing speed and adaptability. This article presents an AI-Enhanced Cloud Data Pipeline (AECDP) framework that combines Deep Learning-based Stream Processing (DLSP) with Adaptive Resource Management (ARM) for real-time data optimization. The framework introduces novel algorithms for stream processing, resource allocation, and quality assurance, including the Adaptive Stream Processing Algorithm (ASPA) and Anomaly Detection and Correction (ADC) system. The implementation utilizes a multi-cloud architecture with containerized microservices, enabling independent scaling and maintenance of pipeline components. Experimental results demonstrate the framework's effectiveness across various industry applications, including e-commerce, financial services, and manufacturing sectors. The system achieves consistent sub-second latency for real-time processing, linear throughput scaling, and optimal resource utilization across cloud instances. Additionally, the framework incorporates advanced security features and automated quality monitoring systems, ensuring robust and reliable data processing. The AECDP framework represents a significant advancement in data pipeline automation, providing organizations with a comprehensive solution for managing complex data processing requirements while maintaining high performance and reliability standards.

Deep learning enhanced quantum holography with undetected photons

Holography is an essential technique of generating three-dimensional images. Recently, quantum holography with undetected photons (QHUP) has emerged as a groundbreaking method capable of capturing complex amplitude images. Despite its potential, the practical application of QHUP has been limited by susceptibility to phase disturbances, low interference visibility, and limited spatial resolution. Deep learning, recognized for its ability in processing complex data, holds significant promise in addressing these challenges. In this report, we present an ample advancement in QHUP achieved by harnessing the power of deep learning to extract images from single-shot holograms, resulting in vastly reduced noise and distortion, alongside a notable enhancement in spatial resolution. The proposed and demonstrated deep learning QHUP (DL-QHUP) methodology offers a transformative solution by delivering high-speed imaging, improved spatial resolution, and superior noise resilience, making it suitable for diverse applications across an array of research fields stretching from biomedical imaging to remote sensing. DL-QHUP signifies a crucial leap forward in the realm of holography, demonstrating its immense potential to revolutionize imaging capabilities and pave the way for advancements in various scientific disciplines. The integration of DL-QHUP promises to unlock new possibilities in imaging applications, transcending existing limitations and offering unparalleled performance in challenging environments.

Hardware-Assisted Low-Latency NPU Virtualization Method for Multi-Sensor AI Systems.

Recently, AI systems such as autonomous driving and smart homes have become integral to daily life. Intelligent multi-sensors, once limited to single data types, now process complex text and image data, demanding faster and more accurate processing. While integrating NPUs and sensors has improved processing speed and accuracy, challenges like low resource utilization and long memory latency remain. This study proposes a method to reduce processing time and improve resource utilization by virtualizing NPUs to simultaneously handle multiple deep-learning models, leveraging a hardware scheduler and data prefetching techniques. Experiments with 30,000 SA resources showed that the hardware scheduler reduced memory cycles by over 10% across all models, with reductions of 30% for NCF and 70% for DLRM. The hardware scheduler effectively minimized memory latency and idle NPU resources in resource-constrained environments with frequent context switching. This approach is particularly valuable for real-time applications like autonomous driving, enabling smooth transitions between tasks such as object detection and route planning. It also enhances multitasking in smart homes by reducing latency when managing diverse data streams. The proposed system is well suited for resource-constrained environments that demand efficient multitasking and low-latency processing.

AI-Driven WSN for Precise Aquatic Pollution Detection Using an Intelligent Monitoring Approach

The proliferation of digital devices, sensors, and interconnected systems has led to an explosion of data. Simple sensors like pH, conductivity, and Turbidity sensors can be used for the classification of gasoline and diesel in water. These sensors are easy to set up and deploy, so they can be installed in vast numbers and can get real-time data from the site. FPGAs are very fast in processing complex data and have low latency time compared to other traditional microcontrollers. But FPGA accepts coding in Hardware Description Languages like Verilog or VHDL, which can be very complex to code complex models that are trained in other high-level languages like Python and C++. Simple classification models in machine learning are implemented using High-Level Synthesis tools, which accept codes written in languages like C, C++, or SystemC, and translate them into hardware-implementable RTL (Register- Transfer Level) code. The data from two major components of oil, gasoline, and petrol are used to train various classification models with widely used libraries in Python. The trained parameters are extracted from the trained model. The parameters are then assembled and then coded in C++ as currently most of the tools support C++. Some modifications need to be made to the original code to make it compatible with the synthesis tool.

ANALISIS EFEKTIFITAS VISUALISASI PERMODELAN DESAIN ARSITEKTUR BERBASIS KECERDASAN BUATAN (AI)

This research focuses on the iteration of architectural modeling using artificial intelligence (AI) as a solution to generate visualization concepts in the architectural iteration process. It examines the development of AI in architecture and highlights AI's position in current architectural practice. AI has revolutionized the way architectural designs are developed, presented, and interpreted. With its ability to process complex data, AI accelerates the design process, enables the exploration of innovative solutions, and enhances understanding of projects before construction begins. AI is used in various applications, from automating visualizations to generating floor plans, opening new potentials in designing and visualizing architectural projects. However, the use of AI in architecture is still in its early stages, and challenges remain in applying AI to conceptual design stages. This research will analyze the effectiveness of AI in architectural modeling and design visualization, evaluating its impact on efficiency, creativity, and design output quality, and exploring limitations and future recommendations for AI use in architectural practice. Based on platform evaluations, MidJourney and DALL·E show superiority in visual quality and design accuracy, each with the highest scores. MidJourney and DALL·E are the top choices for creative and technical architectural visualization, while Leonardo AI and Bing AI show limitations in creativity and flexibility.

Implementasi Deep Learning Untuk Menentukan Harga Buah Sawit

This study aims to analyze the price of palm oil using Convolutional Neural Network (CNN) method in deep learning. CNN was chosen for its ability to process complex data and recognize patterns from diverse data. The stages of research include data analysis, data pre-processing, predictive model design for CNN method, CNN classification model prediction results, CNN method evaluation, and CNN method evaluation results. This study aims to produce a model that can predict the price of oil palm with high accuracy, based on data covering a variety of characteristics of farmers and the quality of oil palm crops. Prediction results were conducted using data from 50 oil palm farmers. From the prediction, as many as 23 data farmers get a price of IDR 2,300, 13 other farmers get a price of IDR 2,000, and the remaining 14 data farmers get a price of IDR 1,800. The results of this prediction are based on data from farmers and the quality of oil palm crops they grow and produce. By utilizing the CNN method, the model can capture various factors that affect the price of palm oil, including the quality of palm fruit and agricultural conditions. Evaluation of the CNN method showed very good results, with almost perfect accuracy. This method managed to predict palm oil prices very precisely, showing that CNN can be an effective tool in the analysis of palm oil prices. The results of this evaluation confirmed that the CNN method can be relied upon to provide accurate predictions, helping farmers and palm oil industry players in determining prices that are in accordance with the quality and condition of the crop.

Integration of Active Learning Techniques for Effective Delivery of Machine Learning Course

Abstract — The field of machine learning (ML) is rapidly advancing and has important implications across a wide variety of industries, spanning from the healthcare industry to the financial sector and beyond. Computers are given the ability to gain information from data and make projections or conclusions through the process of machine learning, which is comprised of complex algorithms, statistical techniques, and data processing procedures. Given the complex nature of machine learning and the fact that it is always evolving, providing it with the appropriate instruction may be a challenging endeavor. In the field of machine learning, the conventional educational methods often consist of theoretical lectures, textbooks, and examples that are predetermined. Despite the fact that these methods provide a fundamental understanding of the concepts underlying machine learning, they typically fail to successfully integrate the theoretical knowledge with the execution of those concepts in the actual world. The fact that this gap exists is a topic of significant concern, given that machine learning is primarily utilized to handle practical concerns and to enhance decision-making that is based on data. It is essential to employ effective instructional and learning strategies such as project-based learning (PBL), flipped classrooms, and case studies-based learning in order to address these challenges and ensure that students are adequately prepared to pursue careers in the field of machine learning. Through an examination of several ML course delivery strategies; this study seeks to address the urgent demand for better ML education. These strategies have the potential to improve the learning outcomes of students by bridging the gap between theoretical comprehension and practical application, as this article demonstrates. Through this exploration, we seek to identify and advocate for teaching practices that can better prepare students for the dynamic and evolving landscape of machine learning. Keywords— Project-Based Learning (PBL); Flipped Classroom; Case Studies-based Learning

Advanced KPI framework for IVF pregnancy prediction models in IVF protocols

The utilization of neural networks in assisted reproductive technology is essential due to their ability to process complex and multidimensional data inherent in IVF procedures, offering opportunities for clinical outcome prediction, personalized treatment implementation, and overall advancement in fertility treatment. The aim of this study was to develop a novel approach to IVF laboratory data analysis, employing deep neural networks to predict the likelihood of clinical pregnancy occurrence within an individual treatment cycle, integrating both key performance indicators and clinical data. We conducted a retrospective analysis spanning 11 years, encompassing 8732 treatment cycles, to extract the most relevant features to our goal and train the model. Internal validation was performed on 1600 preimplantation genetic testing for aneuploidy embryo transfers, while external was conducted across two independent clinics (over 10,000 cases). Leveraging recurrent neural networks, our model demonstrates high accuracy in predicting the likelihood of clinical pregnancy within specific IVF treatment cycles (AUC = 0.68–0.86; test accuracy = 0.78, F1 score = 0.71, sensitivity = 0.62; specificity = 0.86) comparable to time-lapse system but with a simpler approach. Our model facilitates both retrospective analysis of outcomes and prospective evaluation of clinical pregnancy chances, thus presenting a promising avenue for quality management programs and promotes their realization in medical centers.

Image enhancement algorithm combining histogram equalization and bilateral filtering

In the process of image acquisition, transmission, and storage, the image quality is often degraded due to a variety of unfavorable factors, resulting in information loss, which poses certain difficulties for subsequent image processing and analysis. How to enhance the visibility of image details and maintain the naturalness of the image is one of the important challenges in image processing. In response to this challenge, an image enhancement algorithm is proposed based on the advantages of histogram equalization and bilateral filtering. This algorithm organically integrates histogram equalization and bilateral filtering, aiming to improve image quality while reducing noise in the image. Specifically, the study first utilizes an improved histogram equalization strategy to preprocess the image and then applies a bilateral filter for further optimization. The experimental results showed that the optimized histogram equalization could effectively improve the global contrast of the image and avoid excessive enhancement and gray phenomenon of the image. Moreover, its peak signal-to-noise ratio could reach 0.71. However, bilateral filters showed significant advantages in processing complex data sets, and the peak signal-to-noise ratio could reach 0.95. It illustrated that the optimal research method has obvious advantages in improving image quality and reducing noise. The new enhancement strategy not only significantly improves the global contrast of the image but also preserves the naturalness of the image, providing important technical support for image analysis, machine vision, and artificial intelligence applications.

On the design of a scaled rotor for a tethered tidal converter to deploy at sea

Abstract Model scale testing is vital for developing technologies like tidal turbines. Reduced-scale tests follow numerical design and are cost-effective in controlled environments, allowing parameter adjustments. However, controlled tests face drawbacks, such as blockage effects and the inability to replicate real sea conditions. Sea trials, conducted in actual marine environments, offer realistic data and continuous collection, potentially being cost-effective despite environmental variability and complex data processing. This work aims to measure real sea effects, such as marine growth and stream turbulence, on fixed and moving parts, of a scaled model of GEMSTAR tidal stream energy converter, a twin rotors submerged tethered device, designed to tap tidal current energy. The planned model installation site is at the Renewable Marine Energy Laboratory, a natural facility located on the East coast of the Strait of Messina (southern Italy), characterized by tidal currents with a maximum speed of about 1.25 m/s. Due to the low stream speed, in order to get significant data, an up-scaling of the existing model rotor diameter has been necessary, and a specific new blade design has been performed, in order to lower the cut-in speed, increasing in this way the useful power production time of the model. The paper illustrates the different phases of model scale design.

Weight consistency and cluster diversity based concept factorization for multi-view clustering

In the era of information explosion, clustering analysis of multi-view data plays a crucial role in revealing the intrinsic structures of data. Despite the advancements in existing multi-view clustering methods for processing complex data, they often overlook the weight differences among various views and the diversity between clusters. To address the issues, the paper introduces a novel multi-view clustering approach termed weight consistency and cluster diversity based concept factorization for multi-view clustering (MVCF-WD). Specifically, the proposed method automatically learns the weights of the views, and incorporates a cluster diversity term to enhance the discriminability of clusters. Furthermore, to solve the formulated optimization model, an iterative optimization algorithm based on multiplication rules is developed and the convergence is analyzed. Extensive experiments conducted across seven datasets compared with ten state-of-the-art clustering algorithms demonstrate the superior clustering performance of the proposed method.

India’s Forthcoming Rules under the Digital Personal Data Protection Act: An Opportunity to Reduce Gaps in the ‘Notice and Consent’ Framework for Cookies

‘Notice and consent’ is a predominant requirement in data protection ecosystems. This requirement has been recognized by the Digital Personal Data Protection (DPDP) Act, 2023 in India. However, there are several reasons why the notice and consent framework could operate sub-optimally. These are reasons because of the ecosystem in which such a framework functions. For instance, the standard operational practices falling under the ‘notice and consent’ framework do not offer real choices to individuals, they do not succinctly capture complex data processing, and they do not capture the psychosocial aspects of individual decision-making. These problems are amplified in India, which has high digital illiteracy and low privacy awareness. This article suggests that the forthcoming rules under the DPDP Act are a means to address the operational problems that affect the notice and consent framework in India. The article takes cookies as an example to suggest the norms that are crucial to consider while implementing an optimal framework.

The Application of Machine Learning and Intelligent Sensors for Real-Time Air Quality Monitoring: A Literature Review

Air pollution is a global issue that has major consequences for human health and the environment. Accurate air quality prediction plays an important role in mitigating and preventing the negative impacts of air pollution. The thirteen sources analyzed in this literature study show a growing trend in the use of machine learning for air quality prediction, driven by the limitations of traditional methods and machine learning capabilities in efficiently processing complex data. This literature study examines a variety of commonly used machine learning models, such as Support Vector Regression (SVR), Random Forest, Gradient Boosting, and Long Short-Term Memory (LSTM), and evaluates their performance based on metrics such as RMSE, MAE, and R². The sources also highlight the importance of understanding the factors that affect air quality, including concentrations of various pollutants (PM2.5, PM10, NO2, CO, SO2, and ozone), meteorological data (temperature, humidity, wind speed, air pressure, precipitation, and temperature inversion), traffic data, and spatial-temporal variations. The integration of the Internet of Things (IoT) and machine learning is the main focus in the development of real-time air quality monitoring systems. IoT sensors enable the collection of real-time air quality and meteorological data, which are then processed using machine learning models to generate predictions. This literature study identifies several challenges in air quality prediction, such as data limitations, the complexity of air pollution dynamics, and ethical & privacy considerations. However, machine learning offers great potential to improve the accuracy of air quality predictions and monitoring, thus contributing to a healthier and more sustainable environment.