Heterogeneous Data Processing Research Articles

Construction of inherent geometric pattern for equal-area hexagonal discrete global grid systems

ABSTRACT Discrete Global Grid Systems (DGGSs) are hierarchical frameworks that provide seamless global coverage and enable the efficient processing and analysis of heterogeneous geospatial data. As cell resolution becomes finer, similarities in cell attributes, such as cell size and shape, increase. Understanding these shared characteristics enhances our knowledge of DGGSs structures, improving their reliability and accuracy across a range of applications. However, existing methods for quantifying these characteristics are limited, often emphasizing overall analysis and visualization rather than detailed assessment. This paper introduces a geometric pattern quantification method for equal-area hexagonal DGGSs by calculating inter-level attribute similarities to identify a reference level, segmenting the spatial distribution image at this reference level, and fitting the boundaries of these sub-images to create iso-feature lines. The results show that this method improves the accuracy and efficiency of cell attribute calculations, with enhancements of at least 5.6 times for study regions and 13.8 times for sample points at lower levels. This pattern provides a robust basis for grid selection and optimization in regional applications, promoting the broader adoption of DGGSs across various fields.

HMC-FHE: A Heterogeneous Near Data Processing Framework for Homomorphic Encryption

HMC-FHE: A Heterogeneous Near Data Processing Framework for Homomorphic Encryption

Diminished functional segregation and resilience are associated with symptomatic severity and cognitive impairments in schizophrenia: a large-scale study

BackgroundThe research findings on the topological properties of functional connectomes (TP-FCs) in patients with schizophrenia (SZPs) exhibit inconsistencies and contradictions, which can be attributed to limitations such as small sample...

Development of a method for increasing the efficiency of processing heterogeneous data using a metaheuristic algorithm

The problems of processing heterogeneous data are discontinuous, undifferentiated, and multimodal. The most common approaches to processing heterogeneous data are swarm intelligence algorithms (swarm algorithms). Given the above, classical gradient deterministic algorithms are inappropriate for solving the problems of processing heterogeneous data. The problem solved in the study is to increase the efficiency of processing heterogeneous data circulating in information systems, regardless of the number of data sources. The object of the study is hierarchical systems. A method for increasing the efficiency of processing heterogeneous data using a metaheuristic algorithm is proposed. The study is based on the reptile algorithm (RA) for processing heterogeneous data circulating in the system. For RA training, evolving artificial neural networks are used. The originality of the proposed method lies in setting RA taking into account the uncertainty of the initial data, improved global and local search procedures. Also, the originality of the study lies in determining RA feeding locations, which allows prioritizing the search in a given direction. The next element in the originality of the study is the possibility of choosing an RA hunting strategy, which allows a rational use of available system computing resources. Another original element of the study is determining the initial velocity of each RA. This makes it possible to optimize the speed of exploration of each RA in a certain direction. The method provides a 15–19 % increase in data processing efficiency by using additional improved procedures. The proposed method should be used in processing large amounts of data

A Review of Multi-source Heterogeneous Twin Data Processing Methods in Traffic Scenes

As the complexity of intelligent public transportation systems continues to increase and the degree of interconnection of equipment at all levels gradually deepens, building a digital twin system will generate a large amount of multi-source heterogeneous data. Effective processing and in-depth mining of data can provide real-time and intelligent decision-making optimization for the construction of digital twin systems, thereby efficiently handling complex emergencies. This paper conducts a systematic review on the processing methods and technologies of multi-source heterogeneous twin data in traffic dynamic scenarios. First, the content and classification of multi-source heterogeneous twin data in the operation process of intelligent bus systems are clarified; secondly, the multi-source The data processing methods and technologies applied in each stage of data collection, data storage, data fusion and data analysis in heterogeneous twin data processing are analyzed, and the advantages, disadvantages and applications of various methods and technologies are analyzed; finally, corresponding to the multiple This paper summarizes the methods and technologies for processing multi-source heterogeneous twin data, and points out the challenges and development trends faced by the current multi-source heterogeneous twin data processing methods and technologies.

A Federated Social Recommendation Approach with Enhanced Hypergraph Neural Network

In recent years, the development of online social network platforms has led to increased research efforts in social recommendation systems. Unlike traditional recommendation systems, social recommendation systems utilize both user-item interactions and user-user social relations to recommend relevant items, taking into account social homophily and social influence. Graph neural network (GNN) based social recommendation methods have been proposed to model these item interactions and social relations effectively. However, existing GNN-based methods rely on centralized training, which raises privacy concerns and faces challenges in data collection due to regulations and privacy restrictions. Federated learning has emerged as a privacy-preserving alternative. Combining federated learning with GNN-based methods for social recommendation can leverage their respective advantages, but it also introduces new challenges: 1) existing federated recommendation systems often lack the capability to process heterogeneous data, such as user-item interactions and social relations; 2) due to the sparsity of data distributed across different clients, capturing the higher-order relationship information among users becomes challenging and is often overlooked by most federated recommendation systems. To overcome these challenges, we propose a federated social recommendation approach with enhanced hypergraph neural network. We introduce hypergraph graph neural networks (HGNN) to learn user and item embeddings in federated recommendation systems, leveraging the hypergraph structure to address the heterogeneity of data. Based on carefully crafted triangular motifs, we merge user and item nodes to construct hypergraphs on local clients, capturing specific triangular relations. Multiple HGNN channels are used to encode different categories of high-order relations, and an attention mechanism is applied to aggregate the embedded information from these channels. Our experiments on real-world social recommendation datasets demonstrate the effectiveness of the proposed approach. Extensive experiment results on three publicly available datasets validate the effectiveness of the proposed method.

A semantic framework for condition monitoring in Industry 4.0 based on evolving knowledge bases

In Industry 4.0, factory assets and machines are equipped with sensors that collect data for effective condition monitoring. This is a difficult task since it requires the integration and processing of heterogeneous data from different sources, with different temporal resolutions and underlying meanings. Ontologies have emerged as a pertinent method to deal with data integration and to represent manufacturing knowledge in a machine-interpretable way through the construction of semantic models. Ontologies are used to structure knowledge in knowledge bases, which also contain instances and information about these data. Thus, a knowledge base provides a sort of virtual representation of the different elements involved in a manufacturing process. Moreover, the monitoring of industrial processes depends on the dynamic context of their execution. Under these circumstances, the semantic model must provide a way to represent this evolution in order to represent in which situation(s) a resource is in during the execution of its tasks to support decision making. This paper proposes a semantic framework to address the evolution of knowledge bases for condition monitoring in Industry 4.0. To this end, firstly we propose a semantic model (the COInd4 ontology) for the manufacturing domain that represents the resources and processes that are part of a factory, with special emphasis on the context of these resources and processes. Relevant situations that combine sensor observations with domain knowledge are also represented in the model. Secondly, an approach that uses stream reasoning to detect these situations that lead to potential failures is introduced. This approach enriches data collected from sensors with contextual information using the proposed semantic model. The use of stream reasoning facilitates the integration of data from different data sources, different temporal resolutions as well as the processing of these data in real time. This allows to derive high-level situations from lower-level context and sensor information. Detecting situations can trigger actions to adapt the process behavior, and in turn, this change in behavior can lead to the generation of new contexts leading to new situations. These situations can have different levels of severity, and can be nested in different ways. Dealing with the rich relations among situations requires an efficient approach to organize them. Therefore, we propose a method to build a lattice, ordering those situations depending on the constraints they rely on. This lattice represents a road-map of all the situations that can be reached from a given one, normal or abnormal. This helps in decision support, by allowing the identification of the actions that can be taken to correct the abnormality avoiding in this way the interruption of the manufacturing processes. Finally, an industrial application scenario for the proposed approach is described.

Integrated Two-Level New Energy Monitoring and Big Data Centers: A Comprehensive Approach for Urban Sustainability

With the help of the quick development of big data, the Internet of Things, and other related technologies, the construction of new energy monitoring and big data centres aims to enhance the control capability of urban new energy stations. Inefficient operation and maintenance of new energy stations, as well as chaotic production control, are the reasons behind this action. Given the limitations of the current stage of new energy big data platform construction, the construction of group-level new energy monitoring and big data centres to achieve a high degree of integration and sharing of resources and experience has become an inevitable development of the new energy industry. This project aims to achieve multi-level integration in decision-making from the bottom up by using the background of the construction of the Wenzhou New Energy Two-Level Centre as a guide, organising the business processes for new energy production and equipment control, designing the technical architecture of the platform for cloud-side collaboration, providing guidance for the heterogeneous data processing and analysis of new energy assets in the cloud, and acting as a model for the development of the group-level intelligent platform for the new energy industry.

Tensor Recurrent Neural Network With Differential Privacy

Recurrent neural network (RNN), a branch of deep learning, is a powerful model for sequential data that has outstanding performance on a wide range of important Internet of Things (IoT) tasks. This unprecedented growth of RNN model has however encountered both heterogeneous IoT data and privacy issues. Existing RNN model can not deal with heterogeneous sequential data; often the larger datasets used in training of RNN model contain sensitive information. To tackle these challenges and for the first time, this research proposes a novel differentially private tensor-based RNN (DPTRNN) that can be applied in many challenging deep learning sequence tasks for IoT systems. Specifically, to process heterogeneous sequential data, we propose a tensor-based RNN model. To guarantee privacy, we develop a tensor-based back-propagation through time algorithm with perturbation to avoid exposing the sensitive information for training the tensor-based RNN model within the framework of differential privacy. Thorough security analysis shows that the differential private tensor-based RNN efficiently protects the confidentiality of sensitive user information for IoT. Our results from extensive experiments on two challenging large video datasets suggest that our proposed scheme is practical with guarantee of data privacy preservation and acceptable accuracy loss.

Encoding and operation scheme for the rhombic triacontahedron aperture 4 hexagonal discrete global grid system

ABSTRACT Discrete Global Grid System (DGGS) is a hierarchical structure with seamless global coverage, supporting processing and analysis of heterogeneous geospatial data. Compared with the most commonly used icosahedron, the rhombic triacontahedron (RT) approximates Earth better and can improve the accuracy of data modeling and expression. However, current RT DGGS studies have adopted a single-resolution integer coordinate scheme within a single surface that cannot achieve multilevel grid analysis. To this end, three adjacent surfaces are combined and a three-axis integer coordinate system is established to reduce the number of times crossing the surface. Then, the principle for partitioning aperture 4 hexagonal grids and a quadtree hierarchical encoding scheme are proposed. Further, this study establishes a fast transformation between the code and three-axis coordinates, implements hierarchical and neighbor code operations, and designs a transformation between the code and geographic coordinates. Experimental results indicate that, compared with the icosahedral hexagonal DGGS, the proposed scheme has a significant efficiency advantage in the transformation between the code and geographic coordinates, while the neighbor code operation efficiency can reach 40.97 times that of HLQT and 4.35 times that of HHUT. The proposed scheme provides a more suitable framework for organizing and analyzing global geospatial data.

Digital twin of earth: A novel information framework for managing a sustainable earth

<p>As the progress of human society rapidly enters a new stage, sustainable development has become an essential target. It is necessary to have an effective approach to managing the resources and ecosystems of Earth, with the target of promoting sustainable development while meeting the needs and requirements of both present and future generations. Digital twin technology provides a fresh perspective on sustainable management with a better understanding of the interconnectedness of Earth's systems and accurate simulation and prediction of future changes. By integrating various types of data and models, including climate models, geophysical models, and social media data, a digital twin of Earth can simulate and analyze various aspects of the planet. The digital twin of Earth leverages real-time multi-source data integration through advanced IoT and wireless sensor networks, feeding into a data lake architecture that supports the seamless integration and processing of heterogeneous data. The integrated data then informs dynamic models, combining numerical simulation models with intelligent algorithms for enhanced prediction and analysis. This approach emphasizes the integration of environmental, social, and economic considerations into decision-making that balances economic growth, social well-being, and environmental protection to achieve sustainable development. This paper introduces the definition and information framework of the new concept of a digital twin of Earth to promote the current research on Earth management based on digital transformation techniques.</p>

Apache Wayang: A Unified Data Analytics Framework

The large variety of specialized data processing platforms and the increased complexity of data analytics has led to the need for unifying data analytics within a single framework. Such a framework should free users from the burden of (i) choosing the right platform( s) and (ii) gluing code between the different parts of their pipelines. Apache Wayang (Incubating) is the only open-source framework that provides a systematic solution to unified data analytics by integrating multiple heterogeneous data processing platforms. It achieves that by decoupling applications from the underlying platforms and providing an optimizer so that users do not have to specify the platforms on which their pipeline should run. Wayang provides a unified view and processing model, effectively integrating the hodgepodge of heterogeneous platforms into a single framework with increased usability without sacrificing performance and total cost of ownership. In this paper, we present the architecture ofWayang, describe its main components, and give an outlook on future directions.

An MBD-driven order remaining completion time prediction method based on SSA-BiLSTM in the IoT-enabled manufacturing workshop

The transformation of production mode leads to the need to strictly ensure the order in-time delivery. The global control ability of manufacturing workshops has become a necessary capability for enterprises. The real-time accurate Order Remaining Completion Time (ORCT) prediction allows managers to master the production schedule fluctuation, which provides criteria for workshop management. Due to the widespread deployment of Internet of Things (IoT) devices in the workshop, the difficulty of systematic processing and analysis of multi-dimensional heterogeneous data has been the main pain point of ORCT prediction. To tackle these problems, a Manufacturing Big Data (MBD)-driven ORCT prediction method based on Salp Swarm Algorithm (SSA) and Bidirectional Long Short-Term Memory (BiLSTM) is proposed. An attribute selection algorithm combined with max-relevance and min-redundancy and regularisation is used to mine key data from MBD. An SSA-BiLSTM model is proposed to achieve efficient and accurate ORCT prediction. The prediction model is supported by BiLSTM, and the hyperparameters are optimised by SSA. Finally, a case study about ORCT prediction in the IoT-enabled manufacturing workshop is presented. The result verifies that the proposed ORCT prediction method has obvious advantages over the other three traditional methods in accuracy and efficiency.

Heterogeneous Sensor Data Acquisition and Federated Learning for Resource Constrained IoT Devices—A Validation

This paper focused on applying federated learning to process heterogeneous sensor data in a fog computing-enabled smart home environment. The fusion of sensor data from vision sensor, digital ambient sensor and passive infrared sensor is done in a distributed manner and the data processing using federated learning on Raspberry Pi edge nodes. The system uses edge nodes to train the respective machine learning models using federated learning without sending data to a centralized server. Instead, results from the training on multiple edge nodes are aggregated at the central node to produce a final machine-learning model. Each edge node deploys the aggregated model to recognize the subjects in the smart home environment and trigger an alert in case unknown subjects are identified. Linear regression model for temperature prediction and logistic regression model for humidity prediction using the ambient parameters are also trained in a federated way on the same edge nodes. The developed system uses the federated learning Flower framework for training the models. The obtained validation accuracy was 66% for the machine learning models that were built on the resource constraint server and edge clients through the federated learning. The developed system can be used in environments where communication bandwidth is limited, or data privacy concerns exist.

Nonparametric Bayesian Networks as a Tool of Multiscale Time Series Analysis and Remote Sensing Data Integration

Introduction. Nonparametric Bayesian networks are a promising tool for analyzing, visualizing, interpreting and predicting the structural and dynamic characteristics of complex systems. Modern interdisciplinary research involves the complex processing of heterogeneous data obtained using sensors of various physical nature. In the study of the forest fund, both methods of direct dendrological measurements and methods of remote observation using unmanned aerial vehicles are widely used. Information obtained using these methods must be analyzed in conjunction with hydrometeorological monitoring data.Aim. Investigation of the possibility of automating the monitoring of the well-being of the forest fund based on the integration of ground survey data, remote multispectral measurements and hydrometeorological observations using the mathematical apparatus of nonparametric Bayesian networks.Materials and methods. To assess the long-term joint dynamics of natural and climatic indicators and the radial growth of trees, a modified method of multiscale cross-correlation analysis was used with the removal of the background trend described by the moving average model. Relationships between various indicators were estimated based on the unconditional and conditional nonparametric Spearman correlation coefficients, which were used to reconstruct and parameterize the nonparametric Bayesian network.Results. A multiscale nonparametric Bayesian network was constructed to characterize both unconditional and conditional statistical relationships between parameters obtained from remote sensing, hydroclimatic and dendrological measurements. The proposed model showed a good quality of the plant fund state forecasting. The correlation coefficients between the observed and predicted indicators exceed 0.6, with the correlation coefficient comprising 0.77 when predicting the growth trend of annual tree rings.Conclusion. The proposed nonparametric Bayesian network model reflects the relationship between various factors that affect the forest ecosystem. The Bayesian network can be used to assess risks and improve environmental management planning.

Methodology of formation of fuzzy associative rules with weighted attributes from SIEM database for detection of cyber incidents in special information and communication systems

The article presents the method of forming associative rules from the database of the SIEM system for detecting cyber incidents, which is based on the theory of fuzzy sets and methods of data mining. On the basis of the conducted analysis, a conclusion was made about the expediency of detecting cyber incidents in special information and communication systems (SICS) by applying rule-oriented methods. The necessity of applying data mining technologies, in particular, methods of forming associative rules to supplement the knowledge base (KB) of the SIEM system with the aim of improving its characteristics in the process of detecting cyber incidents, is substantiated. For the effective application of cyber incident detection models built on the basis of the theory of fuzzy sets, the use of fuzzy associative rule search methods is proposed, which allow processing heterogeneous data about cyber incidents and are transparent for perception. The mathematical apparatus for forming fuzzy associative rules is considered and examples of its application are given. In order to increase the effectiveness of the methods of searching for fuzzy associative rules from the database of the SIEM it is proposed to use weighting coefficients of attributes that characterize the degree of manifestation of their importance in the fuzzy rule. A formal formulation of the problem of forming fuzzy associative rules with weighted attributes and which are used for the identification of cyber incidents is given. A scheme of their formation and application for identification of cyber incidents is proposed. The method of forming fuzzy associative rules with weighted attributes from the database of the SIEM is given. The problem of determining the weighting coefficients of the relative importance of SIEM system DB attributes is formulated and a method for its solution is proposed. The formulation of the problem of finding sets of elements that have a weighted fuzzy support of at least the given one and are used to form fuzzy associative rules with weighted attributes is given. Methods for its solution are proposed.

Construction of a rural tourism information service management system for multi-source heterogeneous data processing.

This study offers an integrated service management system for rural tourist information based on a cloud platform to address the three main issues of high platform concurrency, difficulty storing and managing data, and trouble sharing data functions. Three levels-data, process, and architecture-are considered in the analysis and design of the platform. The Hadoop data storage system makes possible the collection, storage, administration, and exchange of data functions for large amounts of heterogeneous data from many different sources by utilising Netty data transmission technology, hybrid data storage technology, and the Web Foundation. The results demonstrate that the system's response time is low, and the CPU consumption time and the average utilisation rate meet the actual needs. They resolve issues with the current rural tourism platforms application, such as the difficulty of data collection, the low rate of reuse, the low rate of sharing, the lack of timely updates, and severe island phenomena.

A Visual Analysis Method for Predicting Material Properties Based on Uncertainty

The traditional way of studying fluorinated materials by adjusting parameters throughout multiple trials can no longer meet the needs of the processing and analysis of multi-source, heterogeneous, and numerous complex data. Due to the high confidentiality of fluorinated materials’ data, it is not convenient for the plant to trust the data to third party professionals for processing and analysis. Therefore, this paper introduces a visual analysis method for material performance prediction supporting model selection, MP2-method, which helps with researchers’ independent selection and comparison of different levels of prediction models for different datasets and uses visual analysis to achieve performance prediction of fluorinated materials by adjusting control parameters. In addition, according to the Latin hypercube Markov chain (LHS-MC) model of uncertainty for visual analysis proposed in this paper, the uncertainty of the control-parameter data is reduced, and their prediction accuracy is improved. Finally, the usefulness and reliability of MP2-method are demonstrated through case studies and interviews with domain experts.

Processing of heterogeneous data on the aircraft control system characteristics during its prototypes testing

Processing of heterogeneous data on the aircraft control system characteristics during its prototypes testing

A Data-Science Approach for Creation of a Comprehensive Model to Assess the Impact of Mobile Technologies on Humans

Mobile technologies are an essential part of people’s everyday lives since they are utilized for a variety of purposes, such as communication, entertainment, commerce, and education. However, when these gadgets are misused, the human body is exposed to continuous radiation from the electromagnetic field created by them. The communication services available are improving as mobile technologies advance; however, the problem is becoming more severe as the frequency range of mobile devices expands. To solve this complex case, it is necessary to propose a comprehensive approach that combines and processes data obtained from different types of research and sources of information, such as thermal imaging, electroencephalograms, computer models, and surveys. In the present article, a complex model for the processing and analysis of heterogeneous data is proposed based on mathematical and statistical methods in order to study the problem of electromagnetic radiation from mobile devices in-depth. Data science selection/preprocessing is one of the most important aspects of data and knowledge processing aiming at successful and effective analysis and data fusion from many sources. Special types of logic-based binding and pointing constraints are considered for data/knowledge selection applications. The proposed logic-based statistical modeling method provides both algorithmic as well as data-driven realizations that can be evolutionary. As a result, non-anticipated and collateral data/features can be processed if their role in the selected/constrained area is significant. In this research, the data-driven part does not use artificial neural networks; however, this combination was successfully applied in the past. It is an independent subsystem maintaining control of both the statistical and machine-learning parts. The proposed modeling applies to a wide range of reasoning/smart systems.