Collaborative Data Processing Research Articles

Cloud-edge collaborative high-frequency acquisition data processing for distribution network resilience improvement

To realize transparent monitoring and resilience improvement of low-voltage distribution network, both the data acquisition scope and frequency have been greatly expanded. Cloud-edge collaboration leverages the edge server’s real-time response capabilities and the cloud server’s robust data processing power to enhance the performance of high-frequency data acquisition processing. Nonetheless, it continues to confront challenges such as the entanglement of optimization variables, the presence of uncertain information, and a lack of awareness regarding acquisition frequencies. In this paper, we propose a machine learning-based cloud-edge collaborative data processing optimization algorithm to minimize the weighted sum of data processing delay and device energy consumption for distribution network resilience improvement. The joint optimization problem is decoupled into device-edge data offloading subproblem and edge-cloud data splitting subproblem, which are solved by the proposed upper confidence bound (UCB) based frequency-aware device-edge data offloading optimization algorithm and the exponential-weight algorithm for exploration and exploitation (EXP3) based edge-cloud data splitting optimization algorithm, respectively. Simulation results show that the proposed algorithm is superior to existing algorithms in performances of energy consumption and total processing delay.

HydroRTC: A web-based data transfer and communication library for collaborative data processing and sharing in the hydrological domain

The exponential growth in data generated by satellites, radars, sensors, and analysis and reanalysis from model outputs for the hydrological domain requires efficient real-time data management and distribution mechanisms. This paper introduces HydroRTC, a web-based data transfer and communication library designed to accelerate large-scale data sharing and analysis. Leveraging next-generation web technologies like WebSockets, WebRTC and Node.js, the library enables seamless peer-to-peer sharing, smart data transmission, and large dataset streaming. Three primary scenarios are presented as use cases, demonstrating the potential of HydroRTC as server-to-peer with intelligent data scheduling and large data streaming, peer-to-peer data sharing, and peer-to-server for data exchange. HydroRTC offers a promising solution for collaborative infrastructures in the hydrological and environmental domain, allowing real-time and high-throughput data sharing and transfer for enhancing research efficiency and collaboration capabilities.

Zero-Trust Cryptographic Protocols and Differential Privacy Techniques for Scalable Secure Multi-Party Computation in Big Data Analytics

This research explores the integration of zero-trust cryptographic protocols and differential privacy techniques to establish scalable secure multi-party computation in the context of big data analytics. The study delves into the challenges of collaborative data processing and presents a comprehensive framework that addresses the intricate balance between security, scalability, and privacy. The framework focuses on zero-trust cryptographic protocols, advocating for a fundamental shift in trust assumptions within distributed systems. Differential privacy techniques are then seamlessly integrated to preserve individual privacy during collaborative data analytics. This model employs a layered approach and distributed architecture and leverages serverless and edge computing fusion to enhance scalability and responsiveness in dynamic big data environments. This also explores the optimization of computational resources and real-time processing capabilities through serverless and edge computing fusion. A distributed architecture facilitates efficient collaboration across multiple parties, allowing for seamless data integration, preprocessing, analytics, and visualization. Privacy preservation takes centre stage in the big data privacy component of the framework. Context-aware attribute analysis, distributed federated learning nodes, and Attribute-Based Access Control (ABAC) with cryptographic enforcement are introduced to ensure fine-grained access control, contextual understanding of attributes, and collaborative model training without compromising sensitive information. Smart Multi-Party Computation Protocols (SMPCP) further enhance security, enabling joint computation of functions over private inputs while ensuring the integrity and immutability of data transactions. In essence, the achieved results manifest a paradigm shift where the layered approach, distributed architecture, and advanced privacy techniques converge to heighten data security, drive efficient computation, and robustly preserve privacy in the expansive landscape of big data analytics. Fault tolerance and resource utilization exhibit significant advancements, with fault tolerance experiencing a 10% boost and resource utilization optimizing by 12%. These enhancements underscore the robustness and efficiency of the system's design, ensuring resilience and optimized resource allocation.

Optimizing Data Processing: A Comparative Study of Big Data Platforms in Edge, Fog, and Cloud Layers

Intelligent applications in several areas increasingly rely on big data solutions to improve their efficiency, but the processing and management of big data incur high costs. Although cloud-computing-based big data management and processing offer a promising solution to provide scalable and abundant resources, the current cloud-based big data management platforms do not properly address the high latency, privacy, and bandwidth consumption challenges that arise when sending large volumes of user data to the cloud. Computing in the edge and fog layers is quickly emerging as an extension of cloud computing used to reduce latency and bandwidth consumption, resulting in some of the processing tasks being performed in edge/fog-layer devices. Although these devices are resource-constrained, recent increases in resource capacity provide the potential for collaborative big data processing. We investigated the deployment of data processing platforms based on three different computing paradigms, namely batch processing, stream processing, and function processing, by aggregating the processing power from a diverse set of nodes in the local area. Herein, we demonstrate the efficacy and viability of edge-/fog-layer big data processing across a variety of real-world applications and in comparison to the cloud-native approach in terms of performance.

MP-CLF: An effective Model-Preserving Collaborative deep Learning Framework for mitigating data leakage under the GAN

The development of Internet of Things (IoT) communication technology has accelerated the data transmission between IoT devices, thus facilitating collaborative data processing based on the cloud, such as collaborative deep learning. The collaborative deep learning framework allows local devices to cooperate on training models without sharing private data, which resolves the contradiction of the availability and privacy of data. However, the emergence of the Generative Adversarial Network (GAN) attack has shown that poorly protected local data is vulnerable to being learned by adversaries. In this paper, we aim to address the threat GAN attacks pose to collaborative deep learning. We propose a Model-Preserving Collaborative deep Learning Framework, called MP-CLF, which can effectively resist the GAN attack. Based on fully connected neural network learning, MP-CLF employs a matrix blinding technology to break the local modeling of the GAN attack by blinding specific model parameters and trainers’ data, which is easily implementable and has strong security. Besides, MP-CLF builds a user partition model pre-training to improve training quality and strengthen model protection. Using the MNIST dataset and Fashion-MNIST dataset, we experimentally demonstrate that MP-CLF can completely resist the GAN attack with good computational efficiency.

ProbSAP: A comprehensive and high-performance system for student academic performance prediction

The student academic performance prediction is becoming an indispensable service in the computer supported intelligent education system. But conventional machine learning-based methods can only exploit the sparse discriminative features of student behaviors in imbalanced academic datasets to predict student academic performance (SAP). Furthermore, there is a lack of imbalanced data processing mechanisms that can efficiently capture student characteristics and achievement. Therefore, we propose a comprehensive and high-performance prediction framework to probe SAP characteristics (ProbSAP) on massive educational data, which can resolve imbalanced data issue and improve academic prediction performance for making course final mark prediction. It consists of three main components: collaborative data processing module for enhancing the data quality, scalable metadata clustering module for alleviating the imbalance of academic features, and XGBoost-enhanced SAP prediction module for academic performance forecasting. The collaborative data processing module integrates multi-dimensional academic data, which sustains a good supply for clustering and modeling in the ProbSAP framework. The comparative evaluation results demonstrate that ProbSAP delivers superior accuracy and efficiency improvement for the course final mark prediction of college students over other state-of-the-art methods such as CNN, SVR, RFR, XGBoost, Catboost-SHAP, and AS-SAN. On average, ProbSAP reduces the mean absolute error (MAE) by 84.76%, 72.11%, and 66.49% compared with XGBoost, Catboost-SHAP, and AS-SAN, respectively. It also leads to a better out-sample fit that minimizes prediction errors between 1% and 9% with over 98% of actual samples.

Collaborative Damage Detection Framework for Rail Structures Based on a Multi-Agent System Embedded with Soft Multi-Functional Sensors.

With the rapid growth of railways in China, the focus has changed to the maintenance of large-scale rail structures. Multi-agent systems (MASs) based on wireless sensor network (WSNs) with soft multi-functional sensors (SMFS) are adopted cooperatively for the structural health monitoring of large-scale rail structures. An MAS framework with three layers, namely the sensing data acquisition layer, sensor data processing layer, and application layer, is built here for collaborative data collection and processing for a rail structure. WSN nodes with strain, temperature, and piezoelectric sensor units are developed for the continuous structural health monitoring of the rail structure. The feature data at different levels are extracted for the online monitoring of the rail structure. Experiments carried out at the Rail Transmit Base at East China Jiaotong University verify that the WSN nodes with SMFS are successfully assembled onto a 100-m-long track for damage detection. Based on the sensing data and feature data, a neural network data fusion agent (DFA) is applied to calculate the damage index value of the track for comprehensive decisions regarding rail damage. The use of WSNs with multi-functional sensors and intelligent algorithms is recommended for cooperative structural health monitoring in railways.

Hybrid grasshopper and differential evolution algorithm for prolonging network life expectancy in wireless sensor networks (WSNs)

SummaryIn wireless sensor networks (WSNs), restricted battery power, balanced energy consumption, and collaborative data processing are considered as key challenges that need to be handled with utmost care. Clustering in WSNs is an optimal methodology for conserving energy of sensor nodes and attaining efficient data processing that attributes toward the maximization of network lifetime. An efficient cluster head (CH) selection scheme is essential for achieving superior collaborative data processing in WSNs. Swarm intelligent metaheuristic algorithm‐based CH selection approaches are identified to be better for designing energy‐efficient schemes that select optimal CHs from nodes in a fair way. In this paper, Hybrid Grasshopper and Differential Evolution‐based Optimization Algorithm (HGDEOA) is proposed for targeting on the objective of attaining energy stability and prolonging network lifetime. This HGDEOA incorporates an adaptive strategy into differential evolution (DE) for improving the global searching capability during the process of optimization. It is proposed for improving the convergence efficiency and maintaining population diversity. It also possesses the capability of enhancing the degree of convergence speed and precision in calculation. This integration of GOA into DE prevents the premature convergence of the algorithm, since the deviation amid the scaling individuals induces the population to be more randomly distributed, targeting at the retention of population diversity. The simulation results confirm that the proposed HGDEOA sustains residual energy by 19.32%, improves throughput by 16.21%, prolongs network lifetime by 18.76%, and maintains stability by18.94% when compared to the benchmarked approaches used for investigation.

A Blockchain-based platform for data management and sharing

Data management is the collection, processing, storing, and sharing of data. In today's dispensation, data sharing and collaborative data processing is a necessity for multi-partner organizations as it can lead to the discovery of new insight. Shared data is generally for the purpose of marketing, advertising, and other institutional decision-making reasons. A challenge however is that the collected data (mostly about individuals known as data subjects or producers) is disseminated among organizations without meaningful consent from the data subjects. Hence, data subjects are not aware of what is happening to their data regarding use and misuse. Furthermore, data subjects can hardly determine which third-party institutions have access to their data. In this paper, we opined that data should be managed in a manner that can persuade the trust of data subjects. To achieve this, data subjects should have the rights to be informed about details of their data. Thus, this paper proposes a cloud-based data management and sharing platform to enable data subjects to control who can access their data and consent to its collection and usage based on smart contracts. The proposed system leverages blockchain to enforce accountability, provenance, and auditability of all events. We implemented a dynamic consent management prototype on top of Ethereum blockchain to demonstrate the feasibility of the proposed work.

Blockchain for Transparent Data Management Toward 6G

The wealth of user data acts as a fuel for network intelligence toward the sixth generation wireless networks (6G). Due to data heterogeneity and dynamics, decentralized data management (DM) is desirable for achieving transparent data operations across network domains, and blockchain can be a promising solution. However, the increasing data volume and stringent data privacy-preservation requirements in 6G bring significantly technical challenge to balance transparency, efficiency, and privacy requirements in decentralized blockchain-based DM. In this paper, we investigate blockchain solutions to address the challenge. First, we explore the consensus protocols and scalability mechanisms in blockchains and discuss the roles of DM stakeholders in blockchain architectures. Second, we investigate the authentication and authorization requirements for DM stakeholders. Third, we categorize DM privacy requirements and study blockchain-based mechanisms for collaborative data processing. Subsequently, we present research issues and potential solutions for blockchain-based DM toward 6G from these three perspectives. Finally, we conclude this paper and discuss future research directions.

AI-enabled IoT-Edge Data Analytics for Connected Living

As deep learning, virtual reality, and other technologies become mature, real-time data processing applications running on intelligent terminals are emerging endlessly; meanwhile, edge computing has developed rapidly and has become a popular research direction in the field of distributed computing. Edge computing network is a network computing environment composed of multi-edge computing nodes and data centers. First, the edge computing framework and key technologies are analyzed to improve the performance of real-time data processing applications. In the system scenario where the collaborative deployment tasks of multi-edge nodes and data centers are considered, the stream processing task deployment process is formally described, and an efficient multi-edge node-computing center collaborative task deployment algorithm is proposed, which solves the problem of copy-free task deployment in the task deployment problem. Furthermore, a heterogeneous edge collaborative storage mechanism with tight coupling of computing and data is proposed, which solves the contradiction between the limited computing and storage capabilities of data and intelligent terminals, thereby improving the performance of data processing applications. Here, a Feasible Solution (FS) algorithm is designed to solve the problem of placing copy-free data processing tasks in the system. The FS algorithm has excellent results once considering the overall coordination. Under light load, the V value is reduced by 73% compared to the Only Data Center-available (ODC) algorithm and 41% compared to the Hash algorithm. Under heavy load, the V value is reduced by 66% compared to the ODC algorithm and 35% compared to the Hash algorithm. The algorithm has achieved good results after considering the overall coordination and cooperation and can more effectively use the bandwidth of edge nodes to transmit and process data stream, so that more tasks can be deployed in edge computing nodes, thereby saving time for data transmission to the data centers. The end-to-end collaborative real-time data processing task scheduling mechanism proposed here can effectively avoid the disadvantages of long waiting times and unable to obtain the required data, which significantly improves the success rate of the task and thus ensures the performance of real-time data processing.

Real-Time Imprecise Computation Tasks Mapping for DVFS-Enabled Networked Systems

Networked systems are useful for a wide range of applications, many of which require distributed and collaborative data processing to satisfy real-time requirements. On one hand, networked systems are usually resource constrained, mainly regarding the energy supply of the nodes and their computation and communication abilities. On the other hand, many real-time applications can be executed in an imprecise way, where an approximate result is acceptable as long as the baseline Quality of Service (QoS) is satisfied. Such applications can be modeled through imprecise computation (IC) tasks. To achieve a better tradeoff between QoS and limited system resources, while meeting application requirements, the IC-tasks must be efficiently mapped to the system nodes. To tackle this problem, we first construct an IC-task mapping problem that aims to maximize system QoS subject to real-time and energy constraints. Dynamic voltage and frequency scaling (DVFS) and multipath routing are explored to further enhance real-time performance and reduce energy consumption. Second, based on the problem structure, we propose an optimal approach to perform IC-task mapping and prove its optimality. Furthermore, to enhance the scalability of the proposed approach, we present a heuristic IC-task mapping method with low computation time. Finally, the simulation results demonstrate the effectiveness of the proposed methods in terms of the solution quality and the computation time.

Data Management for Future Wireless Networks: Architecture, Privacy Preservation, and Regulation

Next-generation wireless networks (NGWN) aim to support diversified smart applications that require frequent data exchanges and collaborative data processing among multiple stakeholders. Data management (DM), including data collection, storage, sharing, and computation, plays an essential role in empowering NGWN. However, DM for NGWN faces two significant challenges: stakeholders' data cannot be easily managed across different trust domains under a distributed network architecture; and privacy preservation requirements of personal data become more rigorous under new privacy regulations. To explore possible solutions to address the challenges, we first investigate the state-of-the-art architecture designs for DM and emphasize advantages of a blockchain-based DM architecture. Then we summarize existing privacy-preserving techniques in terms of advantages and challenges when being applied to DM. In addition, we review recent privacy regulations with their impacts on DM and discuss the existing solutions with privacy regulation compliance based on blockchain. Finally, we identify further research directions for achieving DM with privacy preservation.

Distributed Task Privacy for Aggregation Using Linear Codes

Information privacy in modern Internet of Things (IoT) environments is necessary for widespread adoption and public trust of collaborative data processing. Most current distributed privacy results ensure local node observations are not accessible by neighbouring nodes while still solving collaborative tasks. In this work we develop the distinct concept of distributed task privacy in unbounded public networks, where linear codes are used to create arbitrary hard limits on the number of nodes contributing to a distributed task. We accomplish this by wrapping local observations in a linear code and intentionally applying symbol errors or erasures prior to transmission. If many nodes join a distributed task, a proportional number of symbol errors and erasures are introduced into the aggregated code leading to decoding failure if the code’s predefined symbol error/erasure limit is exceeded.

Collaborative data acquisition and processing for post disaster management and surveillance related tasks using UAV-based IoT cloud

Rescue and recovery operations are very critical for post disaster management. For post disaster management and surveillance activities, there is a need of acquiring the information about the ground situation through sensing or observing data and identify the locations, which is a challenging task. These data may be sensed or observed through different types of sensors deployed in the area of interest. This can be achieved through a collaborative way of data acquisition and processing. In this context, this paper introduces the framework for data acquisition based on collaborative processing using unmanned aerial vehicles and internet of things network. The major contribution of this work is that the real time test bed using actual UAVs and IoT devices for data acquisition, clustering of different sensors and devices deployed for it and localising the sensors and different events or situations arising in the region of interest is detailed and demonstrated. The framework is evaluated using real test bed with drones (UAVs) and integrated with cloud platform in IoT-based environment for data storage of the acquired data for further analysis and effective decision making in disaster situations.

Collaborative data acquisition and processing for post disaster management and surveillance related tasks using UAV-based IoT cloud

Collaborative data acquisition and processing for post disaster management and surveillance related tasks using UAV-based IoT cloud

Data Processing in Cyber-Physical-Social Systems Through Edge Computing

Cloud and Fog computing have established a convenient and widely adopted approach for computation offloading, where raw data generated by edge devices in the Internet of Things (IoT) context is collected and processed remotely. This vertical offloading pattern, however, typically does not take into account increasingly pressing time constraints of the emerging IoT scenarios, in which numerous data sources, including human agents (i.e., Social IoT), continuously generate large amounts of data to be processed in a timely manner. Big data solutions could be applied in this respect, provided that networking issues and limitations related to connectivity of edge devices are properly addressed. Although edge devices are traditionally considered to be resource-constrained, main limitations refer to energy, networking, and memory capacities, whereas their ever-growing processing capabilities are already sufficient to be effectively involved in actual (big data) processing. In this context, the role of human agents is no longer limited to passive data generation, but can also include their voluntary involvement in relatively complex computations. This way, users can share their personal computational resources (i.e., mobile phones) to support collaborative data processing, thereby turning the existing IoT into a global cyber-physical-social system (CPSS). To this extent, this paper proposes a novel IoT/CPSS data processing pattern based on the stream processing technology, aiming to distribute the workload among a cluster of edge devices, involving mobile nodes shared by contributors on a voluntary basis, and paving the way for cluster computing at the edge. Experiments on an intelligent surveillance system deployed on an edge device cluster demonstrate the feasibility of the proposed approach, illustrating how its distributed in-memory data processing architecture can be effective.

An Efficient Cluster Head Selection Approach for Collaborative Data Processing in Wireless Sensor Networks

Since wireless sensor networks (WSNs) consist of nodes with limited battery power, collaborative data processing and balanced energy consumption should be considered as the key issue. This paper proposes an efficient cluster head selection approach for collaborative data processing in WSNs. The proposed algorithm designs an effective energy-efficient model to select the optimal cluster heads among all nodes fairly, which helps to reduce the impact of the monitoring scheme on the lifetime of network. Experimental results show that the proposed protocol is able to reduce energy consumption and obtain higher efficiency as well as effectively prolonging the lifetime of network more than a few existing cluster-based routing protocols.

Collaborative Data Processing in WSN Using Voronoi Fuzzy Clustering

In this paper, developed a novel Voronoi Fuzzy Clustering (VF) algorithm for energy efficient collaborative data aggregation in wireless sensor network. VF algorithm is fusion of Voronoi diagram and modified Fuzzy C- Means with respect to distance and Quality of Service. Here throughput, delay time and delivery ratio are considered as QOS parameters. Once clustering of sensor nodes is completed then data management technique such as data aggregation or compression is done for further decision making in sink node. Data mining clustering algorithm reduces overall transmission of data from each sensor to the sink node thus energy spent by individual sensor node is minimized. The cluster heads collects all sensed information from their respective cluster members and performs data aggregation or compression before transmitting the data to the sink node. Finally, the simulations are performed and the results are analyzed within the simulation set up to determine performance of the proposed algorithm in the sensor network. Our proposed approach has achieved 60% efficiency when compare with the K means algorithm.

Adaptive energy-efficient scheduling for hierarchical wireless sensor networks

Most Wireless Sensor Network (WSN) applications require distributed signal and collaborative data processing. One of the critical issues for enabling collaborative processing in WSNs is how to schedule tasks in a systematic way, including assigning tasks to sensor nodes, and determining their execution and communication sequence. Since WSN nodes are very resource constrained, mainly regarding their energy supply, one major concern when scheduling tasks in such environments is to minimize and balance the energy consumption, so that the system operational lifetime is maximized. We propose a heuristic-based three-phase algorithm (TPTS) for allocating tasks to multiple clusters in hierarchical WSNs that aims at finding a scheduling scheme that minimizes the overall energy consumption and balances the workload of the system while meeting the application's deadline. The performance of the proposed algorithm and the effect of several parameters on its behavior were evaluated by simulations, with promising results. The experimental results show that the time and energy performance of TPTS are close to the time and energy of benchmarks in most cases, while load balance is always provided.