Data Exchange Process Research Articles

SATURN: assessing the feasibility of utilising existing registries for real-world evidence data collection to meet patients, regulatory, health technology assessment and payer requirements

BackgroundSATURN (Systematic Accumulation of Treatment practices and Utilisation, Real world evidence, and Natural history data) for the rare condition osteogenesis imperfecta (OI) has the objective to create a common core dataset by utilising existing, well-established data sources to meet the needs of the various stakeholders (physicians, registry/dataset owners, patients and patient associations, OI community leaders, European [EU] policymakers, regulators, health technology assessments [HTA]s, and healthcare systems including payers). This paper describes the steps taken to assess the feasibility of one existing OI registry (i.e., the Registry of OI [ROI]) as a candidate for SATURN. The same methodology will be applied to other existing OI registries in the future and this same concept could be utilised for other rare disease registries.MethodsThe approach to assessing the feasibility of the ROI registry consisted of three steps: (1) an assessment of the registry characteristics using the Registry Evaluation and Quality Standards Tool (REQueST); (2) a gap analysis comparing SATURN required Core Variables to those being captured in the registry’s Case Report Form (CRF); and (3) a compliance check on the data exchange process following the Title 21 of Code of Federal Regulations (CFR) Part 11/EudraLex Annex 11 Compliance Checklist. The first registry that SATURN has assessed is the ROI database at the Istituto Ortopedico Rizzoli (IOR) in Italy.ResultsThe results from the ROI REQueST have demonstrated satisfactory complete responses in terms of methodology, essential standards, interpretability, and interoperability—readiness for data linkage, data sources, and ethics to meet the needs of data customers. However, the ROI data is from a tertiary referral centre which may limit the ability to understand the full patient journey. The gap analysis has revealed that an exact or logical match between SATURN requested variables and the ROI current variables exists for the following items: patient characteristics, treatment of OI (medical and surgical) and treatment of pain (with the exception of frequency of treatment and reasons for discontinuation), fracture history and bone density. However, data on safety was missing. The compliance check has implied that the ROI implemented appropriate controls for the web-based platform (i.e., Genotype–phenotype Data Integration Platform [GeDI]) that is involved in processing the electronic patient data, and GeDI is a validated/compliant application that follows relevant 21 CFR Part 11/EudraLex Annex 11 regulations.ConclusionsThis robust feasibility process highlights potential limitations and opportunities to develop and to refine the collaboration with the ROI as the SATURN programme moves forward. It also ensures that the existing datasets in the rare condition OI are being maximised to respond to the needs of patients, data customers and decision-makers. This feasibility process has allowed SATURN to build a compliant methodology that aligns with the requirements from the European Medicines Agency (EMA) and HTAs. More data variables will continue to be developed and refined along the way with more registries participating in SATURN. As a result, SATURN will become a meaningful and truly collaborative core dataset, which will also contribute to advancing understanding of OI diagnosis, treatment, and care.

SCADA System for Process Data Exchange in Master Slave RF and Iot Network

In every aspect of our lives, we can observe how automation plays a facilitating role in the performance of responsibilities of a comprehensive rank. Thanks to the development of electronics, today's process measurement systems enable the measurement values of process quantities to be visualized on display, sent remotely, and stored in a file, compatible with future user processing. In this paper are represented results of the development, design and practical implementation of the SCADA system, which enables data exchange between SCADA screen, MASTER and SLAVE stations. The MASTER station is installed in one industrial plant and the SLAVE is installed in another plant. The SCADA monitoring system is built into the MASTER station. The SLAVE station is connected by RF (radio frequency connection) to the MASTER station. In the MASTER and SLAVE station two microcomputers units, as well as appropriate RF modules, are installed. The microcomputer in the MASTER station is serially connected to the IoT microcomputer, for the transfer of measurement data in the Internet network. The solution provides visualization, data log file, and transfer to the IoT network of process data from the two industrial plants.

Combination Concept of LSB and PlayFair Cipher for Optimizing Data Security

Data security is now something that is very necessary, with the aim of ensuring that important data and information does not fall into the hands of unauthorized people. The widespread data exchange process provides opportunities for unauthorized parties to take, copy or steal the exchanged data. This is what triggers the importance of securing data when exchange occurs. In the field of computer science, there are many ways that data can be secured. These methods include the concepts of steganography and cryptography. Steganography is a way to hide data in various media, while cryptography is a way to encode data into a form that has no meaning. This research aims to design a system to secure messages in the form of text data using the Playfair cipher cryptography method and Least Significant Bit (LSB) steganography using image media and the image containing the message is not visible to the eye that the image contains a secret message

Moving beyond the Content: 3D Scanning and Post-Processing Analysis of the Cuneiform Tablets of the Turin Collection

This work and manuscript focus on how 3D scanning methodologies and post-processing analyses may help us to gain a deeper investigation of cuneiform tablets beyond the written content. The dataset proposed herein is a key part of the archaeological collection preserved in the Musei Reali of Turin in Italy; these archaeological artefacts enclose further important semantic information extractable through detailed 3D documentation and 3D model filtering. In fact, this scanning process is a fundamental tool for better reading of sealing impressions beneath the cuneiform text, as well as for understanding micrometric evidence of the fingerprints of scribes. Most of the seal impressions were made before the writing (like a watermark), and thus, they are not detectable to the naked eye due to cuneiform signs above them as well as the state of preservation. In this regard, 3D scanning and post-processing analysis could help in the analysis of these nearly invisible features impressed on tablets. For this reason, this work is also based on how 3D analyses may support the identification of the unperceived and almost invisible features concealed in clay tablets. Analysis of fingerprints and the depths of the signs can tell us about the worker’s strategies and the people beyond the artefacts. Three-dimensional models generated inside the Artec 3D ecosystem via Space Spider scanner and Artec Studio software were further investigated by applying specific filters and shaders. Digital light manipulation can reveal, through the dynamic displacement of light and shadows, particular details that can be deeply analysed with specific post-processing operations: for example, the MSII (multi-scale integral invariant) filter is a powerful tool exploited for revealing hidden and unperceived features such as fingerprints and sealing impressions (stratigraphically below cuneiform signs). Finally, the collected data will be handled twofold: in an open-access repository and through a common data environment (CDE) to aid in the data exchange process for project collaborators and common users.

Control modeling and optimization of network information security system based on deep learning data interaction

In order to solve the problem of poor network security protection performance of traditional network information security protection systems, the author proposes a network information security system control modeling based on deep learning data interaction. Adopt CY8C2453 type processor to improve system data exchange and data processing speed, enhance system stability through peripheral backplane, and evaluate network information security, at the same time, the authority management mechanism is set, and the general strategy form of the network attack and defense game is defined as a triple, calculate the attacked path in a certain network area and the defense strategy that the defender should choose, so as to complete the network information security protection based on the attack-defense game model. Experimental results show that: The number of attacked information of the system is 58 less than that of the traditional system; In the webpage attack, the number of attacked information of the designed system is 82 less than that of the traditional system; In the data interception attack, the number of attacked information of the designed system is 77 less than that of the traditional system. ConclusionUnder different attack behaviors, the number of successful attacks on the network information security protection system based on the attack-defense game model designed this time is less than the number of successful attacks on the traditional system, which improves the security of network information.

Physiological-chain: A privacy preserving physiological data sharing ecosystem

Safeguarding the privacy of physiological data has long been a focal point, especially given the increasing concern surrounding the data exchange process and its implications for privacy. In response to these challenges, we propose a two-layer blockchain-based platform comprising the Decentralized Identity Blockchain (DID Chain) and the Physiological Data Blockchain (App Chain). Within the DID chain, we deploy two contracts: the IdentityManager Contract and the PersonalIdentity Contract. Users are empowered to register their decentralized identity, while governments can leverage the IdentityManager Contract to generate decentralized identities. Once users obtain their identity, they can proceed to register the app-chain using that identity. Moreover, users can utilize the AccessControlManager chaincode within the App-chain to share de-identified information generated by their wearable devices, retaining full control over managing the access control rights of their de-identified information. Through the integration of these two chains, our system significantly enhances user data privacy, security, and trust, thereby fostering confidence in the integration of wearable devices within the home healthcare data sharing process.

Dynamic knowledge graph applications for augmented built environments through “The World Avatar”

The proliferation of digital building models in recent years has led to a corresponding rise in specialised, non-interoperable models. These models impede sustainable developments by forming data silos that hinder cross-application data exchange and knowledge discovery processes. Although Semantic Web solutions hold promise in addressing these silos, current approaches primarily focus on developing novel ontologies, yielding similar outcomes. But it is unclear how these methodologies could support broader knowledge discovery processes and application requirements. This paper addresses these research challenges by introducing a dynamic knowledge graph as implemented within The World Avatar for interoperable building models. We demonstrate its value through two distinct applications in urban energy management and laboratory automation. The dynamic knowledge graph revolves around a comprehensive structured knowledge model constructed from ontologies and agents. Ontologies semantically annotate data and represent domain knowledge and their relationships with standardised definitions. When augmented with an agent architecture, the resulting knowledge model can align stakeholder perspectives and accommodate the dynamic and scalable nature of urban data. Moreover, the dynamic knowledge graph fosters innovative human-machine interactions through visualisation interfaces to augment knowledge discovery processes in the built environment for greater efficiencies and innovation. As the knowledge model expands, users gain access to a broader spectrum of private and public data sources and technologies, while reducing integration barriers. This is especially pertinent for smaller and less influential entities like municipal and local governments with limited resources, who can realise substantial benefits at reduced costs.

Analysis and Design of Enterprise Architecture in Cooperatives Using TOGAF Adm Framework (Case Study: Cooperative Mandiri XYZ)

Cooperatives in Indonesia have an important role, one of which can be seen in everyday life, the existence of cooperatives that help build the community's economy. Current digital transformation, it will have an impact on the design of the company's organization, employee management, data collection patterns and transaction patterns. If this Digital transformation is not carried out, there will be a lameness in developing cooperatives. Many cooperatives still use the conventional system, using manual logbooks in depositing cow's milk products by each member of the cooperative and collecting data requirements using excel documents which require more time to process, thus slowing down the performance of the data exchange process in cooperatives. In Maximizing the utilization of the system itself, the Enterprise Architecture design of a cooperative management information system is carried out using the TOGAF ADM approach. Enterprise Architecture can help in aligning various business needs, data, applications and technology. Designing the Enterprise Architecture cooperative management information system at TOGAF ADM, it is carried out with the Preliminary Phase, architecture vision, Business Architecture, Information System Architecture, Technology Architecture, Opportunity and solution and migration planning stages. Final result of this study is in the form of a proposed architecture model for Cooperative XYZ, and blueprint.

Enhancing IT Telecom Operations: The Vital Role of Smart Monitoring Systems By IJISRT

In the digital era, Information Technology (IT) and telecommunications rooms are at the heart of organizational operations, facilitating communication, data exchange, and crucial business processes. The reliability and efficiency of these facilities are paramount, necessitating the adoption of advanced monitoring solutions. Smart monitoring systems, leveraging Internet of Things (IoT) technologies, artificial intelligence (AI), and real-time data analytics, play a critical role in optimizing the performance and reliability of IT telecom environments. This article explores the importance of smart monitoring systems in IT telecom rooms, highlighting their impact on operational efficiency, risk management, and strategic decision-making. By integrating smart monitoring solutions, organizations can significantly enhance the resilience, security, and performance of their telecom infrastructure.

Finite element boundary element numerical simulation study on acoustic vibration coupling problems

Excessive structural resonance response in the acoustic vibration coupling environment can cause adverse consequences such as structural damage, local instability, and component failure. Accurately predicting the vibration response of instruments under external sound fields helps to avoid structural performance degradation or damage caused by vibration, and ensures the reliability and safety of the system, which is of great significance for the effective load of instrument equipment. This paper constructs a finite element boundary element equation for the acoustic vibration coupling problem, and adopts a series of optimization measures for it. In response to the low computational efficiency of finite element boundary element equations and their inability to meet the computational requirements of complex models, this paper introduces the acoustic vibration reciprocity theorem and the fast directional boundary element method to compensate for the aforementioned shortcomings. In addition, the study used the weighted residual method to construct a data exchange algorithm to ensure energy conservation during the transfer process. The research focuses on typical structures of spacecraft and conducts simulation analysis and application research on acoustic vibration coupling problems. The simulation showcases that the introduction of the reciprocity theorem markedly reduces the calculation time and memory occupied by the equation, with a required calculation time of 1.23 h and a data file size of approximately 299.56 MB generated by the calculation; The overall error of the data exchange process is below 10−4. Scanning frequency analysis is conducted on the surface and local openings of the model structure, and it is found that the displacement of the selected excitation point is in good agreement with the analytical solution. This study effectively reduces simulation time and memory requirements through the proposed strategy, ensures energy conservation data exchange algorithms, improves the reliability and safety of structural design, and has important practical value for evaluating the structural performance of key equipment such as spacecraft.

Interoperability Is a Process - The Data Sharing Framework.

Common syntax and data semantics are core components of healthcare interoperability standards. However, interoperable data exchange processes are also needed to enable the integration of existing systems between organizations. While solutions for healthcare delivery processes are available and have been widely adopted, support for processes targeting bio-medical research is limited. Our Data Sharing Framework creates a platform to implement research processes like cohort size estimation, reviews and approvals of research proposals, consent checks, record linkage, pseudonymization and data sharing across organizations. The described framework implements a distributed business process engine for executing BPMN 2.0 processes with synchronization and data exchange using FHIR R4 resources. Our reference implementation has been rolled out to 38 organizations across three research consortia in Germany and is available as open source under the Apache 2.0 license.

Socialization of Registration Information for the Use of E-Book (ADKUH Book) at Puskesmas Colomadu 2

Background: Along with the development of technology and communication that is increasingly sophisticated and rapid, the flow of unlimited information is increasingly accessible, making technology a necessity for humans in carrying out all daily activities. This is also accompanied by the changing times that are increasingly rapid and limitless, bringing world civilization to the era of the fourth revolution. In this era, all activities are carried out by utilizing the existence of a technology, that prioritizes the process of automation and direct data exchange in the form of connectivity between humans, machines, and data covered in the Internet of Things and Artificial Intelligence with a combination of robotic technology and sensors by utilizing the existence of the internet as a link. Information and communication technology is the backbone of this fourth revolution movement, Objectives: The general purpose of this community service activity is to increase knowledge about access to health information and registration status at Colomadu 2 Health Center. Method: This method of community service activities uses socialization methods, tutorial questions and answers, and evaluation. Results: showed that the average knowledge score after ebook registration application education was dukuh (Mean= 15.29; SD= 2.64) is greater than the average knowledge score before education (Mean= 11.10; SD= 3.02), and this result is statistically significant (p<0.001).

OA book usage data exchange

Conceived in 2015 at the Scholarly Communications Institute, the Open Access Book Usage Data Trust (OAEBUDT) has grown into an international effort to develop and pilot a global data trust for OA monograph usage data. It progressed through three stages:• Project 1 (2018-2019): This project documented the challenges and opportunities facing the aggregation, analysis, and communication of OA book usage.• Project 2 (2020-2022): This project documented the data supply chain (Clarke and Ricci, 2021) and use cases related to OA book usage data (Drummond and Hawkins, 2021), developed infrastructure for usage data aggregation and provision, and identified community-based governance models to support a diverse, global data trust for OA monographs usage data.• Project 3 (2022-forward): In partnership with OPERAS and OpenAIRE, this project is developing the Governance Building Blocks for its Industrial Data Space (IDS) infrastructure, with an eye towards how such infrastructure can be extensible to usage and impact data for other scholarly outputs.Since 2015, with support from the Mellon Foundation, stakeholders have worked through the global OAEBUDT effort to foster the secure, multi-party exchange, aggregation and benchmarking of book usage-related data. This work aims to increase trust in usage metrics, improve data quality, and reduce reporting and compliance resource-burdens related to OA usage data. Prior work, such as facilitated stakeholder interviews and virtual design workshops, revealed diverse use-cases and staff roles interested in OA book usage data across libraries, publishers, and publishing platforms and services (Drummond and Hawkins, 2021). It also indicated a shared system-wide need for simplifying usage data curation, aggregation and management given the time and costs individual institutions allocate when combining COUNTER-compliant reporting alongside other usage data dashboards, and non-COUNTER compliant web- analytics. This reinforced the hypothesis that a “Data Trust” model could benefit those working with usage data by facilitating the aggregation and processing of data upstream according to shared, transparent community norms, thereby generating economies of scale (O’Leary and Hawkins, 2019), for example, reducing costs and resources used when aggregating and curating such data.Creating a data exchange ecosystem to unlock public/private usage data sharing would help minimise the time and resources used, plus provide more transparency to all the organisations involved in the data exchange process. While the term “usage data” is often used today to refer to web analytic reports that tally page visits and file downloads, the OABUDT works towards a future where linked usage data analytics regularly inform book publishing and scholarly communications operations. Clear ethical norms, privacy andsecurity guardrails are required to unlock such data exchange for granular analytics.After researching and documenting the challenges, use cases and supply chain for OA book usage data from 2015 to 2021, the current effort is focused on developing governance building blocks for a sustainable infrastructure. These blocks are modelled on a certifiable European IDS that will help ensure that organisations wanting to participate in data sharing meet the highest security standards. In April 2023, an OA Book Usage Data Exchange Use Guidelines and Principles meeting gathered publishers, libraries, standards, and service representatives to draft principles and identify trust mechanisms for such an IDS infrastructure.The aim is to ensure that usage data exchange would be both trusted and support ethical, non-harmful data use.The principles and trust requirements drafted in this meeting are now being shared with the global open science community for consultation to ensure that they meet the needs of diverse OA book stakeholders and the research community at large. This poster will provide background to the OAEBUDT and a preview of emerging participation and ethical data exchange and use guidelines for the data trust.

Personal data protection on the internet under martial law: The case of Ukraine

The objective of this study is to determine the threats to the security of personal data on the Internet as a component of the right to privacy in the conditions of martial law, characteristics and prospects of such legal data protection in the context of Ukraine. Methodology: The study used a set of practical methods, namely: formal and legal, comparative forecasting. These methods were used to examine and classify threats to the security of personal data on the Internet under martial law conditions. Results: The study identified a number of threats to the security of personal data on the Internet under martial law conditions, including: unauthorized access to personal data, alteration or destruction of personal data, disclosure of personal data to third parties without the consent of the data owner, use of personal data for unlawful purposes. The study also examined factors that complicate data protection during cross-border sharing, including: differences in data protection laws and regulations between countries, lack of cooperation between governments on data protection, the complexity of cross-border data exchange processes. Conclusions: The findings of the study suggest that increasing the protection of personal data under martial law conditions requires a comprehensive approach that includes regulatory, organizational and communication measures. In particular, it is necessary to: strengthen data protection legislation under martial law conditions, develop cooperation mechanisms between governments on data protection, promote data protection education and awareness among citizens and organizations.

MULTI-AGENT MODEL OF GROUP CONTROL POINT OF UNMANNED AIRCRAFT AND UNMANNED GROUND VEHICLE WHEN PERFORMING SPECIAL MISSIONS

Unmanned aerial vehicles (UAVs) are becoming very important platforms for conducting combat operations in the russian-ukrainian war, as they significantly reduce the risk of human life during special and combat missions. The UAV group can be used to perform a wide range of military tasks: destruction of enemy equipment, reconnaissance (monitoring) of the area, construction of aerial networks, etc.
 In the conditions of a change in the tactical situation, countermeasures by the enemy, effective management of the UAV group is impossible without an appropriate management system. The UAV group can include both unmanned aerial vehicles (UAVs) and unmanned ground vehicles (UAVs). The following main tasks are assigned to the management system: determination and coordination of the order of joint actions of autonomous UAV groups; formation of flight (ground) tasks and flight (movement) trajectories of the UAV group; dynamic adjustment of the flight task and the role of each UAV during its use; organization of the communication network between the control center and between UAV groups, etc. The tasks and target functions of management of each of their subsystems are determined. The features of the UAV group control cycle are described.
 The main stages of the synthesis of the control system are considered: the choice of the control method (centralized, hierarchical, decentralized and mixed); selection of algorithms (protocols) and organization of the process of data exchange between UAVs. A functional model of the UAV group management system is proposed. Tasks and main management functions of each of the subsystems are defined. Features of the UAV group management cycle are described. The main stages of the synthesis of the control system are considered: the selection of algorithms (protocols) and the organized process of information exchange between UAVs; variants of formation and swarm models of UAV movement in groups; multi-agent decision-making models for the organization of task performance by the UAV group.

Optimal battery energy storage planning and control strategy for grid modernization using improved genetic algorithm

The flexible operation of battery energy storage systems (BESS) to support electricity grid modernization requires optimal planning and an efficient control strategy. This paper proposes the optimal allocation of BESS with photovoltaic systems for microgrids to enhance grid reliability and flexibility. An adaptive BESS control strategy is developed to allow the BESS to operate in both redundant and peak shaving modes. An evolutionary programming-based genetic algorithm implemented with an optimal power flow control is proposed to determine the BESS’s optimal location, sizing, and charge/discharge operation. The objective function is to maximize the benefits of energy loss reduction and peak shaving enhancement while minimizing the installation cost. The proposed approach is conducted with MATLAB and DIgSILENT PowerFactory software, which uses an intelligent automatic data exchange process to solve the optimal solution. A practical 22 kV grid-connected microgrid of Thailand is used as a test system to demonstrate the effectiveness of the planning and control strategy for the BESS installation. The simulation test results show that the proposed optimal BESS allocation and control strategy can significantly reduce the microgrid’s energy loss and peak demand and increase energy shaving leading to enhanced grid resiliency and reliability.

Coordinated Planning of Fixed and Mobile Charging Facilities for Electric Vehicles on Highways

This paper presents a bilevel planning framework to coordinate truck mobile chargers (TMCs) and fixed chargers (FCs) on highways to promote charging flexibility and provide more choices for electric vehicle (EV) users. A collaborative location optimization (CLO) approach is developed at the upper level to optimize the location of charging stations. Furthermore, a collaborative capacity optimization (CCO) approach is formulated at the lower level to optimize the capacity of TMC and FC at candidate stations. In the proposed framework, origin-destination (OD) analysis, Floyd algorithm and Monte Carlo simulation (MCS) are employed to generate the spatial-temporal distribution of charging demand based on historical data. An improved income approach (IIA) is then developed to well capture the heterogeneity of EV users’ charging behavior. The waiting cost of EV users is estimated by their value of time (VOT), which helps them to make a better choice between TMC and FC. To solve the optimal model easily, the big-M method is applied to linearize and convert the nonlinear problem into a mixed-integer linear programming (MILP) model. Meanwhile, the analytical target cascading (ATC) technique is employed to realize the data exchange process between the upper and lower layers. Finally, numerical study demonstrates the effectiveness of the proposed framework and method.

The Influence of the Internet on Study Interests of High School Students in Timor Leste

Several schools in big cities in Timor-Leste already have computers connected to the Internet; these computers are used for school administration and the learning process to facilitate the process of communication and data exchange; each computer must be connected to a wired or wireless network. With a sufficient internet network, I hope students can increase their interest in learning. In this regard, this study aims to explain the positive and negative effects of the Internet on high school students' learning interest in Timor Leste. This research uses a quantitative method, which will be filled with numbers and statistics that will then be described descriptively. Based on the results of this study it is supported by data and analysis of significant conditions, which means that the Internet used is what is expected of the learning interests of high school students in Timor-Leste. Based on the SPSS version 25 data processing test on F count, the results were obtained so that the internet variable (X1) simultaneously had a simultaneous impact on learning interest (Y) for high school students in Timor-Leste with an F test score of 17.614 and a significant level of 0.000. And sourced from the test of the coefficient of determination that the Internet (X1) has on the learning interest of high school students in Timor-Leste. The summary model value of R<sup>2</sup> is 0.317.

Detecting Smart Contract Vulnerabilities with Combined Binary and Multiclass Classification

The development of Distributed Ledger Technology (DLT) is pushing toward automating decentralized data exchange processes. One of the key components of this evolutionary step is facilitating smart contracts that, in turn, come with several additional vulnerabilities. Despite the existing tools for analyzing smart contracts, keeping these systems running and preserving performance while maintaining a decent level of security in a constantly increasing number of contracts becomes challenging. Machine Learning (ML) methods could be utilized for analyzing and detecting vulnerabilities in DLTs. This work proposes a new ML-based two-phase approach for the detection and classification of vulnerabilities in smart contracts. Firstly, the system’s operation is set up to filter the valid contracts. Secondly, it focuses on detecting a vulnerability type, if any. In contrast to existing approaches in this field of research, our algorithm is more focused on vulnerable contracts, which allows to save time and computing resources in the production environment. According to the results, it is possible to detect vulnerability types with an accuracy of 0.9921, F1 score of 0.9902, precision of 0.9883, and recall of 0.9921 within reasonable execution time, which could be suitable for integrating existing DLTs.

Towards Accelerating Data Intensive Application's Shuffle Process Using SmartNICs

The wide adoption of the emerging SmartNIC technology creates new opportunities to offload application-level computation into the networking layer, which frees the burden of host CPUs, leading to performance improvement. Shuffle, the all-to-all data exchange process, is a critical building block for network communication in distributed data-intensive applications and can potentially benefit from SmartNICs. In this paper, we develop SmartShuffle, which accelerates the data-intensive application's shuffle process by offloading various computation tasks into the SmartNIC devices. SmartShuffle supports offloading both low-level network functions, including data partitioning and network transport, and high-level computation tasks, including filtering, aggregation, and sorting. SmartShuffle adopts a coordinated offload architecture to make sender-side and receiver-side SmartNICs jointly contribute to the benefits of shuffle computation offload. SmartShuffle carefully manages the tight and time-varying computation and memory constraints on the device. We propose a liquid offloading approach, which dynamically migrates operators between the host CPU and the SmartNIC at runtime such that resources in both devices are fully utilized. We prototype SmartShuffle on the Stingray SoC SmartNICs and plug it into Spark. Our evaluation shows that SmartShuffle improves host CPU efficiency and I/O efficiency with lower job completion time. SmartShuffle outperforms Spark, and Spark RDMA by up to 40% on TPC-H.