Large Data Transfer Research Articles

Large Data Transfer Optimization for Improved Robustness in Real-Time V2X-Communication

Large Data Transfer Optimization for Improved Robustness in Real-Time V2X-Communication

Performance Analysis of Compression Algorithms on Matrix Data: Data Transfer Optimization in Microservices Architectures

With the rapid proliferation of microservices architectures these days, the efficient and fast transfer of large matrix data between services has become a significant challenge. This study presents an analysis aimed at finding solutions to this challenge. The analysis addresses the compression and decompression of large matrix data, focusing on lossless compression algorithms to optimize data transfer without data loss. The study is implemented on an example scenario. This scenario is taken from a project with a microservice architecture. In the example scenario, an image processing service developed in Python programming language generates 640x480 matrix data. After going through a compression algorithm, this data is periodically transferred to a backend service developed in C# programming language. This data is then stored in a database. In the final stage, decompression operations are performed so that this data can be used for reporting. The performance of various compression algorithms in the data compression, database storage and report generation stages is extensively tested. Within the scope of the study, tests were performed using five different compression algorithms (Gzip, Zlib, Deflate, Brotli and Bz2). The results are obtained through performance tests aimed at determining the most optimized end-to-end solution. Analyzing the performance of the compression algorithms on the example scenario, the Brotli algorithm gives the most optimal result in terms of both speed and compression size. This work makes an important contribution to data transfer optimization in microservice architectures and provides a reference for research in this area by presenting the performance analysis of various compression algorithms.

Globus service enhancements for exascale applications and facilities

Many extreme-scale applications require the movement of large quantities of data to, from, and among leadership computing facilities, as well as other scientific facilities and the home institutions of facility users. These applications, particularly when leadership computing facilities are involved, can touch upon edge cases (e.g., terabyte files) that had not been a focus of previous Globus optimization work, which had emphasized rather the movement of many smaller (megabyte to gigabyte) files. We report here on how automated client-driven chunking can be used to accelerate both the movement of large files and the integrity checking operations that have proven to be essential for large data transfers. We present detailed performance studies that provide insights into the benefits of these modifications in a range of file transfer scenarios.

Comparison of WebSocket and Hypertext Transfer Protocol for Transfer of Electronic Health Records.

Electronic health records (EHR) emerged as a digital record of the data that is generated in the healthcare. In this paper the transfer times of EHRs using the Hypertext Transfer Protocol and WebSocket in both local network and wide area network (WAN) are compared. A python web application to serve Fast Health Interoperability Resources (FHIR) records is created and the transfer times of the EHRs over both HTTP and WebSocket connection are measured. 45000 test Patient resources in 20, 50, 100 and 200 resources per Bundle transfers are used. WebSocket showed much better transfer times of large amount of data. These were 18 s shorter in the local network and 342 s shorter in WAN for the 20 resource per Bundle transfer. RESTful APIs are a convenient way to implement EHR servers; on the other hand, HTTP becomes a bottleneck when transferring large amount of data. WebSocket shows better transfer times and thus its superiority in such situations. The problem can be addressed by developing a new communication protocol or by using network tunneling to handle large data transfer of EHRs.

Federated learning–based global road damage detection

AbstractDeep learning is widely used for road damage detection, but it requires extensive, diverse, and well‐labeled data. Centralized model training can be difficult due to large data transfers, storage needs, and computational resources. Data privacy concerns can also hinder data sharing among clients, leaving them to train models on their own data, leading to less robust models. Federated learning (FL) addresses these problems by training models without data sharing, only exchanging model parameters between clients and the server. This study deploys FL along with YOLOv5l to generate models for single‐ and multi‐country applications. These models gave 21%–25% lesser mean average precision (mAP) than centralized models but outperformed local client models by 1.33%–163% on the global test data.

Publisher Correction to: Leveraging chaos for enhancing encryption and compression in large cloud data transfers

Publisher Correction to: Leveraging chaos for enhancing encryption and compression in large cloud data transfers

Leveraging chaos for enhancing encryption and compression in large cloud data transfers

Leveraging chaos for enhancing encryption and compression in large cloud data transfers

A Rydberg atom-based amplitude-modulated receiver using the dual-tone microwave field

We propose a Rydberg atom-based receiver for amplitude-modulation (AM) reception utilizing a dual-tone microwave field. The pseudo-random binary sequence (PRBS) signal is encoded in the basic microwave field (B-MW) at the frequency of 14.23 GHz. The signal can be decoded by the atomic receiver itself but more obvious with the introduction of an auxiliary microwave (A-MW) field. The receiver’s amplitude variations corresponding to microwave field are simulated by solving density matrices to give this mechanism theoretical support. An appropriate AM frequency is obtained by optimizing the signal-to-noise ratio, guaranteeing both large data transfer capacity (DTC) and high fidelity of the receiver. The power of two MW fields, along with the B-MW field frequency, is studied to acquire larger DTC and wider operating bandwidth. Finally, the readout of PRBS signals is performed by both the proposed and conventional mechanisms, and the comparison proves the obvious increment of DTC with the proposed scheme. This proof-of-principle demonstration exhibits the potential of the dual-tone scheme and offers a novel pathway for Rydberg atom-based microwave communication, which is beneficial for long-distance communication and weak signal perception outside the laboratory.

NOTED: An intelligent network controller to improve the throughput of large data transfers in File Transfer Services by handling dynamic circuits

The NOTED (Network Optimised Transfer of Experimental Data) project has successfully demonstrated the ability to dynamically provision network links to increase the effective bandwidth available for FTS-driven transfers between endpoints, such as WLCG sites, by inspecting on-going data transfers and so identifying those that are bandwidth-limited for a long period of time. Recently, the architecture of NOTED has been improved and the software has been packaged for easy distribution. These improved capabilities and features of NOTED have been tested and demonstrated at various international conferences. For example, during demonstrations at Supercomputing 2022, independent instances of NOTED at CHCERN (Switzerland) and DE-KIT (Germany) monitored large data transfers generated by the ATLAS experiment between these sites and CA-TRIUMF (Canada). We report here on this and other events, highlighting how NOTED can predict link congestion or a notable increase in the network utilisation over an extended period of time and, where appropriate, automatically reconfigure network topology to introduce an additional or an alternative and betterperforming path by using dynamic circuit provisioning systems such as SENSE and AutoGOLE.

BLE Periodic Advertising Based Energy Efficient Sensor Node Operation for Transfer of Large Data in Monitoring Applications

BLE Periodic Advertising Based Energy Efficient Sensor Node Operation for Transfer of Large Data in Monitoring Applications

The role of compression in large scale data transfer and storage of typical biomedical signals at hospitals.

In modern hospitals, monitoring patients' vital signs and other biomedical signals is standard practice. With the advent of data-driven healthcare, Internet of medical things, wearable technologies, and machine learning, we expect this to accelerate and to be used in new and promising ways, including early warning systems and precision diagnostics. Hence, we see an ever-increasing need for retrieving, storing, and managing the large amount of biomedical signal data generated. The popularity of standards, such as HL7 FHIR for interoperability and data transfer, have also resulted in their use as a data storage model, which is inefficient. This article raises concern about the inefficiency of using FHIR for storage of biomedical signals and instead highlights the possibility of a sustainable storage based on data compression. Most reported efforts have focused on ECG signals; however, many other typical biomedical signals are understudied. In this article, we are considering arterial blood pressure, photoplethysmography, and respiration. We focus on simple lossless compression with low implementation complexity, low compression delay, and good compression ratios suitable for wide adoption. Our results show that it is easy to obtain a compression ratio of 2.7:1 for arterial blood pressure, 2.9:1 for photoplethysmography, and 4.1:1 for respiration.

A High Performance and Robust FPGA Implementation of a Driver State Monitoring Application.

A high-performance Driver State Monitoring (DSM) application for the detection of driver drowsiness is presented in this paper. The popular Ensemble of Regression Trees (ERTs) machine learning method has been employed for the alignment of 68 facial landmarks. Open-source implementation of ERTs for facial shape alignment has been ported to different platforms and adapted for the acceleration of the frame processing speed using reconfigurable hardware. Reducing the frame processing latency saves time that can be used to apply frame-to-frame facial shape coherency rules. False face detection and false shape estimations can be ignored for higher robustness and accuracy in the operation of the DSM application without sacrificing the frame processing rate that can reach 65 frames per second. The sensitivity and precision in yawning recognition can reach 93% and 97%, respectively. The implementation of the employed DSM algorithm in reconfigurable hardware is challenging since the kernel arguments require large data transfers and the degree of data reuse in the computational kernel is low. Hence, unconventional hardware acceleration techniques have been employed that can also be useful for the acceleration of several other machine learning applications that require large data transfers to their kernels with low reusability.

Deep Learning Inferencing with High-performance Hardware Accelerators

As computer architectures continue to integrate application-specific hardware, it is critical to understand the relative performance of devices for maximum app acceleration. The goal of benchmarking suites, such as MLPerf for analyzing machine learning (ML) hardware performance, is to standardize a fair comparison of different hardware architectures. However, there are many apps that are not well represented by these standards that require different workloads, such as ML models and datasets, to achieve similar goals. Additionally, many apps, like real-time video processing, are focused on latency of computations rather than strictly on throughput. This research analyzes multiple compute architectures that feature ML-specific hardware on a case study of handwritten Chinese character recognition. Specifically, AlexNet and a custom version of GoogLeNet are benchmarked in terms of their streaming latency and maximum throughput for optical character recognition. Considering that these models are composed of fundamental neural network operations yet architecturally different from each other, these models can stress devices in different yet insightful ways that generalizations of the performance of other models can be drawn from. Many devices featuring ML-specific hardware and optimizations are analyzed including Intel and AMD CPUs, Xilinx and Intel FPGAs, NVIDIA GPUs, and Google TPUs. Overall, ML-oriented hardware added to the Intel Xeon CPUs helps to boost throughput by 3.7× and to reduce latency by up to 34.7×, which makes the latency of Intel Xeon CPUs competitive on more parallel models. The TPU devices were limited in terms of throughput due to large data transfer times and not competitive in terms of latency. The FPGA frameworks showcase the lowest latency on the Xilinx Alveo U200 FPGA achieving 0.48 ms on AlexNet using Mipsology Zebra and 0.39 ms on GoogLeNet using Vitis-AI. Through their custom acceleration datapaths coupled with high-performance SRAM, the FPGAs are able to keep critical model data closer to processing elements for lower latency. The massively parallel and high-memory GPU devices with Tensor Core accelerators achieve the best throughput. The NVIDIA Tesla A100 GPU showcases the highest throughput at 42,513 and 52,484 images/second for AlexNet and GoogLeNet, respectively. 1

A Survey of High Performance Computing (HPC) Infrastructure in Thailand

In the last decade, government organizations and private companies in Thailand have invested considerably in computing resources. Research collaborations typically band together based on shared interests and propose projects to compete for funding. Thus, many institutions frequently use similar computing resources. A bird's eye view of high-performance computer infrastructure in Thailand is important in many ways. To the best of our knowledge and ability, we gathered information on government procurements of HPC resources in the past five years, which cover several organizations and ministries. We list the system specifications and tabulate the target application areas for each system. The aggregated number of cores and storage space for HPC in Thailand, commissioned during the past five years, is 54,838 cores and 21 PB, respectively. We also describe the large data transfers using UniNet for HPC applications. The survey results can be used by academics and decision-makers to build research agendas and national development strategies.

Space-to-speed architecture supporting acceleration on VHR image processing

One of the major focuses in the remote sensing community is the rapid processing of deep neural networks (DNNs) on very high resolution (VHR) aerial images. Few studies have investigated the acceleration of training and prediction by optimizing the architecture of the DNN system rather than designing a lightweight DNN. Parallel processing using multiple graphics processing units (GPUs) increases VHR image processing performance. It drives extremely large and frequent data transfers (input/output(I/O)) from random access memory (RAM) to GPU memory. As a result, the system bus congestion causes the system to hang, resulting in long latency in training/predicting. In this paper, we evaluate the causes of long latency and propose a space-to-speed (S2S) DNN system to overcome the aforementioned challenges. A three-level memory system aiming to reduce data transfer during system operation is presented. Distribution optimization with parallel processing was used to accelerate the training. Training optimizations on VHR images (such as hot-zone searching and image/ground truth queues for data saving) were used to train the VHR images efficiently. Inference optimization was performed to speed up prediction in the release mode. To verify the efficiency of the proposed system, we used aerial image labeling from the Institut National de Recherche en Informatique et en Automatique (INRIA) and benchmarks from the Massachusetts Institute of Technology Aerial Imagery for Roof Segmentation (MITAIRS) to test the system performance and accuracy. Without the loss of accuracy, the S2S system improved prediction speed on the testing dataset by eight GPUs in a normal setting in both the INRIA dataset (from 534 to 72 s) and the MITAIRS dataset (818 to 120 s). With the prediction in half-float (using float-16 data), an 8-GPU parallel processing increased the speed to 38 s in the INRIA dataset and 83 s in the MITAIRS dataset. In a pressure test, our proposed system operated on 18,000 images with a size of 5000 × 5000 from 18.2 to 1.8 h with the prediction in full-float (using float-32 data) and 43 min with the prediction in half-float, increasing the speed by a factor of 9.78 and 25.3, respectively, when compared to system runs without optimization.

Novel Distributed Beamforming Algorithms for Heterogeneous Space Terrestrial Integrated Network

An integrated space-terrestrial network based on the ultradense low-earth-orbit (LEO) satellite constellations has been envisioned in both 5G and beyond 5G (B5G) networks. This approach is a powerful solution to some key challenges from Internet of Things (IoT) services, such as the lack of link capacity to deal with large data transfer or coverage in the remote areas. This article focuses on the beamforming design for the transmissions from multiple LEO satellites, equipped with massive phased array antenna, to a large number of heterogeneous terrestrial terminals. Superposition coding-based beamforming is efficient in dealing with the receiver heterogeneity, but at the cost of higher computational complexity. Based on the dual decomposition theory as well as deep neural networks (DNNs), this article proposes to combine the nonlinear approximation ability of DNNs with distributed algorithms. This combination not only supports advanced nonorthogonal beamforming algorithms for achieving superior throughput performance, but also keeps the overall computational complexity low and enables the beamforming process to be speed up dramatically through parallel computing.

A Batch Processing Technique for Wearable Health Crowd-Sensing in the Internet of Things

The influx of wearable sensor devices has influenced a new paradigm termed wearable health crowd-sensing (WHCS). WHCS enables wearable data collection through active sensing to provide health monitoring to users. Wearable sensing devices capture data and transmit it to the cloud for data processing and analytics. However, data sent to the cloud is vulnerable to on-path attacks. The bandwidth limitation issue is also another major problem during large data transfers. Moreover, the WHCS faces several anonymization issues. In light of this, this article presents a batch processing method to solve the identified issues in WHCS. The proposed batch processing method provides an aggregate authentication and verification approach to resolve bandwidth limitation issues in WHCS. The security of our scheme shows its resistance to forgery and replay attacks, as proved in the random oracle (ROM), while offering anonymity to users. Our performance analysis shows that the proposed scheme achieves a lower computational and communication cost with a reduction in the storage overhead compared to other existing schemes. Finally, the proposed method is more energy-efficient, demonstrating that it is suitable for the WHCS system.

The control mechanism of distributed MQTT brokers for large volume data transfer and its prototype system

The control mechanism of distributed MQTT brokers for large volume data transfer and its prototype system

Theoretical Investigation of an Air-Slot Mode-Size Matcher between Dielectric and MDM Plasmonic Waveguides

Hybrid integration of dielectric and plasmonic waveguides is necessary to reduce the propagation losses due to the metallic interactions and support of nanofabrication of plasmonic devices that deal with large data transfer. In this paper, we propose a direct yet efficient, very short air-slot coupler (ASC) of a length of 36 nm to increase the coupling efficiency between a silicon waveguide and a silver-air-silver plasmonic waveguide. Our numerical simulation results show that having the ASC at the interface makes the fabrication process much easier and ensures that light couples from a dielectric waveguide into and out of a plasmonic waveguide. The proposed coupler works over a broad frequency range achieving a coupling efficiency of 86% from a dielectric waveguide into a metal-dielectric-metal (MDM) plasmonic waveguide and 68% from a dielectric waveguide to an MDM plasmonic waveguide and back into another dielectric waveguide. In addition, we show that even if there are no high-precision fabrication techniques, light couples from a conventional dielectric waveguide (CDW) into an MDM plasmonic waveguide as long as there is an overlap between the CDW and ASC, which reduces the fabrication process tremendously. Our proposed coupler has an impact on the miniaturization of ultracompact nanoplasmonic devices.

Neural networks for on-the-fly single-shot state classification

Neural networks have proven to be efficient for a number of practical applications ranging from image recognition to identifying phase transitions in quantum physics models. In this paper, we investigate the application of neural networks to state classification in a single-shot quantum measurement. We use dispersive readout of a superconducting transmon circuit to demonstrate an increase in assignment fidelity for both two and three state classifications. More importantly, our method is ready for on-the-fly data processing without overhead or need for large data transfer to a hard drive. In addition, we demonstrate the capacity of neural networks to be trained against experimental imperfections, such as phase drift of a local oscillator in a heterodyne detection scheme.