Reduce Data Volume Research Articles

LEAPS data strategy

The continuous evolution of photon sources and their instrumentation enables more and new scientific endeavors at ever increasing pace. This technological evolution is accompanied by an exponential growth of data volumes of increasing complexity, which must be addressed by maximizing efficiency of scientific experiments and automation of workflows covering the entire data lifecycle, aiming to reduce data volumes while producing FAIR and open data of highest reliability. This papers briefly outlines the strategy of the league of European accelerator-based photon sources user facilities to achieve these goals collaboratively in an efficient and sustainable way which will ultimately lead to an increase in the number of publications.

3D target detection and spectral classification for single-photon LiDAR data.

3D single-photon LiDAR imaging has an important role in many applications. However, full deployment of this modality will require the analysis of low signal to noise ratio target returns and very high volume of data. This is particularly evident when imaging through obscurants or in high ambient background light conditions. This paper proposes a multiscale approach for 3D surface detection from the photon timing histogram to permit a significant reduction in data volume. The resulting surfaces are background-free and can be used to infer depth and reflectivity information about the target. We demonstrate this by proposing a hierarchical Bayesian model for 3D reconstruction and spectral classification of multispectral single-photon LiDAR data. The reconstruction method promotes spatial correlation between point-cloud estimates and uses a coordinate gradient descent algorithm for parameter estimation. Results on simulated and real data show the benefits of the proposed target detection and reconstruction approaches when compared to state-of-the-art processing algorithms.

Robust speech recognition based on deep learning for sports game review

Abstract To verify the feasibility of robust speech recognition based on deep learning in sports game review. In this paper, a robust speech recognition model is built based on the generative adversarial network GAN algorithm according to the deep learning model. And the loss function, optimization function and noise reduction front-end are introduced in the model to achieve the optimization of speech extraction features through denoising process to ensure that accurate speech review data can be derived even in the game scene under noisy environment. Finally, the experiments are conducted to verify the four directions of the model algorithm by comparing the speech features MFCC, FBANK and WAVE. The experimental results show that the speech recognition model trained by the GSDNet model algorithm can reach 89% accuracy, 56.24% reduction of auxiliary speech recognition word error rate, 92.61% accuracy of speech feature extraction, about 62.19% reduction of training sample data volume, and 94.75% improvement of speech recognition performance in the speech recognition task under noisy environment. It shows that the robust speech recognition based on deep learning can be applied to sports game reviews, and also can provide accurate voice review information from the noisy sports game scene, and also broaden the application area for deep learning models.

Lossy Image Compression in a Preclinical Multimodal Imaging Study.

The growing use of multimodal high-resolution volumetric data in pre-clinical studies leads to challenges related to the management and handling of the large amount of thesedatasets. Contrarily to the clinical context, currently there are no standard guidelines to regulate the use of image compression in pre-clinical contexts as a potential alleviationof this problem. In this work, the authors study the application of lossy image coding to compress high-resolution volumetric biomedical data. The impact of compression on the metricsand interpretation of volumetric data was quantified for a correlated multimodal imaging study to characterize murine tumor vasculature, using volumetric high-resolution episcopic microscopy (HREM), micro-computed tomography (µCT), and micro-magnetic resonance imaging (µMRI). The effects of compression were assessed by measuring task-specific performances of several biomedical experts who interpreted and labeled multiple data volumes compressed at different degrees. Wedefined trade-offs between data volume reduction and preservation of visual information, which ensured the preservation of relevant vasculature morphology at maximum compression efficiency across scales. Using the Jaccard Index (JI) and the average Hausdorff Distance (HD) after vasculature segmentation, we coulddemonstrate that, in this study, compression that yields toa 256-fold reduction of the data size allowed to keep the error induced by compression below the inter-observer variability, with minimal impact on the assessment of the tumor vasculature across scales.

SZ3: A Modular Framework for Composing Prediction-Based Error-Bounded Lossy Compressors

Today's scientific simulations require a significant reduction of data volume because of extremely large amounts of data they produce and the limited I/O bandwidth and storage space. Error-bounded lossy compression has been considered one of the most effective solutions to the above problem. In practice, however, the best-fit compression method often needs to be customized or optimized in particular because of diverse characteristics in different datasets and various user requirements on the compression quality and performance. In this paper, we address this issue with a novel modular, composable compression framework named SZ3. Our contributions are four-folds. (1) We develop SZ3 which features an innovative modular abstraction for the prediction-based compression framework, such that compression modules can be plugged in easily to create new compressors based on characteristics of data and user requirements. (2) We create a new compression pipeline by SZ3 for GAMESS data, which significantly improves the compression ratios over state-of-the-art compressors. (3) We develop an adaptive compression pipeline by SZ3 for APS data with minimal efforts, which leads to the best rate-distortion among all existing error-bounded lossy compressors for any bit-rate. (4) We compare the sustainability of SZ3 with leading error-bounded prediction-based compressors, and then demonstrate the necessity of diverse pipelines by integrating and evaluating several compression pipelines on diverse scientific datasets from multiple disciplines. Experiments show that SZ3 incurs very limited overhead in compressor integration and our customized compression pipelines lead to up to 20% improvement in compression ratios under the same data distortion, when compared with the best existing approach.

On classifier performance for remote sensing images compressed by different coders

Remote sensing data are widely used in numerous applications. A conventional task solved using remote sensing images is their classification. The classification maps are commonly produced by some pre-trained classifiers applied either to uncompressed or compressed images where lossy compression is often needed and employed in practice due to the necessity to reduce data volume at stages of image transfer and storage. Then, the classification accuracy depends on the characteristics of an image, a classifier, and a coder used. The main subject of this paper is the factors that determine classification accuracy. One of them is compressed image quality. We fix the quality of compressed image quality characterized by the peak signal-to-noise ratio for several coders and rely on the same training approach. Our goal is twofold. First, we would like to consider classification accuracy for two approaches to classifier training: based on undistorted data and images with simulated distortions. Second, our desire is to compare the performance of different techniques of image compression. The task of this paper is to obtain an idea is it worth training the neural network classifier for uncompressed images or images of similar quality to the quality of compressed data to be classified. The coder’s influence on classification results is of special interest as well. The main results are the following. First, the classification accuracy is almost the same for classifiers trained for uncompressed and simulated compressed data for the general distortion model. Second, there is a certain difference in the classification results for different compression techniques studied. Lightly better classification results are observed for data produced by more sophisticated (modern) coders. Experiments have been carried out for two real-life three-channel Sentinel-2 images of Kharkiv and the Kharkiv region having different complexity. Four typical classes have been considered. As a conclusion, it is possible to state that either the general model of distortions must be modified or the classifier training should be performed for data produced by the corresponding compression technique.

A preliminary study on the dispersed classification system for recognizing safety of drivers’ maneuvers

Riskful and aggressive driving are significant problems in road transportation. Numerous studies prove drivers’ behavior is critical in most road accidents and contributes significantly to fuel consumption and emissions. In an attempt to improve road safety, it is important to be able to detect such unskilful and irresponsible drivers on a mass scale. Thus an automated onboard driving safety assessment system is required. Transmitting amounts of telemetric data required to assess driving style is unpractical and economically unjustified, thus some data processing must take place in an Onboard Unit (OBU). As a first step in driving safety assessment is detecting maneuvers, this is the task, that can be done in the OBU. In this paper, we propose a Finite State Machine based algorithm, which could run on OBU and allows marking of basic maneuvers in the telemetry datastream. This allows the calculation of additional features, aggregation, and transmission of data useful for classification the of driving safety, and thus a vast reduction in data volume sent over the air. Also, an example of detecting unsafe maneuver using such aggregated data is given.

SAR Imaging in Frequency Scan Mode: System Optimization and Potentials for Data Volume Reduction

Frequency scanning (FScan) is an innovative acquisition mode for synthetic aperture radar (SAR) systems. The method is based on the frequency-dependent beam pointing capabilities of phased array antennas, artificially increased via the combined use of true time delays and phase shifters within the array antenna. By this, typical limitations of conventional SAR systems in terms of achievable swath width and azimuth resolution can be mitigated, and so a wide swath can be imaged maintaining a fine azimuthal resolution. In the first part of the article, we introduce the theoretical concept, which is necessary to evaluate the reduced echo window length (EWL) with respect to equivalent stripmap data and the implications for the transmit pulse characterization. An FScan sensor flying in a TerraSAR-X-like orbit is shown to be capable of imaging an 80-km wide swath with 1-m azimuth resolution. The resulting time–frequency properties of the recorded raw data make the traditional SAR data compression algorithms such as block-adaptive quantization (BAQ) highly inefficient in this case. Therefore, the second part of the article investigates dedicated quantization methods for efficient data volume reduction in FScan systems. Different solutions are investigated and evaluated through simulations. Various transformations of the raw data have been exploited to optimize the encoding process, including deramping, fast Fourier transform (FFT), and blockwise approaches. Compared with standard BAQ in the time domain, the suggested data compression methods significantly improve the resulting signal-to-quantization noise ratio, allowing for the reduction in the overall data volume by about 60% for the considered system scenario, while maintaining robustness in the presence of inhomogeneous scene characteristics at the cost of a modest complexity increase for its on-board implementation.

Balancing Information Preservation and Data Volume Reduction: Adaptive Flow Aggregation in Flow Metering Systems

The critical role of network traffic measurement and analysis extends across a range of network operations, ensuring quality of service, security, and efficient resource management. Despite the ubiquity of flow-level measurement, the escalating size of flow entries presents significant scalability issues. This study explores the implications of adaptive gradual flow aggre- gation, a solution devised to mitigate these challenges, on flow information distortion. The investigation maintains flow records in buffers of varying aggregation levels, iteratively adjusted based on the changing traffic load mirrored in CPU and memory utilization. Findings underscore the efficiency of adaptive gradual flow aggregation, particularly when applied to a specific buffer, yielding an optimal balance between information preservation and memory utilization. The paper highlights the particular significance of this approach in Internet of Things (IoT) and contrasted environments, characterized by stringent resource constraints. Consequently, it casts light on the imperative for adaptability in flow aggregation methods, the impact of these techniques on information distortion, and their influence on network operations. This research offers a foundation for future studies targeting the development of more adaptive and effective flow measurement techniques in diverse and resource-limited network environments.

IPrivJoin: An ID-Private Data Join Framework for Privacy-Preserving Machine Learning

The world has observed an increasing trend in the development of Privacy-Preserving Machine Learning (PPML) for cross-silo collaborative model training over sensitive data. As the first essential step of cross-silo PPML, it is critical that the parties can align their dataset with privacy assurance, i.e., <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">private data join</i> . However, the existing private data join methods typically leak the ID information in the dataset intersection, which often raises privacy concerns. In this work, we propose iPrivJoin: a novel framework of ID-private data join for PPML. Compared with naively using circuit-based Private Set Intersection (circuit-PSI) for data join, the proposed framework has two advantages. (i) data volume reduction. iPrivJoin utilizes oblivious shuffle to securely trim off the redundant data that is outside the intersection, while the entire dataset needs to be carried to further process in the circuit-PSI based approach. (ii) efficiency improvement. iPrivJoin introduces a new private encoding technique to avoid the expensive circuit evaluation that is needed in circuit-PSI. As a result, compared with directly using circuit-PSI, PPML with iPrivJoin enjoys approximately 3× of speedup. Moreover, we propose a new oblivious shuffle protocol, which may be of independent interest. It achieves 1.44× of speedup to the state-of-the-art in the real-world WAN network setting.

Hyperspectral Image Band Selection Based on CNN Embedded GA (CNNeGA)

Hyperspectral images (HSI) are a powerful source of reliable data in various remote sensing applications. But due to the large number of bands, HSI has information redundancy, and methods are often used to reduce the number of spectral bands. Band selection (BS) is used as a preprocessing solution to reduce data volume, increase processing speed, and improve methodology accuracy. However, most conventional BS approaches are unable to fully explain the interaction between spectral bands and evaluate the representation and redundancy of the selected band subset. This study first examines a supervised band selection method that allows the selection of the required number of bands. A deep network with 3D-convolutional layers embedded in a genetic algorithm (GA) The GA uses embedded 3D-CNN (CNNeGA) as a fitness function. GA also considers the parent check box. The parent check box (parent subbands) is designed to make genetic operators more effective. In addition, the effectiveness of increasing the attention layer to a 3D-CNN and converting this model to spike neural networks (SNNs) has been investigated in terms of accuracy and complexity over time. The evaluation of the proposed method and the obtained results are satisfactory. The accuracy improved from 6 to 21 percent. Accuracy between 90 to 99 percent has been obtained in each evaluation mode.

Dynamic Predictive Quantization for Staggered SAR: Experiments With Real Data

Present and future spaceborne synthetic aperture radar (SAR) missions are designed to acquire an increasingly large amount of onboard data. This is a consequence of the use of large bandwidths, multiple polarizations, and the acquisition of large swath widths at fine spatial resolutions, which result in challenging requirements in terms of onboard memory and downlink capacity. In this scenario, SAR raw data quantization represents an essential aspect, as it affects the volume of data to be stored and transmitted to the ground as well as the quality of the resulting SAR products. Dynamic predictive block-adaptive quantization (DP-BAQ) is a novel technique, recently proposed by the authors, consisting of a low-complexity data compression method, and its application is particularly suitable for staggered SAR systems. DP-BAQ exploits the existing correlation among the azimuth raw data samples by applying linear predictive coding (LPC). This results in a data rate reduction of up to 25% with respect to state-of-the-art SAR quantization methods. In this letter, we test and validate the potential of DP-BAQ on airborne SAR data which emulates the system scenario of Tandem-L, a German Aerospace Center (DLR) mission proposal for a bistatic L-band system. For this purpose, an experimental SAR image has been acquired at the L-band by the airborne DLR flugzeug-SAR (F-SAR) sensor over the Kaufbeuren area, in Southern Germany. In order to simulate the staggered SAR acquisition mode, we implemented a dedicated resampling and filtering of the data. Our analyses confirm the effectiveness of DP-BAQ for efficient data volume reduction, exhibiting a consistent and promising performance when tested on areas characterized by different land cover types and backscatter statistics.

Monitoring of Thermal Comfort and Air Quality for Sustainable Energy Management inside Hospitals Based on Online Analytical Processing and the Internet of Things.

There is a need to ensure comfortable conditions for hospital staff and patients from the point of view of thermal comfort and air quality so that they do not affect their performance. We consider the need for hospital employees and patients to enjoy conditions of greater well-being during their stay. This is understood as a comfortable thermal sensation and adequate air quality, depending on the task they are performing. The contribution of this article is the formulation of the fundamentals of a system and platform for monitoring thermal comfort and Indoor Air Quality (IAQ) in hospitals, based on an Internet of Things platform composed of a low-cost sensor node network that is capable of measuring critical variables such as humidity, temperature, and Carbon Dioxide (CO2). As part of the platform, a multidimensional data model with an On-Line Analytical Processing (OLAP) approach is presented that offers query flexibility, data volume reduction, as well as a significant reduction in query response times. The experimental results confirm the suitability of the platform’s data model, which facilitates operational and strategic decision making in complex hospitals.

Real-time Transformer Vandalism Detection by Application of Tuned Hyper Parameter Deep Learning Model

Vandalism is an illegal act of cannibalism or change of face to a private or public property by human beings for re-sale of parts or to punish the property owner. Initial research findings on transformer Vandalism detection have fallen short of human image recognition of the vandal in real-time but only does detection of activities after the damage is done or as it occurs. Automated real-time systems using sensor feed to a trained deep learning model is a new transformer vandalism detection approach with capabilities of three-dimensional image learning, extracting important image features automatically and temporal output prediction. This paper aims at distinguishing the human object entering a zoned transformer area without permission to take away or modify the established infrastructure, so that the Vandal can be arrested before causing any damage to the transformer. The researchers identified a multiplicative hybrid model combining convolutional neural networks and long short-term memory for application to vandalism problem to detect the image of a vandal as it enters a restricted transformer installation site. The image recognition accuracy can be improved by tuning the model hyper-parameters and the specific hyper-parameters considered in this research work are number of model layers and epochs. The human object is distinguishing by applying the image features taken with Image sensor to a trained deep learning model. The hybrid deep learning method increases the output prediction accuracy from the input data and lowers computational processing complications due to a reduced data volume through pooling. The system is trained and validated using ImageNet dataset. Results achieved by five layers and sixty epochs is 99% recognition accuracy. The performance of the system with an increased number of layers and epochs to five and sixty respectively was the best result as compared with lower layers and epochs. Further increase of these parameters resulted to system overfitting.

Application of Feature Selection and Deep Learning for Cancer Prediction Using DNA Methylation Markers.

DNA methylation is a process that can affect gene accessibility and therefore gene expression. In this study, a machine learning pipeline is proposed for the prediction of breast cancer and the identification of significant genes that contribute to the prediction. The current study utilized breast cancer methylation data from The Cancer Genome Atlas (TCGA), specifically the TCGA-BRCA dataset. Feature engineering techniques have been utilized to reduce data volume and make deep learning scalable. A comparative analysis of the proposed approach on Illumina 27K and 450K methylation data reveals that deep learning methodologies for cancer prediction can be coupled with feature selection models to enhance prediction accuracy. Prediction using 450K methylation markers can be accomplished in less than 13 s with an accuracy of 98.75%. Of the list of 685 genes in the feature selected 27K dataset, 578 were mapped to Ensemble Gene IDs. This reduced set was significantly (FDR < 0.05) enriched in five biological processes and one molecular function. Of the list of 1572 genes in the feature selected 450K data set, 1290 were mapped to Ensemble Gene IDs. This reduced set was significantly (FDR < 0.05) enriched in 95 biological processes and 17 molecular functions. Seven oncogene/tumor suppressor genes were common between the 27K and 450K feature selected gene sets. These genes were RTN4IP1, MYO18B, ANP32A, BRF1, SETBP1, NTRK1, and IGF2R. Our bioinformatics deep learning workflow, incorporating imputation and data balancing methods, is able to identify important methylation markers related to functionally important genes in breast cancer with high accuracy compared to deep learning or statistical models alone.

Continuous on‐board behaviour classification using accelerometry: A case study with a new GPS‐3G‐Bluetooth system in Pacific black ducks

Abstract Over the past two decades, accelerometer (ACC) data have been increasingly used to study animal behaviours and energetics. However, the large amount of raw ACC data can be a burden to device storage and power consumption and in many cases may also require device retrieval for data collection. On‐board data processing to reduce data volume and power consumption for data transmission may hold promise to alleviate these problems and allow for next‐generation, smart trackers. We developed a tracking system processing raw ACC data on‐board of trackers into behaviours using an XGBoost machine learning model. We used this system on six free‐ranging Pacific black ducks Anas superciliosa to study eight behaviours every 2 s. The on‐board XGBoost model for behaviour classification had 92.04% overall accuracy. One day of continuous behaviour records was compressed on‐board to as little as 17.28 kB for routine transmission via the 3G network. We received behaviour data from the six ducks continuously for periods ranging between 56 days and 14 months. On‐board processing of raw ACC data and data transmission proofed highly energy efficient and came at a minimal weight cost to the trackers, providing great potential to open up new areas in ecological and behavioural research.

Blockchain &amp; Multi-Agent System: A New Promising Approach for Cloud Data Integrity Auditing with Deduplication

Recently, data storage represents one of the most important services in Cloud Computing. The cloud provider should ensure two major requirements which are data integrity and storage efficiency. Blockchain data structure and the efficient data deduplication represent possible solutions to address these exigencies. Several approaches have been proposed, some of them implement deduplication in Cloud server side, which involves a lot of computation to eliminate the redundant data and it becomes more and more complex. Therefore, this paper proposed an efficient, reliable and secure approach, in which the authors propose a Multi-Agent System in order to manipulate deduplication technique that permits to reduce data volumes thereby reduce storage overhead. On the other side, the loss of physical control over data introduces security challenges such as data loss, data tampering and data modification. To solve similar problems, the authors also propose Blockchain as a database for storing metadata of client files. This database serves as logging database that ensures data integrity auditing function.

Adaptive Multicale Transformation Run-Length Code-Based Test Data Compression in Benchmark Circuits

Test data volume reduction and power consumption during testing time outlines are two main problems for Very Large Scale Integration (VLSI) gadgets. Most the code-based arrangements have been utilized to diminish test data volume, although the most notable way that test data volume is high. The switching action that happens between the test carriers leads would expand power consumption. This work presents a compression/decompression methodology for limiting the amount of test data that should be kept on a tester and conveyed to each center in a System on a Chip (SOC) during a test utilizing the Adaptive Multiscale Transformation Run Length Code (AMT-RLC). The data transmission capacity between the tester and the SOC is a block when testing modern SOCs with a few centers to check how long the test can run. The proposed work keeps the tester in the SOC and the test vectors in a compacted memory structure. A modest quantity of On-chip equipment is needed to release different test vectors. Due to its variouscharacteristics that decrease the test data volume, the Adaptive Multiscale Transformation Run Length Code is picked for a center test vector framework. These attributes ensure that the code words overhead is decoded (and applied to the center's output exhibit grouping commitment) utilizing an exact cell cluster that occupies less space. The suggested strategy may provide test data compression with much-reduced Area overhead for the decoder and low power consumption, according to the results. The abstract shall be running continuously (not structured) and shall not include reference citations. Abbreviations should be defined in full the first time they appear. Abbreviations could be then used, quoted in-between parentheses.

Virtual Reality Application of the Fortress Al Zubarah in Qatar Including Performance Analysis of Real-Time Visualisation

Technological advancements in the area of Virtual Reality (VR) in the past years have the potential to fundamentally impact our everyday lives. VR makes it possible to explore a digital world with a Head-Mounted Display (HMD) in an immersive, embodied way. In combination with current tools for 3D documentation, modelling and software for creating interactive virtual worlds, VR has the means to play an important role in the conservation and visualisation of cultural heritage (CH) for museums, educational institutions and other cultural areas. Corresponding game engines offer tools for interactive 3D visualisation of CH objects, which makes a new form of knowledge transfer possible with the direct participation of users in the virtual world. However, to ensure smooth and optimal real-time visualisation of the data in the HMD, VR applications should run at 90 frames per second. This frame rate is dependent on several criteria including the amount of data or number of dynamic objects. In this contribution, the performance of a VR application has been investigated using different digital 3D models of the fortress Al Zubarah in Qatar with various resolutions. We demonstrate the influence on real-time performance by the amount of data and the hardware equipment and that developers of VR applications should find a compromise between the amount of data and the available computer hardware, to guarantee a smooth real-time visualisation with approx. 90 fps (frames per second). Therefore, CAD models offer a better performance for real-time VR visualisation than meshed models due to the significant reduced data volume.

Study on various curvilinear data representations and their impact on mask and wafer manufacturing

Inverse lithography technology (ILT) optical proximity correction is going to play a critical role in addressing challenges of optical and EUV lithography as the industry pushes toward advanced nodes. One major barrier in adoption of ILT has been the mask writer’s inability to efficiently write curvilinear patterns. With the introduction of multibeam mask writers, this barrier has been removed and widespread adoption of ILT is imminent. Traditionally, mask writers have accepted only trapezoidal inputs to the tool, though recent trends show that mask writers are adopting newer formats that already reduce file size. However, as the ILT shape complexity and data volume increases further for 5 nm nodes and beyond, the explosion of mask pattern data file size becomes a major concern. Therefore, there is a need for the industry to look toward other compact formats of data representation that will be capable of serving well for multiple generations of mask making. We compare various curvilinear data representation schemes and their value in the curvilinear ILT-based mask manufacturing flow. We demonstrate that given the nature of curvilinear data, representing it using native curve formats has significant value to reduce file size for future mask making flows. The same format may not be applicable for all types of features in the input mask. These options will be discussed. We will compare the value of such exotic representations with regular simplification approaches that reduce data volume using standard methods and discuss the extents and limits of all these techniques. To evaluate practical use of curvilinear representation in place of conventional piecewise linear representation, we manufacture and measure a photomask to evaluate the accuracy of curvilinear representations. Finally, we use EUV AIMS to assess the impact of curvilinear representation on wafer process window.