Compression Tools Research Articles

Communication-Efficient Federated Learning via Predictive Coding

Federated learning can enable remote workers to collaboratively train a shared machine learning model while allowing training data to be kept locally. In the use case of wireless mobile devices, the communication overhead is a critical bottleneck due to limited power and bandwidth. Prior work has utilized various data compression tools such as quantization and sparsification to reduce the overhead. In this paper, we propose a predictive coding based compression scheme for federated learning. The scheme has shared prediction functions among all devices and allows each worker to transmit a compressed residual vector derived from the reference. In each communication round, we select the predictor and quantizer based on the rate–distortion cost, and further reduce the redundancy with entropy coding. Extensive simulations reveal that the communication cost can be reduced up to 99% with even better learning performance when compared with other baseline methods.

Binomial tree method for option pricing: Discrete cosine transform approach

This paper discusses a new pricing method of European options through the binomial tree model using a discrete cosine transform. The discrete cosine transform has been used as a fundamental tool for image compression, including the creation of JPEG files. A discrete cosine transform was also recently used to derive the price of financial options. This method also enables us to derive the option prices using a binomial tree model. Using this approach, we derive the option prices on the classical Black and Scholes, exponential jump–diffusion, and exponential CGMY models. Because we compute the characteristic function numerically, we can derive option prices in various models without knowing the specific form of the characteristic functions. This study can unfold new research areas such as option pricing on various models, including the non-parametric jump and Lévy diffusion models.

Effective Prediction Modes Design for Adaptive Compression With Application in Video Coding.

Adaptive prediction is an important tool for efficient compression of non-stationary signals. A common approach to achieve adaptivity is to switch between a set of prediction modes, designed to capture variations in signal statistics. The design poses several challenges including: i) catastrophic instability due to statistical mismatch driven by propagation through the prediction loop, and ii) severe non-convexity of the cost surface that is often riddled with poor local minima. Motivated by these challenges, this paper presents a near-optimal method for designing prediction modes for adaptive compression. The proposed method builds on a stable, open-loop platform, but with a subterfuge that ensures that the design is asymptotically optimized for closed-loop operation. The non-convexity is handled by deterministic annealing, a powerful optimization tool to avoid poor local minima. To demonstrate the impact of the proposed approach on practical applications, we consider adaptive, transform-domain predictor design for enhancing standard video coding. Experimental results validate the benefits of the proposed design in terms of significant performance gains for both predictive compression systems in general and video coding in particular.

Methodology for the design of didactic tools for the future transport engineers’ competencies formation on the ontology basis

The change in the essence of engineers’ functional activity activates the problem of creating new didactic tools adequate to the conditions of digital transformation of production and university education. The authors consider the ontological-semantic approach based on the use of ontological models and semantic technologies, as well as the principles of interoperability and interdisciplinarity corresponding to this approach as the methodological basis for the development of such tools. The feasibility of developing didactic tools on the ontology basis was determined using the questionnaire survey method, which was conducted among future railway transport engineers from 2016 to 2021. The results of the data analysis indicated a tendency for respondents to use compression tools for better assimilation of new learning material. Among such means students indicated not only tables, drawings and diagrams, but also a fundamentally new form of educational knowledge presentation - ontology. The conclusions made on the results of the questionnaire served as a basis for the development of didactic tools for the formation of communicative competence of future railway transport engineers. The novelty of this educational solution consisted in the complex use of textual, hypertext and ontological forms of branch knowledge representation, including the foreign language studied by students.

Comparative studies on the high-performance compression of SARS-CoV-2 genome collections.

The severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) is fast mutating worldwide. The mutated strains have been timely sequenced by worldwide labs, accumulating a huge amount of viral genome sequences open to public for biomedicine research such as mRNA vaccine design and drug recommendation. It is inefficient to transmit the millions of genome sequences without compression. In this study, we benchmark the performance of reference-free and reference-based compression algorithms on SARS-CoV-2 genome collections extracted from NCBI. Experimental results show that reference-based two-level compression is the most suitable approach to the compression, achieving the best compression ratio 1019.33-fold for compressing 132 372 genomes and 949.73-fold for compressing 416 238 genomes. This enormous file size reduction and efficient decompression have enabled a 5-min download and decompression of $10^5$ SARS-CoV-2 genomes. As compression on datasets containing such big numbers of genomes has been explored seldom before, our comparative analysis of the state-of-the-art compression algorithms provides practical guidance for the selection of compression tools and their parameters such as reference genomes to compress viral genome databases with similar characteristics. We also suggested a genome clustering approach using multiple references for a better compression. It is anticipated that the increased availability of SARS-CoV-2 genome datasets will make biomedicine research more productive.

Versatile Video Coding for 3.0 Next Generation Digital TV in Brazil

In the past few decades, the video broadcast ecosystem has gone through major changes; Originally transmitted using analog signals, it has been more and more transitioned toward digital, leveraging compression technologies and transport protocols, principally developed by MPEG. Along this way, the introduction of new video formats was achieved with standardization of new compression technologies for their better bandwidth preservation. Notably, SD with MPEG-2, HD with H.264, 4K/UHD with HEVC. In Brazil, the successive generations of digital broadcasting systems were developed by the SBTVD Forum, from TV-1.0 to TV-3.0 nowadays. The ambition of TV-3.0 is significantly higher than that of previous generations as it targets the delivery of IPbased signals for applications, such as 8K, HDR, virtual and augmented reality. To deliver such services, compressed video signals shall fit into a limited bandwidth, requiring even more advanced compression technologies. The Versatile Video Coding standard (H.266/VVC), has been finalized by the JVET committee in 2021 and is a relevant candidate to address the TV3.0 requirements. VVC is versatile by nature thanks to its dedicated tools for efficient compression of various formats, from 8K to 360°, and provides around 50% of bitrate saving compared to its predecessor HEVC. This paper presents the VVC-based compression system that has been proposed to the SBTVD call for proposals for TV-3.0. A technical description of VVC and an evaluation of its coding performance is provided. In addition, an end-to-end live transmission chain is demonstrated, supporting 4K real-time encoding and decoding with a low glass-to-glass latency.

Sparsely Connected Autoencoders: A Multi-Purpose Tool for Single Cell omics Analysis.

Background: Biological processes are based on complex networks of cells and molecules. Single cell multi-omics is a new tool aiming to provide new incites in the complex network of events controlling the functionality of the cell. Methods: Since single cell technologies provide many sample measurements, they are the ideal environment for the application of Deep Learning and Machine Learning approaches. An autoencoder is composed of an encoder and a decoder sub-model. An autoencoder is a very powerful tool in data compression and noise removal. However, the decoder model remains a black box from which is impossible to depict the contribution of the single input elements. We have recently developed a new class of autoencoders, called Sparsely Connected Autoencoders (SCA), which have the advantage of providing a controlled association among the input layer and the decoder module. This new architecture has the benefit that the decoder model is not a black box anymore and can be used to depict new biologically interesting features from single cell data. Results: Here, we show that SCA hidden layer can grab new information usually hidden in single cell data, like providing clustering on meta-features difficult, i.e. transcription factors expression, or not technically not possible, i.e. miRNA expression, to depict in single cell RNAseq data. Furthermore, SCA representation of cell clusters has the advantage of simulating a conventional bulk RNAseq, which is a data transformation allowing the identification of similarity among independent experiments. Conclusions: In our opinion, SCA represents the bioinformatics version of a universal “Swiss-knife” for the extraction of hidden knowledgeable features from single cell omics data.

FastqCLS: a FASTQ compressor for long-read sequencing via read reordering using a novel scoring model.

Over the past decades, vast amounts of genome sequencing data have been produced, requiring an enormous level of storage capacity. The time and resources needed to store and transfer such data cause bottlenecks in genome sequencing analysis. To resolve this issue, various compression techniques have been proposed to reduce the size of original FASTQ raw sequencing data, but these remain suboptimal. Long-read sequencing has become dominant in genomics, whereas most existing compression methods focus on short-read sequencing only. We designed a compression algorithm based on read reordering using a novel scoring model for reducing FASTQ file size with no information loss. We integrated all data processing steps into a software package called FastqCLS and provided it as a Docker image for ease of installation and execution to help users easily install and run. We compared our method with existing major FASTQ compression tools using benchmark datasets. We also included new long-read sequencing data in this validation. As a result, FastqCLS outperformed in terms of compression ratios for storing long-read sequencing data. FastqCLS can be downloaded from https://github.com/krlucete/FastqCLS. Supplementary data are available at Bioinformatics online.

Machine Learning at the Network Edge: A Survey

Resource-constrained IoT devices, such as sensors and actuators, have become ubiquitous in recent years. This has led to the generation of large quantities of data in real-time, which is an appealing target for AI systems. However, deploying machine learning models on such end-devices is nearly impossible. A typical solution involves offloading data to external computing systems (such as cloud servers) for further processing but this worsens latency, leads to increased communication costs, and adds to privacy concerns. To address this issue, efforts have been made to place additional computing devices at the edge of the network, i.e., close to the IoT devices where the data is generated. Deploying machine learning systems on such edge computing devices alleviates the above issues by allowing computations to be performed close to the data sources. This survey describes major research efforts where machine learning systems have been deployed at the edge of computer networks, focusing on the operational aspects including compression techniques, tools, frameworks, and hardware used in successful applications of intelligent edge systems.

A Finite Element Study of Large Strain Extrusion Machining Using Modified Zerilli–Armstrong Constitutive Relation

Abstract The objective of this work is to study the performance of modified Zerilli–Armstrong constitutive relation proposed in our previous study for the finite element modeling of a severe plastic deformation technique called large strain extrusion machining. The modified Zerilli–Armstrong constitutive relation is implemented in a finite element model of large strain extrusion machining of Inconel 718 to analyze the influence of process parameters, i.e., the chip compression ratio and tool–chip friction, on deformation, effective strain distribution, and hydrostatic pressure distribution along the extruded chip. The predicted strain values for different chip compression ratios were validated by comparison with those obtained through an analytical model. The finite element predictions also served as a guideline in designing the large strain extrusion-machining setup on which experiments were conducted to generate Inconel 718 foils with superior mechanical properties. The predicted limits of chip compression ratio were in close agreement with experimentally realizable values. Furthermore, the predicted strain distribution through the thickness of the chip was validated with the results of hardness measurement tests. Microstructural characterization of the Inconel 718 foils was carried out by using both optical and transmission-electron microscopic studies in order to reveal the presence of fine-grain structures. The validations showed the effectiveness of the modified Zerilli–Armstrong constitutive relation in modeling large strain extrusion machining—a variant of the conventional machining process.

High-Performance On-Chip Hot-Pressed NdFeB Hard Magnets for MEMS Applications

In order to advance the potential of thick on-chip hard magnets for the micro-electro-mechanical system (MEMS), we investigate a new silicon molding technique to fabricate dry-packed NdFeB magnets, including a silicon compression tool, which enables the pressing step during silicon-compatible processing. This process delivers samples with a remanence of 0.42 T and an energy product of 38 kJ/m <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> . Further studies of metal molding show that, for wax-bonding powder-based NdFeB magnets, the optimum fabrication condition is 300 °C and 425 MPa, giving a remanence of 0.54 T and an energy product of 61.7 kJ/m <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> .

Video Compression With CNN-Based Postprocessing

In recent years, video compression techniques have been significantly challenged by the rapidly increased demands associated with high quality and immersive video content. Among various compression tools, post-processing can be applied on reconstructed video content to mitigate visible compression artefacts and to enhance overall perceptual quality. Inspired by advances in deep learning, we propose a new CNN-based post-processing approach, which has been integrated with two state-of-the-art coding standards, VVC and AV1. The results show consistent coding gains on all tested sequences at various spatial resolutions, with average bit rate savings of 4.0% and 5.8% against original VVC and AV1 respectively (based on the assessment of PSNR). This network has also been trained with perceptually inspired loss functions, which have further improved reconstruction quality based on perceptual quality assessment (VMAF), with average coding gains of 13.9% over VVC and 10.5% against AV1.

Investigation of PCA as a compression pre-processing tool for X-ray image classification

Image classification has rapidly gained interest in the medical field with the ability to assist practitioners to diagnose a variety of conditions. Due to the critical nature of the application, any pre-processing function that may compromise the fitness of the classifier requires careful assessment. Image compression, albeit necessary in terms of volume-based goals, is an example of such a preprocessing function that can deeply affect data veracity. In this work, the trade-off between volume and veracity in bone fracture classification using X-ray images is investigated. The impacts of the dimensionality reduction technique—via Principal Component Analysis—as a compression tool on X-ray image classification are explored. The effects of the compression technique on the detection of fractures are assessed by evaluating how reductions in principal components of the X-ray image, and subsequently its volume, affect the accuracy of the fracture classification. Varying levels of compression are applied to both healthy and fracture image sets with tests conducted using ANFIS, SVM and ANN classifiers. Results indicate that a potentially feasible compression range exists whereby classification accuracy is acceptably diminished, after which further compression yields are marginal and classification accuracies drastically decrease. Overall results demonstrate the suitability of the method which yields compression levels of up to 94% with a corresponding minimal drop in classification accuracy of 2%.

WITHDRAWN: Developing new search pattern in fast motion estimation algorithm used for compress a video

Due to rapid growing of digital video, it was very complicated task for storing, processing and transferring a sequence of video. Motion Estimation (ME) is a primary tool for video compression. This paper comprises new pattern i.e. rectangular search pattern used in place of diamond search pattern in block-based motion estimation algorithm. The performance can be measured in terms of number of points searching in searching area per macro-block and quality of video by using PSNR value.

Deep Learning-Based Compression for Phase-Only Hologram

With the aid of rapid development in optical and display devices, Holography has become an emerging technology to represent 3D objects providing the most natural and realistic depth illusion to the users. To reproduce a light field of objects, a hologram records the interference patterns between the object and reference wavefields. Especially for computer-generated holograms, they contain vast amount of data and must be efficiently compressed for storage or transmission. Although several coding solutions, including standardized compression tools and their variations, have been tried in the literature, the coding efficiency is not satisfactory because holograms have completely different unique properties from those observed in natural images, whereas the legacy coding solutions are optimized for the latter. In this work, we propose a deep-learning based image compression network for phase-only holograms. The proposed network is trained in an end-to-end manner with the help of data augmentation technique and aims at minimizing the entropy of the input hologram. In comprehensive experiments, we demonstrate that our compression network shows the state-of-the-art coding performance that even surpasses the latest compression standard Versatile Video Coding by up to 39% in terms of BD-rate gain. Furthermore, we provide in-depth analysis for experimental results in both holographic and numerical reconstruction domains, such as individual performances of the test codecs, a performance comparison with different objective image fidelity metrics, and subjective quality evaluation results.

Recent developments in code compression techniques for embedded systems

Embedded applications software code is increasingly growing in size. Whereas the entire code of all control panels in a car provided for roughly a few 100 K code lines a decade ago, a single control panel such as the engine control can now have up to 1million code lines. With these help of good approach, common scenarios are developed for other, even for mobiles, embedded systems like PDA’s, cell phones etc. However, increasing software size requires greater memory and can therefore raise the cost of an embedded system considerably. The starting of this pattern was already established in the early 1990 s. The compressed code is created by compressing the binary numbers using a code compression tool (at the time of design) is stored in the instruction memory of the embedded devices. The compressed instructions are decompressed and implemented by the processor at the time of startup. One of the serious challenges is that the tables will become wide in size and therefore decrease the benefits of compressing the code that could be accessed. Although the whole research in this area has mostly concentrated on improving greater code compression without specifically targeting the issue of wide look-up table sizes.

Representation of k-Mer Sets Using Spectrum-Preserving String Sets.

Given the popularity and elegance of k-mer-based tools, finding a space-efficient way to represent a set of k-mers is important for improving the scalability of bioinformatics analyses. One popular approach is to convert the set of k-mers into the more compact set of unitigs. We generalize this approach and formulate it as the problem of finding a smallest spectrum-preserving string set (SPSS) representation. We show that this problem is equivalent to finding a smallest path cover in a compacted de Bruijn graph. Using this reduction, we prove a lower bound on the size of the optimal SPSS and propose a greedy method called UST (Unitig-STitch) that results in a smaller representation than unitigs and is nearly optimal with respect to our lower bound. We demonstrate the usefulness of the SPSS formulation with two applications of UST. The first one is a compression algorithm, UST-Compress, which, we show, can store a set of k-mers by using an order-of-magnitude less disk space than other lossless compression tools. The second one is an exact static k-mer membership index, UST-FM, which, we show, improves index size by 10%-44% compared with other state-of-the-art low-memory indices.

Performance Optimization System for Hadoop and Spark Frameworks

Abstract The optimization of large-scale data sets depends on the technologies and methods used. The MapReduce model, implemented on Apache Hadoop or Spark, allows splitting large data sets into a set of blocks distributed on several machines. Data compression reduces data size and transfer time between disks and memory but requires additional processing. Therefore, finding an optimal tradeoff is a challenge, as a high compression factor may underload Input/Output but overload the processor. The paper aims to present a system enabling the selection of the compression tools and tuning the compression factor to reach the best performance in Apache Hadoop and Spark infrastructures based on simulation analyzes.

A compression method for DNA.

The development of high-throughput sequencing technology has generated huge amounts DNA data. Many general compression algorithms are not ideal for compressing DNA data, such as the LZ77 algorithm. On the basis of Nour and Sharawi’s method,we propose a new, lossless and reference-free method to increase the compression performance. The original sequences are converted into eight intermediate files and six final files. Then, the LZ77 algorithm is used to compress the six final files. The results show that the compression time is decreased by 83% and the decompression time is decreased by 54% on average.The compression rate is almost the same as Nour and Sharawi’s method which is the fastest method so far. What’s more, our method has a wider range of application than Nour and Sharawi’s method. Compared to some very advanced compression tools at present, such as XM and FCM-Mx, the time for compression in our method is much smaller, on average decreasing the time by more than 90%.

Efficient DNA sequence compression with neural networks.

BackgroundThe increasing production of genomic data has led to an intensified need for models that can cope efficiently with the lossless compression of DNA sequences. Important applications include long-term storage and compression-based data analysis. In the literature, only a few recent articles propose the use of neural networks for DNA sequence compression. However, they fall short when compared with specific DNA compression tools, such as GeCo2. This limitation is due to the absence of models specifically designed for DNA sequences. In this work, we combine the power of neural networks with specific DNA models. For this purpose, we created GeCo3, a new genomic sequence compressor that uses neural networks for mixing multiple context and substitution-tolerant context models.FindingsWe benchmark GeCo3 as a reference-free DNA compressor in 5 datasets, including a balanced and comprehensive dataset of DNA sequences, the Y-chromosome and human mitogenome, 2 compilations of archaeal and virus genomes, 4 whole genomes, and 2 collections of FASTQ data of a human virome and ancient DNA. GeCo3 achieves a solid improvement in compression over the previous version (GeCo2) of n}{}2.4%, n}{}7.1%, n}{}6.1%, n}{}5.8%, and n}{}6.0%, respectively. To test its performance as a reference-based DNA compressor, we benchmark GeCo3 in 4 datasets constituted by the pairwise compression of the chromosomes of the genomes of several primates. GeCo3 improves the compression in n}{}12.4%, n}{}11.7%, n}{}10.8%, and n}{}10.1% over the state of the art. The cost of this compression improvement is some additional computational time (1.7–3 times slower than GeCo2). The RAM use is constant, and the tool scales efficiently, independently of the sequence size. Overall, these values outperform the state of the art.ConclusionsGeCo3 is a genomic sequence compressor with a neural network mixing approach that provides additional gains over top specific genomic compressors. The proposed mixing method is portable, requiring only the probabilities of the models as inputs, providing easy adaptation to other data compressors or compression-based data analysis tools. GeCo3 is released under GPLv3 and is available for free download at https://github.com/cobilab/geco3.