Reduction In File Size Research Articles

Efficient Storage and Analysis of Genomic Data: A k-mer Frequency Mapping and Image Representation Method.

k-mer frequencies are crucial for understanding DNA sequence patterns and structure, with applications in motif discovery, genome classification, and short read assembly. However, the exponential increase in the dimension of frequency tables with increasing k-mer length poses storage challenges. In this study, we present a novel method for compressing k-mer data without information loss, aiming to optimize storage and analysis processes. We employed Chaos Game Representation (CGR) to map k-mers to coordinates and used these components to generate raster images of k-mers. The CGR maps were partitioned and labeled based on substrings, with each substring mapped to a subframe, creating a fractal-like structure. The entire k-mer frequency set of each genomic sequence was represented as a single image, with each pixel corresponding to a specific k-mer and its occurrence. This approach reduced file size by up to 16-fold compared to plain text and 3-fold compared to binary format. Furthermore, we demonstrated the feasibility of performing alignment-free similarity analyses on images derived from k-mer frequencies of whole genome sequences from 14 plant species. Our results highlight the potential of this method as a fast and efficient tool for accessing, processing, and analyzing large biological sequence datasets, enabling the retrieval of k-mer frequencies and image reconstruction.

Improvement of Lossless Text Compression Methods using a Hybrid Method by the Integration of RLE, LZW and Huffman Coding Algorithms

In the field of computer science, data compression is essential in the process of data transfer because it reduces file size without losing information. Issues that depend on the characteristics of the text can greatly affect the effectiveness of any compression algorithm. Consequently, each traditional compression algorithm has its own strengths and limitations. Therefore, a highly efficient compression method that achieves higher compression ratio is needed. In this paper, based on the sequential implementation of Huffman, RLE and LZW algorithms, a hybrid data compression method has been proposed. The algorithms have been individually evaluated and compared in terms of compression ratio and compressed file size. We conclude that in terms of compression ratio and compressed file size, LZW is better than Huffman and RLE. The results show that the proposed hybrid algorithm achieves the highest average compression ratio is 2.42 and the lowest average compressed file size is 1138.5 or less compared to the individual implementations of the algorithms. Thus, it can be concluded that the proposed hybrid method enhances lossless text compression in terms of compression ratio and file size according to compression metrics.

A Comparative Study of Text-Based Lossless Compression

Lossless data compression is a critical technique used to reduce file sizes without any loss of information during the encoding and decoding processes. This study presents a comparative analysis of two widely-used lossless compression algorithms: Huffman Encoding and Lempel-Ziv-Welch (LZW). The primary objective is to evaluate the performance of these algorithms in terms of compression ratio, compression time, decompression time, and space savings. The analysis was conducted on 100 files of varying sizes. The results demonstrate that the LZW algorithm outperforms Huffman Encoding, offering superior compression ratios, faster compression and decompression times, and greater disk space savings. These findings highlight the effectiveness of LZW for efficient data compression in practical applications.

A Lightweight Double Compression Detector for HEIF Images Based on Encoding Information.

Extensive research has been conducted in image forensics on the analysis of double-compressed images, particularly in the widely adopted JPEG format. However, there is a lack of methods to detect double compression in the HEIF format, which has recently gained popularity since it allows for reduced file size while maintaining image quality. Traditional JPEG-based techniques do not apply to HEIF due to its distinct encoding algorithms. We previously proposed a method to detect double compression in HEIF images based on Farid's work on coding ghosts in JPEG images. However, this method was limited to scenarios where the quality parameter used for the first encoding was larger than for the second encoding. In this study, we propose a lightweight image classifier to extend the existing model, enabling the identification of double-compressed images without heavily depending on the input image's quantization history. This extended model outperforms the previous approach and, despite its lightness, demonstrates excellent detection accuracy.

DETERMINING OPTIMAL COMPRESSION ALGORITHM FOR FILES OF DIFFERENT FORMATS

The aim of the article is to highlight possible areas of the use of compression algorithms in various fields. The article investigates the efficiency of the compression of files of different formats and provides recommendations on the optimal compression algorithms for specific scenarios. The research was carried out empirically: compression algorithms were implemented at the software level, the programs were used to compress files of various formats, the size of the resulting files was determined, and a comparison of the efficiency of the methods was made. Research results. The article presents a table of file sizes after compression using different compression algorithms, calculates compression ratios for each case, determines the average compression ratios for each compression algorithm, analyses the efficiency of compression algorithms, and identifies the optimal compression algorithms for files of different formats. The scientific novelty of this work is an integrated approach to comparing compression algorithms by compression ratios on different file formats and the study of the combination of Huffman and LZ78 algorithms, which has not been widely studied before. This allows us to gain a deeper understanding of the process of compressing files of different formats and identify effective algorithms for specific data types. The analysis can contribute to the development and improvement of file compression methods and have practical applications in various fields, such as data storage and transmission, file compression, and improving the performance of information processing systems. The practical significance of the work lies in its potential usefulness for various fields. It provides recommendations and conclusions on the selection of efficient file compression algorithms for different file formats. This can have a positive impact on data storage and transmission, data processing speed, software development, and multimedia data. Using optimal compression algorithms helps reduce file size, saves resources, and improves user experience.

Optimizing Digital Image Steganography through Hybridization of LSB and Zstandard Compression

In response to the growing need for secure digital communication globally, this research delves into an innovative strategy for enhancing data transmission security through steganography. This inventive approach involves the integration of the conventional Least Significant Bit (LSB) method with Zstandard (Zstd) compression to elevate the quality of stego images. The study carefully explores how the synergistic use of LSB and Zstd contributes to an improved equilibrium between embedding capacity and visual quality in stego images. This hybrid methodology capitalizes on the efficiency of Zstd in reducing file size, thereby facilitating more effective data concealment using LSB. The experimental outcomes showcase a notable 51.2% increase in embedding capacity, a 4.70% elevation in PSNR value, accompanied by a substantial 51.03% decrease in MSE value. Additionally, SSIM values hover around 0.007%, indicating a perceptually minimal difference between the original and steganographically modified images. These compelling results underscore the efficacy of the proposed method, highlighting its proficiency in preserving and enhancing the quality of stego images generated through the embedding process. This research signifies a significant stride in the realm of secure digital communication, demonstrating a promising fusion of traditional LSB with advanced Zstd compression for optimizing digital image steganographic.

Lightweight Geometric Compression Encoding for Additive Manufacturing

In additive manufacturing (AM), 3D geometric models often contain some repetitive structures. To represent model topology more effectively and reduce storage space, this paper introduces a data structure using the repetition encoding framework and the hash encoding framework. First, we identify repeated meshes within the model and establish transformation groups. Then, we cluster the identical transformation groups to optimize storage space usage by utilizing the geometric relationships among facets. Finally, efficient data storage is achieved using the hash encoding framework. We devise three compatible encodings for prevalent additive manufacturing file formats based on these frameworks. We compare the encoding process with three commercial software solutions regarding the reconstruction efficiency and the storage space requirement. Experimental results exhibit that our proposed methods reduce file sizes by over 70% while preserving model topology information.

A Novel Strategy to Achieve Video Transcoding Using Cloud Computing

One of the fundamental challenges faced in deploying multimedia systems is delivering smooth and uninterrupted audio-visual information anywhere and anytime. In such systems, multimedia content is compressed within a certain format, this requires format conversion for various devices. Thus, a transcoding mechanism is required to make the content adaptive for various devices in the network. Video transcoding converts one digitally encoded format into another, this involves translating any file format containing video and audio at the same time. This is an essential feature for devices that do not support a specific format of media or have limited storage that requires a reduced file size. Through this paper, we provide a novel way of transcoding the block-based video coding schemes using cloud architecture by establishing a video pipelining architecture. The solution discussed in this paper would enable the end users to extract videos in any format and resolution seamlessly, combined with the scalability, reliability, and cost-effectiveness of the cloud. The proposed idea would be lucrative for all the video streaming applications that are currently relying on their legacy infrastructure for video transcoding.

JPEG Reversible Data Hiding Using Dynamic Distortion Optimizing With Frequency Priority Reassignment

JPEG Reversible data hiding (RDH) aims at correctly extracting hidden data and perfectly recovering the original JPEG image. By combining different frequency selection and block ordering manners, appropriate AC coefficients can be determined to achieve an efficient JPEG RDH scheme using histogram shifting (HS) mechanism. Nevertheless, existing methods only rely on rough frequency band distortion model by involving all DCT blocks, ignoring the dynamic influence of block ordering on embedding block selection. As a result, JPEG image containing hidden data may have an obvious distortion and file size expansion due to an inappropriate frequency band selection. To address the aforementioned problem, we design a new JPEG RDH scheme to optimize distortion model with frequency priority reassignment. Our method firstly orders all DCT blocks by referring to zero AC coefficients of each block. Subsequently, based on the given secret data, only partial ordered blocks are selected to calculate the distortion of each DCT frequency, which can further optimize unit distortion caused by shiftable coefficients. Furthermore, we employ the adjustment parameter to adaptively reassign the priority of each frequency band based on its influence on file size expansion, which can significantly reduce file size expansion of the marked JPEG image. Extensive experiments demonstrate that our method outperforms existing JPEG RDH methods with better visual quality and smaller file size increment of the marked image.

Foveated 3D range geometry compression via loss-tolerant variable precision depth encoding.

The capacity of three-dimensional (3D) range geometry acquisition methods to capture high-precision scans at high frame rates increases every year. These improvements have influenced a broadening range of disciplines to implement 3D range geometry capture systems, including telepresence, medicine, the visual arts, and many others. However, its increased popularity, precision, and capture rates have caused mounting pressure on the storage and transmission of 3D range geometry, thus straining their capacities. Compression techniques seek to alleviate this pressure by offering reduced file sizes, while maintaining the levels of precision needed for particular applications. Several such compression methods use sinusoidal modulation approaches to encode floating-point 3D data into conventional 2D red, green, and blue (RGB) images. In some applications, such as telepresence, high precision may only be required in a particular region within a depth scan, thus allowing less important data to be compressed more aggressively. This paper proposes a feature-driven compression method that provides a way to encode regions of interest at higher levels of precision while encoding the remaining data less precisely to reduce file sizes. This method supports both lossless and lossy compression, enabling even greater file-size savings. For example, in the case of a depth scan of a bust, an algorithmically extracted bounding box of the face was used to create a foveated encoding distribution so that the facial region was encoded at higher precisions. When using JPEG 80, the RMS reconstruction error of this novel, to the best of our knowledge, encoding was 0.56mm in the region of interest, compared to a globally fixed higher precision encoding where the error was 0.54mm in the same region. However, the proposed encoding achieved a 26% reduction in overall compressed file size compared to the fixed, higher-precision encoding.

Digital-Image Dimension Reduction Via Analysis of Principal component

An Image with high-resolution is associated with huge size data space because each information of the image is arranged into 2D picture elements' values, each of them containing its associated value of the RGB bits. The depiction of picture data makes it challenging to distribute picture files using the Internet. For Internet users, the time it takes to upload and download photos has all time been the main concern. A high-resolution image takes up more storage space, in addition to the data transit difficulty. The Analysis of Principal Component, or PCA for a brief notation, is a mathematical approach utilized to lessen the data dimensionality. It extracts the main pattern of a linear system using the factoring matrices technique. The objectives of this paper are to see how effective PCA is in reducing digital picture features and to investigate the (feature-reduced) images’ quality on comparison with different values of the variance. As per the synthesizing of the initial research, the dimension or size reduction technique through the Analysis of Principal Component typically involves of 4-important steps: (1) picture-data normalizing (2) matrix of the covariance calculating using picture-data. (3) discovering the picture-data projection (with fewer number of features) to a new basis use the Single Value Decomposition technique (SVD) (4) determining the picture-data projection (with fewer number of characteristics) to a new basis. According to testing results, the PCA approach considerably decreases the size of picture data while sustaining the original picture’s fundamental properties. This approach reduced file size by 35.3 percent for the best feature lowered quality. The upload time of picture files through the Internet has substantially improved, particularly for mobile device downloads.

An iterative reference mapping approach for BIM IFCXML classified content compression

Industry Foundation Classes (IFC) and Industry Foundation Classes XML (IFCXML) are widely accepted data frameworks and formats for information storage and exchange among building information models (BIM). IFCXML are comprehensive and compatible with the major BIM platforms; however, many studies suggest their complex structure often results in redundancy and inflexibility. Researchers have proposed various IFC compression methods to reduce file size and restructure data organization, such as partial model extraction, Solibri IFC Optimizer, and ACC4IFC. However, simplification often results in missing information or data leakage, and there has never been a compressor specific to IFCXML. To overcome these issues, this study proposed a conservative compression method that removes duplicated information while maintaining the data structure of IFCXML through an iterative reference mapping method. Based on the data structure and geometry of IFCXML, the algorithm identified three kinds of duplicate information: independent entity duplication (IED), cross-reference entity duplication (CED), and property-set entity duplication (PED). To validate the proposed compressing method, this study conducted a validation test on six typical BIM models and benchmarked with other existing compressors (Solibri IFC Optimizer and ACC4IFC). The outcomes suggested that the proposed model could compress IFCXML files by 18%-59% without losing information.

SCDeep: Single-Channel Depth Encoding for 3D-Range Geometry Compression Utilizing Deep-Learning Techniques

Recent advances in optics and computing technologies have encouraged many applications to adopt the use of three-dimensional (3D) data for the measurement and visualization of the world around us. Modern 3D-range scanning systems have become much faster than real-time and are able to capture data with incredible precision. However, increasingly fast acquisition speeds and high fidelity data come with increased storage and transmission costs. In order to enable applications that wish to utilize these technologies, efforts must be made to compress the raw data into more manageable formats. One common approach to compressing 3D-range geometry is to encode its depth information within the three color channels of a traditional 24-bit RGB image. To further reduce file sizes, this paper evaluates two novel approaches to the recovery of floating-point 3D range data from only a single-channel 8-bit image using machine learning techniques. Specifically, the recovery of depth data from a single channel is enabled through the use of both semantic image segmentation and end-to-end depth synthesis. These two distinct approaches show that machine learning techniques can be utilized to enable significant file size reduction while maintaining reconstruction accuracy suitable for many applications. For example, a complex set of depth data encoded using the proposed method, stored in the JPG 20 format, and recovered using semantic segmentation techniques was able to achieve an average RMS reconstruction accuracy of 99.18% while achieving an average compression ratio of 106:1 when compared to the raw floating-point data. When end-to-end synthesis techniques were applied to the same encoded dataset, an average reconstruction accuracy of 99.59% was experimentally demonstrated for the same average compression ratio.

"Re-Materialized" Medical Data: Paper-Based Transmission of Structured Medical Data Using QR-Code, for Medical Imaging Reports.

Although paper-based transmission of medical information might seem outdated, it has proven efficient, and remains structurally safe from massive data leaks. As part of the ICIPEMIR project for improving medical imaging report, we explored the idea of structured data storage within a medical report, by embedding the data themselves in a QR-Code (and no URL-to-the-data). Three different datasets from ICIPEMIR were serialized, then encoded in a QR-Code. We compared 4 compression algorithms to reduce file size before QR-Encoding. YAML was the most concise format (character sparing), and allowed for embedding of a 2633-character serialized file within a QR-Code. The best compression rate was obtained with gzip, with a compression ratio of 2.32 in 15.7ms. Data were easily extracted and decompressed from a digital QR-Code using a simple command line. YAML file was also successfully recovered from the printed QR-Code with both Android and iOS smartphone. Minimal detected size was 3*3cm.

Downsampled depth encoding for enhanced 3D range geometry compression.

The use of three-dimensional (3D) range geometry is expanding across a variety of disciplines ranging from medicine to the visual arts. A large amount of information is available in 3D range geometry, causing some applications to be limited in their ability to effectively store or transmit captured data. To help alleviate this constraint, a variety of 3D range data compression techniques have been proposed. One method, multiwavelength depth (MWD) encoding, smoothly encodes 3D range geometry into the three color channels of a 2D RGB image. To the best of our knowledge, we present a novel compression enhancement to further reduce file sizes that employs image downsampling, MWD encoding, and lossless (e.g., PNG) or lossy (e.g., JPEG) compression. Image upsampling is used to return downsampled encodings to their original resolution from which the 3D information is then decoded. The proposed method is robust to various scales of downsampling and levels of lossy compression. For example, when this method was applied with 50% downsampling and JPEG 85 to an encoding of a 3D face scan, a compression ratio of 68.85:1 versus the raw data was achieved with a global RMS reconstruction accuracy of 98.77%. Experimental results demonstrate that the proposed method can provide substantial file size savings at minimal reduction in overall reconstruction accuracy.

Feature-driven 3D range geometry compression via spatially-aware depth encoding

The ever-growing variety of capture methods and applications for 3D range geometry continually increases the need for efficient methods of data storage and transmission. Compression techniques address this need by offering reduced file sizes while maintaining the precision needed for a particular application. Several such compression methods use phase-shifting principles to encode the 3D data into a 2D RGB image. In some applications, such as telepresence, high precision may only be required in a particular region within a scan. This paper proposes a feature-driven compression method that provides a way to encode regions of interest at higher precision while encoding the remaining data at lower precision to reduce file sizes. This method supports both lossless and lossy compression, enabling even greater file size savings and a wider range of applications. In the case of a depth scan of a bust, an extracted bounding box of the face was used to create an encoding distribution such that the facial region was encoded at higher precisions. When using JPEG 80, the global RMS reconstruction accuracy of this novel encoding was 99.72%; however, in the region of interest, the accuracy was 99.88%. This feature-driven encoding achieved a 26% reduction in compressed file size compared to a fixed, high precision encoding.

Dimensionality Reduction of High-throughput Phenotyping Data in Cotton Fields

In this paper, we implement data reduction methods to reduce the size of image datasets from cotton fields for use in a high-throughput phenotyping (HTP) pipeline in order to allow for data transfer more quickly over poor internet connections. We investigate dimensionality reduction methods to accomplish this goal. Specifically, we utilize Principal Component Analysis (PCA) to compress image data into a smaller dimension space, which when uncompressed retains significant variability from the original image. To demonstrate the ability of PCA to produce quality reconstructions, we consider the example use case of detecting cotton bloom flowering patterns with reconstructed images. We employ Open Source Computer Vision (OpenCV) to generate pixel-wise masks which both further reduces the byte size of data and successfully identifies cotton bloom flowering. The results indicate a high amount of data reduction from the original to the reconstructed images; byte sizes reduce 93% through PCA while preserving around 98% variance when using a much smaller number of components. Bitwise masking with OpenCV yields a 99% reduction in file size. The results demonstrate great potential in employing machine learning techniques for the data reduction pre-processing step prior to performing subsequent analysis. This data reduction is a crucial step in developing a field-based HTP big data pipeline.

МЕТОДИ ПІДВИЩЕННЯ ІНФОРМАТИВНОСТІ ТА ЗМЕНШЕННЯ ОБ'ЄМУ ГРАФІЧНИХ ДАНИХ НА ОСНОВІ АНАЛІЗУ ЇХ КОЛІРНОГО ПРОСТОРУ

The constant development of digital technology has led to a sharp increase in the number and volume of media files, including digital images, which make up a significant part of computer network traffic, which reduces the speed of their transmission. The research conducted in this work is based on the provisions and methods of digital image processing, the laws of visual perception, the basics of probability theory and mathematical modeling. The results of theoretical research were verified by simulation. The paper proposes a technology that, through the analysis of the color space of the image and taking into account the laws of visual perception, makes it possible to significantly reduce the size of the image file. This technology is used to solve a number of problems, in particular, the visualization of large files and increase the informativeness of images with complex semantic content. It is established that the reduction of the image file size is achieved through the optimization of the palette and leads to a slight deterioration in the visual quality of image perception. To reduce the visibility of error and create a visual sense of the presence of more different colors in the image than is actually the case, it is proposed to use diffuse pseudo-mixing of colors, which is to model some colors with others. Along with the task of reducing the volume of graphic files based on the optimization of the palette, a similar methodology was developed to increase the informativeness of images through the use of pseudo-colors. By modifying the function of converting the coordinates of color space into color components, a modified approach to the formation of pseudo-color images is proposed, which increases the informativeness of halftone digital images in their visual analysis.

Study on various curvilinear data representations and their impact on mask and wafer manufacturing

Inverse lithography technology (ILT) optical proximity correction is going to play a critical role in addressing challenges of optical and EUV lithography as the industry pushes toward advanced nodes. One major barrier in adoption of ILT has been the mask writer’s inability to efficiently write curvilinear patterns. With the introduction of multibeam mask writers, this barrier has been removed and widespread adoption of ILT is imminent. Traditionally, mask writers have accepted only trapezoidal inputs to the tool, though recent trends show that mask writers are adopting newer formats that already reduce file size. However, as the ILT shape complexity and data volume increases further for 5 nm nodes and beyond, the explosion of mask pattern data file size becomes a major concern. Therefore, there is a need for the industry to look toward other compact formats of data representation that will be capable of serving well for multiple generations of mask making. We compare various curvilinear data representation schemes and their value in the curvilinear ILT-based mask manufacturing flow. We demonstrate that given the nature of curvilinear data, representing it using native curve formats has significant value to reduce file size for future mask making flows. The same format may not be applicable for all types of features in the input mask. These options will be discussed. We will compare the value of such exotic representations with regular simplification approaches that reduce data volume using standard methods and discuss the extents and limits of all these techniques. To evaluate practical use of curvilinear representation in place of conventional piecewise linear representation, we manufacture and measure a photomask to evaluate the accuracy of curvilinear representations. Finally, we use EUV AIMS to assess the impact of curvilinear representation on wafer process window.

Two-channel 3D range geometry compression with primitive depth modification

Modern computing and imaging technologies have allowed for many recent advances to be made in the field of 3D range imaging: range data can now be acquired at speeds much faster than real-time, with sub-millimeter precision. However, these benefits come at the cost of an increased quantity of data being generated by 3D range imaging systems, potentially limiting the number of applications that can take advantage of this technology. One common approach to the compression of 3D range data is to encode it within the three color channels of a traditional 24-bit RGB image. This paper presents a novel method for the modification and compression of 3D range data such that the original depth information can be stored within, and recovered from, only two channels of a traditional 2D RGB image. Storage within a traditional image format allows for further compression to be realized via lossless or lossy image compression techniques. For example, when JPEG 80 is used to store the encoded output image, this method achieves an 18.2% reduction in file size when compared to a similar three-channel, image-base compression method, with only a corresponding 0.17% reduction in global reconstruction accuracy.