Small Files Research Articles

Comparison of image compression techniques using Huffman and Lempel-Ziv-Welch algorithms

Image compression technology is very popular in the field of image analysis because the compressed image is convenient for storage and transmission. In this paper, the Huffman algorithm and Lempel-Ziv-Welch (LZW) algorithm are introduced. They are widely used in the field of image compression, and the compressed image results of the two algorithms are calculated and compared. Based on the four dimensions of Compression Ratio (CR), Mean Square Error (MSE), Peak Signal to Noise Ratio (PSNR) and Bits Per Pixel (BPP), the applicable conditions of the two algorithms in compressing small image files are analysed. The results illustrated that when the source image files are less than 300kb, the Compression Ratio (CR) of Huffman algorithm was better than that of LZW algorithm. However, for Mean Square Error (MSE), Peak Signal to Noise Ratio (PSNR) and Bits Per Pixel (BPP), which are used to represent the compressed images qualities, LZW algorithm gave more satisfactory results.

Small Files Problem Resolution via Hierarchical Clustering Algorithm.

The Small Files Problem in Hadoop Distributed File System (HDFS) is an ongoing challenge that has not yet been solved. However, various approaches have been developed to tackle the obstacles this problem creates. Properly managing the size of blocks in a file system is essential as it saves memory and computing time and may reduce bottlenecks. In this article, a new approach using a Hierarchical Clustering Algorithm is suggested for dealing with small files. The proposed method identifies the files by their structure and via a special Dendrogram analysis, and then recommends which files can be merged. As a simulation, the proposed algorithm was applied via 100 CSV files with different structures, containing 2-4 columns with different data types (integer, decimal and text). Also, 20 files that were not CSV files were created to demonstrate that the algorithm only works on CSV files. All data were analyzed via a machine learning hierarchical clustering method, and a Dendrogram was created. According to the merge process that was performed, seven files from the Dendrogram analysis were chosen as appropriate files to be merged. This reduced the memory space in the HDFS. Furthermore, the results showed that using the suggested algorithm led to efficient file management.

HDF5eis: A storage and input/output solution for big multidimensional time series data from environmental sensors

Modern high-performance computing (HPC) tasks overwhelm conventional geophysical data formats. We describe a new data schema called HDF5eis (read H-D-F-size) for handling big multidimensional time series data from environmental sensors in HPC applications and implement a freely available Python application programming interface (API) for building and processing HDF5eis files. HDF5eis augments the popular Hierarchical Data Format 5 with a minimal set of additional conventions that facilitate fast and flexible data input and output protocols for regularly sampled (in time) data with any number of dimensions. HDF5eis supports arbitrary ancillary data (e.g., metadata) storage in columnar format or as UTF-8 encoded byte streams alongside time series data. Our HDF5eis API enables simple and efficient access to big data sets distributed across a potentially large number of small heterogeneous files through a single point of access. HDF5eis outperforms conventional seismic data formats by up to two orders of magnitude in terms of random read access times. We contribute HDF5eis as an operational tool and an experimental draft proposal that will help establish the next generation of data standards in the earth sciences.

Small files access efficiency in hadoop distributed file system a case study performed on British library text files

Small files access efficiency in hadoop distributed file system a case study performed on British library text files

PepTraq: a toolbox for in silico data mining and fast sequence filtering.

The development of de novo sequencing tools has led to the massive production of genomes and transcriptomes from many unconventional animal models. To tackle this huge flow of data, PepTraq brings together many functionalities generally scattered in multiple tools, so that sequences can be filtered on the basis of multiple criteria. It is particularly suitable for the identification of non-annotated transcripts, re-annotation, extraction of secretomes, neuropeptidomes, targeted search for peptides and proteins, preparing specific proteomics/peptidomics fasta files for mass spectrometry (MS) applications, MS data processing, etc. PepTraq is developed in Java, and is available as a desktop application that can be downloaded from https://peptraq.greyc.fr . It is also available as a web application at the same URL for processing small files (10-20MB). The source code is open under a CeCILL-B licence.

‘Compression makes every movie a melodrama’: A conversation with Laura U. Marks on the Small File Media Festival

This article presents a conversation with Laura U. Marks, a distinguished scholar of film and new media who launched the Small File Media Festival in 2020. Marks’ message is that large-file streaming is harmful to the planet and that we need to act against its growing yet under-the-radar carbon footprint. The festival’s goal is to demonstrate that movies can be captivating and immersive even if they are not large in file size, and encourages creators to experiment with low-energy technologies and techniques. Given the high stakes, the festival’s organizers hope that small-file media will become increasingly influential as an aesthetic movement.

Editor’s Introduction

The ‘Editor’s introduction’ notes the two interviews and five critical/analytical articles included in issue 13.1, which include attention to the energy-focused Small File Media Festival; Richard Raskin’s book, The Yin and Yang of Short Film Storytelling (); Thai filmmaker Apichatpong Weerasethakul’s short films; two historical short film adaptations of the Ambrose Bierce story, ‘An Occurrence at Owl Creek Bridge’; the experimental short, Return to Forms; the Malaysian fiction short, Nor from the Second Level, and Portuguese science-fiction short Flores.

Detecting Parallel Covert Data Transmission Channels in Video Conferencing Using Machine Learning

Covert communication channels are a concept in which a policy-breaking method is used in order to covertly transmit data from inside an organization to an external or accessible point. VoIP and Video systems are exposed to such attacks on different layers, such as the underlying real-time transport protocol (RTP) which uses Transmission Control Protocol (TCP) or User Datagram Protocol (UDP) packet streams to punch a hole through Network address translation (NAT). This paper presents different innovative attack methods utilizing covert communication and RTP channels to spread malware or to create a data leak channel between different organizations. The demonstrated attacks are based on a UDP punch hole created using Skype peer-to-peer video conferencing communication. The different attack methods were successfully able to transmit a small text file in an undetectable manner by observing the communication channel, and without causing interruption to the audio/video channels or creating a noticeable disturbance to the quality. While these attacks are hard to detect by the eye, we show that applying classical Machine Learning algorithms to detect these covert channels on statistical features sampled from the communication channel is effective for one type of attack.

Zgli: A Pipeline for Clustering by Compression with Application to Patient Stratification in Spondyloarthritis.

The normalized compression distance (NCD) is a similarity measure between a pair of finite objects based on compression. Clustering methods usually use distances (e.g., Euclidean distance, Manhattan distance) to measure the similarity between objects. The NCD is yet another distance with particular characteristics that can be used to build the starting distance matrix for methods such as hierarchical clustering or K-medoids. In this work, we propose Zgli, a novel Python module that enables the user to compute the NCD between files inside a given folder. Inspired by the CompLearn Linux command line tool, this module iterates on it by providing new text file compressors, a new compression-by-column option for tabular data, such as CSV files, and an encoder for small files made up of categorical data. Our results demonstrate that compression by column can yield better results than previous methods in the literature when clustering tabular data. Additionally, the categorical encoder shows that it can augment categorical data, allowing the use of the NCD for new data types. One of the advantages is that using this new feature does not require knowledge or context of the data. Furthermore, the fact that the new proposed module is written in Python, one of the most popular programming languages for machine learning, potentiates its use by developers to tackle problems with a new approach based on compression. This pipeline was tested in clinical data and proved a promising computational strategy by providing patient stratification via clusters aiding in precision medicine.

ATAC-DEA: A Web-Based ATAC-Seq Data Differential Peak and Annotation Analysis Application.

Assay for transposase-accessible chromatin sequencing (ATAC-seq) has become one of the most widely used sequencing methods in studies of gene regulation, aiming to identify open chromatin sites and decipher how chromatin accessibility regulates gene expression. However, due to a lack of programming experience or minimal bioinformatics training, it is difficult for biologists to fully explore and interpret ATAC-seq results. Despite several available programs or websites that allow researchers to analyze and visualize ATAC-seq data, several limitations exist. ATAC-seq data differential expression analysis (ATAC-DEA), a web application that facilitates the exploration and visualization of differential peak analysis and annotation from ATAC-seq data, was developed (www.atac-dea.xyz:3838/ATAC-DEA). ATAC-DEA uses DiffBind and ChIPpeakAnno to process differential peak and annotation analysis results. ATAC-DEA has five features: (1) runs on a web server; (2) processes three files into one small file, which is used as the input for ATAC-DEA; (3) availability of various downloadable plots; (4) multifactor analysis and customized contrast model; and (5) annotates individual, overlapped, and differential peaks. It provides an easy-to-use user interface (UI) design for users to explore the data and modify the parameters interactively based on experimental purposes. ATAC-DEA allows biologists to generate user-friendly visual results from ATAC-seq downstream analysis.

Near-Lossless MPI Tracing and Proxy Application Autogeneration

Traces of MPI communications are used by many performance analysis and visualization tools. Storing exhaustive traces of large-scale MPI applications is infeasible, however, because of their large volume. Aggregated or lossy MPI traces are smaller but provide much less information. In this paper we present Pilgrim, a near-lossless MPI tracing tool that, by using sophisticated compression techniques, generates small trace files at large scales and incurs only moderate overheads. We perform comprehensive studies of various compression techniques used for storing timestamps associated with each call. This timing information is essential for analysis purposes such as skews study. To demonstrate the usefulness of the detailed information stored by Pilgrim, we present a proxy application generator that can generate proxy apps that preserve original communication patterns from the Pilgrim traces.

FACHS: Adaptive Hybrid Storage Strategy Based on File Access Characteristics

With the widespread use of distributed systems and the development of big data technology, the storage redundancy, data availability and persistence of distributed systems have become a concern. Traditional multi-copy storage may cause large storage redundancy, thus wasting storage space. The erasure code is considered as the best alternative to the replica strategy. However, the existing methods do not fully consider the access characteristics of files, so that the data with high access popularity are stored using computationally complex erasure codes, resulting in poor parallel read and write performance. Moreover, due to insufficient consideration of file size, storage redundancy cannot be minimized, thus wasting storage space. Therefore, we propose an adaptive hybrid storage strategy based on file access characteristics called FACHS. For cold files, namely the files with low access frequency, we use RS Code (Reed Solomon Code) to store them. RS Code has low computing and storage costs. We use multi-copy and LRC code (local reconstruction code) to store small and large size hot files respectively. Multi-copy ensures the efficiency of file read/write, as well as the ability of parallel read/write. The recovery cost of LRC code in case of node failure is very low. The experimental results show that compared with the existing methods, FACHS can reduce the storage space occupation by 12% for cold files, and improve the read/write speed by 8% and the recovery efficiency by 29% for hot files.

JPEG Image Encryption With Adaptive DC Coefficient Prediction and RS Pair Permutation

JPEG image encryption aims at effectively converting the original JPEG image into a noise-like image that does not contain any useful information of original image. Existing schemes for JPEG image encryption, however, may not attain a good balance in terms of file size increment and encryption security. To address the problem, we design a novel JPEG image encryption scheme. Different from existing schemes, we first predict DC coefficients by an adaptive prediction method. Subsequently, the histogram of DC coefficient prediction errors is encrypted by combining the prediction errors and random integers to reduce the encoded length, which can ensure a very small increment of file size. Furthermore, we construct the RS (run/size) pairs in each DCT block and then implement the permutation for both RS pairs extracted from the upper left corner of each DCT block and all DCT blocks excluding DC coefficients, which can further distort the image contents. Extensive experiments demonstrate that, compared with existing JPEG image encryption schemes, our scheme can ensure not only the JPEG format compatibility for encrypted image, but also keep a very small file size increment and the superior security performance.

Enhanced Best Fit Algorithm for Merging Small Files

In the Big Data era, numerous sources and environments generate massive amounts of data. This enormous amount of data necessitates specialized advanced tools and procedures that effectively evaluate the information and anticipate decisions for future changes. Hadoop is used to process this kind of data. It is known to handle vast volumes of data more efficiently than tiny amounts, which results in inefficiency in the framework. This study proposes a novel solution to the problem by applying the Enhanced Best Fit Merging algorithm (EBFM) that merges files depending on predefined parameters (type and size). Implementing this algorithm will ensure that the maximum amount of the block size and the generated file size will be in the same range. Its primary goal is to dynamically merge files with the stated criteria based on the file type to guarantee the efficacy and efficiency of the established system. This procedure takes place before the files are available for the Hadoop framework. Additionally, the files generated by the system are named with specific keywords to ensure there is no data loss (file overwrite). The proposed approach guarantees the generation of the fewest possible large files, which reduces the input/output memory burden and corresponds to the Hadoop framework’s effectiveness. The findings show that the proposed technique enhances the framework’s performance by approximately 64% while comparing all other potential performance-impairing variables. The proposed approach is implementable in any environment that uses the Hadoop framework, not limited to smart cities, real-time data analysis, etc.

Public monitoring as a tool for assessing the state of the air basin in the Ukrainian Regions

Over the recent years automated observation systems that work continuously and allow obtaining information at any time have been widely implemented. Such systems include public monitoring systems in order to receive more complete information about the quality characteristics of the air basin over Ukraine. Examples of public monitoring organizations include SaveEcoBot, EcoCity, SaveDnipro.
 The information presented on the websites of the public monitoring system contains large amount of data that are difficult to process and analyze. In order to optimize data processing, a program called PublicMonitoringParser was developed. It transforms large volumes of data presented on the Internet resource SaveEcoBot.com into compact, convenient, small files containing average monthly concentrations of pollutants and indicators of meteorological parameters. The program is written using the C# programming language and the .NET platform. The structure of the program and the principles of its operation are presented in this research.
 According to the data of the website SavEecoBot.com the developed program was applied for assessment of the air basin state of Lviv Region in 2021. In general, for most substances, monitoring data were available only at the station in the city of Lviv at the following address: 13 Plastova Str. Monitoring of PM2.5 and PM10 content was carried out across the whole region.
 It was detected that the content of NO2, SO2, NH3 exceeded the limit value. Significant excesses were also noted for the NH3 content (up to 9 – 16 MPC). It was established that the location of the public monitoring station was surrounded by industrial buildings, numerous transport industry enterprises, an oil depot for OKKO gas stations and city sewage treatment facilities. A comparison of the data obtained at stationary and automated points that monitor individual pollutants showed that the results are comparable only for the NO2 content.
 Data from automated observation points complement the network of stationary points. Currently, the existing stationary network of atmospheric pollution monitoring across most regions of Ukraine is outdated and needs to be re-equipped. In general, it is necessary to further implement automated systems that are able to characterize the state of the air basin across individual populated places in a more complete and detailed manner.

IMPLEMENTASI ALGORITMA BYTE PAIR ENCODING UNTUK KOMPRESI FILE

ABSTRACT Data compression is a field that focuses on forming a small output file from a large file. The need for compression has given birth to several methods that can be implemented in communication activities on the network as well as on storage activities. The byte pair encoding algorithm is a compression algorithm that takes advantage of the emergence of repeated or frequent character byte pairs. The repeating character byte pairs are in the data and are substituted with a wildcard byte where the wildcard byte is not in the character byte series of the file. One of the well-known compression methods is the byte pair encoding compression method. The implementation of the system uses the Visual Basic 2010 programming language and the method used in this research is an applied research method. 
 Keywords: Compression, Byte Pair Encoding, Document.

Pretrained Configuration of Power-Quality Grayscale-Image Dataset for Sensor Improvement in Smart-Grid Transmission

The primary source of the various power-quality-disruption (PQD) concerns in smart grids is the large number of sensors, intelligent electronic devices (IEDs), remote terminal units, smart meters, measurement units, and computers that are linked by a large network. Because real-time data exchange via a network of various sensors demands a small file size without an adverse effect on the information quality, one measure of the power-quality monitoring in a smart grid is restricted by the vast volume of the data collection. In order to provide dependable and bandwidth-friendly data transfer, the data-processing techniques’ effectiveness was evaluated for precise power-quality monitoring in wireless sensor networks (WSNs) using grayscale PQD image data and employing pretrained PQD data with deep-learning techniques, such as ResNet50, MobileNet, and EfficientNetB0. The suggested layers, added between the pretrained base model and the classifier, modify the pretrained approaches. The result shows that advanced MobileNet is a fairly good-fitting model. This model outperforms the other pretraining methods, with 99.32% accuracy, the smallest file size, and the fastest computation time. The preprocessed data’s output is anticipated to allow for reliable and bandwidth-friendly data-packet transmission in WSNs.

Moving small files in a networked environment

Globally distributed computing infrastructures, such as clouds and supercomputers, are currently used to manage data that is generated with an unprecedented speed from a variety of resources. Coping with this trend, the volume of data exchanged across distant sites increases substantially. To accelerate data transfer, high-speed networks are provided to connect remote sites. Most existing data movement solutions are optimized for moving large files. However, it is still challenging to transfer a large number of small files across networks. This disadvantage not only lowers data transfer performance, but also decreases overall system utilization. We identify that moving small files is mainly constrained by degraded file system throughput, not just network performance as might be suspected. We have built a data transfer pipeline model to analyze the impact of small network I/O and storage I/O on data movement. Extending one of the widely used open source data movement solutions, GridFTP, we demonstrate several appropriate engineering approaches that mitigate the bottleneck and increase data transfer efficiency. We show optimizations that improve data transfer performance more than 5 times. In comparison to existing solutions, our approaches can save a significant amount of system resources for moving lots of small files.

Which Data Format To Store Scientific Data Should I Use? A Performance Analysis

Abstract A lot of scientific work is dedicated to the analysis of data. Most of the analyzed data, like data from space missions, are structured. The choice of data format can affect various characteristics - read/write speed of standard files, read/write speed of small files and read/write speed of compressed data formats. In this paper, we analyze binary data formats, proposed types of the tests and testing methods, and compare their performance with human-readable text format. We also discuss compressed and uncompressed modes available for data formats like HDF5 and netCDF. When disregarding precision, the best data format from the size perspective is lossy HDF5 without compression. Losless HDF5 without compression show the best speed performance. Lossy HDF5 without compression is the best balance between size reduction and speed. However, for specific criteria and types of files, there might be better candidates as detailed in the conclusion.

Reactive Direct Bonding of Digital Light Process Components

AbstractA major restriction in additive manufacturing is the limited build volume of the printers. If components with larger dimensions are to be manufactured additively, the STL‐file is often split up into smaller files and manufactured in several jobs and joined afterwards. A frequently used joining process is adhesive bonding. High strength and durability of bonded joints can be achieved with chemically curing adhesives. In chemically cured adhesives, monomers or oligomers are present, which react as a function of time, temperature, and pressure to form a polymeric adhesive layer. One‐component systems do not react without activation. Triggers for the reaction can be water, the absence of oxygen, radiation (e.g., UV light), or heat. In the Digital Light Processing (DLP), the component is also built up via a chemical reaction. This is triggered, layer by layer, by UV light and is comparable to the reaction of a UV curing adhesive. Directly after manufacturing, the DLP components are not fully cured. This can be used for a reactive direct bonding of the DLP components without any additional adhesives. In this study, a procedure is developed for reactive direct bonding and the strength of the bonding joint is investigated. Different resins are considered and also a suitability for material combinations is investigated. The results of this study show that reactive direct bonding of DLP components is possible and that high bond strengths can be achieved.