Disk Usage Research Articles

ОПТИМІЗАЦІЯ ОБСЯГУ НЕРЕЛЯЦІЙНОЇ БАЗИ ДАНИХ ТА ЧАСУ ДОСТУПУ ДО ДАННИХ ПРИ МНОЖИННИХ ЧАСТОТАХ РЕЄСТРАЦІЇ ПАРАМЕТРІВ

A non-relational database was developed at JSC Element for the recording operating parameters of GTE and systems during the test flight. The database is modernized version of one which is a part of the GTE test bench. The purpose of the database is the recording and analysis of GTE and systems operating parameters. The parameters are measured (collected) with multiple frequencies in range from fractions of Hz to tens of kHz. Number of recorded operating parameters is 1000+. Correspondingly the database file is huge and may reach tens of Gbytes. Number of different parameters recording frequencies is 10…20 (e.g. 20 parameters – 10 Hz, 30 parameters – 50 Hz, 20 parameters – 10000 Hz). Data reading and chart displaying should be possible as right after the test flight as just in parallel with database recording. There should be no significant delays in getting the data displayed independently from the zoom level chosen. The user should see the result in time comparable with time of the displayed chart perception making the user feel the smooth application operating and not interrupting the process of test flight result comprehension – main purpose of our product.However, min-max chart displaying (minimal and maximal value are found to render one-pixel vertical lines) requires processing of all the data for the corresponding time interval. Correspondingly, time to process data depends on that. In order to provide data quickly on user’s actions the sample is read according to the chart horizontal resolution. Then whole the data is processed in background mode to build min-max chart. This dual approach allows to surf the database quickly losing no details when the surfing is stopped.Modern computers and SSD drivers provides high I/O performance. However huge files can be neither quickly read nor stored in RAM. Therefore, to read data in parallel with the recording only the choice is to store the data blocks which are either indexed by the time stamp or fixed size. Our solution is based on fixed size approach with the constant offset of each parameter storing. Size of the block is calculated as least common multiple of all the frequencies the parameters are recorded with. According to our statistic the disk usage rate is 95%. As result the time to display the sample for 2000 timestamps is less than 0.5 sec independently for the database size and the displayed duration of the test flight.

A benchmark study of k-mer counting methods for high-throughput sequencing.

The rapid development of high-throughput sequencing technologies means that hundreds of gigabytes of sequencing data can be produced in a single study. Many bioinformatics tools require counts of substrings of length k in DNA/RNA sequencing reads obtained for applications such as genome and transcriptome assembly, error correction, multiple sequence alignment, and repeat detection. Recently, several techniques have been developed to count k-mers in large sequencing datasets, with a trade-off between the time and memory required to perform this function. We assessed several k-mer counting programs and evaluated their relative performance, primarily on the basis of runtime and memory usage. We also considered additional parameters such as disk usage, accuracy, parallelism, the impact of compressed input, performance in terms of counting large k values and the scalability of the application to larger datasets.We make specific recommendations for the setup of a current state-of-the-art program and suggestions for further development.

Impact Live Migration on Cloud Performance

Hypervisor-based virtualization and containerization offer abstraction capabilities that make applications independent of the cloud's hardware infrastructure for migration. Cloud Elasticity is the ability to adapt to workload changes by provisioning and de-provisioning resources. There are two types of elasticity: horizontal and vertical also live migration. The later plays a big role in cloud elasticity. Virtual machine (VM) and container migration in Dynamic Resource Management (DRM) is a crucial factor in minimizing the cost of datacenter operation. Live migration is a widely used technology for load balancing, fault tolerance and energy saving in Cloud Datacenters. In this paper, we will be converging the two types of virtualization (Container and VMs together) into a hybrid architecture that promotes their complementarities without altering or degrading the quality of service (QoS) of the Cloud infrastructure. For this, we studied the live migration of a monolithic application and precisely the video streaming installed in a container nested in a virtual machine. We examined the contribution of this new type of virtualization in terms of processor utilization, disk usage, total migration time and downtime. The results presented show the advantages of Hybrid virtualization especially in terms of total migration time. Indeed, the absence of a real virtualization layer in the containers leads to a rapid interaction with the operating system processes, which affects the total migration time. To finish we will explore the live migration in the case of multiCloud, we show that live migration allows better management of material and energy resources.

A study of software pools for seismogeology-related software based on the Docker technique

ABSTRACTIn oil exploration data centers, the seismic exploration process employs multiple applications that are highly specific and replaceable. In addition, the various software versions depend on a particular version of the operating system. As the server purchase from different times, the operating system is also different, with the server scrapped, the newly purchased server operating system version is higher than the original server, resulting in the original application software that cannot be used further. To solve this problem, the software life cycle needs to be extended an independent application software resource pool with a hardware system that is required to be built. Herein, the Docker containerization technology has been used to construct seismic and geologic software pools in a virtualized manner. Furthermore, the hardware and software have been separated from each other, and the hardware resources have been allocated to each container using a scheduling algorithm. By runtime start speed comparison, the physical disk usage comparison test proves that the seismic software has the advantage of being lightweight through the container, and by the analysis of effect of application on conventional module, it proves that the software running in the container has a good effect and is available independence and concurrency. The analysis of the resource-scheduling effect of the scheduling algorithm works well. Thus, we confirmed that the proposed method is effective and has a good promotional value.

R-SHT: A state history tree with R-Tree properties for analysis and visualization of highly parallel system traces

Abstract Understanding the behaviour of distributed computer systems with many threads and resources is a challenging task. Dynamic analysis tools such as tracers have been developed to assist programmers in debugging and optimizing the performance of such systems. However, complex systems can generate huge traces, with billions of events, which are hard to analyze manually. Trace visualization and analysis programs aim to solve this problem. Such software needs fast access to data, which a linear search through the trace cannot provide. Several programs have resorted to stateful analysis to rearrange data into more query friendly structures. In previous work, we suggested modifications to the State History Tree (SHT) data structure to correct its disk and memory usage. While the improved structure, eSHT, made near optimal disk usage and had reduced memory usage, we found that query performance, while twice as fast, exhibited scaling limitations. In this paper, we proposed a new structure using R-Tree techniques to improve query performance. We explain the hybrid scheme and algorithms used to optimize the structure to model the expected behaviour. Finally, we benchmark the data structure on highly parallel traces and on a demanding trace visualization use case. Our results show that the hybrid R-SHT structure retains the eSHT’s optimal disk usage properties while providing several orders of magnitude speed up to queries on highly parallel traces.

An adaptive dynamic feedback load balancing algorithm based on QoS in distributed file system

An adaptive dynamic load balancing algorithm based on QoS is proposed to improve the performance of load balancing in distributed file system, combining the advantages of a variety of load balancing algorithms. The new algorithm uses a tuple containing the number of files and the total file size as the QoS measure for the requested task. The master node sets a threshold for the requested task based on the QoS to filter storage nodes that meet the requirements of the task. In order to guarantee the reliability of the new algorithm, we consider the impact of CPU utilization, memory usage, disk IO occupancy rate, network bandwidth usage and hard disk usage on load balancing performance when calculating the real-time load balancing of storage nodes. The heterogeneity of the network is considered when the master node schedule task assignments to ensure the fairness of the algorithm. The comprehensive evaluation value is determined based the performance load ratio, which is calculated from the real-time load value of the storage node and a performance value after normalization. The master node assigns tasks to the storage node with the highest comprehensive evaluation value. The storage nodes provide adaptive feedback based on changes in the degree of connectivity, rather than periodic update of the load information. The actual distributed file system environment is set up on the server cluster, the performance of the new algorithm is tested through a contrast experiment. The experimental results show that the new algorithm can effectively reduce the average response time of the system, improve throughput, and enable the system load to reach a good balance.

Performance analysis of open‐source distributed file systems for practical large‐scale molecular ab initio, density functional theory, and GW + BSE calculations

AbstractApplication of conventional post‐Hartree–Fock theory, density functional theory, and GW + BSE theory, on chemical and biological problems of growing molecular scale, has created a need for innovative designs in high‐performance computing infrastructures and associated middleware technologies. The growing trend toward petascale computing creates an enormous demand for highly efficient storage solutions. Notably, in reaching limitations of the scale‐up model, technological progress, and continual growth in demand has evolved into solutions that alternatively scale‐out. This investigation involves an analysis of commonly used open source storage solutions for high demand ab initio calculations. Comparisons are shown for the most competitive solutions for a broad range of workloads, following disk usage of key ab initio algorithms in the widely used open source electronic structure theory software, GAMESS. Assessment is based on results of read/write speeds and ratios, scaling ability, and impact of block‐size on overall performance of the distributed storage. GlusterFS is shown to generally outperform CEPH and local disk file systems, providing the most reasonable alternative to local storage, and is therefore recommended for large‐scale molecular calculations.

Maximising resource utilisation and minimising hardware cost in specific cloud

Cloud computing is a new computing model, which provides various services through networks. Cloud customers can obtain resource and service on demand. There are a large number of storage services that can be used to manage the needs of customers in the cloud computing environment. The cloud storage services can let users in a convenient and reliable way store and share data from anywhere, on any device, and at any time. However, a lot of resources and energies are wasted in cloud computing environment. Especially, there is huge waste when it is a low-IO demand cloud system. We seek to save the resource and save the hardware cost at the same time. Nowadays, the important problem is how to save the resources and energies. It is hard to handle it in a high-IO demand system. But we find a way to handle it in a low-IO system. Maximising resource utilisation is the most popular approach which saves the resources and energies in cloud computing. In this paper, we propose to maximise resource utilisation of storage in the cloud environment. We assign the data which is stored in chunk devices of cloud by our data placement algorithms. Our algorithms are running on a central 'decision centre' which holds all devices' status of cloud. As the experiment has shown, minimising the disk usage can reduce the waste of the resources and energies. Finally, our experiment shows that our approach is efficient.

The Edge Weight Computation with MapReduce for Extracting Weighted Graphs

Automated weighted graph construction from massive data is essential to weighted graph theory based data mining processes, where the edge weight computation is time consuming or even fails to complete on a single machine when necessary resources are exhausted. In addition, existing work lacks of the measurement on the accuracy of the edge weights, which represents the graph accuracy and affects the following data mining results. This paper describes the classification, implementation and evaluation of edge weight computation algorithms with MapReduce Framework, which is a powerful parallel and distributed processing model. First, a classification of the edge weight computation algorithms is developed and how they can be applied on MapReduce is also discussed. Then we propose comprehensive measurements on the edge weight accuracy in terms of the number of edges, strength distribution, community structure, Hop-plot and effective diameters. Finally, a performance study has been conducted to evaluate these algorithms in terms of memory and disk usage, execution time and accuracy using a real massive social network application dataset. The results are presented and discussed. Our comparison results can help find out the most effective parallel and distributed edge weight computation algorithm for constructing a weighted graph for a given massive dataset.

The Debsources Dataset: two decades of free and open source software

We present the Debsources Dataset: source code and related metadata spanning two decades of Free and Open Source Software (FOSS) history, seen through the lens of the Debian distribution. The dataset spans more than 3 billion lines of source code as well as metadata about them such as: size metrics (lines of code, disk usage), developer-defined symbols (ctags), file-level checksums (SHA1, SHA256, TLSH), file media types (MIME), release information (which version of which package containing which source code files has been released when), and license information (GPL, BSD, etc). The Debsources Dataset comes as a set of tarballs containing deduplicated unique source code files organized by their SHA1 checksums (the source code), plus a portable PostgreSQL database dump (the metadata). A case study is run to show how the Debsources Dataset can be used to easily and efficiently instrument very long-term analyses of the evolution of Debian from various angles (size, granularity, licensing, etc.), getting a grasp of major FOSS trends of the past two decades. The Debsources Dataset is Open Data, released under the terms of the CC BY-SA 4.0 license, and available for download from Zenodo with DOI reference 10.5281/zenodo.61089.

GRAbB: Selective Assembly of Genomic Regions, a New Niche for Genomic Research.

GRAbB (Genomic Region Assembly by Baiting) is a new program that is dedicated to assemble specific genomic regions from NGS data. This approach is especially useful when dealing with multi copy regions, such as mitochondrial genome and the rDNA repeat region, parts of the genome that are often neglected or poorly assembled, although they contain interesting information from phylogenetic or epidemiologic perspectives, but also single copy regions can be assembled. The program is capable of targeting multiple regions within a single run. Furthermore, GRAbB can be used to extract specific loci from NGS data, based on homology, like sequences that are used for barcoding. To make the assembly specific, a known part of the region, such as the sequence of a PCR amplicon or a homologous sequence from a related species must be specified. By assembling only the region of interest, the assembly process is computationally much less demanding and may lead to assemblies of better quality. In this study the different applications and functionalities of the program are demonstrated such as: exhaustive assembly (rDNA region and mitochondrial genome), extracting homologous regions or genes (IGS, RPB1, RPB2 and TEF1a), as well as extracting multiple regions within a single run. The program is also compared with MITObim, which is meant for the exhaustive assembly of a single target based on a similar query sequence. GRAbB is shown to be more efficient than MITObim in terms of speed, memory and disk usage. The other functionalities (handling multiple targets simultaneously and extracting homologous regions) of the new program are not matched by other programs. The program is available with explanatory documentation at https://github.com/b-brankovics/grabb. GRAbB has been tested on Ubuntu (12.04 and 14.04), Fedora (23), CentOS (7.1.1503) and Mac OS X (10.7). Furthermore, GRAbB is available as a docker repository: brankovics/grabb (https://hub.docker.com/r/brankovics/grabb/).

Compact implicit surface reconstruction via low-rank tensor approximation

Implicit representations have gained an increasing popularity in geometric modeling and computer graphics due to their ability to represent shapes with complicated geometry and topology. However, the storage requirement, e.g. memory or disk usage, for implicit representations of complex models is relatively large. In this paper, we propose a compact representation for multilevel rational algebraic spline (MRAS) surfaces using low-rank tensor approximation technique, and exploit its applications in surface reconstruction. Given a set of 3D points equipped with oriented normals, we first fit them with an algebraic spline surface defined on a box that bounds the point cloud. We split the bounding box into eight sub-cells if the fitting error is greater than a given threshold. Then for each sub-cell over which the fitting error is greater than the threshold, an offset function represented by an algebraic spline function of low rank is computed by locally solving a convex optimization problem. An algorithm is presented to solve the optimization problem based on the alternating direction method of multipliers (ADMM) and the CANDECOMP/PARAFAC (CP) decomposition of tensors. The procedure is recursively performed until a certain accuracy is achieved. To ensure the global continuity of the MRAS surface, quadratic B-spline weight functions are used to blend the offset functions. Numerous experiments show that our approach can greatly reduce the storage of the reconstructed implicit surface while preserve the fitting accuracy compared with the state-of-the-art methods. Furthermore, our method has good adaptability and is able to produce reconstruction results with high quality.

Energy Efficient in Virtual Infrastructure and Green Cloud Computing: A Review

Background: Cloud computing has been widely used to refer different technologies, services and concepts. It is associated with virtualized infrastructure or hardware on-demand. There exist many challenges especially in energy management on datacenter. To analyze the energy consumptions in datacenter, monitor the types of services which are offered by cloud computing to users. Statistical analysis: The parameters are analyzed and calculate the power usages in the different states, like server level, cluster level in the datacenter and are recorded in real time. The results are obtained at different level and helps to optimize the servers in data center, the server level optimization, consolidation and load balancing between the servers are achieved which helps to consume less in power in the cloud. Findings: Based on the statistical analysis, the server consolidation and optimization are required in the cloud environment. This paper mainly focused on application virtualization and desktop virtualization in cloud environment. From the identified the gap between server consolidation and optimization, created an experimental setup and continuously monitor the CPU usage, memory usage, Disk and network usage parameters. The Power consumption was calculated based on the above mentioned parameters which help to reduce the energy consumption in data center. Applications/Improvements: The analysis parameters are experimented in the laboratory which are helped to consume less power and reduced the carbon emission in the data center. This helps to achieves green cloud environment in data center.

Survey on Data Deduplication for Cloud Storage to Reduce Fragmentation

Deduplication is an important technique which provides better result to store more information with less space. Cost and maintenance of Information backup storage system for major enterprises can be minimized by storing it on Cloud Storage. Data redundancy between different kinds of data storage gets minimal by utilizing data deduplication method. By giving each application differently and storing the associated information distinctly the overall disk usage can be enhanced to a great level. Cloud backup systems uses data deduplication to eliminate duplicate chunks that are present in multiple files. The duplicate chunks are substituted with the references to already present chunks through deduplication, without storing it again on cloud storage. The successive chunks are actually stored in scattered form in backup system in numerous segments (the storage unit of cloud). KeywordsBackup, Data deduplication, Fragmentation.

Disk usage prediction based on an improved weighted moving average method

Disk usage prediction based on an improved weighted moving average method

Sharing lattice QCD data over a widely distributed file system

JLDG is a data-grid for the lattice QCD (LQCD) community in Japan. Several large research groups in Japan have been working on lattice QCD simulations using supercomputers distributed over distant sites. The JLDG provides such collaborations with an efficient method of data management and sharing.File servers installed on 9 sites are connected to the NII SINET VPN and are bound into a single file system with the GFarm. The file system looks the same from any sites, so that users can do analyses on a supercomputer on a site, using data generated and stored in the JLDG at a different site.We present a brief description of hardware and software of the JLDG, including a recently developed subsystem for cooperating with the HPCI shared storage, and report performance and statistics of the JLDG. As of April 2015, 15 research groups (61 users) store their daily research data of 4.7PB including replica and 68 million files in total. Number of publications for works which used the JLDG is 98. The large number of publications and recent rapid increase of disk usage convince us that the JLDG has grown up into a useful infrastructure for LQCD community in Japan.

SSDExplorer: A Virtual Platform for Performance/Reliability-Oriented Fine-Grained Design Space Exploration of Solid State Drives

Currently available electronic design automation tools for design space exploration of solid state drives (SSDs) are not able to assess: 1) the device architecture inefficiencies; 2) architecture overdesign for a target performance; and 3) performance degradation caused by the disk usage. These tools feature either an overly high abstraction modeling strategy or lack the required flexibility to perform design exploration. To overcome these problems, this paper proposes SSDExplorer, a tool for fine-grained yet reasonably fast design space exploration of different SSD architectures highlighting possible bottlenecks. To prove its accuracy SSDExplorer has been validated with two real SSDs. SSDExplorer efficiency has been assessed by evaluating the impact of the NAND flash read retry algorithm impact on the SSD performance as a function of its internal architecture.

Next step toward the automation of screening for cervical cancer.

The simulations have always been of a great importance as they substitute the real processes when those are too expensive to be performed or impossible to be observed. The latter case is typical for optical microscopy. Here, we observe fixed or living cells under assumption, that the optical system and the attached electronic acquisition device do not affect the quality of the original specimen too much. Even though we can measure the most of optical aberrations and estimate the dominant sources of noise that together cause the final observed image to be blurred and noisy, we are still not able to reveal the original unaffected image how it would appear without any damage. Although several deconvolution methods are capable of inverting this degradation process, they can improve the quality of the data only to some extent. As there exists no exact knowledge, how the microscopic specimens look like, it is very difficult to evaluate the quality of new emerging segmentation and tracking algorithms that are of a great importance in medicine and biology. The same issue arises when one wants to tune-up their parameters. In the past, the only available quality measurement of the algorithms was an expert’s knowledge. The expert either classified the results of selected algorithm or provided an annotation of some real image dataset that was further used for evaluation purposes. Both ways, however, suffer from two main issues. First, the expert’s evaluation is nondeterministic. Second, for higher dimensional data (sequences of 2D or 3D images) the handmade annotation is impractical or even impossible. For this reason, the synthetic data, naturally accompanied by their ground truth, have started to appear. In the very beginning, only the basic geometric shapes like spheres or disc without any texture representing the internal structure of the observed cells were used. Since the late 90s, computer generated images have started to be more complex as the computer capabilities rose and allowed for calculations that required higher performance and extensive memory and disk usage. Namely, in the last 10 years, several simulation frameworks able to gener- ate for example cells with detailed description of subcellular components, large cell populations, and time-lapse image sequences emerged.

Fuzzy Pool Balance: An algorithm to achieve a two dimensional balance in distribute storage systems

The limitation of scheduling modules and the gradual addition of disk pools in distributed storage systems often result in imbalances among their disk pools in terms of both disk usage and file count. This can cause various problems to the storage system such as single point of failure, low system throughput and imbalanced resource utilization and system loads. An algorithm named Fuzzy Pool Balance (FPB) is proposed here to solve this problem. The input of FPB is the current file distribution among disk pools and the output is a file migration plan indicating what files are to be migrated to which pools. FPB uses an array to classify the files by their sizes. The file classification array is dynamically calculated with a defined threshold named Tmax that defines the allowed pool disk usage deviations. File classification is the basis of file migration. FPB also defines the Immigration Pool (IP) and Emigration Pool (EP) according to the pool disk usage and File Quantity Ratio (FQR) that indicates the percentage of each category of files in each disk pool, so files with higher FQR in an EP will be migrated to IP(s) with a lower FQR of this file category. To verify this algorithm, we implemented FPB on an ATLAS Tier2 dCache production system. The results show that FPB can achieve a very good balance in both free space and file counts, and adjusting the threshold value Tmax and the correction factor to the average FQR can achieve a tradeoff between free space and file count.

Predicting Utilization of Server Resources from Log Data

 Abstract—In this paper, we propose a model to predict resource utilization matrix for a given workload by mining the information residing in application as well as system logs for resource utilization. Unlike regression based or queuing network based approaches, our mechanism neither requires estimating per-function resource utilization nor does it require to benchmark individual business transactions in order to derive resource utilization matrix for the desired workload. In our experimental analysis, we have tried to predict the utilization of server resources like cpu, memory, disk and network usage based on several workload pattern. Across all experiments, we find the average absolute error in predicting utilization of all resources was less than 6%. This model becomes particularly helpful in the scenario where there are only few data-points available for system running with light workload and it is essential to analyze the impact of any change in workload pattern demanding heavy resource usage. Our model is not only useful for resource provisioning and what-if analysis to assess the impact of any workload change but also can be used for bottleneck analysis and early alert generating engine.