Storage Layouts Research Articles

Optimising the design and real-time operation of systems of distributed stormwater storages to reduce urban flooding at the catchment scale

The impact of urban flooding is increasing due to the effects of increasing urbanisation and climate change. The use of storages is a relatively well-established approach to reduce peak flows and therefore reduce the need for costly upgrades to stormwater conveyance infrastructure. Recently, real-time control (RTC) has been considered as a means of increasing the performance of these storages. However, previous RTC studies have generally focussed on evaluating RTC with pre-determined storage layouts, i.e., storage locations and volumes, and have ignored the impact of optimising the storage layout. Hence, the objectives of this paper are (i) to introduce a two-step approach to minimising peak flows at the catchment scale by firstly optimising the layout of distributed storages and then optimising their RTC strategies, and (ii) to test the effectiveness of this approach on a real catchment in Adelaide, South Australia. Results show that distributed storage with optimised layouts can achieve significantly higher peak flow reductions than more commonly used end-of-system storage. These range from 5% for 100 m3 of system storage up to 40% for 700 m3 of system storage – whereas end-of-system storage achieved no peak flow reduction for up to 700 m3 of storage. The addition of optimised RTC to distributed storages is able to achieve an additional 10% in peak flow reduction. This increase in performance due to RTC is more significant for smaller system storage volumes. Analysis of the flood hydrographs shows the system peak flow reductions are achieved by attenuating hydrographs at individual storages, as well as delaying hydrographs to reduce their coincidence at the confluence of sub-catchments. These results highlight the potential for using optimised distributed storage layout and RTC as an alternative approach to end-of-system storage to reduce flood peaks.

Towards instance-optimized data systems

In recent years, we have seen increased interest in applying machine learning to system problems. For example, there has been work on applying machine learning to improve query optimization, indexing, storage layouts, scheduling, log-structured merge trees, sorting, compression, and sketches, among many other data management tasks. Arguably, the ideas behind these techniques are similar: machine learning is used to model the data and/or workload in order to derive a more efficient algorithm or data structure. Ultimately, these techniques will allow us to build "instance-optimized" systems: that is, systems that self-adjust to a given workload and data distribution to provide unprecedented performance without the need for tuning by an administrator. While many of these techniques promise orders-of-magnitude better performance in lab settings, there is still general skepticism about how practical the current techniques really are. The following is intended as a progress report on ML for Systems and its readiness for real-world deployments, with a focus on our projects done as part of the Data Systems and AI Lab (DSAIL) at MIT By no means is it a comprehensive overview of all existing work, which has been steadily growing over the past several years not only in the database community but also in the systems, networking, theory, PL, and many other adjacent communities.

Understanding and Optimizing Conjunctive Predicates Under Memory-Efficient Storage Layouts

Database queries can contain multiple predicates. The optimization of conjunctive predicates is still vital to the overall performance of analytic data processing tasks. Prior work proposes several memory-efficient storage layouts, e.g., BitWeaving and ByteSlice, to significantly accelerate predicate evaluation, as circuit-level intra-cycle parallelism available in modern CPUs can be exploited such that the total number of instructions can be dramatically reduced. However, the performance potential of conjunctive predicates has not been harvested yet under such storage layouts as there is no accurate cost model to provide necessary insights that guide the optimization process. In this paper, we propose a hybrid empirical/analytical cost model (Understanding) to unveil the performance characteristics of such storage layouts when applying to predicate evaluation. Our cost model takes into account effect of non-linear factors, e.g., cache miss and branch misprediction, and easily applies to different CPUs. The main finding from our cost model is to distinguish high-cost instruction (which suffers from cache miss and/or branch misprediction) from low-cost instruction (which enjoys cache hit and correct branch prediction) in the context of predicate evaluation under these storage layouts. Guided by such a finding, we propose a simple execution scheme Hebe (Optimizing), which is order-oblivious while maintaining high performance. Hebe is attractive to the query optimizer (QO), as the QO does not need to go through a sampling process to decide the optimal evaluation order in advance. The intuition behind Hebe is to significantly reduce the number of high-cost instructions while keeping low-cost instructions unchanged. Our finding from Hebe sheds light on the importance of accurate cost model that guide us to derive an efficient execution scheme for query processing on modern CPUs.

Impact of mixed precision and storage layout on additive Schwarz smoothers

AbstractThe growing discrepancy between CPU computing power and memory bandwidth drives more and more numerical algorithms into a bandwidth‐bound regime. One example is the overlapping Schwarz smoother, a highly effective building block for iterative multigrid solution of elliptic equations with higher order finite elements. Two options of reducing the required memory bandwidth are sparsity exploiting storage layouts and representing matrix entries with reduced precision in floating point or fixed point format. We investigate the impact of several options on storage demand and contraction rate, both analytically in the context of subspace correction methods and numerically at an example of solid mechanics. Both perspectives agree on the favourite scheme: fixed point representation of Cholesky factors in nested dissection storage.

Efficient storage of heterogeneous geospatial data in spatial databases

The no-schema approach of NoSQL document stores is a tempting solution for importing heterogenous geospatial data to a spatial database. However, this approach means sacrificing the benefits of RDBMSes, such as existing integrations and the ACID principle. Previous comparisons of the document-store and table-based layout for storing geospatial data favours the document-store approach but does not consider importing data that can be segmented into homogenous datasets. In this paper we propose “The Heterogeneous Open Geodata Storage (HOGS)” system. HOGS is a command line utility that automates the process of importing geospatial data to a PostgreSQL/PostGIS database. It is developed in order to compare the performance of a traditional storage layout adhering to the ACID principle, and a NoSQL-inspired document store. A collection of eight open geospatial datasets comprising 15 million features was imported and queried in order to compare the differences between the two storage layouts. The results from a quantitative experiment are presented and shows that large amounts of open geospatial data can be stored using traditional RDBMSes using a table-based layout without any performance penalties.

Analysis of two heat storage integrations for an Organic Rankine Cycle Parabolic trough solar power plant

Among the concentrated solar power technologies, those based on Organic Rankine Cycles have a very low market presence. However they have favorable characteristics for applications with low temperature and small/medium size (<10MW), such as off-grid applications or distributed power generation.In this paper is analyzed a 5MW parabolic trough plant integrated with an Organic Rankine cycle power block and thermal storage. On this purpose, two different thermal storage integrations are analyzed. They are based on two different heat storage layouts: direct system using Hitec XL both as Heat Transfer Fluid and as storage medium; indirect system using Therminol VP-1 as Heat Transfer Fluid and Hitec XL as storage medium.Full system performance at rated and off-design conditions is presented operating with different organic working fluids. Its potential application and main challenges for its development are discussed in terms of performance and costs. Among the analyzed working fluids, the best results were obtained for the cycle working with Toluene with an efficiency at the power block of 31.5% and an estimated power block cost of 825€/kW. The indirect storage layout was the most interesting from the point of view of Levelized Electricity Cost (16.19c€/kW) and productivity (28.2GWh/y for a 5MWel plant) for 10h of storage However, it results in a storage tanks volume 26% greater than the obtained for the equivalent direct storage layout. The results show the competitiveness and the potential of the proposed integrated small size parabolic trough designs for isolated applications as mines or for some distributed generation uses where grid capacity is limited.

Automated Storage and Retrieval Systems (ASRS): Research on Warehouse Configuration and Performance Studies

Automated storage and retrieval systems (ASRSs) are used for loading and retrievaling products in the companies' warehouses. A variety of automated storage and retrieval vehicles are used in different storage layouts in ASRSs. The loading and retrievaling process is usually performed with aisle robots (ARs) and shuttle robots (SRs) at these systems. While the placement and retrievaling of products on/from the shelves, the operator intervention is minimized. In this paper, a literature overview has been given about ASRSs' types, applied various storage installations and solutions and optimization studies used while loading and retrievaling products.

A principled approach to bridging the gap between graph data and their schemas

Although RDF graph data often come with an associated schema, recent studies have proven that real RDF data rarely conform to their perceived schemas. Since a number of data management decisions, including storage layouts, indexing, and efficient query processing, use schemas to guide the decision making, it is imperative to have an accurate description of the structuredness of the data at hand (how well the data conform to the schema). In this paper, we have approached the study of the structuredness of an RDF graph in a principled way: we propose a framework for specifying structuredness functions, which gauge the degree to which an RDF graph conforms to a schema. In particular, we first define a formal language for specifying structuredness functions with expressions we call rules. This language allows a user to state a rule to which an RDF graph may fully or partially conform. Then we consider the issue of discovering a refinement of a sort (type) by partitioning the dataset into subsets whose structuredness is over a specified threshold. In particular, we prove that the natural decision problem associated to this refinement problem is NP-complete, and we provide a natural translation of this problem into Integer Linear Programming (ILP). Finally, we test this ILP solution with three real world datasets and three different and intuitive rules, which gauge the structuredness in different ways. We show that the rules give meaningful refinements of the datasets, showing that our language can be a powerful tool for understanding the structure of RDF data, and we show that the ILP solution is practical for a large fraction of existing data.

ImgLib2—generic image processing in Java

Summary: ImgLib2 is an open-source Java library for n-dimensional data representation and manipulation with focus on image processing. It aims at minimizing code duplication by cleanly separating pixel-algebra, data access and data representation in memory. Algorithms can be implemented for classes of pixel types and generic access patterns by which they become independent of the specific dimensionality, pixel type and data representation. ImgLib2 illustrates that an elegant high-level programming interface can be achieved without sacrificing performance. It provides efficient implementations of common data types, storage layouts and algorithms. It is the data model underlying ImageJ2, the KNIME Image Processing toolbox and an increasing number of Fiji-Plugins.Availability: ImgLib2 is licensed under BSD. Documentation and source code are available at http://imglib2.net and in a public repository at https://github.com/imagej/imglib.Supplementary Information: Supplementary data are available at Bioinformatics Online.Contact: saalfeld@mpi-cbg.de

Comparative evaluation of overlap strategies with study of I/O overlap in MPI-IO

Many scientific applications use parallel I/O to meet the low latency and high bandwidth I/O requirement. Among many available parallel I/O operations, collective I/O is one of the most popular methods when the storage layouts and access patterns of data do not match. The implementation of collective I/O typically involves disk I/O operations followed by interprocessor communications. Also, in many I/O-intensive applications, parallel I/O operations are usually followed by parallel computations. This paper presents a comparative study of different overlap strategies in parallel applications. We have experimented with four different overlap strategies 1) Overlapping I/O and communication; 2) Overlapping I/O and computation; 3) Overlapping computation and communication; and 4) Overlapping I/O, communication, and computation. All experiments have been conducted on a Linux Cluster and the performance results obtained are very encouraging. On an average, we have enhanced the performance of a generic collective read call by 38%, the MxM benchmark by 26%, and the FFT benchmark by 34%.

Continuous program optimization

Much of the software in everyday operation is not making optimal use of the hardware on which it actually runs. Among the reasons for this discrepancy are hardware/software mismatches, modularization overheads introduced by software engineering considerations, and the inability of systems to adapt to users' behaviors.A solution to these problems is to delay code generation until load time. This is the earliest point at which a piece of software can be fine-tuned to the actual capabilities of the hardware on which it is about to be executed, and also the earliest point at wich modularization overheads can be overcome by global optimization.A still better match between software and hardware can be achieved by replacing the already executing software at regular intervals by new versions constructed on-the-fly using a background code re-optimizer. This not only enables the use of live profiling data to guide optimization decisions, but also facilitates adaptation to changing usage patterns and the late addition of dynamic link libraries.This paper presents a system that provides code generation at load-time and continuous program optimization at run-time. First, the architecture of the system is presented. Then, two optimization techniques are discussed that were developed specifically in the context of continuous optimization. The first of these optimizations continually adjusts the storage layouts of dynamic data structures to maximize data cache locality, while the second performs profile-driven instruction re-scheduling to increase instruction-level parallelism. These two optimizations have very different cost/benefit ratios, presented in a series of benchmarks. The paper concludes with an outlook to future research directions and an enumeration of some remaining research problems.The empirical results presented in this paper make a case in favor of continuous optimization, but indicate that it needs to be applied judiciously. In many situations, the costs of dynamic optimizations outweigh their benefit, so that no break-even point is ever reached. In favorable circumstances, on the other hand, speed-ups of over 120% have been observed. It appears as if the main beneficiaries of continuous optimization are shared libraries, which at different times can be optimized in the context of the currently dominant client application.

High-performance linear algebra algorithms using new generalized data structures for matrices

We present a novel way to produce dense linear algebra factorization algorithms. The current state-of-the-art (SOA) dense linear algebra algorithms have a performance inefficiency, and thus they give suboptimal performance for most LAPACK factorizations. We show that using standard Fortran and C two-dimensional arrays is the main source of this inefficiency. For the other standard format (packed one-dimensional arrays for symmetric and/or triangular matrices), the situation is much worse. We show how to correct these performance inefficiencies by using new data structures (NDS) along with so-called kernel routines. The NDS generalize the current storage layouts for both standard formats. We use the concept of Equivalence and Elementary Matrices along with coordinate (linear) transformations to prove that our method works for an entire class of dense linear algebra algorithms. Also, we use the Algorithms and Architecture approach to explain why our new method gives higher efficiency. The simplest forms of the new factorization algorithms are a direct generalization of the commonly used LINPACK algorithms. On IBM platforms they can be generated from simple, textbook-type codes by the XLF Fortran compiler. On the IBM POWER3 processor, our implementation of Cholesky factorization achieves 92% of peak performance, whereas conventional SOA full-format LAPACK DPOTRF achieves 77% of peak performance. All programming for our NDS can be accomplished in standard Fortran through the use of three- and four-dimensional arrays. Thus, no new compiler support is necessary. Finally, we describe block hybrid formats (BHF). BHF allow one to use no additional storage over conventional (full and packed) matrix storage. This means that new algorithms based on BHF can be used as a backward-compatible replacement for LAPACK or LINPACK algorithms.

A geometrical approach to computing expected cycle times for zonebased storage layouts in AS/RS

An exact, geometry-based analytical model is presented that can be used to compute the expected cycle time for a storage/retrieval (S/R) machine, executing single-commands, dual-commands, or both, in a rack structure that has been laid out in pre-specified storage zones for classes of goods. The rack may be either square-in-time or non-square-in-time. The approach is intuitively appealing, and it does not assume any certain layout shape, such as traditional 'L-shaped' class layouts. The model has been coded in Turbo Pascal, and can be used by designers as a tool for quickly evaluating alternative layout configurations with respect to expected S/R cycle time in an AS/RS, and thereby the throughput of an automated warehouse over time. This model has been successfully applied in a major manufacturing plant in Europe to evaluate reconfigurations of their rack storage layouts over the past five years.

Cost-Benefit Analysis of Long-Term Cabbage Storage

The costs of storing locally grown cabbage for off-season marketing were investigated. Building and equipment costs as well as annual costs associated with the operation of six different hypothetical storage layouts, all with same holding capacity of 600 t, were reviewed. Long term storage of cabbage for marketing during the non-producing seasons on the Montreal market was found to be economically sound since the internal rates of return (IRR) computed were positive and ranged from 17% to 27%. The best time to market the cabbage greatly depended on the storage conditions. The best marketing time for cabbage stored in a room ventilated with outside air (VA), in a room equipped with mechanical refrigeration (RA) and in a room operated under controlled atmosphere (CA) were, respectively, January, March and May. It was also shown that the upgrading of an RA storage room to CA increased the IRR.