Out-of-core Research Articles

Research on Linear Static Water Resources IO Model Optimization Based on General Equilibrium Method

The static input-output model is limited to analyzing and studying the production process in a certain period, and it is weak in explaining the process of social reproduction. At the same time, the linear assumption of the model is contrary to the actual production. Based on the evolution of the input-output model, the linear static water resources I-O model is extended to the computable nonlinear dynamic water resources I-O model, which overcomes the irreplaceable elements of the input model of the linear water resources I-O model and the CGE model can only solve the market static equilibrium point Shortcomings, more realistic description of the regional economic system and the relationship between water resources system.

Generalized enhanced suffix array construction in external memory

BackgroundSuffix arrays, augmented by additional data structures, allow solving efficiently many string processing problems. The external memory construction of the generalized suffix array for a string collection is a fundamental task when the size of the input collection or the data structure exceeds the available internal memory.ResultsIn this article we present and analyze mathsf {eGSA} [introduced in CPM (External memory generalized suffix and mathsf {LCP} arrays construction. In: Proceedings of CPM. pp 201–10, 2013)], the first external memory algorithm to construct generalized suffix arrays augmented with the longest common prefix array for a string collection. Our algorithm relies on a combination of buffers, induced sorting and a heap to avoid direct string comparisons. We performed experiments that covered different aspects of our algorithm, including running time, efficiency, external memory access, internal phases and the influence of different optimization strategies. On real datasets of size up to 24 GB and using 2 GB of internal memory, mathsf {eGSA} showed a competitive performance when compared to mathsf {eSAIS} and mathsf {SAscan}, which are efficient algorithms for a single string according to the related literature. We also show the effect of disk caching managed by the operating system on our algorithm.ConclusionsThe proposed algorithm was validated through performance tests using real datasets from different domains, in various combinations, and showed a competitive performance. Our algorithm can also construct the generalized Burrows-Wheeler transform of a string collection with no additional cost except by the output time.

On approximating the accelerator part in dynamic input–output models

We release the limitations of previous studies and instead of setting the crucial parameters of the dynamic endogenous input–output model with layers of techniques on an arbitrary basis we propose a new optimization-based approach to approximating of the elements of capital matrices on the basis of recent historical data. Using recent IO data we first formally prove that in comparison to arbitrarily adjusted dynamic IO models the new theoretical approach allows one to obtain a significantly better fit to the historical data in the short-run. This result has also some implications for the long-run analyses as it suggests that using the new approach for typical empirical applications of dynamic IO models with respect to modelling future behavior of economies seems relatively much more reasonable. Having this remark in mind, in the empirical part of the paper we use the new methodological approach in a particular case study. In an illustrative empirical application we try to forecast the possible evolution of sectoral classification in the Polish economy over the next 40 years.

Multi-regional economic impacts of recreational fisheries: Analysis of Small Sea Ranch in Gyeong-Nam Province, Korea

The economic impacts of recreational fishing in Small Sea Ranch in Gyeong-Nam (GN) province, Korea, are calculated using a multi-regional input-output (MRIO) model to overcome the weakness of a single-region IO model that ignores the spread effects in other provinces. First, multiplier decomposition analysis is conducted to substantiate the existence of strong economic linkages between GN and other provinces by calculating intra-regional effects, net closed-loop effects, net open-loop effects and spillover coefficients. Subsequently, the economic impacts from spending by anglers’ visits to Small Sea Ranch are computed using a 161-sector MRIO model for each of the sixteen provinces in Korea. The results show that there exist strong relationships between the economy of GN and those of the other provinces, and that only around 68% of the economic impacts of anglers’ spending on the total output in Korea accrues in GN, with the remaining impacts (32%) being accounted for in non-GN provinces. This indicates the importance of using a multi-regional framework to estimate the economic impacts of recreational fisheries for a region whose economy relies heavily on the economies of other regions.

Privacy-Preserving Computation with Trusted Computing via Scramble-then-Compute

AbstractWe consider privacy-preserving computation of big data using trusted computing primitives with limited private memory. Simply ensuring that the data remains encrypted outside the trusted computing environment is insufficient to preserve data privacy, for data movement observed during computation could leak information. While it is possible to thwart such leakage using generic solution such as ORAM [42], designing efficient privacy-preserving algorithms is challenging. Besides computation efficiency, it is critical to keep trusted code bases lean, for large ones are unwieldy to vet and verify. In this paper, we advocate a simple approach wherein many basic algorithms (e.g., sorting) can be made privacy-preserving by adding a step that securely scrambles the data before feeding it to the original algorithms. We call this approachScramble-then-Compute(StC), and give a sufficient condition whereby existing external memory algorithms can be made privacy-preserving via StC. This approach facilitates code-reuse, and its simplicity contributes to a smaller trusted code base. It is also general, allowing algorithm designers to leverage an extensive body of known efficient algorithms for better performance. Our experiments show that StC could offer up to 4.1× speedups over known, application-specific alternatives.

Competition and environmental quality as conflicting objectives: the case of the European municipal waste industry

The article focuses on the European Union (EU) municipal waste (MW) industry, examining the conjoint functioning of EU self sufficiency principle and proximity principle (SSP/PP). Using standard IO models, we derive the quota of recycled MW under different kinds of market competition. We show that the percentage of unsorted MW, and the connected environmental problems, increases with competition, so that social welfare is lower as compared to SSP/PP implementation. Nonetheless, higher market competition in the MW industry matched with Pigouvian taxation improves social welfare upon SSP/PP.

Representation of the direct correlation function of the hard-sphere fluid.

An accurate representation of the structural pair correlation functions of the hard sphere (HS) fluid up to the freezing density is proposed which combines the pole expression for the total correlation function h(r), the Ornstein-Zernike equation, and molecular dynamics (MD) computer simulation data. In the scheme, h(r) is expressed in terms of a set of pole parameters, which reveals how the tail of the Fourier transform of h(r) contains information on the discontinuities in the derivatives of the direct correlation function (DCF). This formulation leads to a DCF expressed as the sum of a numerically obtainable part and an analytic part which consists of elementary integral terms, some of which are found to give rise to the discontinuities. An exact formula for the magnitude of these discontinuities is derived, which indicates that there is a particular density (ρ≅0.133) below which the magnitude of the discontinuities decreases with increasing order of the derivative. With the accurate MD simulation data the set of parameters that specifies h(r) was determined. These can be used to obtain the different structural functions of the HS fluid, and following the calculation stages of the pole structure scheme the DCF is obtained. From this route to the DCF the second pole of the HS fluid can be determined and the non-negligible role of the "out of core" part of the DCF at high densities is revealed. The density-dependent separation range where the two pole approximation represents well the h(r) has been determined.

USEEIO: A new and transparent United States environmentally-extended input-output model

National-scope environmental life cycle models of goods and services may be used for many purposes, not limited to quantifying impacts of production and consumption of nations, assessing organization-wide impacts, identifying purchasing hotspots, analyzing environmental impacts of policies, and performing streamlined life cycle assessment. USEEIO is a new environmentally-extended input-output model of the United States fit for such purposes and other sustainable materials management applications. USEEIO melds data on economic transactions between 389 industry sectors with environmental data for these sectors covering land, water, energy and mineral usage and emissions of greenhouse gases, criteria air pollutants, nutrients and toxics, to build a life cycle model of 385 US goods and services. In comparison with existing US models, USEEIO is more current with most data representing year 2013, more extensive in its coverage of resources and emissions, more deliberate and detailed in its interpretation and combination of data sources, and includes formal data quality evaluation and description. USEEIO is assembled with a new Python module called the IO Model Builder capable of assembling and calculating results of user-defined input-output models and exporting the models into LCA software. The model and data quality evaluation capabilities are demonstrated with an analysis of the environmental performance of an average hospital in the US. All USEEIO files are publicly available bringing a new level of transparency for environmentally-extended input-output models.

EmMAW: computing minimal absent words in external memory.

The biological significance of minimal absent words has been investigated in genomes of organisms from all domains of life. For instance, three minimal absent words of the human genome were found in Ebola virus genomes. There exists an O(n) -time and O(n) -space algorithm for computing all minimal absent words of a sequence of length n on a fixed-sized alphabet based on suffix arrays. A standard implementation of this algorithm, when applied to a large sequence of length n , requires more than 20 n bytes of RAM. Such memory requirements are a significant hurdle to the computation of minimal absent words in large datasets. We present emMAW, the first external-memory algorithm for computing minimal absent words. A free open-source implementation of our algorithm is made available. This allows for computation of minimal absent words on far bigger data sets than was previously possible. Our implementation requires less than 3 h on a standard workstation to process the full human genome when as little as 1 GB of RAM is made available. We stress that our implementation, despite making use of external memory, is fast; indeed, even on relatively smaller datasets when enough RAM is available to hold all necessary data structures, it is less than two times slower than state-of-the-art internal-memory implementations. https://github.com/solonas13/maw (free software under the terms of the GNU GPL). alice.heliou@lix.polytechnique.fr or solon.pissis@kcl.ac.uk. Supplementary data are available at Bioinformatics online.

Faster Average Case Low Memory Semi-external Construction of the Burrows\u2013Wheeler Transform

The Burrows–Wheeler transform has applications in data compression as well as full text indexing. Despite its important applications and various existing algorithmic approaches the construction of the transform for large data sets is still challenging. In this paper we present a new semi external memory algorithm for constructing the Burrows–Wheeler transform. It is capable of constructing the transform for an input text of length n over a finite alphabet in time O(nlog ^2log n) on average, if sufficient internal memory is available to hold a fixed fraction of the input text. In the worst case the run-time is O(nlog n log log n). The amount of space used by the algorithm in external memory is O(n) bits. Based on the serial version we also present a shared memory parallel algorithm running in time O(frac{n}{p}max {log ^2log n+log p}) on average when p processors are available.

On the limited usability of the inoperability IO model

ABSTRACTThis note shows that the inoperability input–output model (IIM) estimates only a part of mainly the negative indirect economic impacts of disasters, whereas it neglects most of the positive indirect impacts. This means that the IIM is not suited to prioritize industries for policy interventions that aim at reducing the negative impacts of such disasters. Besides, this note shows that the application of the IIM is problematic and tends to overestimate the subset of impacts that the model is able to quantify. Finally, we identify two approaches that much better capture the variety of different disaster impacts.

Structural change versus turnpike optimality: A Polish perspective

Using a modified dynamic IO model for Poland which allows taking into account actual trends observed in recently available statistical data we compare the rate of economic growth calculated for different growth paths resulting from the model. The goal of the research was to examine the distance between the actual structure of production and the structure on the turnpike and its impact on the economic growth of the economy under study. The results of the study indicate that the impact of structural change on output takes place in three general stages. The benefits of structural change do not outbalance the corresponding costs immediately, since it takes several periods until the growth rate of those paths which are closer to the von Neumann ray become larger than the corresponding growth rate of the benchmark growth path.

A network approach for assembling and linking input–output models

ABSTRACTInput–output (IO) models, describing trade between different sectors and regions, are widely used to study the environmental repercussions of human activities. A frequent challenge in assembling an IO model or linking several such models is the absence of flow data with the same level of detail for all components. Such problems can be addressed using proportional allocation, which is a form of algebraic transformations. In this paper, we propose a novel approach whereby the IO system is viewed as a network, the topology of which is transformed with the addition of virtual nodes so that available empirical flow data can be mapped directly to existing links, with no additional estimation required, and no impact on results. As IO systems become increasingly disaggregated, and coupled to adjacent databases and models, the adaptability of IO frameworks becomes increasingly important. We show that topological transformations also offer large advantages in terms of transparency, modularity and increasingly importantly for global IO models, efficiency. We illustrate the results in the context of trade linking, multi-scale integration and other applications.

Modeling the water circulation in the North Atlantic in the scope of the CORE-II experiment

A numerical experiment on the reproduction of the variability in the state of North Atlantic water in 1948–2007 with a spatial resolution of 0.25° has been performed using the global ocean model developed at Institute of Numerical Mathematics, Russian Academy of Sciences (INM RAS), and the Shirshov Institute of Oceanology (IO RAS) (the INM–IO model). The data on the state of the atmosphere, radiation fluxes, and bulk formulas of the CORE-II protocol are used as boundary conditions. Five successive 60-year calculation cycles have been performed in order to obtain the quasi-equilibrium state of a model ocean. For the last 20 years, the main elements of large-scale ocean circulation have been analyzed and compared with the WOA09 atlas data and the results of other models.

I/O-efficient join dependency testing, Loomis–Whitney join, and triangle enumeration

We revisit two fundamental problems in database theory. The join-dependency (JD) testing problem is to determine whether a given JD holds on a relation r. We prove that the problem is NP-hard even if the JD involves only relations each of which has only two attributes. The JD-existence testing problem is to determine if there exists any non-trivial JD satisfied by r. We present an I/O-efficient algorithm in the external memory model, which in fact settles the closely related Loomis–Whitney enumeration problem. As a side product, we solve the triangle enumeration problem with the optimal I/O-complexity, improving a recent result of Pagh and Silvestri in PODS'14.

An External-Memory Algorithm for String Graph Construction

Some recent results (Bauer et al. in Algorithms in bioinformatics, Springer, Berlin, pp 326---337, 2012; Cox et al. in Algorithms in bioinformatics, Springer, Berlin, pp. 214---224, 2012; Rosone and Sciortino in The nature of computation. Logic, algorithms, applications, Springer, Berlin, pp 353---364, 2013) have introduced external-memory algorithms to compute self-indexes of a set of strings, mainly via computing the Burrows---Wheeler transform of the input strings. The motivations for those results stem from Bioinformatics, where a large number of short strings (called reads) are routinely produced and analyzed. In that field, a fundamental problem is to assemble a genome from a large set of much shorter samples extracted from the unknown genome. The approaches that are currently used to tackle this problem are memory-intensive. This fact does not bode well with the ongoing increase in the availability of genomic data. A data structure that is used in genome assembly is the string graph, where vertices correspond to samples and arcs represent two overlapping samples. In this paper we address an open problem of Simpson and Durbin (Bioinformatics 26(12):i367---i373, 2010): to design an external-memory algorithm to compute the string graph.

Truss decomposition of uncertain graphs

The k-truss of a graph is the largest edge-induced subgraph such that every edge is contained in at least k triangles within the subgraph, where a triangle is a cycle consisting of three vertices. As a new notion of cohesive subgraphs, truss has recently attracted a lot of research attentions in the database and data mining fields. At the same time, uncertainty is an intrinsic property of massive graph data, and truss decomposition (i.e., finding all k-trusses of a graph) has become a key primitive on uncertain graphs. In this paper, we study the truss decomposition problem on uncertain graphs, that is, finding all highly probable k-trusses of an uncertain graph. We first give an formal statement of the truss decomposition problem on uncertain graphs. Then, we prove that the truss decomposition of an uncertain graph attains two elegant properties, namely uniqueness and hierarchy. We show that the truss decomposition of an uncertain graph can be found in $$O(m^{1.5}Q)$$O(m1.5Q) time by proposing an in-memory algorithm called $$\mathtt {TD_{mem}}$$TDmem, where m is the number of edges of the uncertain graph, and Q is at most the maximum number of common neighbors of the endpoints of an edge. When an uncertain graph is too large to fit into main memory, we propose an external-memory algorithm $$\mathtt {TD_{I/O}}$$TDI/O to find the truss decomposition of the uncertain graph. Extensive experiments have been carried out to evaluate the practical performance of the proposed algorithms. The experimental results verify that both $$\mathtt {TD_{mem}}$$TDmem and $$\mathtt {TD_{I/O}}$$TDI/O are efficient when an uncertain graph is small enough to fit into main memory, and that $$\mathtt {TD_{I/O}}$$TDI/O is much faster than $$\mathtt {TD_{mem}}$$TDmem when the graph is too large to fit into main memory.

LCP Array Construction in External Memory

One of the most important data structures for string processing—the suffix array—needs to be augmented with the longest-common-prefix (LCP) array in numerous applications. We describe the first external memory algorithm for constructing the LCP array given the suffix array as input. The only previous way to compute the LCP array for data that is bigger than the RAM is to use an external memory suffix array construction algorithm (SACA) with complex modifications to produce the LCP array as a by-product. Compared to the best prior method, our algorithm needs much less disk space (by more than a factor of three) and is significantly faster. Furthermore, our algorithm can be combined with any SACA, including a better one developed in the future.

LSG: An External-Memory Tool to Compute String Graphs for Next-Generation Sequencing Data Assembly.

The large amount of short read data that has to be assembled in future applications, such as in metagenomics or cancer genomics, strongly motivates the investigation of disk-based approaches to index next-generation sequencing (NGS) data. Positive results in this direction stimulate the investigation of efficient external memory algorithms for de novo assembly from NGS data. Our article is also motivated by the open problem of designing a space-efficient algorithm to compute a string graph using an indexing procedure based on the Burrows-Wheeler transform (BWT). We have developed a disk-based algorithm for computing string graphs in external memory: the light string graph (LSG). LSG relies on a new representation of the FM-index that is exploited to use an amount of main memory requirement that is independent from the size of the data set. Moreover, we have developed a pipeline for genome assembly from NGS data that integrates LSG with the assembly step of SGA (Simpson and Durbin, 2012 ), a state-of-the-art string graph-based assembler, and uses BEETL for indexing the input data. LSG is open source software and is available online. We have analyzed our implementation on a 875-million read whole-genome dataset, on which LSG has built the string graph using only 1GB of main memory (reducing the memory occupation by a factor of 50 with respect to SGA), while requiring slightly more than twice the time than SGA. The analysis of the entire pipeline shows an important decrease in memory usage, while managing to have only a moderate increase in the running time.

Simulating evolution of interindustry linkages in endogenous dynamic IO model with layers of techniques

AbstractIn this article, we propose a new version of dynamic input–output model in which both technological progress and deployment are endogenous, and where sector‐specific outlays on R&D speed up the development of new technologies and the installation of capital stock. In this two‐technology model, the new and old technical processes within a sector exchange their relative weights in production. We use the model to obtain projections of the interindustry linkages of sectors in the Polish economy over the next 50 years. The results of this simulation suggest an ongoing change of the composition of the set of key sectors of the Polish economy. In general, one may expect to see an ongoing drop in the importance of agriculture‐ and heavy‐industry‐related sectors on the one hand, and a rise in the importance of services‐related ones on the other.