Memory Data Structures Research Articles

Causal Inconsistencies Are Normal in Windows Memory Dumps (Too)

Main memory contains valuable information for criminal investigations, e.g., process information or keys for disk encryption. Taking snapshots of memory is therefore common practice during a digital forensic examination. Inconsistencies in such memory dumps can, however, hamper their analysis. In this article, we perform a systematic assessment of causal inconsistencies in memory dumps taken on a Windows 10 machine using the kernel-level acquisition tool WinPmem. We use two approaches to measure the quantity of inconsistencies in Windows 10: (1) causal inconsistencies within self-injected memory data structures using a known methodology transferred from the Linux operating system, and (2) inconsistencies in the memory management data structures of the Windows kernel using a novel measurement technique based on properties of the virtual address descriptor (VAD) tree. Our evaluation is based on a dataset of more than 180 memory dumps. As a central result, both types of inconsistency measurement reveal that a high number of inconsistencies is the norm rather than the exception. We also correlate workload and execution time of the memory acquisition tool to the number of inconsistencies in the respective memory snapshot. By controlling these factors it is possible to (somewhat) control the level of inconsistencies in Windows memory dumps.

Vector-deductive Memory-based Transactions for Fault-as-address Simulation

The main idea is to create logic-free vector computing, using only read-write transactions on address memory. The strategic goal is to create a deterministic vector-quantum computing using photons for read-write transactions on stable subatomic memory elements. The main task is to implement new vector computing models and methods based on primitive read-write transactions in vector flexible interpretive fault modeling and simulation technology, where data is used as addresses for processing the data itself. The essence of vector computing is read-write transactions on vector data structures in address memory. Vector computing is a computational process based on elementary read-write transactions over cells of binary vectors that are stored in address memory and form a functionality where the input data to be processed is the addresses of these cells. The advantages of a vector universal model for a compact description of ordered processes, phenomena, functions, and structures are defined for the purpose of their parallel analysis. Analytical expressions of logic, which require algorithmically complex calculators, are replaced by output state vectors of elements and digital circuits, focused on the parallelism of register logical procedures on regular data structures. A vector-deductive method for formula synthesis for propagating input lists (data) of faults is proposed, which has a quadratic computational complexity of register operations. A new matrix of deductive vectors has been synthesized, which is characterized by the following properties: compactness, parallel data processing based on a single read-write transaction in memory, elimination of traditional logic from fault simulation procedures, full automation of its synthesis process, and focus on technological solving all problems of technical diagnosis. In the work, the transition to vector logic in the organization of computing and the elimination of traditional logic presented in the form of tables and analytical expressions were carried out. The use of read-write transactions on memory in the absence of a command system focuses the new vector-logic computing towards deterministic quantum architectures based on stable subatomic memory particles.

Compact and indexed representation for LiDAR point clouds

ABSTRACT LiDAR devices are capable of acquiring clouds of 3D points reflecting any object around them, and adding additional attributes to each point such as color, position, time, etc. LiDAR datasets are usually large, and compressed data formats (e.g. LAZ) have been proposed over the years. These formats are capable of transparently decompressing portions of the data, but they are not focused on solving general queries over the data. In contrast to that traditional approach, a new recent research line focuses on designing data structures that combine compression and indexation, allowing directly querying the compressed data. Compression is used to fit the data structure in main memory all the time, thus getting rid of disk accesses, and indexation is used to query the compressed data as fast as querying the uncompressed data. In this paper, we present the first data structure capable of losslessly compressing point clouds that have attributes and jointly indexing all three dimensions of space and attribute values. Our method is able to run range queries and attribute queries up to 100 times faster than previous methods.

Persistent Summaries

A persistent data structure , also known as a multiversion data structure in the database literature, is a data structure that preserves all its previous versions as it is updated over time. Every update (inserting, deleting, or changing a data record) to the data structure creates a new version, while all the versions are kept in the data structure so that any previous version can still be queried. Persistent data structures aim at recording all versions accurately, which results in a space requirement that is at least linear to the number of updates. In many of today’s big data applications, in particular, for high-speed streaming data, the volume and velocity of the data are so high that we cannot afford to store everything. Therefore, streaming algorithms have received a lot of attention in the research community, which uses only sublinear space by sacrificing slightly on accuracy. All streaming algorithms work by maintaining a small data structure in memory, which is usually called a sketch , summary , or synopsis . The summary is updated upon the arrival of every element in the stream, thus it is ephemeral , meaning that it can only answer queries about the current status of the stream. In this article, we aim at designing persistent summaries, thereby giving streaming algorithms the ability to answer queries about the stream at any prior time.

Block-encoding-based quantum algorithm for linear systems with displacement structures

Matrices with the displacement structures of circulant, Toeplitz, and Hankel types as well as matrices with structures generalizing these types are omnipresent in computations of sciences and engineering. In this paper we present efficient and memory-reduced quantum algorithms for solving linear systems with such structures by devising an approach to implement the block-encodings of these structured matrices. More specifically, by decomposing $n\ifmmode\times\else\texttimes\fi{}n$ dense matrices into linear combinations of displacement matrices, we first deduce the parametrized representations of the matrices with displacement structures so that they can be treated similarly. With such representations, we then construct $\ensuremath{\epsilon}$-approximate block-encodings of these structured matrices in two different data access models, i.e., the black-box model and the quantum random access memory (QRAM) data structure model. It is shown the quantum linear system solvers based on the proposed block-encodings provide a quadratic speedup with respect to the dimension over classical algorithms in the black-box model and an exponential speedup in the QRAM data structure model. In particular, these linear system solvers subsume known results with significant improvements and also can motivate new instances where there was no specialized quantum algorithm before. As an application, one of the quantum linear system solvers is applied to the linear prediction of time series, which justifies the claimed quantum speedup is achievable for problems of practical interest.

Verifying concurrent multicopy search structures

Multicopy search structures such as log-structured merge (LSM) trees are optimized for high insert/update/delete (collectively known as upsert) performance. In such data structures, an upsert on key k , which adds ( k , v ) where v can be a value or a tombstone, is added to the root node even if k is already present in other nodes. Thus there may be multiple copies of k in the search structure. A search on k aims to return the value associated with the most recent upsert. We present a general framework for verifying linearizability of concurrent multicopy search structures that abstracts from the underlying representation of the data structure in memory, enabling proof-reuse across diverse implementations. Based on our framework, we propose template algorithms for (a) LSM structures forming arbitrary directed acyclic graphs and (b) differential file structures, and formally verify these templates in the concurrent separation logic Iris. We also instantiate the LSM template to obtain the first verified concurrent in-memory LSM tree implementation.

The Local Power Set in Transaction Scanning for Efficient Frequent Item sets Mining

Discovering frequent item sets has been known among researchers as the most computationally expensive task in association rules mining. As a result of a technique used in the traditional algorithm, Apriori, generating new candidate (k+1)-item sets by performing self-join operation over elements of the frequent (k)-item sets which needs to scan dataset multiple times. Although, many modern algorithms have been proposed to the problems, most of them encounter with wasting the times to manipulate other data structures in memory. In this paper, we propose an algorithm utilizing concept of local power set enumeration and hash table. The proposed is not only able to escape self-join operation but also able to reduce CPU-times in computation when compared to the related algorithm that also applied concept of power set. To get better performance, we also modify the algorithm to two new versions applying an intersection-set operation in the phase of removing useless item sets from the dataset, instead of checking item elements of transactions one by one. Many experiments are conducted to evaluate performance of all our proposed. The results expressed that all modified versions of the algorithms provide better performance in term of reducing computational CPU-time and take less amount of scanning.

I/O-efficient 2-d orthogonal range skyline and attrition priority queues

We present the first static and dynamic external memory data structures for variants of 2-d orthogonal range skyline reporting with worst-case logarithmic query and update I/O-complexity. The results are obtained by using persistent data structures and by extending the attrition priorities queues of Sundar (1989) [26] to also support real-time concatenation, a result of independent interest. We show that the problem is as hard as standard 2-d orthogonal range reporting in the indexability model by a lower bound on the I/O-complexity of 2-d orthogonal anti-dominance skyline reporting queries.

A quantum algorithm for simulating non-sparse Hamiltonians

We present a quantum algorithm for simulating the dynamics of Hamiltonians that are not necessarily sparse. Our algorithm is based on the input model where the entries of the Hamiltonian are stored in a data structure in a quantum random access memory (qRAM) which allows for the efficient preparation of states that encode the rows of the Hamiltonian. We use a linear combination of quantum walks to achieve poly-logarithmic dependence on precision. The time complexity of our algorithm, measured in terms of the circuit depth, is O(t\sqrt{N}\norm{H}\,\polylog(N, t\norm{H}, 1/\epsilon)), where t is the evolution time, $N$ is the dimension of the system, and $\epsilon$ is the error in the final state, which we call precision. Our algorithm can be directly applied as a subroutine for unitary implementation and quantum linear systems solvers, achieving \widetilde{O}(\sqrt{N}) dependence for both applications.

RETRACTED ARTICLE: An improved memory adaptive up-growth to mine high utility itemsets from large transaction databases

High utility itemset (HUI) mining identifies an interesting pattern from transactional databases by taking into consider the utility of each in the transaction for instance, margins or profits. Many candidates are generated for HUIs which reduces the mining performance. Frequent pattern-growth (FP-Growth) algorithm was widely used to discover the frequent itemsets using FP-Tree. But, UP-Tree was constructed to store high utility item set. The mining of UP-Tree by FP-Growth extracts high utility itemset and generates too many candidates. So, UP-Growth and UP-Growth+ was proposed to shorten the candidate itemsets. In UP-Growth, two tactics such as discarding local unpromising items (DLU) and decreasing local node (DLN) were used in FP-Growth. In UP-Growth+, two strategies such as DLU and its estimated node utilities (DNU) and DLN tools for the nodes of internal UP-Tree by evaluated the descendant nodes (DNN) were incorporated in FP-Growth. However, the UP-Growth and UP-Growth+ has the problem of poor spatial and temporal localities. Initially, the UP-Tree is created in available main memory then extended to secondary memory when the transaction is large. The storage of data structure in secondary memory and accessibility is not clearly derived in existing algorithms. In this paper, the spatial locality issues of UP-Growth and UP-Growth+ is solved by rearranging nodes of UP-Tree in a depth first manner and temporal locality issues of UP-Growth and UP-Growth+ is solved by page blocking technique which reorganizes the execution part of UP-Growth and UP-Growth+. The computational part is rearranged. In addition to, the memory management strategy is introduced to minimize the space requirement of high utility itemsets. Thus the proposed Improved Adaptive UP-Growth (IAUP-Growth) and Improved Adaptive UP-Growth+ (IAUP-Growth+) overcomes the spatial and temporal locality problem and effectively reduces the memory usage.

OMiCroN – Oblique Multipass Hierarchy Creation while Navigating

Rendering large point clouds ordinarily requires building a hierarchical data structure for accessing the points that best represent the object for a given viewing frustum and level-of-detail. The building of such data structures frequently represents a large portion of the cost of the rendering pipeline both in terms of time and space complexity, especially when rendering is done for inspection purposes only. This problem has been addressed in the past by incremental construction approaches, but these either result in low quality hierarchies or in longer construction times. In this work we present OMiCroN – Oblique Multipass Hierarchy Creation while Navigating – which is the first algorithm capable of immediately displaying partial renders of the geometry, provided the cloud is made available sorted in Morton order. OMiCroN is fast, being capable of building the entire data structure in memory spending an amount of time that is comparable to that of just reading the cloud from disk. Thus, there is no need for storing an expensive hierarchy, nor for delaying the rendering until the whole hierarchy is read from disk. In fact, a pipeline coupling OMiCroN with an incremental sorting algorithm running in parallel can start rendering as soon as the first sorted prefix is produced, making this setup very convenient for streamed viewing.

External memory BWT and LCP computation for sequence collections with applications

BackgroundSequencing technologies produce larger and larger collections of biosequences that have to be stored in compressed indices supporting fast search operations. Many compressed indices are based on the Burrows–Wheeler Transform (BWT) and the longest common prefix (LCP) array. Because of the sheer size of the input it is important to build these data structures in external memory and time using in the best possible way the available RAM.ResultsWe propose a space-efficient algorithm to compute the BWT and LCP array for a collection of sequences in the external or semi-external memory setting. Our algorithm splits the input collection into subcollections sufficiently small that it can compute their BWT in RAM using an optimal linear time algorithm. Next, it merges the partial BWTs in external or semi-external memory and in the process it also computes the LCP values. Our algorithm can be modified to output two additional arrays that, combined with the BWT and LCP array, provide simple, scan-based, external memory algorithms for three well known problems in bioinformatics: the computation of maximal repeats, the all pairs suffix–prefix overlaps, and the construction of succinct de Bruijn graphs.ConclusionsWe prove that our algorithm performs {mathcal {O}}(n, mathsf {maxlcp}) sequential I/Os, where n is the total length of the collection and mathsf {maxlcp} is the maximum LCP value. The experimental results show that our algorithm is only slightly slower than the state of the art for short sequences but it is up to 40 times faster for longer sequences or when the available RAM is at least equal to the size of the input.

Building an Optimal Point-Location Structure in $$O( sort (n))$$ O ( s o r t ( n ) ) I/Os

We revisit the problem of constructing an external memory data structure on a planar subdivision formed by n segments to answer point location queries optimally in $$O(\log _B n)$$ I/Os. The objective is to achieve the I/O cost of $$ sort (n) = O(\frac{n}{B} \log _{M/B} \frac{n}{B})$$ , where B is the number of words in a disk block, and M being the number of words in memory. The previous algorithms are able to achieve this either in expectation or under the tall cache assumption of $$M \ge B^2$$ . We present the first algorithm that solves the problem deterministically for all values of M and B satisfying $$M \ge 2B$$ .

Hybrid Lighting for faster rendering of scenes with many lights

There is growing interest in rendering scenes with many lights, where scenes typically contain hundreds to thousands of lights. Each light illuminates geometry within a finite extent called a light volume. A key aspect of performance is determining which lights apply to what geometry, and then applying those lights efficiently. We present a GPU-based approach using spatial data structures, binning lights by depth analytically while also taking advantage of hardware rasterization. This improves light binning performance by 3–6 $$\times $$ . We also present a GPU memory and cache friendly data structure that takes two passes to build, giving 4–10 $$\times $$ improved performance when applying lighting and an overall improvement of 1.3–4 $$\times $$ for total frametime.

A sound abstract memory model for static analysis of C programs

Abstract memory model plays an important role in the static analysis of programs. This paper proposes a region-based symbolic three-valued logic (RSTVL) to guarantee the soundness of static analysis, which utilises abstract regions to simulate blocks of the concrete memory. RSTVL applies symbolic expressions to express the value of memory objects, and the interval domain to describe the value of each symbol of symbolic expressions. Various operations for memory objects can be mapped to operations about regions. As a sound abstract memory model, RSTVL can describe the shape information of data structure in memory and the storage state of memory objects for C programs, and a variety of associative addressable expressions, including the point-to relations, hierarchical and valued logic relations. We have built a prototype tool DTSC_RSTVL that detects code level defects in C programs. Five popular C programs are analysed, the results indicate that the analysis is sufficiently sound to detect code level defects with zero false negative rate.

A Loop-Based Methodology for Reducing Computational Redundancy in Workload Sets

The design of general purpose processors relies heavily on a workload gathering step in which representative programs are collected from various application domains. Processor performance, when running the workload set, is profiled using simulators that model the targeted processor architecture. However, simulating the entire workload set is prohibitively time-consuming, which precludes considering a large number of programs. To reduce simulation time, several techniques in the literature have exploited the internal program repetitiveness to extract and execute only representative code segments. Existing solutions are based on reducing cross-program computational redundancy or on eliminating internal-program redundancy to decrease execution time. In this paper, we propose an orthogonal and complementary loop-centric methodology that targets loop-dominant programs by exploiting internal-program characteristics to reduce cross-program computational redundancy. The approach employs a newly developed framework that extracts and analyzes core loops within workloads. The collected characteristics model memory behavior, computational complexity, and data structures of a program, and are used to construct a signature vector for each program. From these vectors, cross-workload similarity metrics are extracted, which are processed by a novel heuristic to exclude similar programs and reduce redundancy within the set. Finally, a reverse engineering approach that synthesizes executable micro-benchmarks having the same instruction mix as the loops in the original workload is introduced. A tool that automates the flow steps of the proposed methodology is developed. Simulation results demonstrate that applying the proposed methodology to a set of workloads reduces the set size by half, while preserving the main characterizations of the initial workloads.

Hiding the Long Latency of Persist Barriers Using Speculative Execution

Byte-addressable non-volatile memory technology is emerging as an alternative for DRAM for main memory. This new Non-Volatile Main Memory (NVMM) allows programmers to store important data in data structures in memory instead of serializing it to the file system, thereby providing a substantial performance boost. However, modern systems reorder memory operations and utilize volatile caches for better performance, making it difficult to ensure a consistent state in NVMM. Intel recently announced a new set of persistence instructions, clflushopt, clwb, and pcommit. These new instructions make it possible to implement fail-safe code on NVMM, but few workloads have been written or characterized using these new instructions. In this work, we describe how these instructions work and how they can be used to implement write-ahead logging based transactions. We implement several common data structures and kernels and evaluate the performance overhead incurred over traditional non-persistent implementations. In particular, we find that persistence instructions occur in clusters along with expensive fence operations, they have long latency, and they add a significant execution time overhead, on average by 20.3% over code with logging but without fence instructions to order persists. To deal with this overhead and alleviate the performance bottleneck, we propose to speculate past long latency persistency operations using checkpoint-based processing. Our speculative persistence architecture reduces the execution time overheads to only 3.6%.

CONSTRUCTIVE MODEL OF ADAPTATION OF DATA STRUCTURES IN RAM. PART II. CONSTRUCTORS OF SCENARIOS AND ADAPTATION PROCESSES

Purpose.The second part of the paper completes presentation of constructive and the productive structures (CPS), modeling adaptation of data structures in memory (RAM). The purpose of the second part in the research is to develop a model of process of adaptation data in a RAM functioning in different hardware and software environments and scenarios of data processing. Methodology. The methodology of mathematical and algorithmic constructionism was applied. In this part of the paper, changes were developed the constructors of scenarios and adaptation processes based on a generalized CPS through its transformational conversions. Constructors are interpreted, specialized CPS. Were highlighted the terminal alphabets of the constructor scenarios in the form of data processing algorithms and the constructor of adaptation – in the form of algorithmic components of the adaptation process. The methodology involves the development of substitution rules that determine the output process of the relevant structures. Findings. In the second part of the paper, system is represented by CPS modeling adaptation data placement in the RAM, namely, constructors of scenarios and of adaptation processes. The result of the implementation of constructor of scenarios is a set of data processing operations in the form of text in the language of programming C#, constructor of the adaptation processes – a process of adaptation, and the result the process of adaptation – the adapted binary code of processing data structures. Originality. For the first time proposed the constructive model of data processing – the scenario that takes into account the order and number of calls to the various elements of data structures and adaptation of data structures to the different hardware and software environments. At the same the placement of data in RAM and processing algorithms are adapted. Constructionism application in modeling allows to link data models and algorithms for their processing with the performance criteria in the framework of unified approach and applied means. The developed models allow us to study the process of adaptation and control it. Practical value. The developed model and methods allow automatically changing the data placement in the RAM and their algorithmic connection depending on the operational requirements, the design features of the hardware and software operating environment.

Fast, multicore-scalable, low-fragmentation memory allocation through large virtual memory and global data structures

We demonstrate that general-purpose memory allocation involving many threads on many cores can be done with high performance, multicore scalability, and low memory consumption. For this purpose, we have designed and implemented scalloc, a concurrent allocator that generally performs and scales in our experiments better than other allocators while using less memory, and is still competitive otherwise. The main ideas behind the design of scalloc are: uniform treatment of small and big objects through so-called virtual spans, efficiently and effectively reclaiming free memory through fast and scalable global data structures, and constant-time (modulo synchronization) allocation and deallocation operations that trade off memory reuse and spatial locality without being subject to false sharing.

LiquidHaskell

Haskell has many delightful features. Perhaps the one most beloved by its users is its type system that allows developers to specify and verify a variety of program properties at compile time. However, many properties, typically those that depend on relationships between program values are impossible, or at the very least, cumbersome to encode within the existing type system. Many such properties can be verified using a combination of Refinement Types and external SMT solvers. We describe the refinement type checker liquidHaskell, which we have used to specify and verify a variety of properties of over 10,000 lines of Haskell code from various popular libraries, including containers, hscolour, bytestring, text, vector-algorithms and xmonad. First, we present a high-level overview of liquidHaskell, through a tour of its features. Second, we present a qualitative discussion of the kinds of properties that can be checked -- ranging from generic application independent criteria like totality and termination, to application specific concerns like memory safety and data structure correctness invariants. Finally, we present a quantitative evaluation of the approach, with a view towards measuring the efficiency and programmer effort required for verification, and discuss the limitations of the approach.