Internal Data Structures Research Articles

AFEIC: Advanced forensic Ext4 inode carving

In forensic computing, especially in the field of postmortem file system forensics, the reconstruction of lost or deleted files plays a major role. The techniques that can be applied to this end strongly depend on the specifics of the file system in question. Various file systems are already well-investigated, such as FAT16/32, NTFS for Microsoft Windows systems and Ext2/3 as the most relevant file system for Linux systems. There also exist tools, such as the famous Sleuthkit (Carrier), that provide file recovery features for those file systems by interpreting the file system internal data structures. In case of an Ext file system, the interpretation of the so-called superblock is essential to interpret the data. The Ext4 file system can mainly be analyzed with the tools and techniques that have been developed for its predecessor Ext3, because most principles and internal structures remained unchanged. However, a few innovations have been implemented that have to be considered for file recovery. In this paper, we investigate those changes with respect to forensic file recovery and develop a novel approach to identify files in an Ext4 file system even in cases where the superblock is corrupted or overwritten, e.g. because of a re-formatting of the volume. Our approach applies heuristic search patterns for utilizing methods of file carving and combines them with metadata analysis. We implemented our approach as a proof of concept and integrated it into the Sleuthkit framework.

TCASM: An asynchronous shared memory interface for high-performance application composition

This paper addresses the growing need for mechanisms supporting intra-node application composition in high-performance computing (HPC) systems. It provides a novel shared memory interface that allows composite applications, two or more coupled applications, to share internal data structures without blocking. This allows independent progress of the applications such that they can proceed in a parallel, overlapped fashion. Composite applications using in-node shared memory can reduce the amount of data to be communicated between nodes, allowing checkpointing and data reduction or analytics to be performed locally and in parallel. The approach is implemented in Linux, and evaluated using benchmarks that represent typical composite applications on a large HPC testbed. The results show that the proposed approach significantly outperforms the traditional ones (up to a 15-fold speed increase on a 200 node machine).

Checkmate: Fast Argument Checks for Defensive R Programming

Dynamically typed programming languages like R allow programmers to write generic, flexible and concise code and to interact with the language using an interactive Read-eval-print-loop (REPL). However, this flexibility has its price: As the R interpreter has no information about the expected variable type, many base functions automatically convert the input instead of raising an exception. Unfortunately, this frequently leads to runtime errors deeper down the call stack which obfuscates the original problem and renders debugging challenging. Even worse, unwanted conversions can remain undetected and skew or invalidate the results of a statistical analysis. As a resort, assertions can be employed to detect unexpected input during runtime and to signal understandable and traceable errors. The package "checkmate" provides a plethora of functions to check the type and related properties of the most frequently used R objects and variable types. The package is mostly written in C to avoid any unnecessary performance overhead. Thus, the programmer can conveniently write concise, well-tested assertions which outperforms custom R code for many applications. Furthermore, checkmate simplifies writing unit tests using the framework "testthat" by extending it with plenty of additional expectation functions, and registered C routines are available for package developers to perform assertions on arbitrary SEXPs (internal data structure for R objects implemented as struct in C) in compiled code.

Cubrick

This paper describes the architecture and design of Cubrick, a distributed multidimensional in-memory DBMS suited for interactive analytics over highly dynamic datasets. Cubrick has a strictly multidimensional data model composed of cubes, dimensions and metrics, supporting sub-second OLAP operations such as slice and dice, roll-up and drill-down over terabytes of data. All data stored in Cubrick is range partitioned by every dimension and stored within containers called bricks in an unordered and sparse fashion, providing high data ingestion rates and indexed access through any combination of dimensions. In this paper, we describe details about Cubrick's internal data structures, distributed model, query execution engine and a few details about the current implementation. Finally, we present results from a thorough experimental evaluation that leveraged datasets and queries collected from a few internal Cubrick deployments at Facebook.

METHODIQA - Development of a Quality Assurance Methodology for Renewable Heat Systems Based on Intelligent Operational Monitoring

The continuous monitoring and analysis of renewable heat installations is the key for successful long-term plant operation. Since an ongoing detailed monitoring process requires high efforts regarding time and human resources, it is too costly and thus not feasible for smaller plants. The METHODIQA project aims at developing the technical and scientific methodology for a quality assurance system based on intelligent operational monitoring which is cost-effective, meets high quality standards and exhibits a high degree of automation. The core parts of the methodology comprise the data acquisition and data pre-processing, the analysis and diagnostics based on mathematical algorithms as well as an automated reporting and notification system. METHOTHIQA is not fully automated yet, since the required plant modeling and internal data structure are currently still under development. Nevertheless, the system is functional and a proof-of-concept could be achieved: a broad range of algorithms have already been implemented, and each algorithm is assigned to several graphical outputs (charts), aiming at a straightforward, intuitive interpretation of the algorithms’ results. Example algorithms include the calculation of basic key figures and several complex algorithms which e.g. calculate the collector field efficiency based on intervals with stationary operating conditions. First results generated from the METHODIQA system are promising and indicate the high potential of the methodology. This paper focuses on the methodic approach chosen in the data pre-processing and on the algorithm based analyses.

Cloning DRASiW systems via memory transfer

DRASiW is an extension of the WiSARD Weightless Neural Network (WNN) model with the capability of storing the frequencies of seen patterns during the training phase in an internal data structure called “mental image” (MI). Due to this capability, in a previous work it was demonstrated how to reversely process MIs in order to generate synthetic prototypes. Then, a training set composed of synthetic prototypes can be used to train new DRASiW systems (clones) with different architectures. In this paper we present a methodology to transfer memory between DRASiW systems, and we show how it is possible to generate clones of DRASiW systems with good classification capabilities within an acceptable loss of accuracy.

Multiprocessor Ada platform based on MaRTE OS and GNAT

This paper presents the implementation of a multiprocessor Ada platform based on MaRTE OS and GNAT. The paper describes the modifications required to adapt MaRTE OS to a multiprocessor architecture, namely the protection of the internal data structures against concurrent access and the management of several ready queues. We also describe the adaptation of the GNAT run-time library in order to be used on top of the multiprocessor services provided by MaRTE OS. The developed MaRTE/GNAT platform provides full Ada tasking functionality and the multiprocessor support defined in the Ravenscar profile. Two multiprocessor architectures are supported: the multiprocessor version of the XtratuM hypervisor for Intel x86 architecture and the Linux operating system (with MaRTE taking the role of a Pthreads multiprocessor library).

Parallel factor analysis for monitoring data from a grape harvest in Qualified Designation of Origin Rioja including spatial and temporal variability

The proper ripening of grapes is the key to obtain a high-quality wine. Ripening is a temporal process that is influenced, in addition to uncontrollable climate factors, by the spatial distribution of the vineyard and planted variety. It is a complex process that cannot be characterized by a single parameter; rather, it is a modification of the profile of the compounds of the grape. This paper analyzes the joint evolution of twelve physicochemical parameters determined in red grapes from four different varieties, in sixteen representative (in both geographical and edaphic point of view) plots belonging to the Qualified Designation of Origin (DOC) Rioja. Samples were collected in September 2009 during four consecutive weeks prior to harvest.The data are naturally structured in a three-dimensional cube of data (named tensor) with these profiles: sample (plot/variety), physicochemical, and temporal. For the first time, studies are made for jointly analyzing the effect of the spatial and temporal distribution on the physicochemical profile of the grape. A PARAFAC model shows an internal data structure that accurately reproduces the ripening process with its distinctive features according to plot/variety, evolution of the physicochemical parameter of grapes, and ripening time.PARAFAC analysis reveals the different grape maturity depending on the vineyard's situation (zone and altitude) and the grape variety (garnacha, tempranillo, graciano, mazuelo). The obtained information allows evaluating the attainable quality in the grape related to the potential of the zone and its variety.

ImgFS: a transparent cryptography for stored images using a filesystem in userspace

Real-time encryption and decryption of digital images stored on end-user devices is a challenging task due to the inherent features of the images. Traditional software encryption applications generally suffered from the expense of user convenience, performance efficiency, and the level of security provided. To overcome these limitations, the concept of transparent encryption has been proposed. This type of encryption mechanism can be implemented most efficiently with kernel file systems. However, this approach has some disadvantages since developing a new file system and attaching it in the kernel level requires a deep understanding of the kernel internal data structure. A filesystem in userspace (FUSE) can be used to bridge the gap. Nevertheless, current implementations of cryptographic FUSE-based file systems suffered from several weaknesses that make them less than ideal for deployment. This paper describes the design and implementation of ImgFS, a fully transparent cryptographic file system that resides on user space. ImgFS can provide a sophisticated way to access, manage, and monitor all encryption and key management operations for image files stored on the local disk without any interaction from the user. The development of ImgFS has managed to solve weaknesses that have been identified on cryptographic FUSE-based implementations. Experiments were carried out to measure the performance of ImgFS over image files’ read and write against the cryptographic service, and the results indicated that while ImgFS has managed to provide higher level of security and transparency, its performance was competitive with other established cryptographic FUSE-based schemes of high performance.

Towards a forensic-aware database solution: Using a secured database replication protocol and transaction management for digital investigations

Databases contain an enormous amount of structured data. While the use of forensic analysis on the file system level for creating (partial) timelines, recovering deleted data and revealing concealed activities is very popular and multiple forensic toolsets exist, the systematic analysis of database management systems has only recently begun. Databases contain a large amount of temporary data files and metadata which are used by internal mechanisms. These data structures are maintained in order to ensure transaction authenticity, to perform rollbacks, or to set back the database to a predefined earlier state in case of e.g. an inconsistent state or a hardware failure. However, these data structures are intended to be used by the internal system methods only and are in general not human-readable.In this work we present a novel approach for a forensic-aware database management system using transaction- and replication sources. We use these internal data structures as a vital baseline to reconstruct evidence during a forensic investigation. The overall benefit of our method is that no additional logs (such as administrator logs) are needed. Furthermore, our approach is invariant to retroactive malicious modifications by an attacker. This assures the authenticity of the evidence and strengthens the chain of custody. To evaluate our approach, we present a formal description, a prototype implementation in MySQL alongside and a comprehensive security evaluation with respect to the most relevant attack scenarios.

Non-volatile main memory management methods based on a file system.

There are upcoming non-volatile (NV) memory technologies that provide byte addressability and high performance. PCM, MRAM, and STT-RAM are such examples. Such NV memory can be used as storage because of its data persistency without power supply while it can be used as main memory because of its high performance that matches up with DRAM. There are a number of researches that investigated its uses for main memory and storage. They were, however, conducted independently. This paper presents the methods that enables the integration of the main memory and file system management for NV memory. Such integration makes NV memory simultaneously utilized as both main memory and storage. The presented methods use a file system as their basis for the NV memory management. We implemented the proposed methods in the Linux kernel, and performed the evaluation on the QEMU system emulator. The evaluation results show that 1) the proposed methods can perform comparably to the existing DRAM memory allocator and significantly better than the page swapping, 2) their performance is affected by the internal data structures of a file system, and 3) the data structures appropriate for traditional hard disk drives do not always work effectively for byte addressable NV memory. We also performed the evaluation of the effects caused by the longer access latency of NV memory by cycle-accurate full-system simulation. The results show that the effect on page allocation cost is limited if the increase of latency is moderate.

Integrated cluster analysis and artificial neural network modeling for steam-assisted gravity drainage performance prediction in heterogeneous reservoirs

Evaluation of steam-assisted gravity drainage (SAGD) performance that involves detailed compositional simulations is usually deterministic, cumbersome, expensive (manpower and time consuming), and not quite suitable for practical decision making and forecasting, particularly when dealing with high-dimensional data space consisting of large number of operational and geological parameters. Data-driven modeling techniques, which entail comprehensive data analysis and implementation of machine learning methods for system forecast, provide an attractive alternative.In this paper, artificial neural network (ANN) is employed to predict SAGD production in heterogeneous reservoirs, an important application that is lacking in existing literature. Numerical flow simulations are performed to construct a training data set consists of various attributes describing characteristics associated with reservoir heterogeneities and other relevant operating parameters. Empirical Arps decline parameters are tested successfully for parameterization of cumulative production profile and considered as outputs of the ANN models. Sensitivity studies on network configurations are also investigated. Principal components analysis (PCA) is performed to reduce the dimensionality of the input vector, improve prediction quality, and limit over-fitting. In a case study, reservoirs with distinct heterogeneity distributions are fed to the model. It is shown that robustness and accuracy of the prediction capability are greatly enhanced when cluster analysis are performed to identify internal data structures and groupings prior to ANN modeling. Both deterministic and fuzzy-based clustering techniques are compared, and separate ANN model is constructed for each cluster. The model is then tested using a validation data set (cases that have not been used during the training stage).The proposed approach can be integrated directly into most existing reservoir management routines. In addition, incorporating techniques for dimensionality reduction and clustering with ANN demonstrates the viability of this approach for analyzing large field data set. Given that quantitative ranking of operating areas, robust forecasting, and optimization of heavy oil recovery processes are major challenges faced by the industry, the proposed research highlights the significant potential of applying effective data-driven modeling approaches in analyzing other solvent-additive steam injection projects.

Research on the 840D NC Machine Tool Production Process Monitoring and Management

According to the communication principle of 840D numerical control (NC) system and internal data structure, the PLC program is developed to make 840D system state data transmit to the PC with the MPI channels; production status data real-time monitoring system based on 840D NC machine tools is developed, meanwhile, the monitoring and data management for NC machine tool production process status is realized.

'바다지도' 데이터 입력 모듈 설계 및 구현

본 논문은 '바다지도' 데이터를 이용한 지도 기반의 응용시스템을 위한 '바다지도' 데이터 입력 모듈의 설계 및 구현에 관한 것이다. '바다지도' 데이터 입력 모듈은 응용시스템에서 '바다지도' 데이터를 시스템의 내부 자료 구조 및 형식으로 쉽게 변경하여 사용할 수 있도록 일관된 형식으로 '바다지도' 데이터를 입력하여 제공한다. '바다지도' 데이터 입력 모듈의 설계는 '바다지도' 데이터 특성 분석을 바탕으로 하였다. 국제수로기구의 S-57 전자해도 표준의 데이터 형식과 '바다지도' 데이터의 데이터 형식을 비교 분석하고, 이를 바탕으로 전자해도 표준에 따른 전자해도 데이터의 디스플레이 표준인 S-52 표준의 적용이 가능하도록 '바다지도' 데이터를 위한 자료구조를 정의하였다. 그리고 배포형식의 '바다지도' 데이터 셋을 해도 응용시스템에서 이용할 수 있도록 '바다지도' 데이터를 파싱하여 객체를 중심으로 정의한 자료구조로 변환하는 입력 모듈을 설계하고 구현하였다. 또한 '바다지도' 데이터 입력 모듈의 테스트를 위해 '바다지도' 뷰어를 구현하여, 입력 모듈을 테스트하였다. This paper is about design and implementation of 'SeaMap' importer module which is for map-based application using 'SeaMap' data. 'SeaMap' data importer is a module that reads 'SeaMap' data in consistent form and offers using easily change the format and the internal data structure in the application. Design of data importer module is based on analyzing characteristic of 'SeaMap'. The comparative analysis between the data type of 'SeaMap' and standard S-57 Electronic Navigation Chart (ENC) of the International Hydrographic Organization (IHO), based on this, to be applicable of S-52 standards 'SeaMap' data is defined as a structure of data. The importer module is designed and converted to allow to use distribution type of 'SeaMap' data sets in map application, parsing 'SeaMap' data around the object defining transformation data structure. In addition, we implemented a 'SeaMap' data viewer in order to test our 'SeaMap' data importer module.

Interpreter for Open Architecture CNC System: A Conceptual Model

Conventional Computer Numerical Control (CNC) machines are operated by controllers. Each controller has software model inside known as interpreter. The function of interpreter is to translate input data model code into CNC understandable internal data structure. These conventional CNC controllers are found to be closed in nature and are dependent on vendor specifications. In order to overcome these problems open architecture control systems are introduced. This paper presents an interpreter for open architecture CNC control system. The developed interpreter takes current available Computer Aided Design (CAD)/ Computer Aided Manufacturing (CAM) systems generated International Standard Organization (ISO) 6983 file in text (.txt) format as input and extracts the position, feed rate, spindle etc data from code. Then finally it writes interpreted code in user defined .txt and EXtensible Markup Language (.xml) formats.

Weak Scalability Analysis of the Distributed-Memory Parallel MLFMA

Distributed-memory parallelization of the multilevel fast multipole algorithm (MLFMA) relies on the partitioning of the internal data structures of the MLFMA among the local memories of networked machines. For three existing data partitioning schemes (spatial, hybrid and hierarchical partitioning), the weak scalability, i.e., the asymptotic behavior for proportionally increasing problem size and number of parallel processes, is analyzed. It is demonstrated that none of these schemes are weakly scalable. A nontrivial change to the hierarchical scheme is proposed, yielding a parallel MLFMA that does exhibit weak scalability. It is shown that, even for modest problem sizes and a modest number of parallel processes, the memory requirements of the proposed scheme are already significantly lower, compared to existing schemes. Additionally, the proposed scheme is used to perform full-wave simulations of a canonical example, where the number of unknowns and CPU cores are proportionally increased up to more than 200 millions of unknowns and 1024 CPU cores. The time per matrix-vector multiplication for an increasing number of unknowns and CPU cores corresponds very well to the theoretical time complexity.

The BIM 'Families' Based Environmental Performance Design Research

As far as the core of internal data structure is concerned, the elements of Building Information Modeling (BIM) platform relate to each other as components of families, independent of their materiality, parametric dependencies, geometrical structure, or their physical connection to other elements. Considering a building as an organism, this research aims at developing an evaluation methodology for buildings, based on their environmental performance. The methodology to be established is multi-objective, uses different weights and values assigned to the parameters of the families members, and checks and balances the system characteristics in order to come up with a dynamic criteria meant for guiding in the overall design and building process. This paper establishes a solid set of methodological tools that can be used for assessing a buildings environmental performance, and develops independent researches for architectural designs, alongside incorporating the ecological thoughts and building.

NGS++: a library for rapid prototyping of epigenomics software tools

The development of computational tools to enable testing and analysis of high-throughput-sequencing data is essential to modern genomics research. However, although multiple frameworks have been developed to facilitate access to these tools, comparatively little effort has been made at implementing low-level programming libraries to increase the speed and ease of their development. We propose NGS++, a programming library in C++11 specialized in manipulating both next-generation sequencing (NGS) datasets and genomic information files. This library allows easy integration of new formats and rapid prototyping of new functionalities with a focus on the analysis of genomic regions and features. It offers a powerful, yet versatile and easily extensible interface to read, write and manipulate multiple genomic file formats. By standardizing the internal data structures and presenting a common interface to the data parser, NGS++ offers an effective framework for epigenomics tool development. NGS++ was written in C++ using the C++11 standard. It requires minimal efforts to build and is well-documented via a complete docXygen guide, online documentation and tutorials. Source code, tests, code examples and documentation are available via the website at http://www.ngsplusplus.ca and the github repository at https://github.com/NGS-lib/NGSplusplus. nicolas.gevry@usherbrooke.ca or arnaud.droit@crchuq.ulaval.ca.

No-Schema Algorithm for Extracting a Partial Model from an IFC Instance Model

AbstractThis paper proposes an algorithm for extracting a partial model from an Industry Foundation Classes (IFC) instance model without an IFC schema or a complete IFC model view definition (MVD). The methods developed in previous studies require either an IFC schema or MVD and software applications, such as an IFC model server or a building information modeling (BIM) authoring tool, to extract a partial IFC instance model. The algorithm proposed in this paper generates a partial model by recursively extracting IFC data instances in referential relations directly from an IFC instance model file, and it relies solely on the internal data structure of an IFC instance model, without an IFC schema or a MVD. The algorithm extracts physical and nonphysical data instances relevant to the user’s selection of building elements by recursively iterating through data instances based on the rules specified in the algorithm. A set of required building elements is not defined on the spot; rather, a set of building elem...

Acceleration of iterative Navier-Stokes solvers on graphics processing units

While new power-efficient computer architectures exhibit spectacular theoretical peak performance, they require specific conditions to operate efficiently, which makes porting complex algorithms a challenge. Here, we report results of the semi-implicit method for pressure linked equations (SIMPLE) and the pressure implicit with operator splitting (PISO) methods implemented on the graphics processing unit (GPU). We examine the advantages and disadvantages of the full porting over a partial acceleration of these algorithms run on unstructured meshes. We found that the full-port strategy requires adjusting the internal data structures to the new hardware and proposed a convenient format for storing internal data structures on GPUs. Our implementation is validated on standard steady and unsteady problems and its computational efficiency is checked by comparing its results and run times with those of some standard software (OpenFOAM) run on central processing unit (CPU). The results show that a server-class GPU outperforms a server-class dual-socket multi-core CPU system running essentially the same algorithm by up to a factor of 4.