Ensure Data Consistency Research Articles

WOJ

Journaling is a commonly used technique to ensure data consistency in file systems, such as ext3 and ext4. With journaling technique, file system updates are first recorded in a journal (in the commit phase) and later applied to their home locations in the file system (in the checkpoint phase). Based on the contents recorded in the journal, file sytem can be either in data or metadata journaling mode. With data journaling mode enabled, all file system (data and metadata) updates are written to the journal before being written to the files later on. In contrast, with metadata journaling mode, only updated metadata are written to and protected by the journal, while data are written directly to their home locations in the files. File system users are usually reluctant to use the data journaling mode as every modification (data and metadata) to the file system is written twice, and instead resort to metadata journaling for its fast speed.

Avoiding Data Corruption in Drop Computing Mobile Networks

Drop computing is a network paradigm that aims to address the issues of the mobile cloud computing model, which has started to show limitations especially since the advent of the Internet of Things and the increase in the number of connected devices. In drop computing, nodes are able to offload data and computations to the cloud, to edge devices, or to the social-based opportunistic network composed of other nodes located nearby. In this paper, we focus on the lowest layer of drop computing, where mobile nodes offload tasks and data to and from each other through close-range protocols, based on their social connections. In such a scenario, where the data can circulate in the mobile network on multiple paths (and through multiple other devices), consistency issues may appear due to data corruption or malicious intent. Since there is no central entity that can control the way information is spread and its correctness, alternative methods need to be employed. In this paper, we propose several mechanisms for ensuring data consistency in drop computing, ranging from a rating system to careful analysis of the data received. Through thorough experimentation, we show that our proposed solution is able to maximize the amount of correct (i.e., uncorrupted) data exchanged in the network, with percentages as high as 100%.

The Effect of Duration of the Rivet Process on the Strength of Single Lap Joining

Good airplane joining supports aircraft strength. The general joining process is using rivet as joining method. The parameter that affects the strength during the riveting process is the duration of the rivet head formation. The purpose was to determine the effect of the duration of rivet head formation on the quality of the joint. The research was carried out with the help of homemade tools, to ensure data consistency. The selected riveting duration is 3, 4, 5, 6 seconds. After connection, each connection was tested for strength through tensile testing and macro photo images from the joining. Results show that the 5-second duration provides the best connection with the tensile test results of 165 MPa. Based on the macro photo image, the 5 second rivet process produces the best connection because it produces the biggest rivet body so that it provides the greatest strength compared to the others.

Design and Development of a Technology Platform for DNA-Encoded Library Production and Affinity Selection

DNA-encoded libraries (DELs) have emerged as an efficient and cost-effective drug discovery tool for the exploration and screening of very large chemical space using small-molecule collections of unprecedented size. Herein, we report an integrated automation and informatics system designed to enhance the quality, efficiency, and throughput of the production and affinity selection of these libraries. The platform is governed by software developed according to a database-centric architecture to ensure data consistency, integrity, and availability. Through its versatile protocol management functionalities, this application captures the wide diversity of experimental processes involved with DEL technology, keeps track of working protocols in the database, and uses them to command robotic liquid handlers for the synthesis of libraries. This approach provides full traceability of building-blocks and DNA tags in each split-and-pool cycle. Affinity selection experiments and high-throughput sequencing reads are also captured in the database, and the results are automatically deconvoluted and visualized in customizable representations. Researchers can compare results of different experiments and use machine learning methods to discover patterns in data. As of this writing, the platform has been validated through the generation and affinity selection of various libraries, and it has become the cornerstone of the DEL production effort at Lilly.

UMFS: An efficient user-space file system for non-volatile memory

Abstract Emerging non-volatile memory (NVM) is expected to be a mainstream storage media in embedded systems for its low-power consumption, near-DRAM speed, high density, and byte-addressability. In-memory file systems are proposed to achieve high-performance file accesses by storing files in NVM. Existing in-memory file systems, such as NOVA and EXT4-DAX, operate in kernel space and have additional overhead caused by kernel layers and mode change. In this paper, we propose a new design of User-space in-Memory File System (UMFS) to improve file access speed by minimizing the overhead of kernel. We implement UMFS in Linux system to verify the proposed design. In open operation, UMFS exposes a file into user-space in constant time independent from the file size. Then, UMFS can achieve high-performance file accesses taking advantages of user virtual address space and existing address translation hardware in processors. We also propose an efficient user-space journaling to ensure data consistency while minimizing kernel cost. Extensive experiments are conducted on standard benchmarks to compare UMFS with NOVA, EXT4-DAX, and SIMFS, the state-of-the-art in-memory file system. The experimental results show that UMFS outperforms any of existing in-memory file systems.

Integration and formal representation in civil engineering supervision based on data-driven

In the civil engineering supervision, the management processes about personnel, materials, quality, safety, schedule and etc. are complex, which include large amount of data, involve more participants, and timely coordination and feedback is difficult. In this paper, we propose the data integration idea based on data-driven, which could guide the whole life cycle of civil engineering supervision from the top-level data organization, supervision of business processes, the interactive units and the users, the four-elements method TDTM and so on. Then, the data integration algorithms of civil engineering supervision are put forward to integrate the civil engineering supervision data from outside to inside and from coarse-grained to fine-grained, eliminate contradictions and redundancy and ensure data consistency. Finally, the civil engineering supervision data entities are determined, and we further analyze and discuss its query and report, data maintenance and conversion, and compare the functions in the different supervision platform. The civil engineering supervision unified data platform proposed could maintain the independence of the data, have the good scalability and support the more functions.

Automated production data integration for energy-oriented process chain design

Energy efficiency is an important aspect in automotive industry. While energy demand occurs during the operation phase of a production system, it is determined during the planning process. To enable a forecast of energy consumption during process chain design and a demand specific dimensioning of machine components, a modular framework consisting of specialized planning tools and an integrated database is developed. A seamless information flow based on OPC UA and AutomationML connects shop floor and planning environment and ensures data consistency. A database integrates available data sources and condenses acquired raw data to relevant reference data. The concept is prototypically implemented in a crank shaft production line.

Comparison of Event Choreography and Orchestration Techniques in Microservice Architecture

Microservice Architecture (MSA) is an architectural design pattern which was introduced to solve the challenges involved in achieving the horizontal scalability, high availability, modularity and infrastructure agility for the traditional monolithic applications. Though MSA comes with a large set of benefits, it is challenging to design isolated services using independent Database per Service pattern. We observed that with each micro service having its own database, when transactions span across multiple services, it becomes challenging to ensure data consistency across databases, particularly in case of roll backs. In case of monolithic applications using RDBMS databases, these distributed transactions and roll backs can be handled efficiently using 2 phase commit techniques. These techniques cannot be applied for isolated No-SQL databases in micro services. This research paper aims to address three things: 1) elucidate the challenges with distributed transactions and rollbacks in isolated No-SQL databases with dependent collections in MSA, 2) examine the application of event choreography and orchestration techniques for the Saga pattern implementation, and 3) present the fact-based recommendations on the saga pattern implementations for the use cases.

Testing of transactional services in NoSQL key-value databases

Transactional services guarantee the consistency of shared data during the concurrent execution of multiple applications. They have been used in various domains ranging from classical databases through to service-oriented computing systems to NoSQL databases and cloud. Though transactional services aim to ensure data consistency, NoSQL databases prioritize efficiency/availability over data consistency. In order to address these issues various transaction models and protocols have been proposed in the literature. However, testing of transactions in NoSQL database has not been addressed. In this paper, we investigate into the testing of transactional services in NoSQL databases in order to test and analyse the data consistency by taking into account the characteristics of NoSQL databases such as efficiency, velocity, etc. Accordingly, we develop a framework for testing transactional services in NoSQL databases. The novelty and contributions are that we develop a context-aware transactional model that takes into account contextual requirements of NoSQL clients and the system level setting in relation to the data consistency. This can assist NoSQL application developers in choosing between transactional and non-transactional services based on their requirements of the level of data consistency. The framework also provides ways to analyse the impact of the big data requirements and characteristics (e.g., velocity, efficiency) on the data consistency of NoSQL databases. The evaluation and testing are carried out using a widely used NoSQL key/value database, Riak, and a real (open) and big data from the Council of London for public transportation of the London bus services.

Data Dissemination and Parallel Processing Techniques Research Based on Massively Parallel Processing

In the current distributed database system architecture enterprise-class, the massively parallel processing architecture is used frequently. This method can be used to carry out large-scale analysis of data through distributed across multiple nodes and storage and query process, from its scope of application produce simple reports to perform complex analytics workloads. However, due to the characteristics of shared-nothing MPP technology, to carry out large-scale data analysis query and maintain data consistency there are some difficulties. In this paper, a relational SQL-based query parsing distributed MPP data distribution and parallel processing technology, the goal is to maintain and improve the consistency of distributed data query speed. First SQL query analysis section, according to the syntax analysis, semantic analysis and sentence parsing steps such order; in the form of work distribution node/data node in the data distribution phase, all tasks emanating from the work of a distribution node, all need to treated results are returned to the node; when parallel processing, each node needs to store a copy of the lookup table, and on each node concurrent execution of SQL statements for each query. Experimental results show that the proposed MPP data distribution and parallel processing scheme can support large volume of data processing, ensuring data consistency in the premise of improving query processing speed.

Research and Application of Satellite Integration Design Based on MBD

As satellite integration design and implementation become more and more complex and difficult, a MBD model structure suit for satellite integration design is proposed in this paper, to ensure data consistency and the design work more efficiency. In order to help designers build normative MBD model efficient, a rapid MBD modeling tool has been developed and already been validated in several satellite development projects.

Getting the Data Flowing: Lessons Learned from Existing Reporting Systems in the Forestry Sector in Indonesia for REDD+ MRV.

In the context of REDD+, Measurement, Reporting and Verification (MRV) is one way to manage forest change information. A national carbon and non-carbon database will be used in REDD+ to negotiate compensation schemes with the international community. Much of this data will be collected at the local level, thus a reporting system that can integrate these locally collected data into the national database is crucial. In this paper we compare and draw lessons from three existing local to national reporting systems that include the participation of local communities: 1) the government extension services, 2) the government owned forestry company, and 3) a private logging company in Indonesia, and provide recommendations for REDD+ reporting systems. The results suggest that the main desired conditions for effective data flow are: benefits to motivate local participation, based on contributions to reporting activities; simple data format and reporting procedures to allow local participation in the reporting process, and to support data aggregation at the national level; a facilitator to mediate data aggregation at the village level to ensure data consistency, completeness and accuracy; and a transparent and clear data flow. Under these conditions, continuous, accountable and consistent data flow from the local level will reach the national level where it can be fully utilized.

Cancer incidence in Poland – an approach toward ensuring data consistency

Cancer incidence in Poland – an approach toward ensuring data consistency

Harmonised high throughput microsomal stability assay

IntroductionPrediction of human pharmacokinetics from in vitro assays and pre-clinical data is an integral part of drug discovery. In vitro stability metabolic studies can provide an estimate of in vivo hepatic intrinsic clearance through inclusion of biological scaling factors. Many labs have personalised stability protocols including marker compounds and have adopted QC criteria and assay limits to ensure data integrity. Contract research organisations (CRO's) provide integrated drug discovery support to academic and pharmaceutical/biotechnology institutions to progress their in-house projects. The majority of these clients have established in-house protocols with associated criteria to ensure data consistency between in-house and external labs. MethodsIn this study, numerous assay variables were condensed into one harmonised assay format and a range of compounds with diverse physicochemical properties were evaluated. The protocols were diverse with respect to the following attributes: buffer, microsomal concentration and species strain. ResultsComparison of human lots in vitro CLint between the traditional and consolidated assay formats showed a good correlation with no significant difference. A clear relationship was demonstrated between strains. Interpretation of in vitro intrinsic clearance between the strains for each species was consistent. Using strict classes of in vitro hepatic intrinsic clearance values (<50, 50–100, >150μL/min/mg protein) comparisons across different conditions such as, assay variables, human lots, mouse and rat strains showed >80% agreement. DiscussionA high throughput assay was developed that enables the simultaneous measurement of CLint using mouse, rat and human hepatic microsomes (consolidated assay). By condensing all possible variables into one assay format, consistent data were obtained during head to head tests.

Data Virtualization and Digital Agility

Organizations must increasingly deal with never-ending, complex, and costly requirements for using all of their data sources to produce value. Big data must be integrated with small data, but trying to bring varying data sources together without a set of structures and processes supported by the right technology leads to the proverbial "data swamp." One technique is to establish a master data management (MDM) program to coordinate different data sources, which goes hand in hand with data virtualization to ensure data consistency and accessibility.

Using Linked Data with Information Standards for Interoperability in Production Engineering

Design and verification of factory layout and material flow is a multidisciplinary, knowledge-intensive task which requires a collaborative framework where all specialists involved can communicate, interact, manage and visualize different models. However, the communication of digital models comes with challenges. The information resides in various systems and applications, in different formats and with various levels of detail and viewpoints. This makes the communication and sharing of information among different actors and application, challenging.To deal with the data exchange and integration problem, the information standards ISO 10303 (STEP) has shown a strong capability to represent rich information models in a wide variety of industrial domains for the purpose of exchanging data.On the other hand, the Open Services for Lifecycle Collaboration (OSLC) initiative provides a minimalistic set of standardized information models, focusing on the most common concepts within a particular domain. Assuming a loosely-coupled distributed architecture of tools and services, OSLC adopts the Linked Data (LD) approach to ensure data consistency across the data resources.How can we combine STEP's rich information model for data exchange, with OSLC's minimalistic approach for data integration The aim of this work is to show the applicability of using these two complementary paradigms – and their corresponding standards to support interoperability and data integration in a heterogeneous IT environment for material flow analysis and layout design.

Integration of Digital Factory with Smart Factory Based on Internet of Things

Internet of things (IoT) in manufacturing can be defined as a future where every day physical objects in the shop floor, people and systems (things) are connected by the Internet to build services critical to the manufacturing. Smart factory is a way towards a factory-of-things, which is very much aligned with IoT. IoT not only deals with smart connections between physical objects but also with the interaction with different IT tools used within the digital factory. Data and information come from heterogeneous IT systems and from different domains, viewpoints, levels of granularity and life cycle phases causing potential inconsistencies in the data sharing, preventing interoperability. Hence, our aim is to investigate approaches and principles when integrating the digital factory, IT tools and IoT in manufacturing in a heterogeneous IT environment to ensure data consistency. In particular this paper suggests an approach to identify what, when and how information should be integrated. Secondly it suggests integration between IoT and PLM platforms using semantic web technologies and Open Services for Lifecycle Collaboration (OSLC) standard on tool interoperability.

The Stream‐Catchment (StreamCat) Dataset: A Database of Watershed Metrics for the Conterminous United States

AbstractWe developed an extensive database of landscape metrics for ~2.65 million stream segments, and their associated catchments, within the conterminous United States (U.S.): The Stream‐Catchment (StreamCat) Dataset. These data are publically available (http://www2.epa.gov/national-aquatic-resource-surveys/streamcat) and greatly reduce the specialized geospatial expertise needed by researchers and managers to acquire landscape information for both catchments (i.e., the nearby landscape flowing directly into streams) and full upstream watersheds of specific stream reaches. When combined with an existing geospatial framework of the Nation's rivers and streams (National Hydrography Dataset Plus Version 2), the distribution of catchment and watershed characteristics can be visualized for the conterminous U.S. In this article, we document the development and main features of this dataset, including the suite of landscape features that were used to develop the data, scripts and algorithms used to accumulate and produce watershed summaries of landscape features, and the quality assurance procedures used to ensure data consistency. The StreamCat Dataset provides an important tool for stream researchers and managers to understand and characterize the Nation's rivers and streams.

A new concurrency control mechanism for multi-threaded environment using transactional memory

Software transactional memory (STM) is one of the techniques used towards achieving non-blocking process synchronization in multi-threaded computing environment. In spite of its high potential, one of the major limitations of transactional memory (TM) is that in order to ensure data consistency as well as progress condition, TM often forces transactions to abort. This paper proposes a new concurrency control mechanism. It starts with the existing TM implementations for obstruction freedom and eventually builds a new STM methodology. The primary objective is to reduce aborting of transactions in some typical scenarios. A programming model is described for a chain of update transactions that share the same data object among themselves. Using the proposed approach, any new update transaction appended in this chain need not wait for the earlier transactions to finish. The proposed STM allows wait-free, non-blocking implementation of a mix of read and multiple update transactions on the same shared data object with higher throughput.

Adaptive Replica Synchronization for Distributed File Systems

This paper presents an adaptive replica synchronization mechanism among storage servers (SSs) without the interference from the metadata server (MDS) in a distributed file system. This mechanism employs a chunk list data structure, which holds the information about the relevant chunk replicas and is stored on the associated SSs corresponding to the replicas. Combined with version-based update replay technique, our proposed mechanism can ensure data consistency from the viewpoint of clients and help the SSs to conduct replica synchronization proactively with a per-chunk granularity. In contrast to conventional approaches of replica synchronization that are reliant on the MDS to trigger the synchronization, our proposed mechanism can side step the cumbersome centralized synchronization process to accomplish the same synchronization job. Moreover, a deferred update approach for replica synchronization has been adopted by the newly proposed mechanism for contributing to I/O data rate in write-intensive workloads. The evaluation experiments show that this newly presented mechanism can achieve attractive I/O performance enhancement with less synchronization overhead in specific application contexts.