High Data Availability Research Articles

SeqArray-a storage-efficient high-performance data format for WGS variant calls.

Whole-genome sequencing (WGS) data are being generated at an unprecedented rate. Analysis of WGS data requires a flexible data format to store the different types of DNA variation. Variant call format (VCF) is a general text-based format developed to store variant genotypes and their annotations. However, VCF files are large and data retrieval is relatively slow. Here we introduce a new WGS variant data format implemented in the R/Bioconductor package 'SeqArray' for storing variant calls in an array-oriented manner which provides the same capabilities as VCF, but with multiple high compression options and data access using high-performance parallel computing. Benchmarks using 1000 Genomes Phase 3 data show file sizes are 14.0 Gb (VCF), 12.3 Gb (BCF, binary VCF), 3.5 Gb (BGT) and 2.6 Gb (SeqArray) respectively. Reading genotypes in the SeqArray package are two to three times faster compared with the htslib C library using BCF files. For the allele frequency calculation, the implementation in the SeqArray package is over 5 times faster than PLINK v1.9 with VCF and BCF files, and over 16 times faster than vcftools. When used in conjunction with R/Bioconductor packages, the SeqArray package provides users a flexible, feature-rich, high-performance programming environment for analysis of WGS variant data. http://www.bioconductor.org/packages/SeqArray. zhengx@u.washington.edu. Supplementary data are available at Bioinformatics online.

Quantifying Interdependent Risks in Genomic Privacy

The rapid progress in human-genome sequencing is leading to a high availability of genomic data. These data is notoriously very sensitive and stable in time, and highly correlated among relatives. In this article, we study the implications of these familial correlations on kin genomic privacy. We formalize the problem and detail efficient reconstruction attacks based on graphical models and belief propagation. With our approach, an attacker can infer the genomes of the relatives of an individual whose genome or phenotype are observed by notably relying on Mendel’s Laws, statistical relationships between the genomic variants, and between the genome and the phenotype. We evaluate the effect of these dependencies on privacy with respect to the amount of observed variants and the relatives sharing them. We also study how the algorithmic performance evolves when we take these various relationships into account. Furthermore, to quantify the level of genomic privacy as a result of the proposed inference attack, we discuss possible definitions of genomic privacy metrics, and compare their values and evolution. Genomic data reveals Mendelian disorders and the likelihood of developing severe diseases, such as Alzheimer’s. We also introduce the quantification of health privacy , specifically, the measure of how well the predisposition to a disease is concealed from an attacker. We evaluate our approach on actual genomic data from a pedigree and show the threat extent by combining data gathered from a genome-sharing website as well as an online social network.

From Data to Information: Does the Internet Oversupply the Market with Data?

In this study I examine how the amount of publicly available data influences price discovery and information asymmetry. Increased accessibility of firm-specific data may speed price discovery and decrease information asymmetry in capital markets through decreased search and discovery costs. Too much data, however, may slow price discovery and increase information asymmetry because of increased friction arising from market participants being unable to promptly transform this overabundance of data into useful information. Using a novel measure of firm data availability, I examine the relation between it and price formation and information asymmetry around earnings announcements. I find that abnormally high data availability is negatively associated with the speed at which earnings information is impounded into price (i.e. slower price discovery) and positively associated with information asymmetry measures around earnings announcements, and these findings are driven by large firms. Conversely, traditional news coverage and investor search for information are associated with faster price discovery and less information asymmetry around earnings announcements (consistent with prior research). I also find evidence that the relation between data availability and price discovery is concave, suggesting that more data initially speeds price discovery but at some level begins to act as a market friction. Overall, my findings suggest that an overabundance of publicly available data “muddies the water” and impedes the impounding of information into stock prices.

Production and use of estimates for monitoring progress in the health sector: the case of Bangladesh

ABSTRACTBackground: In order to support the progress towards the post-2015 development agenda for the health sector, the importance of high-quality and timely estimates has become evident both globally and at the country level.Objective and Methods: Based on desk review, key informant interviews and expert panel discussions, the paper critically reviews health estimates from both the local (i.e. nationally generated information by the government and other agencies) and the global sources (which are mostly modeled or interpolated estimates developed by international organizations based on different sources of information), and assesses the country capacity and monitoring strategies to meet the increasing data demand in the coming years. Primarily, this paper provides a situation analysis of Bangladesh in terms of production and use of health estimates for monitoring progress towards the post-2015 development goals for the health sector.Results: The analysis reveals that Bangladesh is data rich, particularly from household surveys and health facility assessments. Practices of data utilization also exist, with wide acceptability of survey results for informing policy, programme review and course corrections. Despite high data availability from multiple sources, the country capacity for providing regular updates of major global health estimates/indicators remains low. Major challenges also include limited human resources, capacity to generate quality data and multiplicity of data sources, where discrepancy and lack of linkages among different data sources (local sources and between local and global estimates) present emerging challenges for interpretation of the resulting estimates.Conclusion: To fulfill the increased data requirement for the post-2015 era, Bangladesh needs to invest more in electronic data capture and routine health information systems. Streamlining of data sources, integration of parallel information systems into a common platform, and capacity building for data generation and analysis are recommended as priority actions for Bangladesh in the coming years. In addition to automation of routine health information systems, establishing an Indicator Reference Group for Bangladesh to analyze data; building country capacity in data quality assessment and triangulation; and feeding into global, inter-agency estimates for better reporting would address a number of mentioned challenges in the short- and long-run.

On the Dynamic Maintenance of Data Replicas based on Access Patterns in A Multi-Cloud Environment

Cloud computing provides services and infrastructures to enable end-users to access, modify and share massive geographically distributed data. There are increasing interests in developing data-intensive (big data) applications in this computing environment that need to access huge datasets. Accessing such data in an efficient way is deterred with factors such as dynamic changes in resource availability, provision of diverse service quality by different cloud providers. Data replication has already been proven to be an effective technique to overcome these challenges. Replication offers reduced response time in data access, higher data availability and improved system load balancing. Once the replicas are created in a multi-cloud environment, it is of utmost importance to continuously support maintenance of these replicas dynamically. This is to ensure that replicas are located in optimal data center locations to minimize replication cost and to meet specific user and system requirements. First, this paper proposes a novel approach to distributed placement of static replicas in appropriate data center locations. Secondly, motivated by the fact that a multi-cloud environment is highly dynamic, the paper presents a dynamic replica maintenance technique that re-allocates replicas to new data center locations upon significant performance degradation. The evaluation results demonstrate the effectiveness of the presented dynamic maintenance technique with static placement decisions in a multi-cloud environment

Toward a General Distributed Messaging Framework for Online Transaction Processing Applications

This paper presents a general distributed messaging framework for online transaction processing applications in e-business industry, and we name it as Metamorphosis (MetaQ). Specifically, this messaging framework has features of high availability, scalability, and high data throughput. The current implementation of MetaQ supports distributed XA transactions, asynchronous messaging, and multiple ways for storing message offsets. As a consequence, it is suited to the application contexts having a large quantity of messages, transaction-support, and real-time requirements. More important, another branch of MetaQ implementation, i.e., RocketMQ has been deployed and used in Taobao.com and Alipay.com. The real usages in both typical online transaction applications have proven that the nature of MetaQ can perform well for such big Internet applications.

A Reliable and Efficient Data Migration Method Based On GlusterFS

Big data storage and high speed data access have become the main performance bottleneck for many big data applications. The higher speed data access and lower cost for storage must be required than some applications having small-scale data. Distributed hierarchical storage provides a good storage way to speed data access and lower cost. But it a data migration method which you choose decide the performance of distributed hierarchical storage system because data migration occurs frequently in hierarchical storage systems. There are many data migration methods, which most of those cannot ensure data utterly integrity after data migration. In this paper, we invent a reliable and efficient data migration method to ensure the utterly integrity of migrated data by MD5 checksum and improve performance of data migration by the pipeline technology. Through adjusting parameters, we get the best performance of data migration by using pipeline in our storage system which is a hierarchical storage system based on GlusterFS.

Single Disk Recovery and Load Balancing Using Parity Declustering

In the recent scenario, capacity of the storage system increases significantly due to increasing demands. Data availability is a major concern in the context of storage systems. In order to achieve high data availability, a Maximum Distance Separable (MDS) code such as Row-Diagonal Parity (RDP) is used in RAID 6 storage systems. In order to ensure continuous availability of data, even a failure in a single disk cannot be tolerated and hence it must be recovered at the earliest. Existing RDOR-RDP, a modified scheme of RDP that achieves minimal disk I/O access and reduced recovery time when compared with conventional scheme but yet fails to achieve load balancing during the recovery process. In this work, Parity declustering based load balancing is proposed, which provides optimal recovery time when compared to RDOR-RDP.

Optimal Scheduling of Data-Intensive Applications in Cloud-Based Video Distribution Services

Cloud computing opens a new door for designing the next-generation video distribution platform. As video services move to the cloud, some related data-intensive applications, such as recommender system, automatic scoring mechanism, and prediction algorithm, should also be cloud-based. Since traditional cloud file systems like MapReduce/Hadoop exhibit cost disadvantages in data accessing, a Cache A Replica On Modification (CAROM) cloud file system is designed to achieve high data availability and low storage cost, which provides resiliency in cloud file systems with high efficiency. In this paper, we focus on moving data-intensive applications to the CAROM cloud file system, for optimizing access latencies while maintaining the benefit of low storage cost. To achieve this, we propose a novel scheduling mechanism as a lubricant between CAROM and data-intensive applications. Our scheme consists of three parts. First, tripartite graph is employed to describe the relationships among tasks, computation nodes, and data nodes. Second, we give a 1:1:1 framework based on the situation that the data of task have been stored in the cache, and introduce two variant frameworks 1:1:M and 1:N:M with the consideration of limitations of cache size and performance of task. Finally, a ${k}$ -list algorithm is proposed as an approximation algorithm, and its mathematical definitions and proofs are given in detail. We conduct simulations to evaluate our scheme and the results prove that the performance of our algorithm is significantly better than that of the general two-layer algorithm.

A HIGH AVAILABILITY CLUSTER-BASED REPLICA CONTROL PROTOCOL IN DATA GRID

Data replication is widely used to provide high data availability, and increase the performance of the distributed systems. Many replica control protocols have been proposed in distributed and grid environments that achieved both high performance and availability. However, the previously proposed protocols still require a bigger number of replicas for read and write operations which are not suitable for a large scale system such as data grid. In this paper, a new replica control protocol called Clusteringbased Hybrid (CBH) has been proposed for managing the data in grid environments. We analyzed the communication cost and data availability for the operations and compared CBH protocol with recently proposed replica control protocols called Dynamic Hybrid (DH) protocol and Diagonal Replication in 2D Mesh (DR2M) protocol. To evaluate CBH protocol, a simulation model was implemented using Java. Our results show that for the read operations, CBH protocol improves the performance of communication cost and data availability compared to the DH and DR2M protocols.

Dynamic replication to reduce access latency based on fuzzy logic system

In a distributed environment, limited available bandwidth resources lead a high data access latency. Replication is a popular method that can upgrade the access performance and increase the data availability. However, unreasonable replication would cause over-consumption of system resources and finally a further deterioration on data access latency. So, In this paper, a theoretical model of access latency optimization with replication is presented firstly, which complement the blank space, and then a well-designed dynamic replication strategy is proposed, which consists of three algorithms: replica selection algorithm, replica layout algorithm and replica replacement algorithm. Replica selection algorithm selects the optimal replica with a hierarchical time cost based on the derivation of the theoretical model. Replica layout algorithm selects the optimal node for placing the replica based on the spatio-temporal locality of data access. Replica replacement algorithm, in which the fuzzy logic system is introduced originally, deletes replica when the available storage space is insufficient. FLSDR is tested by OptorSim and experimental results show that FLSDR achieves better performance in comparison with other algorithms in terms of mean job execution time, computing resource usage, number of data scheduling between clusters and number of replicas.

Assessment of wind resources in two parts of Northeast Brazil with the use of numerical models

Assessment of wind resources in two parts of Northeast Brazil with the use of numerical models

Research of privacy preservation method based on data coloring

In the cloud computing environment, outsourcing service mode of data storage causes the security problem, the reliability of data cannot be guaranteed, and the privacy preservation problem has aroused wide concern. In order to solve the problem of inefficiency and high-complexity caused by traditional privacy preservation methods such as data encryption and access control technology, a privacy preservation method based on data coloring is proposed. The data coloring model is established and the coloring mechanism is adopted to deal with the sensitive data of numerical attributes, and the cloud model similarity measurement based on arithmetic average least-approximability is adopted to authenticate the ownership of privacy data. On the premise of high availability of data, the method strengthens the security of the privacy information. Then, the performance, validity and the parameter errors of the algorithm are quantitatively analyzed by the experiments using the UCI dataset. Under the same conditions of privacy preservation requirements, the proposed method can track privacy leakage efficiently and reduce privacy leakage risks. Compared with the k-anonymity approach, the proposed method enhances the computational time efficiency by 18.5%.

Mapping rice extent and cropping scheme in the Mekong Delta using Sentinel-1A data

ABSTRACTSynthetic aperture radar (SAR)-based multi-temporal backscatter analysis is a common approach that has been widely used for rice mapping. Co-polarized C-band microwave backscatter (HH or VV) and the co-polarization ratio (HH/VV) are the most commonly used data sets for rice mapping due to their high data availability while the utilization of cross-polarized backscatter (HV or VH) has received less attention. In this study, Sentinel 1A time series – acquired in the dual-polarized (VV/VH) Interferometric Wide (IW) swath mode during the spring growing season (October 2015 to March 2016) in the Mekong Delta – were used to analyse the relationship between the growing cycle of rice plants and the temporal variation of SAR backscatter at different polarizations. Results show that VH backscatter is more sensitive to rice growth than VV backscatter. Several vegetation phenological parameters including beginning date, heading date and the length of the growing season were extracted from the VH backscatter time series. A decision tree approach was applied to delineate rice-cultivated areas based on seasonal phenological parameters. The classification result was validated against a 2015 land use map. The overall classification accuracy is 87.2% (kappa coefficient – κ = 0.71). In addition, the SAR-derived rice area was compared against ground statistical data at the provincial level (coefficient of determination R2 = 0.98).

Tuning the Performance of Pt/HfO2/Ti/Pt ReRAM Devices Obtained from Plasma-Enhanced Atomic Layer Deposition for HfO2 Thin Films

The market of portable electronics and Green IT solutions demands for non-volatile memory circuits with high speed data access, high reliability, and low power consumption. Resistive switching random access memories (ReRAM) based on the valence change mechanism (VCM) represent a promising concept. VCM-type memristive devices are generally based on transition metal oxide thin films sandwiched between metal electrodes that differ in respect of work function and oxidation enthalpy. ReRAM cells built from insulating metal oxides with a high band gap, like e.g. TiO2, HfO2, Al2O3, typically require an electroforming step to enable resistive switching. Decisive for a stable device operation are a homogeneous microstructure and a controlled defect density in the oxide films of a few nm in thickness. Atomic layer deposition (ALD) can fulfill these requirements by its unique surface-reaction controlled self-limiting growth enabling a control of the film defect structure even for thicknesses in the nm-regime and for low growth temperatures. During the electroforming process, an electronically conducting filament is formed a significantly increased density of oxygen vacancies. The stable VCM-type resistive switching is related to drift/diffusion processes of oxygen vacancies (or oxygen ions, respectively) which depend on the oxidation/reduction enthalpies of the materials involved in the switching event. In this presentation we are going to discuss possibilities to tune the resistive switching performance of nanostructured ReRAM devices by means of a variation of metal oxide bilayer structures with respect to the layer thickness and the stack sequence. This study was performed on various bilayers based on combinations of TiO2/Al2O3 [1] and TiO2/HfO2 [2] with a total thickness < 10 nm integrated into nano-crossbar structures with a size of 100 x 100 nm2. Addressing the ReRAM device performance we discuss effects on the electroforming voltage, necessary SET- and RESET-voltages, achievable resistance ratios, switching stability and variability. In addition, changes in the resistive switching polarity, i.e. from bipolar to complementary-type, which have been observed for certain stacks, are discussed. This work was supported in part by the Deutsche Forschungsgemeinschaft (SFB917).

E2FS: an elastic storage system for cloud computing

In cloud storage, replication technologies are essential to fault tolerance and high availability of data. While achieving the goal of high availability, replication brings extra number of active servers to the storage system. Extra active servers mean extra power consumption and capital expenditure. Furthermore, the lack of classification of data makes replication scheme fixed at the very beginning. This paper proposes an elastic and efficient file storage called E2FS for big data applications. E2FS can dynamically scale in/out the storage system based on real-time demands of big data applications. We adopt a novel replication scheme based on data blocks, which provides a fine-grained maintenance of the data in the storage system. E2FS analyzes features of data and makes dynamic replication decision to balance the cost and performance of cloud storage. To evaluate the performance of proposed work, we implement a prototype of E2FS and compare it with HDFS. Our experiments show E2FS can outperform HDFS in elasticity while achieving guaranteed performance for big data applications.

Perancangan Sistem Cluster Server untuk Jaminan Ketersediaan Layanan Tinggi pada Lingkungan Virtual

Computer based services are now used in various fields, such as business, health, and education. The objective is to improve the performance of the company, institution, or organization. Services and data are stored in a server machine. Therefore, the server machine becomes an important thing that supports the availability of services, beside network infrastructure and electricity. The server machine may have hardware failure or software crash. The server machine can be down because of a power failure, human error, or disaster. The server machine sometimes must be turned off for upgrades or maintenance purposes. When a server is down, services running on the server will stop and important data can be lost. The objective of this research is to build a server cluster system that supports high availability services and data integrity over a virtual environment. The result of this research is a cluster server system that supports high availability services over the virtual environment and guarantee data integrity using server virtualization Proxmox VE 3.4 on two computer machines, FreeNAS x86 9.2.1.9 as a NAS server, two units of the Cisco Catalyst 2960 switch, and DRBD for data synchronization.

Improving remote sensing crop classification by argumentation-based conflict resolution in ensemble learning

The acquisition of data through remote sensing has become of great importance in precision agriculture, as it covers large geographical areas faster and cheaper than ground inspections. The challenge is to develop technical solutions that can benefit from both huge amounts of raw data extracted from satellite images, but also from the robust amount of knowledge refined during centuries of agricultural practice. Aiming to accurately classify crops from satellite images, we developed a hybrid intelligent system that can exploit both agricultural expert knowledge and machine learning algorithms. As the crop raw data is characterized by heterogeneity, we drive our attention to ensemble learners, while expert knowledge is encapsulated within a rule-based system. Vote-based methods for solving conflicts between ensemble’s base learners have difficulties in classifying exceptional cases correctly and also to give the rationale behind their decision. The conceptual research question is on conflict resolution in ensemble learning. To deal with debatable cases in ensemble learning and to increase transparency in such debatable decisions, our hypothesis is that argumentation could be more effective than voting-based methods. The main contribution is that voting system in ensemble learning is substituted by an argumentation-base conflict resolutor. Prospective decisions of base classifiers are presented to an argumentative system based on defeasible logic that performs dialectical reasoning on pros and cons against a classification decision. The system computes a recommendation considering both the rules extracted from base learners and the available expert knowledge. The investigated case study deals with crop classification into four classes: corn, soybean, cotton, and rice. The test site used for the experiment is an area of 20 square kilometers in the New Madrid County, southeast of the Missouri State, USA. The results show that our approach increases classification accuracy compared to the voting-based method for conflict resolution in an ensemble learner comprising of three base classifiers: a decision tree, a neural network, and a support vector machine algorithm. We also argue that combining ensemble learning and argumentation fits the decision patterns of human agents, who first collect various opinions and then perform dialectical reasoning on these opinions. We think that the people who can benefit from the conceptual instrumentation presented in this work are decision makers in domains characterized by high data availability, robust expert knowledge, and a need for justifying the rationale behind decisions.

Risk assessment of the Groningen geothermal potential: From seismic to reservoir uncertainty using a discrete parameter analysis

Geothermal exploitation is subject to several uncertainties, even in settings with high data availability, adding to project risk. Uncertainty can stem from the reservoir’s initial state, as well as from the geological and operational parameters. The interplay between these aspects entails irreducible risk prior to exploration drilling. Consequently it is difficult to construct an indicative qualitative and quantitative depiction of the most prominent facets (e.g. pressure, permeability). This paper shows the classification of known unknowns to risks, while also providing numerical results. Starting from seismic data and arriving at a reservoir model using a discrete parameter analysis we assess the risks and uncertainties of a geothermal project near the city of Groningen (NE Netherlands). By simulating all combinations of the considered parameters, their relative importance can be mapped out. Findings suggest that the unique regime of possible pressure depletion due to neighbouring gas production can highly impact the feasibility of the project. Results demonstrate how an in depth analysis at the exploration phase can direct future efforts towards the most significant elements. Although the numerical results are field specific, the methodology can be readily applied to different locations.

Adaptive data replication strategy in cloud computing for performance improvement

Cloud computing is becoming a very popular word in industry and is receiving a large amount of attention from the research community. Replica management is one of the most important issues in the cloud, which can offer fast data access time, high data availability and reliability. By keeping all replicas active, the replicas may enhance system task successful execution rate if the replicas and requests are reasonably distributed. However, appropriate replica placement in a large-scale, dynamically scalable and totally virtualized data centers is much more complicated. To provide cost-effective availability, minimize the response time of applications and make load balancing for cloud storage, a new replica placement is proposed. The replica placement is based on five important parameters: mean service time, failure probability, load variance, latency and storage usage. However, replication should be used wisely because the storage size of each site is limited. Thus, the site must keep only the important replicas.We also present a new replica replacement strategy based on the availability of the file, the last time the replica was requested, number of access, and size of replica. We evaluate our algorithm using the CloudSim simulator and find that it offers better performance in comparison with other algorithms in terms of mean response time, effective network usage, load balancing, replication frequency, and storage usage.