Hundreds Of Terabytes Research Articles

Robust Cell Detection for Large-Scale 3D Microscopy Using GPU-Accelerated Iterative Voting.

High-throughput imaging techniques, such as Knife-Edge Scanning Microscopy (KESM),are capable of acquiring three-dimensional whole-organ images at sub-micrometer resolution. These images are challenging to segment since they can exceed several terabytes (TB) in size, requiring extremely fast and fully automated algorithms. Staining techniques are limited to contrast agents that can be applied to large samples and imaged in a single pass. This requires maximizing the number of structures labeled in a single channel, resulting in images that are densely packed with spatial features. In this paper, we propose a three-dimensional approach for locating cells based on iterative voting. Due to the computational complexity of this algorithm, a highly efficient GPU implementation is required to make it practical on large data sets. The proposed algorithm has a limited number of input parameters and is highly parallel.

Improved Search of Large Transcriptomic Sequencing Databases Using Split Sequence Bloom Trees.

Enormous databases of short-read RNA-seq experiments such as the NIH Sequencing Read Archive are now available. These databases could answer many questions about condition-specific expression or population variation, and this resource is only going to grow over time. However, these collections remain difficult to use due to the inability to search for a particular expressed sequence. Although some progress has been made on this problem, it is still not feasible to search collections of hundreds of terabytes of short-read sequencing experiments. We introduce an indexing scheme called split sequence bloom trees (SSBTs) to support sequence-based querying of terabyte scale collections of thousands of short-read sequencing experiments. SSBT is an improvement over the sequence bloom tree (SBT) data structure for the same task. We apply SSBTs to the problem of finding conditions under which query transcripts are expressed. Our experiments are conducted on a set of 2652 publicly available RNA-seq experiments for the breast, blood, and brain tissues. We demonstrate that this SSBT index can be queried for a 1000 nt sequence in <4 minutes using a single thread and can be stored in just 39 GB, a fivefold improvement in search and storage costs compared with SBT.

Task scheduling and virtual resource optimising in Hadoop YARN-based cloud computing environment

Big data is being generated everywhere around us at all times by cameras, mobile devices, sensors, and software logs with large amount of data in units of hundreds of terabytes to petabytes. Therefore, to analyse these massive data, new skills, intensive applications and storage clusters are needed. Apache Hadoop is one of the most recently popular tools developed for big data processing. The main purpose in this paper is to analyse different scheduling algorithms that can help to achieve better performance, efficiency and reliability of Hadoop YARN environment. We describe some task schedulers which consider different levels of Hadoop such as first in first out (FIFO) scheduler, fair scheduler, delay scheduler, deadline constraint scheduler, dynamic priority scheduling, capacity scheduler, and we analyse the performance of these widely used Hadoop task schedulers based on the following elements: makespan; turnaround time; and throughput. To conclude this paper, the experimental results were given.

Towards Accurate Statistical Analysis of Security Margins: New Searching Strategies for Differential Attacks

In today's world of the internet, billions of computer systems are connected to one another in a global network. The internet provides an unsecured channel in which hundreds of terabytes of data is being transmitted daily. Computer and software systems rely on encryption algorithms such as block ciphers to ensure that sensitive data remains confidential and secure. However, adversaries can leverage the statistical behavior of underlying ciphers to recover encryption keys. Accurate evaluation of the security margins of these encryption algorithms remains to be a big challenge. In this paper, we tackle this issue by introducing several searching strategies based on differential cryptanalysis. By clustering differential paths, the searching algorithm derives more accurate distinguishers as compared to examining individual paths, which in turn provides a more accurate estimation of cipher security margins. We verify the effectiveness of this technique on ciphers with the generalized Feistel and SPN structures, whereby the best distinguishers for each of the investigated ciphers were obtained by discovering clusters with thousands of paths. With the KATAN block cipher family as a test case, we also show how to apply the searching algorithm alongside other cryptanalysis techniques such as the boomerang attack and related-key model to obtain the best cryptanalytic results. This also depicts the flexibility of the proposed searching scheme, which can be tailored to improve upon other differential attack variants. In short, the proposed searching strategy realizes an automated security evaluation tool with higher accuracy compared to previous techniques. In addition, it is applicable to a wide range of encryption schemes which makes it a flexible tool for both academic research and industrial purposes.

TDat: An Efficient Platform for Processing Petabyte-Scale Whole-Brain Volumetric Images.

Three-dimensional imaging of whole mammalian brains at single-neuron resolution has generated terabyte (TB)- and even petabyte (PB)-sized datasets. Due to their size, processing these massive image datasets can be hindered by the computer hardware and software typically found in biological laboratories. To fill this gap, we have developed an efficient platform named TDat, which adopts a novel data reformatting strategy by reading cuboid data and employing parallel computing. In data reformatting, TDat is more efficient than any other software. In data accessing, we adopted parallelization to fully explore the capability for data transmission in computers. We applied TDat in large-volume data rigid registration and neuron tracing in whole-brain data with single-neuron resolution, which has never been demonstrated in other studies. We also showed its compatibility with various computing platforms, image processing software and imaging systems.

Backup to the Future

What if something horrible happens in the future? Great Scott! Do we have a backup? Is a backup even enough? At the Kepler Asteroseismic Science Operations Center (KASOC) we have a goal that all data and information from Kepler and KASC is preserved for the future. The benchmark is that the data should be useful for, at least, the next 50 years. But how do we ensure that hundreds of terabytes of data are understandable or even readable in half a century?

A fully coupled hybrid computational aeroacoustics method on hierarchical Cartesian meshes

Abstract Hybrid computational fluid dynamics (CFD) – computational aeroacoustics (CAA) schemes are the standard method for aeroacoustics simulations. This approach requires the exchange of information between the CFD and the CAA step, which is usually accomplished by storing acoustic source data. This data exchange procedure, however, poses two problems when such hybrid methods are used for large-scale problems with O ( 10 9 ) degrees of freedom: On the one hand, the required disk space becomes large and reaches hundreds of terabytes for a single simulation. On the other hand, the parallel scalability of the overall numerical scheme is limited by the available I/O bandwidth, which typically peaks between 5,000 and 10,000 cores. To avoid these problems, a highly scalable direct-hybrid scheme is presented, in which both the flow and the acoustics simulations run simultaneously. That is, all data between the two solvers is transferred in-memory, avoiding the restrictions of the I/O subsystem. Both solvers operate on a joint hierarchical Cartesian grid, which enables efficient parallelization and dynamic load balancing and inherently supports local mesh refinement. To demonstrate the capabilities of the new scheme, the aeroacoustic field of a co-rotating vortex pair is computed. The results show that the direct-hybrid method is able to efficiently predict the acoustic pressure field and that it is suitable for highly parallel simulations. Furthermore, in comparison to the hybrid method with data exchange via disk I/O, the novel approach shows superior performance when scaling to thousands of cores.

Toward large-scale crop production forecasts for global food security

Predicting crop production plays a critical role in food price forecasting and mitigating potential food shortages. Crop models may require parameters from, for example, weather, crop genotype, farm management, and soil. Sources for these data are often found in very different places. Researchers spend a significant amount of time to collect and curate them. In addition, in order to scale yield forecasts from the single-farm level up to the continental scale, crop models have to be coupled with a geospatial big data platform to provide the required data inputs. In a proof-of-concept case study, we investigate the coupling of a scalable geospatial big data platform, Physical Analytics Integrated Repository and Services (PAIRS), to the Decision Support System for Agrotechnology Transfer (DSSAT) crop model. We envision running this system on a global scale. For geospatial analytics, PAIRS provides curation of heterogeneous data sources to simulate crop models using hundreds of terabytes of data.

Large scale research data archiving: Training for an inconvenient technology

At small scales, storage is straightforward to afford and to use, but at large scales – from several Terabytes (TB) to many Petabytes (PB) and soon Exabytes (EB) – tradeoffs must be made between cost and convenience, and training for use of such resources needs to take such inconveniences into account. A large scale, long term (over 10 year) institutional research data storage archive is described, focusing on both hardware and software. The technology choices give rise to inconveniences, which in turn not only lead to a crucial requirement for training on the proper use of the archive, but also inform the specifics of that training, as does each individual use case.

Archiving and retrieval of experimental data using SAN based centralized storage system for SST-1

SAN (Storage Area Network, a high-speed, block level storage device) based centralized data storage system of SST-1 (Steady State superconducting Tokamak) has envisaged to address the need of availability of SST-1 operation & experimental data centrally for archival as well as retrieval [2]. Considering the initial data volume requirement, ∼10TB (Terabytes) capacity of SAN based data storage system has configured/installed with optical fiber backbone with compatibility considerations of existing Ethernet network of SST-1.The SAN based data storage system has been designed/configured with 3-tiered architecture and GFS (Global File System) cluster file system with multipath support. Tier-1 is of ∼3TB (frequent access and low data storage capacity) comprises of Fiber channel (FC) based hard disks for optimum throughput. Tier-2 is of ∼6TB (less frequent access and high data storage capacity) comprises of SATA based hard disks. Tier-3 will be planned later to store offline historical data. In the SAN configuration two tightly coupled storage servers (with cluster configuration) are working together to achieve increase performance, reliability, distribute workload and provides access to the files from any server regardless of the physical location of the file. Different RAID (Redundant Array of Independent Disks) groups are created with both, FC and SATA hard disks to increase reliability, security and performance of the storage system.The adopted SAN based data storage for SST-1 is a modular, robust, and allows future expandability. The storage modules can be added as and when required without changing the existing storage architecture. The data read/write time of the configured system is adequate enough to cater the present throughput requirements of individual subsystems. Sufficient redundancy in terms of hardware has been incorporated to assure the uninterrupted experimental data availability 24×7 within intranet. The installed storage system can be expanded up to ∼100TB of capacity with the existing controller pair presents in the SAN configuration.

The role of geophytes in stone-age hunter-–gatherer diets

Identification and quantification of the characteristics of stem cell preparations is critical for understanding stem cell biology and for the development and manufacturing of stem cell based therapies. We have developed image analysis and visualization software that allows effective use of time-lapse microscopy to provide spatial and dynamic information from large numbers of human embryonic stem cell colonies. To achieve statistically relevant sampling, we examined > 680 colonies from 3 different preparations of cells over 5 days each, generating a total experimental dataset of 0.9 terabyte (TB). The 0.5 Giga-pixel images at each time point were represented by multi-resolution pyramids and visualized using the Deep Zoom Javascript library extended to support viewing Giga-pixel images over time and extracting data on individual colonies. We present a methodology that enables quantification of variations in nominally-identical preparations and between colonies, correlation of colony characteristics with Oct4 expression, and identification of rare events.

Large-scale time-lapse microscopy of Oct4 expression in human embryonic stem cell colonies

Identification and quantification of the characteristics of stem cell preparations is critical for understanding stem cell biology and for the development and manufacturing of stem cell based therapies. We have developed image analysis and visualization software that allows effective use of time-lapse microscopy to provide spatial and dynamic information from large numbers of human embryonic stem cell colonies. To achieve statistically relevant sampling, we examined >680 colonies from 3 different preparations of cells over 5days each, generating a total experimental dataset of 0.9 terabyte (TB). The 0.5 Giga-pixel images at each time point were represented by multi-resolution pyramids and visualized using the Deep Zoom Javascript library extended to support viewing Giga-pixel images over time and extracting data on individual colonies. We present a methodology that enables quantification of variations in nominally-identical preparations and between colonies, correlation of colony characteristics with Oct4 expression, and identification of rare events.

Study on Hadoop Cluster

In today's world, require Data Recovery system is most challenging aspects in the internet or World Wide Webapplications. Now a day's evens a Tera Bytes (TB) and Peta Bytes(PB) of data is not enough for storing large chunks of database(DB). Hence IT industries use concept is known as Hadoop in their applications. This approach has been adopted in Cloud computing environment for unstructured data. Hadoop is an open source distributed computing framework based on java and supports large set of distributed data processing.

Open Polar Server (OPS)—An Open Source Infrastructure for the Cryosphere Community

The Center for Remote Sensing of Ice Sheets (CReSIS) at the University of Kansas has collected approximately 1000 terabytes (TB) of radar depth sounding data over the Arctic and Antarctic ice sheets since 1993 in an effort to map the thickness of the ice sheets and ultimately understand the impacts of climate change and sea level rise. In addition to data collection, the storage, management, and public distribution of the dataset are also primary roles of the CReSIS. The Open Polar Server (OPS) project developed a free and open source infrastructure to store, manage, analyze, and distribute the data collected by CReSIS in an effort to replace its current data storage and distribution approach. The OPS infrastructure includes a spatial database management system (DBMS), map and web server, JavaScript geoportal, and MATLAB application programming interface (API) for the inclusion of data created by the cryosphere community. Open source software including GeoServer, PostgreSQL, PostGIS, OpenLayers, ExtJS, GeoEXT and others are used to build a system that modernizes the CReSIS data distribution for the entire cryosphere community and creates a flexible platform for future development. Usability analysis demonstrates the OPS infrastructure provides an improved end user experience. In addition, interpolating glacier topography is provided as an application example of the system.

NeuroBlocks--Visual Tracking of Segmentation and Proofreading for Large Connectomics Projects.

In the field of connectomics, neuroscientists acquire electron microscopy volumes at nanometer resolution in order to reconstruct a detailed wiring diagram of the neurons in the brain. The resulting image volumes, which often are hundreds of terabytes in size, need to be segmented to identify cell boundaries, synapses, and important cell organelles. However, the segmentation process of a single volume is very complex, time-intensive, and usually performed using a diverse set of tools and many users. To tackle the associated challenges, this paper presents NeuroBlocks, which is a novel visualization system for tracking the state, progress, and evolution of very large volumetric segmentation data in neuroscience. NeuroBlocks is a multi-user web-based application that seamlessly integrates the diverse set of tools that neuroscientists currently use for manual and semi-automatic segmentation, proofreading, visualization, and analysis. NeuroBlocks is the first system that integrates this heterogeneous tool set, providing crucial support for the management, provenance, accountability, and auditing of large-scale segmentations. We describe the design of NeuroBlocks, starting with an analysis of the domain-specific tasks, their inherent challenges, and our subsequent task abstraction and visual representation. We demonstrate the utility of our design based on two case studies that focus on different user roles and their respective requirements for performing and tracking the progress of segmentation and proofreading in a large real-world connectomics project.

A Web services accessible database of turbulent channel flow and its use for testing a new integral wall model for LES

abstractThe output from a direct numerical simulation (DNS) of turbulent channel flow at Reτ ≈ 1000 is used to construct a publicly and Web services accessible, spatio-temporal database for this flow. The simulated channel has a size of 8πh × 2h × 3πh, where h is the channel half-height. Data are stored at 2048 × 512 × 1536 spatial grid points for a total of 4000 time samples every 5 time steps of the DNS. These cover an entire channel flow-through time, i.e. the time it takes to traverse the entire channel length 8πh at the mean velocity of the bulk flow. Users can access the database through an interface that is based on the Web services model and perform numerical experiments on the slightly over 100 terabytes (TB) DNS data on their remote platforms, such as laptops or local desktops. Additional technical details about the pressure calculation, database interpolation, and differentiation tools are provided in several appendices. As a sample application of the channel flow database, we use it to conduct an a-priori test of a recently introduced integral wall model for large eddy simulation of wall-bounded turbulent flow. The results are compared with those of the equilibrium wall model, showing the strengths of the integral wall model as compared to the equilibrium model.

Operational experience running Hadoop XRootD Fallback

In April of 2014, the UCSD T2 Center deployed hdfs-xrootd-fallback, a UCSD- developed software system that interfaces Hadoop with XRootD to increase reliability of the Hadoop file system. The hdfs-xrootd-fallback system allows a site to depend less on local file replication and more on global replication provided by the XRootD federation to ensure data redundancy. Deploying the software has allowed us to reduce Hadoop replication on a significant subset of files in our cluster, freeing hundreds of terabytes in our local storage, and to recover HDFS blocks lost due to storage degradation. An overview of the architecture of the hdfs-xrootd-fallback system will be presented, as well as details of our experience operating the service over the past year.

Data correlation‐based analysis methods for automatic memory forensic

AbstractMemory forensics is an important technique for protecting network security and fighting against computer crimes. It has developed greatly in the past decade, because memory can provide more reliable information that other evidence sources do not contain. However, nowadays, when investigating network criminal cases, the Gigabyte (GB) and even Terabyte (TB) level memory and many such dumps have made memory analysis a difficult task. And investigators usually have to deal with complex operating system (OS) data structures, which they have little knowledge of. So how to analyze memory evidence automatically so as to find the hidden criminal behavior and reconstruct the scenario in an understandable way has become an important problem. This paper presents an automatic memory analysis methodology based on data correlation. Through analyzing key OS data structures and utilizing a clustering algorithm, this methodology can discover the relationships among processes, files, users, Dynamic‐link library (DLLs), and network connections. By describing these relationships as correlation graphs, our methods can reorganize these independent memory evidences and disclose their meanings in a high semantic level. Experiments have proved that these correlation graphs can help investigators find hidden criminal behavior and reconstruct the criminal scenarios. And as we know, now, little work is in this field. Copyright © 2015 John Wiley &amp; Sons, Ltd.

Abstract LB-308: A novel data safe haven approach to bring analyses to the International Cancer Genome Consortium data

Abstract To target and personalize cancer therapies to the genomic aberrations present in a particular patient's tumor, researchers need to identify the genes that drive the progression of malignant tumors. This requires analysis of somatic mutations from large samples of patients to identify driver mutations up to the “tail end” of the frequency distribution. Community genomics data sets from the TCGA and ICGC projects represent a valuable resource to which researchers can add their own data to gain statistical power in their analyzes. The current issue to this methodology is the highly fragmented storage of public and private data and the inefficient access to public data. Researchers spend weeks to months downloading hundreds of terabytes of data from central repositories before computations can begin. What is needed is a data “safe haven” where researchers can bring compute to the reference data without the need to incur in bulky data transfers or duplicative storage costs, in an environment that protects the privacy of the patients’ data. In collaboration with the International Cancer genome Consortium, we developed ShareSeq, a genomic data safe haven platform that provides an informatics solution for storing, handling and analyzing protected identifiable genomic data. This resource leverages Annai-GNOS, the technology which we developed to create and manage the CGHub TCGA repository together with UCSC, and that is being used in the ICGC Pan Cancer Analysis of Whole Genomes project, and combines it with a high-performance compute environment and an array of tools to process and analyze genomic data. Built using a walled garden approach, where the data is stored, processed and managed within the security of the system, ShareSeq avoids the complexity of assured end point encryption. GeneTorrent, our fast and secure file transfer mechanism, enables researchers’ private information to be transferred into the walled garden simply and securely to combine it with the public datasets. ShareSeq differs dramatically from the traditional cloud in two features: (i) formal mechanisms and a service level agreement to store protected identifiable genomic data securely and safely, built into the system from the ground up; (ii) the system is specifically designed for genomic computing over large shared data sets supporting common bioinformatics workflow tools; (iii) Fast download and access to raw genomic information and its metadata; and (iv) access controls leveraging federated authentication systems that Data Access Committees utilize to authorize access to the restricted data. ShareSeq is initially hosting raw, normalized, and processed data from the ICGC, but we envision that over time it will host an increasing number of high value reference genomic public datasets and add standards-based interfaces promoted by the Global Alliance of Genomes and Health to allow broader data discovery and sharing. Citation Format: Francisco M. De La Vega, Ying Wu, Tal Shmaya, Thomas Schlumpberger, James Wiley, Akshay Patel, Raja Hayek. A novel data safe haven approach to bring analyses to the International Cancer Genome Consortium data. [abstract]. In: Proceedings of the 106th Annual Meeting of the American Association for Cancer Research; 2015 Apr 18-22; Philadelphia, PA. Philadelphia (PA): AACR; Cancer Res 2015;75(15 Suppl):Abstract nr LB-308. doi:10.1158/1538-7445.AM2015-LB-308

Data compression for the first G-APD Cherenkov Telescope

The First Geiger-mode Avalanche photodiode (G-APD) Cherenkov Telescope (FACT) has been operating on the Canary island of La Palma since October 2011. Operations were automated so that the system can be operated remotely. Manual interaction is required only when the observation schedule is modified due to weather conditions or in case of unexpected events such as a mechanical failure. Automatic operations enabled high data taking efficiency, which resulted in up to two terabytes of FITS files being recorded nightly and transferred from La Palma to the FACT archive at ISDC in Switzerland. Since long term storage of hundreds of terabytes of observations data is costly, data compression is mandatory. This paper discusses the design choices that were made to increase the compression ratio and speed of writing of the data with respect to existing compression algorithms. Following a more detailed motivation, the FACT compression algorithm along with the associated I/O layer is discussed. Eventually, the performances of the algorithm is compared to other approaches.